SSL certificates from Let’s Encrypt for Kubernetes Private Ingress via Terraform

Image by skylarvision from Pixabay

There are various ways on how to achieve SSL certificates for Kubernetes ingresses. One of them is using cert-manager.io together with Let’s Encrypt. This was my choice moving forward as automation is my prime goal when it comes to Kubernetes and its services.

Kubernetes is all about automation, so let’s apply it for expiring SSL certificates too, and no more need for manually renewing them. Let’s Encrypt certificates are valid for only 3 months.

This is not something that we would like to repeat every 3 months. and uploading a manually purchased SSL certificate is not satisfactory since it requires manual intervention from time to time to ensure certificates are always up to date.

However, while migrating our services from on-premise to cloud, one of the challenges that I was facing was to activate a valid SSL certificate for our private ingress domains.

For ACME validation, Cert-manager supports two types of validations, namely DNS01 and HTTP01. DNS validation method modifies DNS to add a DNS record asked by Let’s Encrypt to validate your own control of domain name. HTTP validation method requests a file with a specific name to be uploaded into the acme-challenge folder of your domain name for Let’s encrypt to validate your domain ownership.

SSL Certificate via YAML

In a cloud environment, we want it to be automatic, which is where cert-manager.io comes to the rescue.

Install cert-manager first:

kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.1.0/cert-manager.yaml

While the HTTP01 method worked flawlessly for public ingresses, it failed hard for private ingresses. for obvious reason, Private ingresses are only accessible using Private IP, so Let’s encrypt have no way of accessing the website to verify if the file is uploaded there.

Following works perfectly for public ingress domain names:

apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
name: letsencrypt
spec:
acme:
server: https://acme-v02.api.letsencrypt.org/directory
email: youremail@example.com
privateKeySecretRef:
name: letsencrypt
solvers:
- selector: {}
http01:
ingress:
class: nginx

But now it leaves us with the DNS01 method for private ingresses. DNS01 method works by giving our DNS server tokens to cert-manager.io configurations so that it can access and update the given domain name records. so that It can fetch the required DNS record from Let’s Encrypt and add it to our DNS, wait for Let’s encrypt to verify domain ownership and then remove that verification DNS record.

If you are using Akamai, AzureDNS, Cloudflare, DigitalOcean, Google CloudDNS or Route53 odds are on your side. You can generate a token for cert-manager.io and it will handle the SSL certificates for you.

Unfortunately, the odds were not on my side! we were using a DNS service not listed under cert-manager.io supported DNS01 providers. There was an abandoned GitHub project aiming to support our DNS provider but it never finds its way to production.

So as the first step, I had to move our DNS service to one of the supported providers. Instead of AWS, Azure, Google, or DigitalOcean DNS providers, let’s focus on Cloudflare since it is not linked to any cloud service provider.

I’ve created an API token (not to be confused with an API key) that allows access to DNS edit, and Zone read and have access to All zone resources. API token allows limited access to a specific zone, where API key is global and can be considered a security issue if you use them.

This API token must be stored in a secret:

apiVersion: v1
kind: Secret
metadata:
name: cloudflare-api-token-secret
type: Opaque
stringData:
api-token: <API Token>

So now we can use it in our cert-manager issuer:

apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
name: example-issuer
spec:
acme:
...
solvers:
- dns01:
cloudflare:
email: youremail@example.com
apiTokenSecretRef:
name: cloudflare-api-token-secret
key: api-token

And that’s it for the SSL. All will work and you can see cert-manager is issuing HTTPS/SSL certificate for your private ingresses successfully now.

SSL Certificates via Terraform in AWS EKS Cluster

Terraform is great for infrastructure as code and I always use it to simplify the re-creation of similar resources, avoid human error and have transparent visibility on how everything is set up and connected together.

Looking above for SSL certificate generation, it looks straight forward. but how do we achieve this via Terraform exactly?

I am going to skip HTTP01 method and only focus on DNS01 over Cloudflare here.

We need to create our cloudflare-api-token-secret secret first which accepts cloudflare_token as a variable:

variable "cloudflare_token" {
sensitive = true
description = "Your Cloudflare API token"
}
resource "kubernetes_secret" "cloudflare_api_token_secret" {
metadata {
namespace = "cert-manager"
name = "cloudflare-api-token-secret"
}
type = "Opaque"
data = {
"api-token" = var.cloudflare_token
}
}

Here comes a challenge. How do we exactly create a resource Kind of “Issuer” or (in my case), “ClusterIssuer” via terraform? Terraform’s Kubernetes provider does not have any resources to cover that. Terraform has gone forward and issued a Kubernetes-alpha provider which solves this. You can use kubernetes_manifest where you can specify custom Kinds to resources.

But I choose a different approach. I wanted to experiment with having full kubectl control while provisioning some specific types of YAMLs that terraform currently does not support or have a module so that I do not get limited to the current feature set of Terraform and I’ve used a null_resource to execute kubectl for me.

I didn’t want to spend the whole day creating a cert-manager module for terraform by converting their YAML file into terraforming. instead, I wanted to be able to apply that YAML file directly via kubectl.

And in my case, I am using Terraform Cloud. so local_exec does not really run on my own computer. kubectl is not available and if it is, it has no access to the Kubernetes cluster so it needs to be configured. It should be another article by itself but it is necessary for me to show it here so we can achieve our goal of generating SSL certificates here.

First, configure the kubectl command to work. Please note that I am using Terraform EKS module to generate the EKS cluster. So I get generated kubeconfig file from EKS module and copy it into ~/.kube/config, then download aws-iam-authenticator and kubectl:

resource "null_resource" "kubeconfig" {  provisioner "local-exec" {
command = <<EOF
set -e
mkdir -p ~/.kube/
cp ${module.eks.kubeconfig_filename} ~/.kube/config
curl -o aws-iam-authenticator https://amazon-eks.s3.us-west-2.amazonaws.com/1.18.9/2020-11-02/bin/linux/amd64/aws-iam-authenticator
chmod +x aws-iam-authenticator
mkdir -p $HOME/bin && cp ./aws-iam-authenticator $HOME/bin/aws-iam-authenticator
curl -LO https://storage.googleapis.com/kubernetes-release/release/v1.20.0/bin/linux/amd64/kubectl
chmod +x kubectl
cp ./kubectl $HOME/bin/
EOF }
}

And now we can use kubectl to access our EKS cluster. so now installing cert-manager and configuring our Issuer/ClusterIssuer is trivial:

variable "cloudflare_email" {
description = "Your Cloudflare Email Address"
}
resource "null_resource" "certmanager" { provisioner "local-exec" { command = <<EOT
set -e
export PATH=$PATH:$HOME/bin
./kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.1.0/cert-manager.yaml
cat <<EOF | kubectl apply -f -
apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
name: letsencrypt
spec:
acme:
server: https://acme-v02.api.letsencrypt.org/directory
email: youremail@example.com
privateKeySecretRef:
name: letsencrypt
solvers:
- selector: {}
dns01:
cloudflare:
email: ${var.cloudflare_email}
apiKeySecretRef:
name: cloudflare-api-token-secret
key: api-token
EOF
EOT
}
}

If I get some time in the future, I might create a cert-manager module for Terraform and provide it in Terraform Registery, and update this kubectl method for the kubernetes_alpha provider.

Activation of SSL in an Ingress

Now that we have configured our cert-manager correctly, all we need to do is to tell our ingress to use it.

When defining an ingress, we need to specify the following anonation:

"cert-manager.io/cluster-issuer" = "letsencrypt"

and in spec section of ingress:

tls {
hosts = ["sub.example.com"]
secret_name = "sub.example.com"
}

so a full ingress in terraform would look like this:

resource "kubernetes_ingress" "this" {
metadata {
name = "sub.example.com"
annotations = {
"cert-manager.io/cluster-issuer" = "letsencrypt"
}
}
spec {
rule {
host = "sub.example.com"
http {
path {
backend {
service_name = yourservice
service_port = 80
}
path = "/"
}
}
}
tls {
hosts = ["sub.example.com"]
secret_name = "sub.example.com"
}
}
}

Coders changed the world. To be continued…

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store