Set up cert-manager in Kubernetes for TLS

Welcome to another part of my N part series on setting up a scalable WordPress installation on Kubernetes, specifically a Digital Ocean cluster (DOKS).

We’ll base this on the setup we had in the last tutorial when we installed nginx-ingress. In that version we didn’t have any TLS, a.k.a. encryption on your communication, a.k.a. the little lock in the address bar.

You might not think it is a big deal, but consider that when you log in to your site you’ll be sending your password in the clear over the public internet, and anyone with visibility of those packets can see your password being sent in the clear. So lets get started!

This is a bit of a cross between a couple of the official documentation pages (here and here), but I make some changes to the issuers that make things a bit easier to manage in a single tenant environment.

Configure DNS

First up we need to configure DNS. The certificates we will generate/use will be tied to a domain name which we obviously need to have control of. It would be crazy if we could generate a certificate for google.com. So, get the IP of your load balancer in Digital Ocean either using the web UI, or by running the following doctl command.

doctl compute load-balancer list --format Name,IP

You should see your load balancer name and IP, the name will probably be a random hex string, the IP will be…an IP.

Create an A record in you DNS management tool of choice pointing to your Load Balancer IP.

Update Ingress to include TLS

It is possible to add the TLS fields to our ingress before we have actually generated a certificate, the result will be that it will use a self signed certificate and you’ll get a warning in your browser. At this stage it isn’t an issue – we don’t expect it to work.

We’ll use the same manifest we did in the previous part, but we’ll update the Ingress so that it looks like this:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
  name: doks-example
  namespace: website-1
spec:
  tls:
  - hosts:
    - sample-app.goruncode.com
    secretName: website-1-kube-cert
  rules:
    - host: sample-app.goruncode.com
      http:
        paths:
          - backend:
              serviceName: doks-example
              servicePort: 80
            path: /

There are four new lines here:

tls:
  - hosts:
    - sample-app.goruncode.com
    secretName: website-1-sample-app-cert

The domain should obviously match the domain you’ll be using.

The secretName is just a name that will be used for the certificate. Make it whatever you want, but something that explains what it is is generally helpful.

Now apply the manifest to make the changes:

$ kubectl apply -f manifest.yaml

And inspect the Ingress to see that it is now listening on port 443:

$ kubectl get ingress -A

You’ll notice that under ports 80 and 443 are now listed.

Install cert-manager

This is actually really easy, and can be done in two commands:

kubectl create namespace cert-manager
kubectl apply --validate=false -f https://github.com/jetstack/cert-manager/releases/download/v0.13.0/cert-manager.yaml

The first creates a namespace in which everything will live (nice!).

The second command creates everything required to run cert-manager in your cluster. It might be simple to install but a lot goes on when you run this command.

Check that the cert-manager pods are running:

kubectl get pods --namespace cert-manager

You should see 3 pods running, something along the lines of this:

NAME                                       READY   STATUS    RESTARTS   AGE
cert-manager-6f57ae4565-x8bsg              1/1     Running   0          4d3h
cert-manager-cainjector-75b9cc7b8b-fkn8p   1/1     Running   0          4d3h
cert-manager-webhook-8444c4fa77-nw8lw      1/1     Running   0          4d3h

Create Certificate Issuers

Cert-manager needs to know how to get certificates issued, we’ll be using LetsEncrypt, but you could be using some internal CA. We’ll create two issuers, one for staging and one for prod. This is useful because LetsEncrypt has strict rate limiting on requests, so if you mess anything up it is better to do so with staging where you won’t be blocked.

Note that this is where i diverge a bit from the official documentation. I prefer to create a ClusterIssuer rather than an Issuer. The main difference is the ClusterIssuer doesn’t belong to a specific namespace and can be referenced by certificates from multiple namespaces. If you need a specific Issuer per namespace for some reason, then use an Issuer. This could be, for example, because you issue certificates using one contact email in one namespace, and another contact email in another. If you will just be using LetsEncrypt, with a single contact email for all certificates, a ClusterIssuer is probably the best choice.

So, to create your staging issuer, create a file called staging-issuer.yaml with the following contents:

---
apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
  name: letsencrypt-staging
spec:
  acme:
    # The ACME server URL
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    # Email address used for ACME registration
    email: you@your-domain.com
    # Name of a secret used to store the ACME account private key
    privateKeySecretRef:
      name: letsencrypt-staging
    # Enable the HTTP-01 challenge provider
    solvers:
    - http01:
        ingress:
          class:  nginx

Be sure to update the email address, then create the ClusterIssuer:

kubectl apply -f staging-issuer.yaml

Create a production issuer in the same way, but use the production LetsEncrypt API:

---
apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    # The ACME server URL
    server: https://acme-v02.api.letsencrypt.org/directory
    # Email address used for ACME registration
    email: you@your-domain.com
    # Name of a secret used to store the ACME account private key
    privateKeySecretRef:
      name: letsencrypt-prod
    # Enable the HTTP-01 challenge provider
    solvers:
    - http01:
        ingress:
          class:  nginx

And apply:

kubectl apply -f staging-issuer.yaml

Everything should now be in place to generate and start using a TLS certificate!

Test the setup using staging

We’re going to add another annotation to our Ingress now, this time it will cause cert-manager to do its thing and automate the TLS cert generation. This includes talking to nginx to do everything it needs to do to convince LetsEncrypt that we deserve a certificate.

The change is a single line:

cert-manager.io/cluster-issuer: "letsencrypt-staging"

A quick note, this again varies from the official doco in that we are using a cluster-issuer rather than an issuer. I have been caught out by this before so learn from my mistakes.

The whole ingress will now look like this:

---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
    cert-manager.io/cluster-issuer: "letsencrypt-staging"
  name: doks-example
  namespace: website-1
spec:
  tls:
  - hosts:
    - sample-app.goruncode.com
    secretName: website-1-kube-cert
  rules:
    - host: sample-app.goruncode.com
      http:
        paths:
          - backend:
              serviceName: doks-example
              servicePort: 80
            path: /

Apply the result:

kubectl apply -f manifest.yaml

A bunch of stuff will start happening in the background. A common issue i’ve experienced is DNS not being propogated, but it is generally pretty quick.

Have a look at what is going on. Rather than showing you, try running some of the following commands:

kubectl get certificates -n website-1
kubectl get secrets -n website-1
kubectl get certificaterequests -n website-1

Also try kubectl describe ... to look at the details of each of the above, you should be able to see the request get made and certificates get generated.

Eventually, when you run kubectl get certificates -n website-1 you should see the READY column switch to true. This means you’ve successfully been assigned a cert by the staging issuer. Note that this certificate doesn’t work, but it does mean the setup is correct.

Create a proper certificate

Alright, you’ve installed everything and the test setup worked! Now lets create the real deal certificate! Update our ingress to change the cluster-issuer to letsencypt-prod. The result should be:

---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
  name: doks-example
  namespace: website-1
spec:
  tls:
  - hosts:
    - sample-app.goruncode.com
    secretName: website-1-kube-cert
  rules:
    - host: sample-app.goruncode.com
      http:
        paths:
          - backend:
              serviceName: doks-example
              servicePort: 80
            path: /

I’ve run into issues directly applying the new config, so I recommend deleting the existing ingress, then re-applying the manifest. But I do tend to be risk averse. So feel free to skip the delete command…

kubectl delete ingress -n website-1 doks-example
kubectl apply -f manifest.yaml

Once again watch as your certificate gets requested and generated, then eventually, if all goes well, browse to https://your.domain.com and enjoy a fully encrypted browsing experience!

Things will probably go wrong

As versions change and API’s change with them, things constantly stop working. Try to figure it out, or write a comment and maybe I or someone else will be able to help.

Leave a Reply

Your email address will not be published.