This Ghost Blog is now running with Let's Encrypt in a cheap bare-metal Kubernetes Cluster (on Hetzner Cloud) — Part 2/3

• Part 1
• Part 3

About storage and persistence

A service like a blog needs to run in a stateful manner, so all articles are persisted. This blog was initially set up with a MySQL database along with local on-disk persistence for images and themes. In order to run on Kubernetes, it is therefore needed to rethink of how persistence would be managed, due to the volatile nature of containers in Kubernetes. Containers may come and go at any time, but state needs to be persisted somewhere.

I tested the approaches depicted below, but ended up using the first method (Container Storage Interface). There are however many other options for handling storage that I might explore later on.

Container Storage Interface (CSI)

CSI is a standard meant to be implemented by most Kubernetes providers, so cloud-native applications could be easily portable. In a nutshell, this specification allows users to claim some classes of volumes (via for example a PersistentVolumeClaim), which could then be mapped under the hood by persistent volumes auto-provisioned by the cloud provider in which the containers are running.

Hetzner Cloud provides an Open-Source CSI-compatible driver: Github://hetznercloud/csi-driver.

Installation

• Grab your personal account token under your Hetzner Account. Open "Cloud Console > Access > API Tokens > Generate API Token"
• Create a secret with the token above
• Apply the secret file created above
• Now we can deploy the storage driver in our Kubernetes Cluster:

apiVersion: v1
kind: Secret
metadata:
  name: hcloud-csi
  namespace: kube-system
stringData:
  token: <MY_TOKEN>

kubectl apply -f <secret.yml>

kubectl apply -f https://raw.githubusercontent.com/hetznercloud/csi-driver/v1.2.0/deploy/kubernetes/hcloud-csi.yml

Example Usage

First a YML descriptor for the PersistentVolumeClaim (say saved as my-blog-pvc,yml), as follows:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: rm3l-org-content
spec:
  storageClassName: hcloud-volumes
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi

As you can see, the storage class name is what actually what makes Kubernetes leverage the CSI driver for Hetzner.

Next, we can apply it to our Kubernetes cluster, which should result in a new volume being provisioned in your Hetzner Account:

kubectl apply -f my-blog-pvc.yml

We can now see in our Hetzner Cloud Web Console that a volume (10GB is the minimum at this time) is created:

Network File System (NFS) share, using Rook

Rook is a Cloud Native Computing Foundation incubating project (at the time of writing), with interesting features such as storage orchestration for Kubernetes.

The steps to use it are very well documented here, but unfortunately, I didn't explore this any further, due to port 111 (PortMapper / rpcbind) being open in my nodes and flagged as unsecured traffic by folks at Hetzner Cloud.

In fact, in addition to being potentially abused for DDoS reflection attacks, the Portmapper service may be used by attackers to obtain information on target networks, like available RPC services or network shares (as is the case with NFS). We could mitigate the risk by adding strict firewall rules (ufw - the uncomplicated firewall might help for this) to make sure this port is not exposed publicly.

Managing auto-renewable SSL certificates with Let's Encrypt

HTTPS is now becoming more and more important to any website, and Let's Encrypt greatly helps democratize its adoption by:

• providing free SSL/TLS signed certificates and a (relatively simple) API to generate these certificates.
• and being a Certificate Authority recognized by most modern tools, including browsers.

By default, pods of Kubernetes services are not accessible from the external network, but only by other pods within the Kubernetes cluster. Kubernetes has a built‑in configuration for HTTP load balancing, called Ingress, that defines rules for external connectivity to Kubernetes services.

If the remote cloud provider of choice supports it, you can even request a load balanced auto-managed IP address or hostname for the Service. This is something that can quickly add up to the usage bill, depending on the traffic.

At this time, Hetzner Cloud does not provide a fully-managed Kubernetes cluster, but we may still set up basic routing rules to different services.

The advantage of using Ingress resources is that we can easily configure for typical load-balancing use cases, such as SSL/TLS Termination or rewrite rules.

Ingress can be backed by different implementations through the use of different Ingress Controllers. The most popular of these is the NGINX Ingress Controller; however there are other options available such as Traefik or Rancher. Each controller should support a basic configuration, but can even expose other features (rewrite rules, authentication modes) via annotations.

In the scope of this work, we will use NGINX Ingress Controller. Installation instructions are provided here, but one command to apply is:

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/mandatory.yaml

Once the mandatory setup is done, we need to define an NGINX load-balancer configuration file (e.g.: ingress-nginx.k8s.yaml) and instruct Kubernetes to create it:

kind: Service
apiVersion: v1
metadata:
  name: ingress-nginx
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
spec:
  type: LoadBalancer
  selector:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
  ports:
    - name: http
      port: 80
      targetPort: http
    - name: https
      port: 443
      targetPort: https

We can now apply it and check that the service is actually up and running:

kubectl apply -f ingress-nginx.k8s.yaml 

We should have a Service of type LoadBalancer, with Cluster-IP and External-IP picked from the pool of IP addresses added in the "About Networking" section above (via MetalLB), e.g:

❯ kubectl get svc --namespace ingress-nginx
Alias tip: kgs --namespace ingress-nginx
NAME            TYPE           CLUSTER-IP     EXTERNAL-IP       PORT(S)                      AGE
ingress-nginx   LoadBalancer   10.99.77.237   116.203.238.174   80:32255/TCP,443:32464/TCP   64d

Next step is now to create a dedicated namespace in which cert-manager configuration will live, as depicted below. For reference, cert-manager is a native Kubernetes certificate management controller. It can help with issuing certificates from a variety of sources, such as Let’s Encrypt, HashiCorp Vault, Venafi, a simple signing key pair, or self signed. It also ensures certificates are valid and up to date, and attempts to renew certificates at a configured time before expiry.

# Create a namespace to run cert-manager in
kubectl create namespace cert-manager

# Disable resource validation on the cert-manager namespace
kubectl label namespace cert-manager certmanager.k8s.io/disable-validation=true

# Install the CustomResourceDefinitions and cert-manager itself
kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v0.10.1/cert-manager.yaml

Now that this is done, we need to create certificate issuers resource. Since Let's Encrypt enforces rate limits to ensure a fair usage, it is recommended to first test this against their staging environment first, and once confident, switch to the Production environment.

This covers the staging environment, but can be easily translated to Production.

Below is an example of a YAML file to create a Certificate Issuer against the staging environment:

   apiVersion: certmanager.k8s.io/v1alpha1
   kind: ClusterIssuer
   metadata:
     name: letsencrypt-staging
   spec:
     acme:
       # The ACME server URL
       server: https://acme-staging-v02.api.letsencrypt.org/directory
       # Email address used for ACME registration
       email: [email protected]
       # Name of a secret used to store the ACME account private key
       privateKeySecretRef:
         name: letsencrypt-staging
       # Enable the HTTP-01 challenge provider
       solvers:
       - http01:
           ingress:
             class:  nginx

Now we can apply it with kubectl apply -f ...