Configuring Cloudflare Tunnels in k3s

As part of setting up a home k3s cluster I had expected to be able to expose applications to the internet by using a Dynamic Domain Name System (DDNS) to keep my dynamic home IP address up-to-date with my domain name provider and to then port forward traffic through my router to the appropriate services.

However it turns out that Community Fibre, my internet provider, uses CGNAT which doesn’t allow port forwarding. This meant that I needed another method of exposing internal services. The two main options seemed to be Tailscale Funnel and Cloudflare Tunnel, Cloudflare Tunnel seemed to be better documented so that’s the solution I went with.

Setting up a Cloudflare Tunnel was pretty straightforward (described in the docs linked above) but it took me a bit longer to get things working on the k3s side.

apiVersion: v1
kind: ConfigMap
metadata:
  name: cloudflared
  namespace: kube-system
data:
  config.yml: |
    tunnel: k3s-tunnel

    ingress:
      - hostname: bitscry.com
        service: http://traefik.kube-system.svc.cluster.local:80
      - hostname: "*.bitscry.com"
        service: http://traefik.kube-system.svc.cluster.local:80

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: cloudflared
  name: cloudflared
  namespace: kube-system
spec:
  replicas: 1
  selector:
    matchLabels:
      pod: cloudflared
  template:
    metadata:
      creationTimestamp: null
      labels:
        pod: cloudflared
    spec:
      containers:
        - command:
            - cloudflared
            - tunnel
            - --no-autoupdate
            - --metrics
            - 0.0.0.0:2000
            - run
          args:
            - --token
            - $(CLOUDFLARED_TOKEN)
          image: cloudflare/cloudflared:latest
          name: cloudflared
          env:
            - name: CLOUDFLARED_TOKEN
              valueFrom:
                secretKeyRef:
                  name: cloudflare
                  key: cloudflared-token
          livenessProbe:
            httpGet:
              path: /ready
              port: 2000
            failureThreshold: 1
            initialDelaySeconds: 10
            periodSeconds: 10
          volumeMounts:
          - name: config
            mountPath: /etc/cloudflared
      volumes:
      - name: config
        configMap:
          name: cloudflared

Once I had Cloudflare Tunnel working successfully I then added an external-dns service which creates DNS entries for my exposed ingress resources with my DNS provider (Cloudflare in this case).

apiVersion: v1
kind: ServiceAccount
metadata:
  name: external-dns
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: external-dns
rules:
- apiGroups: [""]
  resources: ["services", "endpoints", "pods"]
  verbs: ["get", "watch", "list"]
- apiGroups: ["extensions", "networking.k8s.io"]
  resources: ["ingresses"]
  verbs: ["get", "watch", "list"]
- apiGroups: ["networking.k8s.io"]
  resources: ["ingressclasses"]
  verbs: ["get", "watch", "list"]
- apiGroups: [""]
  resources: ["nodes"]
  verbs: ["list", "watch"]
- apiGroups: ["networking.k8s.io"]
  resources: ["ingresses/status"]
  verbs: ["update"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: external-dns
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: external-dns
subjects:
- kind: ServiceAccount
  name: external-dns
  namespace: kube-system

apiVersion: apps/v1
kind: Deployment
metadata:
  name: external-dns
  namespace: kube-system
  labels:
    app: external-dns
spec:
  replicas: 1
  selector:
    matchLabels:
      app: external-dns
  template:
    metadata:
      labels:
        app: external-dns
    spec:
      serviceAccountName: external-dns
      containers:
      - name: external-dns
        image: k8s.gcr.io/external-dns/external-dns:v0.10.1
        args:
        - --source=service
        - --source=ingress
        - --provider=cloudflare
        - --cloudflare-proxied
        - --policy=upsert-only
        env:
        - name: CF_API_TOKEN
          valueFrom:
            secretKeyRef:
              name: cloudflare
              key: k3s-external-dns-api-token