# linsnotes.com > Kuibin Lin's personal blog, on raspberry pi, networking, homelab and AI ## Pages ### [Categories](https://linsnotes.com/categories/) ### [Tags](https://linsnotes.com/tags/) ### [Archives](https://linsnotes.com/archives/) ### [Projects](https://linsnotes.com/projects/) > These are web-based tools for learning Chinese, built using HTML and JavaScript. {: .prompt-tip } > AI Playground [**AI Playground**](/ai-playground/) — Experiment with 6 AI models: text-to-image, chat, vision, sentiment analysis, summarization, and translation. {: .prompt-info } > Attention Visualiser [**Attention Visualiser**](/attention-visualiser/) — Learn how transformer attention works step by step. {: .prompt-info } > Typing [**Typing**](https://kuibinlin.github.io/typing/) {: .prompt-info } > Language Puzzle [**Language Puzzle**](https://kuibinlin.github.io/puzzle/) {: .prompt-info } > Language Marksman [**Language Marksman**](https://kuibinlin.github.io/shooting/) {: .prompt-info } > Quiz Lah [**Quiz Lah**](https://kuibinlin.github.io/quiz/) {: .prompt-info } > flashcard [**Flashcard**](https://kuibinlin.github.io/flashcard/) {: .prompt-info } > tingxie [**tingxie**](https://kuibinlin.github.io/tingxie/) {: .prompt-info } > Pinyin Reader [**Pinyin Reader**](https://kuibinlin.github.io/pinyin/) {: .prompt-info } > Daily Quotes [**Daily Quotes**](https://kuibinlin.github.io/daily/) {: .prompt-info } > e-Calligraphy [**e-Calligraphy**](https://kuibinlin.github.io/e-calligraphy/) {: .prompt-info } > xizi [**xizi**](https://kuibinlin.github.io/xizi/) {: .prompt-info } > MyDeck [**MyDeck**](https://linsnotes.com/mydeck/) — Flashcards & Challenges with leaderboards {: .prompt-info } ### [About](https://linsnotes.com/about/) ## Hi. I’m Kuibin Lin. Nice to meet you! I'm an AI engineer, and before that I spent seven years teaching Chinese. I came to technology fairly late, and mostly through self-study. I can't remember exactly when it started, but at some point I realised that even a basic understanding of programming isn't just useful, it's part of the foundation of how the modern world works and how new ideas become reality. Back at university, I always wanted to take a programming module, but it never fit into my timetable, so I never had the chance to study it formally. After I started working, I took a short data science course that used Python. My learning has been on and off ever since, but my curiosity about programming, AI, and computer hardware has never faded. Since bootcamps and formal courses can be expensive, I taught myself using resources like YouTube, Coursera, and Udemy. Later, I enrolled in an [AI Singapore](https://aisingapore.org/) course when it was offered at a promotional price with a complimentary DataCamp subscription. I didn't expect much beyond learning a few new skills, but that course eventually led me to become an Associate AI Engineer in AI Singapore's AI Apprenticeship Programme. The arrival of tools like ChatGPT made learning even more accessible. By combining online tutorials with AI, I've been able to explore a wide range of technologies, including building PCs, Python, HTML, CSS, JavaScript, Jekyll, GitHub Pages, Raspberry Pi, Cloudflare DNS, WireGuard VPN, Docker, Nginx, Proxmox VE, TrueNAS, OpenWrt, Azure, and more. I originally started this blog as a way to learn Jekyll and GitHub Pages. Over time, it has become a place where I document what I'm learning, share the projects I'm working on, and write about things I think others might find useful. You can browse the [tools](/tags) I've used or explore the different topics I've written about. You'll also find links to my GitHub, LinkedIn, and email in the sidebar. If anything here resonates with you, or you're working on something similar, I'd be happy to connect. ## Posts ### [Building a Kubernetes Lab with k3s and LXD Part 8 Monitoring with Prometheus and Grafana](https://linsnotes.com/posts/building-a-kubernetes-lab-with-k3s-and-lxd-part-8-monitoring-with-prometheus-and-grafana/) > Install kube-prometheus-stack through Argo CD, put Grafana behind HTTPS, read the numbers that matter for this lab, then stop a node and watch an alert fire. This is **Part 8** of a series on building a Kubernetes lab with k3s and LXD, and it is the last part of this arc. If you are just arriving, start at the beginning: [**Part 1, Preparing the Ubuntu Host**](/posts/building-a-kubernetes-lab-with-k3s-and-lxd-part-1-preparing-the-ubuntu-host/), [**Part 2, Building the Five Nodes**](/posts/building-a-kubernetes-lab-with-k3s-and-lxd-part-2-building-the-five-nodes/), [**Part 3, Installing k3s**](/posts/building-a-kubernetes-lab-with-k3s-and-lxd-part-3-installing-k3s/), [**Part 4, Adding Persistent Storage**](/posts/building-a-kubernetes-lab-with-k3s-and-lxd-part-4-adding-persistent-storage/), [**Part 5, Exposing Applications**](/posts/building-a-kubernetes-lab-with-k3s-and-lxd-part-5-exposing-applications/), [**Part 6, GitOps with Argo CD**](/posts/building-a-kubernetes-lab-with-k3s-and-lxd-part-6-gitops-with-argo-cd/), and [**Part 7, Securing with TLS and cert-manager**](/posts/building-a-kubernetes-lab-with-k3s-and-lxd-part-7-securing-with-tls-and-cert-manager/). **Where you are.** You have a five node cluster that heals when a master dies, keeps data when a worker dies, exposes applications by hostname through ingress-nginx on a MetalLB address, deploys itself from a Git repository through Argo CD, and serves real, trusted HTTPS certificates from Let's Encrypt. Argo CD itself is at `https://argocd.k3s.linsnotes.com` with a certificate a browser is happy with. There is one thing missing, and it is the thing you would need first on any real platform: **you cannot see what your cluster is doing.** **What this part does.** It installs Prometheus, Grafana, Alertmanager and their friends, through the Git repository from Part 6, puts Grafana behind HTTPS using the ClusterIssuer from Part 7, then teaches you to read the numbers that actually matter for this particular lab. Finally it stops a node and makes an alert fire, so you see the whole chain from a machine going away to a red box on a screen. **The main point, in one line.** Kubernetes tells you a pod is `Running`. It does not tell you whether that pod is healthy, slow, or about to be killed for using too much memory, and monitoring is the thing that closes that gap. **What this part does not do.** It does not collect logs. Metrics and logs are different problems with different tools (Loki, or Elasticsearch, or something else), and mixing them in would double the length of this part and halve the clarity. It also does not send alerts anywhere real. Alertmanager will be running and you will watch an alert reach it, but wiring up email or Slack is configuration, not understanding. **What you need before starting.** The `post-tls` snapshot from Part 7, all five nodes Ready, Argo CD healthy, cert-manager running, and **at least 4 GB of free memory on the host**. Step 1 checks all of it, and the memory point is not decoration. Read the first of the three things below before you install anything. --- ## Words you'll need These build on Parts 2 to 7. Here are the new ones. | Word | What it means | |---|---| | **metric** | One number that describes one thing at one moment. "This node is using 3.1 GB of memory." That is a metric. | | **time series** | The same metric, recorded over and over, with a timestamp on each reading. A metric is a number; a time series is that number's history. Prometheus stores time series, which is why it can draw graphs and you cannot get a graph out of `kubectl`. | | **label** | A name and value attached to a time series to say which thing it describes, for example `node="worker1"`. One metric name plus different labels is many separate time series. | | **scrape** | How Prometheus collects. It asks each target for its current numbers over HTTP, every so often (15 or 30 seconds is typical), and stores the answer. Nothing pushes to Prometheus; Prometheus pulls. | | **exporter** | A small program that turns something's internal state into numbers Prometheus can scrape. If a thing does not speak Prometheus itself, you put an exporter in front of it. | | **node-exporter** | The exporter for a machine. Memory, CPU, disk, network, uptime. One copy runs on every node. | | **kube-state-metrics** | An exporter for *Kubernetes objects*, not machines. How many pods exist, which are Ready, what each one asked for, how many replicas a Deployment wants versus has. It reads the API server and turns the answers into metrics. | | **PromQL** | The query language you type into Prometheus or Grafana to ask a question of your time series. | | **CRD** | Custom Resource Definition. A way to teach Kubernetes a brand new kind of object, so that `kubectl get ` works as if it had always been built in. Part 5's MetalLB `IPAddressPool` and Part 7's `ClusterIssuer` were both CRDs. | | **Operator** | A program that runs inside the cluster, watches for custom objects, and does the real work of setting them up. You describe what you want in an object; the Operator makes it exist and keeps it that way. | | **ServiceMonitor** | A custom object that means "scrape the pods behind this Service." You write one of these instead of editing a Prometheus config file, and the Prometheus Operator rewrites the config for you. | | **Alertmanager** | The program that receives alerts from Prometheus, groups them, silences the ones you asked it to, and sends them somewhere. Prometheus decides *whether* something is wrong; Alertmanager decides *who hears about it*. | | **alert rule** | A PromQL query plus a condition plus a waiting period. If the query stays true for that long, the alert fires. | | **dashboard** | A saved page of graphs in Grafana. The queries are already written; you just pick the cluster or node you want to look at. | | **retention** | How long Prometheus keeps its history before deleting the old data. Longer history costs disk. | | **cardinality** | How many separate time series you have. One metric with a label that takes 100,000 different values is 100,000 time series, and that is how people accidentally run Prometheus out of memory. | --- ## Three things to know before you start ### 1. This is the first part that can genuinely run your host out of memory Every previous part added something small. This one adds a real workload, and it is worth doing the arithmetic before you install rather than after. Here is what you have. From Part 2's Step 5.3, on a 32 GB host: | Nodes | Each | Total | |---|---|---| | master1, master2, master3 | 4 GiB | 12 GiB | | worker1, worker2 | 6 GiB | 12 GiB | | | | **24 GiB schedulable** | That leaves about 8 GB for the host itself, which also has to feed the ZFS cache. Here is roughly what kube-prometheus-stack wants, on a five node cluster with nothing much running on it. These are rough shapes rather than promises, because the real numbers depend on how many time series you end up with: | Piece | Rough memory | |---|---| | Prometheus | 1 to 2 GiB | | Grafana | 150 to 300 MiB | | Alertmanager | 50 to 150 MiB | | Prometheus Operator | 50 to 150 MiB | | kube-state-metrics | 40 to 150 MiB | | node-exporter, five copies | about 150 MiB in total | | | **roughly 1.5 to 3 GiB** | Two to three gigabytes out of twenty four is comfortable. **It is comfortable only because your masters are not tainted.** Part 3's Step 6 offered you an optional taint that stops ordinary workloads landing on the three masters, and said plainly that taking it cuts your schedulable nodes from five to two. In memory terms, taking that taint drops you from 24 GiB to 12 GiB, and now a stack that wants up to 3 GiB is a quarter of everything you have. > **If you took Part 3's optional master taint, read this before you install.** Prometheus is the single biggest thing you have ever asked this cluster to run. With only the two workers schedulable, install it with explicit limits and a shorter history, or it will be the thing that teaches you what an out of memory kill looks like. The trimmed values are in Step 3. {: .prompt-warning } Even with untainted masters, do two things. Give Prometheus an explicit memory **request** so the scheduler knows to reserve room for it, and give it an explicit **limit** so a runaway query cannot eat a whole node. A pod with no request looks free to the scheduler, which is how you overbook a node that then kills something at random. ### 2. Prometheus must not store its data on NFS This is the most important sentence in Part 8, and it directly reuses Part 4. Prometheus' own storage documentation says its local storage is not compatible with non-POSIX filesystems, and names NFS specifically as not supported, because many NFS implementations do not do file locking properly and there is no reliable way to detect that from inside Prometheus. The failure mode is not a clean error. It is database corruption. So Part 4 gave you two StorageClasses, and for once the network one is the wrong answer: | StorageClass | From | Use it for Prometheus? | |---|---|---| | `nfs-client` | Part 4, Step 6 | **No.** Network filesystem, not supported, can corrupt the database. | | `local-path` | k3s built in, Part 4 Step 2 | **Yes.** A real directory on a real node's real disk. | Now be honest about what that costs, because Part 4 spent an entire step on exactly this weakness. `local-path` is node-locked. The PersistentVolume it creates has a node affinity rule pinning it to one machine, and any pod that wants it can only run on that machine. So: - **Your Prometheus pod is now pinned to one node.** It cannot move. - **If that node dies, Prometheus stays `Pending` until the node comes back**, exactly like `reader2` did in Part 4's Step 3. - **If that node is destroyed, the history is gone.** That is a real trade, not a technicality, and it is the correct trade here. Losing a week of lab metrics is annoying. A silently corrupted metrics database that reports plausible but wrong numbers is much worse, because you will believe it. The production answer is the one Part 4 already named at the end: replicated block storage such as Longhorn, which gives you a POSIX filesystem that survives a node dying. You do not have that, so you accept a pinned Prometheus and you know why. Remember this in Step 6, when you stop a node on purpose. **Check which node Prometheus landed on first, and stop a different one**, or your monitoring goes down at exactly the moment you wanted to watch it work. ### 3. `Running` does not mean healthy, and that gap is the whole point Part 4 made the same shape of argument about data: "the pod is Running" and "the data is safe" are two separate claims, and Kubernetes reports the first loudly while saying nothing about the second. Monitoring is that argument again, aimed at behaviour instead of storage. `kubectl get pods` shows `Running`, and `Running` only means the container's main process has not exited. It says nothing about whether: - the program is answering requests, or answering them slowly - it is at 95 percent of its memory limit and one request away from being killed - it has restarted eleven times in the last hour and happens to be up right now - the node underneath it is out of disk - it has been quietly failing one request in twenty for three days None of that is visible in `kubectl`. All of it is visible in a graph. That is what you are installing. --- ## Step 1 — Check Part 7 is still good **Why:** the same reason as every part. If something drifted, find out now, while a fault still looks like itself. **Safe to run:** everything here only reads. ```bash lxc info | head -3 # wakes LXD if it's asleep, same trick as before sleep 3 lxc list # expect five RUNNING rows, plus the STOPPED template kubectl get nodes # Expect: five Ready. kubectl get storageclass # Expect two: local-path (default) and nfs-client. You want local-path today. kubectl get pods -n ingress-nginx # Expect: one controller pod, Running. This is Part 5's front door. kubectl get pods -n cert-manager # Expect: three pods Running (controller, webhook, cainjector). Part 7's work. kubectl get clusterissuer # Expect: your self-signed CA issuer and your Let's Encrypt production issuer, # both READY True. Note the exact NAME of the Let's Encrypt one; you need it # in Step 4. This guide calls it letsencrypt-prod. ``` Now check Argo CD, because everything in this part goes through it: ```bash kubectl get pods -n argocd # Expect: all Running. kubectl get applications -n argocd # Expect: your existing Applications, every one Synced and Healthy. ``` **If anything is already `OutOfSync` or `Degraded`, fix that before adding a ninth thing on top.** An Application that was unhappy before you started will look like it was caused by monitoring, and you will spend an hour on the wrong problem. Finally, prove HTTPS still works end to end, which is the one check that exercises Parts 5, 6 and 7 in a single line: ```bash curl --max-time 10 -sI https://argocd.k3s.linsnotes.com | head -1 # Expect: HTTP/2 200. No -k flag, no certificate warning. If this needs -k, # your certificate is not trusted and Step 4 will inherit that problem. ``` And check the host has room, since Step 3 is the first thing in this series that can genuinely fill it: ```bash free -h # Expect: several GB available. If the host is already close to full, shrink # something before installing, not after. sudo zpool list default df -h / ``` --- ## Step 2 — Understand what you are about to install, before you install it `kube-prometheus-stack` is a Helm chart, but calling it "a chart" undersells it. It is a **bundle** of six separate programs plus a large pile of pre-written alert rules and Grafana dashboards, wired together so they find each other. Installing the pieces one at a time is a genuinely miserable afternoon, which is why this bundle exists and why almost everyone uses it. Here is every piece and what it does. Read this table before you run anything, because in Step 5 you will be querying numbers and it matters that you know which program produced them. | Piece | What it is | What it does here | |---|---|---| | **Prometheus Operator** | A controller | Watches for `Prometheus`, `Alertmanager` and `ServiceMonitor` objects and builds the real configuration from them. You never edit a Prometheus config file by hand. | | **Prometheus** | The metrics database | Scrapes every target every 30 seconds, stores the history, evaluates alert rules. This is the piece that needs memory and disk. | | **Alertmanager** | The notifier | Receives firing alerts from Prometheus, groups related ones, applies silences, and would send them onward if you configured a destination. | | **Grafana** | The screen | Draws graphs from Prometheus. Ships with dozens of ready-made Kubernetes dashboards so you do not start from a blank page. | | **node-exporter** | A DaemonSet | One pod per node, reporting that machine's memory, CPU, disk and network. Machine level, not Kubernetes level. | | **kube-state-metrics** | A Deployment | Reads the Kubernetes API and reports the state of *objects*: how many pods, which are Ready, what each asked for. Kubernetes level, not machine level. | **The last two are the pair people mix up, and the difference is worth ten seconds now.** `node-exporter` answers "how much memory is worker1 using?" `kube-state-metrics` answers "how many pods on worker1 are in `CrashLoopBackOff`?" One looks at the machine, one looks at Kubernetes' records of the machine. You need both, and in Step 5 you will use both. ### The Operator pattern, and why `ServiceMonitor` makes everything else make sense Plain Prometheus is configured with a YAML file listing what to scrape. That works fine when your targets are fixed servers. It works badly in Kubernetes, where pods appear and disappear constantly and nobody wants to edit a config file each time. The Operator pattern solves this by turning configuration into objects: ``` You create: A ServiceMonitor object saying "scrape the pods behind Service X, on port Y, every 30s" | The Prometheus Operator: notices it, regenerates Prometheus' real config, and reloads Prometheus | Prometheus: starts scraping. You never touched a config file. ``` That single indirection is why the whole stack is manageable. When you install something new later, you do not reconfigure Prometheus. You ship a `ServiceMonitor` next to your application, and monitoring turns itself on. Most well-behaved Helm charts have a `serviceMonitor.enabled` value that does exactly that. `ServiceMonitor` is a **CRD**, the same mechanism as Part 5's `IPAddressPool` and Part 7's `ClusterIssuer`. By now you have met this pattern three times, which is the point: a Kubernetes platform is mostly Operators watching custom objects. --- ## Step 3 — Install it through Git, not through Helm Part 6 turned deployment into a Git problem, and turned on automated sync with `selfHeal: true` and `prune: true`. That decision has a consequence you need to respect from here on: > **Do not run `helm install` for this.** With self-heal on, anything you create by hand that Argo CD does not know about is unmanaged, and anything you edit by hand inside a managed Application gets reverted within minutes. This is Part 6 working correctly, not Argo CD being difficult. The repository is the only way in now. {: .prompt-warning } So the install is: write one file, commit it, push it, and watch. ### 3a. Pick a chart version, and do not trust this page for it ```bash helm repo add prometheus-community https://prometheus-community.github.io/helm-charts helm repo update helm search repo prometheus-community/kube-prometheus-stack --versions | head -5 # Shows the newest chart versions available right now. ``` This guide was written against chart version **87.17.0**. That number will be out of date by the time you read it. Use whatever the command above shows you, and pin it explicitly in the Application rather than tracking whatever is newest, so that a chart release six months from now cannot change your cluster while you are not looking. This is the same reasoning Part 5 gave for the MetalLB manifest URL. While you are here, look at the values file, because you are about to set eight of its keys and there are several thousand: ```bash helm show values prometheus-community/kube-prometheus-stack --version 87.17.0 > /tmp/kps-values.yaml wc -l /tmp/kps-values.yaml # Expect: several thousand lines. This is the file to grep when a key name # in this post does not match your chart version. ``` That file is the honest answer to any "is this key still called that?" question. Chart values do get renamed between major versions, and grepping the real file beats trusting a blog post, including this one. ### 3b. The k3s problem nobody warns you about Before you write the values, know what you are about to see, because otherwise you will think you broke something. On a normal Kubernetes cluster built with kubeadm, the control plane is four separate programs running as four separate pods: `kube-apiserver`, `kube-controller-manager`, `kube-scheduler` and `kube-proxy`, plus `etcd`. The chart ships scrape configuration for all of them, because on a kubeadm cluster they are all there and all reachable. **k3s does not work that way.** k3s runs the control plane inside a single process, as Part 3's Step 8 already told you when it explained that k3s embeds etcd rather than running it as a pod. There are no `kube-controller-manager` pods to find. There are no `kube-scheduler` pods to find. The metrics for those components do exist, but they are served by the one k3s process, and by default they are bound to `127.0.0.1`, which means nothing outside that node can scrape them. So if you install the chart with its defaults, you get this: | Component | What the chart expects | What k3s gives you | |---|---|---| | kube-apiserver | Reachable via the `kubernetes` Service | Works. Scrapes fine. | | kubelet and cAdvisor | Port 10250 on every node | Works. Scrapes fine. This is where your container metrics come from. | | kube-controller-manager | A pod labelled `component=kube-controller-manager` | No such pod. Metrics on `127.0.0.1:10257`. | | kube-scheduler | A pod labelled `component=kube-scheduler` | No such pod. Metrics on `127.0.0.1:10259`. | | kube-proxy | A pod labelled `k8s-app=kube-proxy` | Embedded. Metrics on `127.0.0.1:10249`. | | etcd | A pod labelled `component=etcd` | Embedded in k3s. Metrics off by default. | The visible result is scrape jobs with no targets, or targets that will not connect, and a handful of alerts firing with names like `KubeControllerManagerDown`, `KubeSchedulerDown`, `KubeProxyDown` and `etcdMembersDown`. **Nothing is broken.** This is Part 2's "suspect the check before you suspect your machine" in its most expensive form yet, because here the check is not one command, it is a whole set of alert rules written for a cluster shaped differently from yours. You have two honest options. **Option A, turn those scrape jobs and their alert rules off.** This is what this guide does. It is the right default for a lab, because scheduler and controller-manager internals are not what you are here to learn, and an alert that fires forever trains you to ignore alerts, which is worse than having none. **Option B, expose the metrics properly.** k3s can bind those endpoints to all addresses instead of just localhost, and can expose etcd's metrics. You would add flags to `/etc/rancher/k3s/config.yaml` on the three masters, the same file Part 5 used to disable ServiceLB and Traefik. Remember Part 5's warning: **rewrite that file whole, never append a second `disable:` key**, or k3s will refuse to start on invalid YAML. ```bash # OPTION B ONLY. Skip this if you are taking Option A. # Rewrite the file whole, keeping the disable list from Part 5. for n in master1 master2 master3; do lxc exec "$n" -- bash -c 'mkdir -p /etc/rancher/k3s && cat > /etc/rancher/k3s/config.yaml **`local-path` does not enforce the size you asked for.** The `20Gi` in the values is a request Kubernetes records, but the local-path provisioner just creates a directory on the node's disk; there is no quota behind it. So Prometheus can grow past 20 GB and fill the node. That is what `retentionSize: 15GB` is for: Prometheus policing its own disk use because the storage layer will not. It is a lab-grade answer, and it is honest about being one. {: .prompt-info } ### If the Application will not go Healthy Three things account for almost all of it: - **`Too long: must have at most 262144 bytes`.** You missed `ServerSideApply=true`. Add it, commit, push. - **Prometheus stuck `Pending`.** Read `kubectl describe pod -n monitoring prometheus-...-0`. Either no node has 1 GiB free (the memory arithmetic from point 1), or the PVC never bound. - **Permanently `OutOfSync` while everything is Running.** Usually the Operator or an admission webhook writing to a field Argo CD then sees as drift. Use `kubectl get application -n argocd kube-prometheus-stack -o yaml` or the UI's diff view to see which field, then either accept it with an `ignoreDifferences` entry or leave it. A cosmetically OutOfSync app with everything Healthy is not urgent, but do find out which field it is rather than shrugging. --- ## Step 4 — Reach Grafana over HTTPS This is the shortest step in the whole series, and that is the point of it. ### The two lines that used to be a whole part Look again at the `grafana.ingress` block from Step 3c. Strip out the hostname and the class, and here is everything that makes it HTTPS: ```yaml annotations: cert-manager.io/cluster-issuer: letsencrypt-prod tls: - secretName: grafana-tls hosts: - grafana.k3s.linsnotes.com ``` **One annotation and one `tls:` block.** That is it. There is no certificate to request, no key to generate, no file to copy anywhere, and no renewal to remember. Here is what those five lines set in motion, and it is worth spelling out because it is three parts of this series firing at once: 1. ingress-nginx (Part 5) claims the Ingress and starts listening for the hostname. 2. cert-manager (Part 7) sees the annotation, creates a `Certificate` object, and asks Let's Encrypt for a certificate for `grafana.k3s.linsnotes.com`. 3. Let's Encrypt asks it to prove it controls the name. cert-manager answers with the DNS-01 challenge through Cloudflare, writes a TXT record, waits for it to propagate, and cleans it up afterwards. 4. cert-manager stores the signed certificate in a Secret called `grafana-tls`. 5. ingress-nginx picks up the Secret and serves HTTPS. 6. In about sixty days, cert-manager renews it, and nobody notices. Parts 5, 6 and 7 each cost you a long afternoon. The whole payoff is that the eighth thing you deploy costs you five lines, and the ninth will too. That is what a platform is: the third application is cheap because the first one was expensive. ### Watch the certificate arrive ```bash kubectl get certificate -n monitoring # Expect: grafana-tls, READY True. DNS-01 through Cloudflare usually takes # one to three minutes because it waits for the TXT record to propagate. # READY False for the first couple of minutes is normal, not a fault. kubectl describe certificate -n monitoring grafana-tls | tail -20 # If it is still False after five minutes, the Events at the bottom name the # reason: a Cloudflare API token problem, or a DNS propagation wait. kubectl get ingress -n monitoring # Expect: CLASS nginx, HOSTS grafana.k3s.linsnotes.com, ADDRESS filling in # with your ingress-nginx address from Part 5, PORTS 80, 443. ``` ### Add the name Same pattern as every hostname since Part 5. Find the ingress-nginx address and add the name to the host's hosts file: ```bash NGINX_IP=$(kubectl get svc -n ingress-nginx ingress-nginx-controller \ -o jsonpath='{.status.loadBalancer.ingress[0].ip}') echo "ingress-nginx is on $NGINX_IP" echo "$NGINX_IP grafana.k3s.linsnotes.com" | sudo tee -a /etc/hosts curl --max-time 10 -sI https://grafana.k3s.linsnotes.com | head -1 # Expect: HTTP/2 200 or HTTP/2 302. No -k flag. If this needs -k, the # certificate is not the Let's Encrypt one; check the ClusterIssuer name # in your annotation matches the one from Step 1. ``` Add the same line to your laptop's hosts file if you did Part 5's Step 5. ### Get in Grafana's admin password is generated into a Secret. Read it out: ```bash kubectl get secret -n monitoring kube-prometheus-stack-grafana \ -o jsonpath='{.data.admin-user}' | base64 -d; echo # Expect: admin kubectl get secret -n monitoring kube-prometheus-stack-grafana \ -o jsonpath='{.data.admin-password}' | base64 -d; echo # Prints the password. Do not print it into a screenshot or a blog post. ``` Some chart versions use a fixed default password rather than a generated one. Reading the Secret works either way, which is why the guide reads it instead of telling you a value. > **Do not change the password in Grafana's own UI and expect it to stick.** Part 6's `selfHeal: true` is watching. If you want a password you chose, set `grafana.adminPassword` in the values file, or better, point `grafana.admin.existingSecret` at a Secret you created separately so a real credential never sits in a public Git repository. This is the first time in the series where the GitOps decision from Part 6 makes you think about secrets, and the honest answer is that this lab has no secrets management. That gap is named again in the wrap-up. {: .prompt-warning } Open `https://grafana.k3s.linsnotes.com` and log in. ### Three dashboards worth opening first The chart ships a large collection of dashboards under **Dashboards**, and the first look is overwhelming. Open these three, in this order. **1. `Kubernetes / Compute Resources / Cluster`.** The whole cluster on one page: CPU and memory used against what is available, broken down by namespace. This is the "am I about to run out" screen, and after installing this stack it is the one you should actually check. You will see `monitoring` sitting near the top of the memory list, which is a fair reflection of point 1 of "Three things to know". **2. `Kubernetes / Compute Resources / Node (Pods)`.** Pick a node from the dropdown and see every pod on it, with its CPU and memory. This is where you find out that master2 is doing more work than you thought, or that one pod is using nine tenths of a node. **3. `Node Exporter / Nodes`.** The machine level view: memory, CPU, disk, network per node. **Read this one with the suspicion Part 2 taught you**, because your nodes are LXD containers sharing one kernel, and the numbers here may be the host's rather than the node's. There is a check for exactly that in Step 5. One more, so it is not a surprise: **the `etcd` dashboard will be empty.** That is Step 3b, not a fault. You turned that scrape job off because k3s does not expose those metrics by default. If you took Option B, it may have data. --- ## Step 5 — Read the numbers that matter for THIS lab Dashboards are somebody else's questions. Writing your own queries is how you ask yours, and this lab has some very specific ones, because a five node cluster inside LXD containers on one host is not the cluster those dashboards were designed for. Use Grafana's **Explore** page (the compass icon), pick the Prometheus data source, and paste these in one at a time. Everything below only reads. ### 5a. Did Part 2's memory limits actually reach Kubernetes? Part 2's Step 5.3 set `limits.memory=4GiB` on the masters and `6GiB` on the workers, and warned at length that if those limits do not reach kubelet, Kubernetes will believe every node has the host's full 32 GB and cheerfully overbook a machine five times over. Part 3's Step 1 made you check it with `free -h`. Now you can check the same thing from Kubernetes' own point of view, which is the number the scheduler actually uses: ```promql kube_node_status_capacity{resource="memory"} / 1024 / 1024 / 1024 ``` **Expect: about 4 on the three masters, about 6 on the two workers.** Five results, one per node. If any node comes back at about 31, that node has no LXD memory limit and Kubernetes is being lied to. Go back to Part 2's Step 5.3 and set it. This is the same fault Part 2 predicted and Part 3 re-checked, now visible as a graph you can leave on a screen instead of a command you have to remember to run. That metric comes from **kube-state-metrics**, which reads Kubernetes' own records. That is exactly why it is trustworthy for this question: it tells you what the scheduler believes, which is the thing that decides where pods go. ### 5b. How close are you to filling a node? Two different questions here, and people confuse them constantly. **What has been promised** (this is what the scheduler cares about): ```promql 100 * sum by (node) (kube_pod_container_resource_requests{resource="memory"}) / on (node) sum by (node) (kube_node_status_allocatable{resource="memory"}) ``` This is the percentage of each node's memory that has been **requested** by the pods on it. When this reaches 100 on every node, new pods stay `Pending` no matter how much memory is actually free, because the scheduler books by request, not by use. If you took Part 3's master taint, watch this number on the two workers. **What is actually being used** (this is what the kernel cares about): ```promql sum by (node) (container_memory_working_set_bytes{container!=""}) / 1024 / 1024 / 1024 ``` This comes from cAdvisor inside kubelet, and it reads cgroups, which is the same accounting the kernel uses to decide what to kill. It is the number to trust in this lab. Run both and compare. Requested is usually far higher than used, because most charts request more than they need. That gap is normal, and understanding that it is normal is most of what capacity planning is. > **If `node` comes back empty on the cAdvisor query, your chart version labels those series differently.** Type `container_memory_working_set_bytes` on its own in Explore, look at the labels on a returned series, and use whichever one names the machine (often `instance`). This is a five second check, and it beats copying a query from a blog post that assumed a different chart. Suspect the query before you suspect the cluster. {: .prompt-tip } ### 5c. Does node-exporter agree, or is it seeing the host? Here is a question specific to running Kubernetes inside LXD, and one worth answering yourself rather than trusting anyone about. Your nodes share the host's kernel. Part 2's Step 5.3 explained that LXD uses lxcfs to make each container see its own limits when it reads `/proc/meminfo`, which is why `free -h` inside master1 says 4 GiB rather than 32 GB. `node-exporter` reads that same `/proc`, but it reads it through a mount from inside a pod, and whether the lxcfs view survives that trip is not something to assume. So check: ```promql node_memory_MemTotal_bytes / 1024 / 1024 / 1024 ``` Compare each result against 5a's answer for the same node. - **If they agree** (about 4 on masters, about 6 on workers), lxcfs's view reached node-exporter and the `Node Exporter / Nodes` dashboard is telling you the truth about your nodes. - **If node-exporter reports about 31 on every node**, it is seeing straight through to the host. The dashboard is then showing you five copies of one machine's memory, which is real information about the host but not about the node. In that case, prefer 5a and 5b, which come from kube-state-metrics and cAdvisor and are cgroup-based, and read the node-exporter dashboard as a host dashboard. Either answer is fine. The failure would be not knowing which one you have, and then making a capacity decision from a number that means something other than you think. This is the shared-kernel trade-off from Part 1 showing up one last time, in the monitoring layer. ### 5d. CPU ```promql sum by (node) (rate(container_cpu_usage_seconds_total{container!=""}[5m])) ``` The result is in **cores**. A value of `0.4` means that node is using four tenths of one core, averaged over the last five minutes. Compare against Part 2's `limits.cpu` of 2 on masters and 4 on workers. `rate(...[5m])` is the PromQL idea worth taking away from this part. `container_cpu_usage_seconds_total` only ever counts upwards, forever, which is useless to look at directly. `rate` converts that ever-rising counter into "how fast is it rising per second", which is the thing you actually wanted. Almost any metric whose name ends in `_total` needs `rate` wrapped around it. ### 5e. Pods per node, against the ceiling nobody mentions ```promql count by (node) (kube_pod_info) ``` and the limit it is heading towards: ```promql kube_node_status_capacity{resource="pods"} ``` **Expect 110 per node**, which is Kubernetes' default maximum and has nothing to do with your memory. It is a separate ceiling, and on a small lab you will hit memory long before you hit it. Worth knowing it exists, because "the node has plenty of memory and pods still will not schedule" is otherwise a genuinely confusing hour. ### 5f. etcd health across the three masters Part 3 spent a long time on quorum, leader election, and what happens when a majority is lost. The natural next question is whether you can watch that on a graph. **Honest answer: only if you took Option B in Step 3b.** k3s does not expose etcd's metrics by default, and this guide turned that scrape job off, so `etcd_*` metrics do not exist in your Prometheus. Confirm it for yourself rather than taking my word: ```promql etcd_server_has_leader ``` Empty result means the metrics are not there, which is the expected outcome of Option A. If you did take Option B, these are the three to know: | Query | What it answers | |---|---| | `etcd_server_has_leader` | 1 means this member can see a leader. A 0 on any master is Part 3's quorum failure, live. | | `increase(etcd_server_leader_changes_seen_total[1h])` | How many elections happened in the last hour. Occasional is normal, constant is a sign of a struggling member. | | `histogram_quantile(0.99, rate(etcd_disk_wal_fsync_duration_seconds_bucket[5m]))` | How long etcd waits for the disk. This is the number that goes bad first when etcd is unhappy, and on a lab sharing one ZFS pool between five members it is a genuinely interesting thing to watch. | What you **can** see with Option A is the cluster's view of its own masters, which is most of what Part 3 was actually about: ```promql kube_node_status_condition{condition="Ready", status="true"} ``` One line per node, 1 for Ready and 0 for not. Keep this one; Step 6 uses it. ### 5g. Is anything restarting behind your back? The single most useful query on this page, and the clearest example of "Running does not mean healthy": ```promql sum by (namespace, pod) (increase(kube_pod_container_status_restarts_total[1h])) ``` Any pod with a number above zero has restarted in the last hour. A pod that crashes every four minutes and restarts successfully shows `Running` in `kubectl` almost every time you look. This query is how you find it. On a healthy lab, everything here should be zero. --- ## Step 6 — Break it on purpose Every part of this series ends by taking something away. Part 3 killed a master to watch quorum protect itself. Part 4 killed a worker to watch data survive. Part 5 killed the node holding the MetalLB address to watch it move. Here you kill a worker and watch the whole monitoring chain react, from a machine disappearing to a red box on a screen. ### First, do not kill the node Prometheus is on This is point 2 of "Three things to know", collecting its debt. ```bash kubectl get pods -n monitoring -o wide | grep -E 'prometheus-|grafana' # Note which node Prometheus is on. Because of local-path, it CANNOT move. ``` If Prometheus is on `worker1`, stop `worker2` instead, and swap the names throughout this step. If you stop the node Prometheus is pinned to, Prometheus goes down, stays `Pending` until the node returns, and you have no monitoring with which to watch your monitoring experiment. That would be an instructive mistake, but you would rather choose it than trip over it. ### Set up the watch Open two things before you break anything: - **Grafana**, on the `Kubernetes / Compute Resources / Cluster` dashboard, or on Explore running `kube_node_status_condition{condition="Ready", status="true"}`. - **Prometheus' own UI.** It has no Ingress, so port-forward to it: ```bash kubectl port-forward -n monitoring svc/kube-prometheus-stack-prometheus 9090:9090 # Leave this running. Open http://127.0.0.1:9090 in the host's browser. # The Alerts tab is what you want. The Status > Targets tab is worth a look # too: this is where you SEE the k3s scrape jobs from Step 3b, or rather # see that they are absent because you turned them off. ``` On the Alerts tab you will notice one alert already firing, called **`Watchdog`**. That is deliberate and it ships that way. It is an alert designed to fire always, so that a monitoring system that has quietly died is detectable: if you ever stop receiving `Watchdog`, the thing that sends alerts is broken. An always-firing alert is the only honest way to monitor the monitor. ### Stop the node ```bash lxc stop worker2 # or whichever worker is NOT running Prometheus ``` Now watch, in this order. The timings matter and they are the lesson. **Within seconds:** `lxc list` shows the container `STOPPED`. This is the truth, and it is the only source that is instant. Everything else lags, exactly as Parts 3 and 4 warned. **After about a minute:** `kubectl get nodes` shows `NotReady`. Your Grafana query flips that node's line from 1 to 0. This is the control plane deciding the node has missed enough heartbeats. **After a few minutes:** the pods on that node are evicted and recreated on the surviving nodes, exactly as Part 4's Step 8 showed. Watch it: ```bash kubectl get pods -A -o wide -w | grep -v Running # Ctrl-C when it settles. ``` The `node-exporter` pod for the stopped node does **not** get rescheduled, and that is correct. It is a DaemonSet, meaning "one copy per node", so with the node gone there is nowhere for it to go. It comes back when the node does. **Now go to the Alerts tab in Prometheus and watch what does not happen for a while.** ### The `for:` duration, which is the single most confusing thing here You stopped a node. Kubernetes knows within a minute. And the `KubeNodeNotReady` alert does not fire. **Nothing is broken.** Look at the alert's state. It will be **Pending**, not **Firing**, and Pending is a real state with a specific meaning. Every alert rule has three parts: ```yaml - alert: KubeNodeNotReady expr: kube_node_status_condition{condition="Ready",status="true"} == 0 for: 15m # THIS is the part people miss ``` - **`expr`** is the condition. It is true right now. - **`for`** is how long the condition must stay true before the alert fires. - Between the condition becoming true and the `for` elapsing, the alert is **Pending**. So the sequence is: condition true, alert Pending, fifteen minutes pass, alert Firing, Alertmanager notified. **Why wait at all?** Because without the wait, every alert would be useless. A node that is `NotReady` for twenty seconds during a restart is not an incident, it is a Tuesday. A node that is `NotReady` for fifteen minutes is an incident. The `for:` duration is the entire difference between an alerting system people act on and one people mute. Check the real duration on your chart version rather than trusting this page, because the shipped rules do change: ```bash kubectl get prometheusrule -n monitoring -o yaml | grep -B2 -A6 'alert: KubeNodeNotReady' # Expect: the expr, and a "for:" line. 15m at time of writing. ``` Fifteen minutes is a long time to sit and stare. Two options: **Option 1: wait it out** and use the time productively. Watch the pods reschedule. Watch `TargetDown` go Pending too, because Prometheus can no longer scrape the kubelet on that node. Look at the Cluster dashboard and see the memory of the surviving nodes go up as they take on the evicted pods. **Option 2: add your own fast alert rule**, which is a better use of the time because you learn how to write one. Add a file to the Git repository, because Part 6 means that is now the only way in. Create `infra/monitoring/lab-alerts.yaml`: ```yaml apiVersion: monitoring.coreos.com/v1 kind: PrometheusRule metadata: name: lab-alerts namespace: monitoring labels: release: kube-prometheus-stack # this label is how Prometheus finds it spec: groups: - name: lab rules: - alert: LabNodeDownFast expr: kube_node_status_condition{condition="Ready", status="true"} == 0 for: 1m labels: severity: warning annotations: summary: "Node {{ $labels.node }} has been NotReady for 1 minute" ``` **That `release:` label is the whole trick, and it is worth understanding.** The Prometheus Operator does not pick up every `PrometheusRule` in the cluster. It picks up the ones matching a label selector, and the chart sets that selector to `release: `. Get the label wrong and your rule is applied to the cluster successfully, sits there looking perfectly healthy, and is never loaded. That is a silent failure of exactly the shape Part 2 warned about, so check rather than assume: ```bash # Confirm the Operator actually loaded it, in the Prometheus UI: # Status > Rules, and look for the "lab" group. # Or from the command line: kubectl get prometheusrule -n monitoring ``` If your rule does not appear in Prometheus' Rules page after a minute or two, read the selector the chart used and match it: ```bash kubectl get prometheus -n monitoring -o jsonpath='{.items[0].spec.ruleSelector}'; echo ``` Commit and push it, and you will need a second Argo CD Application pointing at `infra/monitoring/` if Part 6's setup does not already sync that directory. With the rule loaded, stop the node again and the alert goes Pending immediately and Firing one minute later, which is short enough to actually watch. ### Watch it reach Alertmanager Once something is Firing, it leaves Prometheus and arrives at Alertmanager. Look at it there: ```bash kubectl port-forward -n monitoring svc/kube-prometheus-stack-alertmanager 9093:9093 # Open http://127.0.0.1:9093 in the host's browser. ``` You will see the firing alert, grouped with any others that share its labels. This is the division of labour from Step 2 made concrete: **Prometheus decided something is wrong; Alertmanager decides who hears about it and how often.** With no receivers configured, "who hears about it" is nobody, which is why this is a lab and not a pager. Try the **Silence** button on the alert. A silence is how you say "yes, I know, I am doing maintenance, stop telling me until Tuesday." It is the single most used feature of Alertmanager in real life, and it costs nothing to try once. ### Bring the node back ```bash lxc start worker2 sleep 30 kubectl get nodes # Expect: five Ready again within a minute or two. ``` Watch the alert **resolve**. It does not vanish instantly. The `expr` becomes false as soon as the node reports Ready, the alert leaves the Firing state in Prometheus, and Alertmanager marks it resolved after its own short delay. In Grafana, the line goes back to 1. **That is the whole chain, seen once, end to end:** a machine went away, kubelet stopped reporting, kube-state-metrics recorded the change, Prometheus scraped it, an alert rule matched, a timer ran, an alert fired, Alertmanager received it, the machine came back, and everything unwound in the same order. Nothing in that chain is magic, and now you have watched every link of it. --- ## Step 7 — Clean up and save a restore point Remove the experiment, keep the platform. Same rule as every part: test things go, infrastructure stays. ```bash # If you added the fast alert rule and want it gone, remove it from Git. # Argo CD's prune: true will delete it from the cluster on the next sync. # git rm infra/monitoring/lab-alerts.yaml && git commit && git push # Or keep it. It is small, correct, and occasionally useful. # Stop any port-forwards you left running (Ctrl-C in their terminals). kubectl get all -n default # Expect ONE line: service/kubernetes. The familiar clean baseline. kubectl get pods -n monitoring # Expect: everything Running. This is infrastructure now, and it stays. kubectl get application -n argocd # Expect: every Application Synced and Healthy, including the new one. ``` Snapshot all five, stopped and together, exactly as Parts 3 to 7 taught: ```bash for n in worker1 worker2 master1 master2 master3; do lxc stop "$n"; done for n in master1 master2 master3 worker1 worker2; do lxc snapshot "$n" post-monitoring done for n in master1 master2 master3 worker1 worker2; do lxc start "$n"; done sleep 30 kubectl get nodes # expect: five Ready kubectl get pods -n monitoring # expect: everything back Running ``` Give Prometheus a minute after the restart. It replays its write-ahead log on startup, so the pod can sit in `Running` but not `Ready` for a short while, which is correct behaviour and not a fault. ### What is not in this snapshot The list keeps growing, and it is worth keeping straight, because a snapshot that you think holds more than it does is worse than no snapshot: | Not in the node snapshots | Where it actually lives | |---|---| | The NFS server and its share | The host, `/srv/nfs/k8s` (Part 4) | | Your `/etc/hosts` entries | The host, and your laptop (Part 5) | | The router's static route | Your router (Part 5) | | **The Git repository** | GitHub (Part 6). This is now the real source of truth for what runs. | | Your Cloudflare API token and DNS records | Cloudflare (Part 7) | That fourth row deserves a moment. Since Part 6, an LXD snapshot is no longer the most important backup you have. **Roll the nodes back to `post-monitoring` and Argo CD will immediately reconcile them against whatever is in Git right now, not whatever was in Git when you took the snapshot.** That is GitOps working exactly as designed, and it means the repository, not the snapshot, is the thing you would be sad to lose. --- ## Where you should be | Thing | State | |---|---| | Monitoring stack | kube-prometheus-stack in namespace `monitoring`, deployed by Argo CD from Git | | Prometheus | Running, 7 day retention, on `local-path`, pinned to one node and you know which | | Grafana | `https://grafana.k3s.linsnotes.com`, real Let's Encrypt certificate, five lines of config | | Alertmanager | Running, receiving alerts, no receivers configured | | Exporters | node-exporter on all five nodes, kube-state-metrics in the cluster | | k3s control plane targets | Deliberately disabled, with their alert rules, and you know why | | Alerting | Seen going Pending, then Firing, then resolving, with the `for:` delay understood | | Snapshots | `post-monitoring` on all five, taken stopped and together | ### Things worth carrying forward - **Prometheus must not go on NFS.** Prometheus' own documentation says so, and the failure is corruption rather than an error. Use block or local storage, and accept that on `local-path` your Prometheus is pinned to one node and loses its history if that node is destroyed. Replicated block storage (Longhorn) is the fix, and it is the same fix Part 4 named. - **A default scrape configuration assumes a cluster shaped like the author's, not yours.** On k3s, the control plane is one process, so the chart's controller-manager, scheduler, kube-proxy and etcd targets find nothing and their alert rules fire forever. Turning them off is the correct action, not a workaround. Suspect the check before the machine, one last time. - **`for:` is why your alert is not firing.** An alert with a condition that is true but a timer that has not elapsed sits in Pending, and Pending is normal. Read the `for:` before you conclude that alerting is broken. - **An always-firing alert (`Watchdog`) is the only way to monitor the monitor.** If it ever stops arriving, the alerting path itself has failed. - **kube-state-metrics and node-exporter answer different questions.** One is about Kubernetes objects, one is about machines. In LXD, check whether node-exporter is seeing your node or your host, and prefer the cgroup-based numbers when they disagree. - **Wrap `rate()` around anything ending in `_total`.** Counters only rise; the useful number is how fast. - **Size it before you install it.** This is the first workload in the series big enough to matter. Set requests so the scheduler reserves room, and limits so one query cannot take a node down. - **Once Argo CD has self-heal on, the repository is the only way in.** Everything in this part arrived through a commit, including a change of password you have to think about differently now. --- ## The series, wrapped up Eight parts ago you had one Ubuntu machine and no plan. Here is what you have now, said plainly. **What you built.** - **A highly available cluster.** Three masters share an etcd database with a quorum of two. You killed one and it shrugged. You killed two and watched it refuse to accept writes rather than risk disagreeing with itself, which is the feature and not the bug. - **Storage that survives a node dying.** Two StorageClasses, and more importantly the knowledge of which to reach for. You watched a pod get stranded by local storage and then watched the same test pass on network storage. - **A front door.** MetalLB gives out real addresses, ingress-nginx splits one address between many hostnames, and you watched the address move to another node in seconds when its holder died. - **A deployment process you did not have to remember.** Argo CD keeps the cluster matching a Git repository, and it puts things back when you delete them. - **Real HTTPS, automatically.** cert-manager gets certificates from Let's Encrypt through a DNS-01 challenge and renews them without being asked. By Part 8, adding HTTPS to a new service was five lines. - **A cluster that reports on its own health.** Prometheus scrapes, Grafana draws, Alertmanager receives, and you have watched an alert travel the entire path. That is a genuinely working small Kubernetes platform, and every piece of it is one you would meet in a real job. **What it is not.** This matters as much, and most tutorials stop before saying it. - **No replicated block storage.** NFS is a single point of failure and `local-path` is node-locked. A production platform runs Longhorn, Ceph, or a cloud provider's disks. Part 4 named this and Part 8 paid for it with a pinned Prometheus. - **No CI.** You commit to Git and Argo CD deploys it. Nothing tests your manifests, nothing builds an image, nothing stops you pushing something broken. Real platforms have a pipeline between the commit and the cluster. - **No secrets management.** Every secret in this lab was created by hand with `kubectl` or lives in a values file. Nothing is encrypted at rest in Git, nothing is rotated, and the Grafana password problem in Step 4 was the first time it pinched. The usual answers are Sealed Secrets, External Secrets Operator, or Vault. - **No logging.** You can see that a pod restarted eleven times. You cannot see *why* without `kubectl logs`, and that is gone the moment the pod is. A logging stack (Loki is the natural partner to this one) is the missing half of observability. - **No backups leaving the host.** k3s takes etcd snapshots and you take LXD snapshots, and both live on the same disk as the thing they protect. That is not a backup, it is a convenience. - **One host, which is one point of failure.** The whole reason this lab is affordable is the whole reason it is fragile. Three masters protect you from a master failing. Nothing protects you from the machine underneath all five failing, and that machine is also the NFS server. Be honest about this when you talk about it: you built a real HA control plane on top of a single point of failure, on purpose, because the alternative was five computers. Knowing where the edges are is worth more than pretending there are none. Every gap above is a thing a real platform team spends real time on, and now you know what each one is for. --- ## Where this goes next I am working towards an MLOps role, and the point of building this platform was never Kubernetes for its own sake. It was to have somewhere real to run machine learning workloads, on infrastructure I understand all the way down, so that when something breaks I know whether the problem is the model, the pipeline, or the cluster. The next arc, if I write it, builds on this exact platform rather than starting over. Everything below plugs into what Parts 1 to 8 already produced: it deploys through Argo CD from the same repository, it gets a hostname and HTTPS from the same ingress and the same ClusterIssuer, and it reports into the same Grafana. - **MinIO** for artifact storage, giving the cluster an S3-compatible bucket without an S3 bill. Datasets, model files, and anything else too big to belong in Git. - **MLflow** for experiment tracking, with Postgres behind it for metadata and MinIO behind it for artifacts. This is where "which run produced this model, and with what parameters" stops being a folder of filenames. - **Training runs as Kubernetes Jobs**, so a training run is a scheduled workload with requests, limits, retries and logs, rather than a script somebody ran on a laptop and then closed. - **Model serving behind the same Ingress**, so a trained model becomes an HTTPS endpoint at a hostname, using the five lines from Part 8's Step 4. - **Model metrics on the same Grafana**, because a served model exports latency and error rate like anything else, and prediction drift is just another time series. That is the shape. Whether it becomes Part 9 depends on how the first attempt goes, and I would rather write it after building it than before. If you have followed all eight parts: you built a Kubernetes cluster from an empty machine, broke it eight different ways on purpose, and understood why each recovery worked. That is a better foundation than most people get. ### [Building a Kubernetes Lab with k3s and LXD Part 7 Securing with TLS and cert manager](https://linsnotes.com/posts/building-a-kubernetes-lab-with-k3s-and-lxd-part-7-securing-with-tls-and-cert-manager/) > Put real HTTPS on a private lab. Build your own certificate authority first, then get trusted Let's Encrypt certificates through a Cloudflare DNS-01 challenge. This is **Part 7** of a series on building a Kubernetes lab with k3s and LXD. If you are just arriving, start with [**Part 1, Preparing the Ubuntu Host**](/posts/building-a-kubernetes-lab-with-k3s-and-lxd-part-1-preparing-the-ubuntu-host/), then [**Part 2, Building the Five Nodes**](/posts/building-a-kubernetes-lab-with-k3s-and-lxd-part-2-building-the-five-nodes/), [**Part 3, Installing k3s**](/posts/building-a-kubernetes-lab-with-k3s-and-lxd-part-3-installing-k3s/), [**Part 4, Adding Persistent Storage**](/posts/building-a-kubernetes-lab-with-k3s-and-lxd-part-4-adding-persistent-storage/), [**Part 5, Exposing Applications**](/posts/building-a-kubernetes-lab-with-k3s-and-lxd-part-5-exposing-applications/), and [**Part 6, GitOps with Argo CD**](/posts/building-a-kubernetes-lab-with-k3s-and-lxd-part-6-gitops-with-argo-cd/). This part picks up where Part 6 left off. **Where you are.** You have a five node cluster that survives a master dying and keeps data when a worker dies. MetalLB hands out addresses, ingress-nginx routes by hostname, and Argo CD keeps the whole thing matching a Git repository. Type a hostname into a browser and an application answers. It answers over plain HTTP. Every byte, in both directions, travels as readable text, and the browser says so with a "Not secure" label in the address bar. **What this part does.** It puts real certificates on your lab. First certificates you sign yourself, so you can watch every piece of the machinery with nothing outside your house involved. Then genuinely trusted certificates from Let's Encrypt, on a lab that has no public address and accepts no inbound traffic at all. **The main point, in one line.** A certificate is trusted because something the browser already trusts signed it, and the only hard part of getting one is proving you own the name. **What this part does not do.** It does not encrypt traffic between the Ingress and the pods behind it. That is a real thing (people call it mutual TLS, or a service mesh) and it is a much bigger subject. Here, TLS stops at the Ingress, which is where the vast majority of real clusters put it too. **What you need before starting.** The `post-gitops` snapshot from Part 6, all five nodes Ready, Argo CD healthy, and a domain name whose DNS you control. This guide uses `linsnotes.com` with DNS hosted at Cloudflare. Step 1 checks the cluster. Step 4 is the part that needs the domain, and Step 3 works completely offline if you do not have one. --- ## Words you'll need These build on Parts 2 to 6. Here are the new ones. | Word | What it means | |---|---| | **TLS** | The thing that makes a connection private. It encrypts the traffic and proves the server is who it claims to be. The `S` in HTTPS. You will still see it called SSL, which is the old name for the same job. | | **certificate** | A small file saying "this public key belongs to this name," signed by somebody. It is public. You hand it to every visitor. | | **private key** | The secret half. The certificate is useless to an impostor without it. This never leaves the server. | | **CA (certificate authority)** | Something that signs certificates for other people. Its own certificate is the one browsers are shipped with. | | **chain of trust** | Your certificate was signed by an intermediate, which was signed by a root that your browser already trusts. Follow the chain to something known and the certificate is accepted. | | **self-signed** | A certificate that signed itself, with no CA above it. Nothing trusts it by default, because trusting it means trusting whoever made it, and nobody has agreed to that. | | **CSR** | Certificate Signing Request. "Here is my public key and the name I want. Please sign it." What you send to a CA. | | **ACME** | The protocol for getting a certificate automatically, with no human involved. Let's Encrypt invented it. cert-manager speaks it for you. | | **Let's Encrypt** | A free, public CA that issues certificates over ACME. Its root is in every browser and operating system. | | **challenge** | The test a CA sets to prove you actually control the name you asked for. No proof, no certificate. | | **HTTP-01** | A challenge where the CA fetches a file from `http://yourname/.well-known/...`. Needs your server to be reachable from the public internet. | | **DNS-01** | A challenge where you put a specific TXT record in your domain's DNS. Needs nothing to be reachable, only that you control the domain. | | **Issuer vs ClusterIssuer** | Both say "here is how to get certificates." An `Issuer` works in one namespace. A `ClusterIssuer` works everywhere. Same fields otherwise. | | **Certificate (the resource)** | A Kubernetes object saying "I want a certificate for these names, from this issuer, stored in this Secret." cert-manager makes it real and keeps it real. | | **Secret** | Where the finished certificate and its private key are stored. TLS ones hold two keys: `tls.crt` and `tls.key`. | | **SAN** | Subject Alternative Name. The list of hostnames a certificate is actually valid for. Browsers read this list and ignore the older common name field. | | **wildcard certificate** | One certificate covering `*.k3s.linsnotes.com`, so every name under it works. Only obtainable with DNS-01. | | **staging vs production** | Let's Encrypt runs two systems. Staging is for practice: generous limits, and certificates nothing trusts. Production is the real one, with limits worth respecting. | --- ## Three things to know before you start ### 1. HTTP-01 cannot work in this lab, and that fact is the whole reason DNS-01 exists This is the single most valuable idea in this part, so it gets the most space. When you ask a certificate authority for a certificate for `whoami.k3s.linsnotes.com`, it will not just hand one over. It has to know you control that name, or anyone could ask for a certificate for your bank. So it sets a challenge, and there are two common kinds. **HTTP-01** works like this. Let's Encrypt gives your machine a random string. You put that string at a fixed path on the web server that name points to. Then Let's Encrypt, from its own servers on the public internet, fetches `http://whoami.k3s.linsnotes.com/.well-known/acme-challenge/` and checks it got the right string back. If it did, you clearly control whatever machine that name resolves to, so you get your certificate. It is elegant, it is the default nearly everywhere, and **it cannot possibly work here.** Look at what it needs and compare it to what you built: | HTTP-01 needs | Your lab has | |---|---| | The name to resolve publicly, for everyone | A line in `/etc/hosts` on one machine, and maybe a laptop | | A path from the public internet to your Ingress | A private network on `10.99.99.0/24`, behind NAT, with nothing forwarded in | | Port 80 reachable from outside | Port 80 on a MetalLB address that exists only inside your house | Let's Encrypt's servers would look up `whoami.k3s.linsnotes.com`, get nothing back, and fail. Even if you added a public DNS record pointing at `10.99.99.240`, that address is private and unroutable, so the fetch would go nowhere. There is no clever workaround. HTTP-01 is out. **DNS-01 changes the question.** Instead of "can I fetch something from your web server," it asks "can you put a specific value into your domain's DNS?" You prove ownership by writing a TXT record at `_acme-challenge.whoami.k3s.linsnotes.com`, Let's Encrypt looks that record up, and if the value matches, you get your certificate. Read that again and notice what is missing. **Nothing has to be reachable.** No inbound port. No public address. No DNS record pointing at your lab at all. The only thing that becomes public is one TXT record, and cert-manager deletes it as soon as the challenge is done. That is why this part goes to the trouble of a Cloudflare API token instead of the two lines that HTTP-01 would need. It is not a preference, it is the only door that opens. It also buys you something HTTP-01 could never give you: **wildcard certificates**, because there is no single web server to fetch a file from for a name that means "all of them." ### 2. A certificate is trusted because something you already trust signed it Strip away the acronyms and a certificate is a signed statement: "the holder of this public key is allowed to call itself `whoami.k3s.linsnotes.com`." Your browser believes it for exactly one reason: it checks who signed it, and follows that chain upward until it hits one of the few hundred root certificates it was shipped with. Let's Encrypt's root is one of those. That is the entire basis of the padlock. **Self-signed means you become the thing at the top of the chain.** You create a root CA, you decide what it signs, and then you tell your own machines to trust it. Nobody else's machine will, which is why you would never use it on a public site. But in a lab it is perfect, because every piece is visible and nothing outside your house is involved. You will see a tool reject a certificate, then add one file to your host, then watch the same certificate be accepted. So Step 3 builds a CA and Step 4 replaces it with a real one. The objects you write barely change between them, and that similarity is the point. ### 3. Let's Encrypt has real rate limits, and hitting one costs you a week Let's Encrypt is free and public, which means it has to protect itself. The limits that matter to you: - **50 certificates per registered domain every 7 days.** That is per `linsnotes.com`, across every account, not per subdomain. - **5 certificates for the exact same set of names every 7 days.** This is the one that catches people learning. Get your `ClusterIssuer` slightly wrong, fix it, retry, get it wrong again, and you can burn through five attempts in an afternoon and then wait a week. - **5 authorisation failures per name per hour.** A misconfigured Cloudflare token trips this quickly. These reset on a rolling schedule, not at midnight. There is no support desk you can talk into resetting them. The answer is **staging.** Let's Encrypt runs a second, identical system at a different address with far higher limits, aimed exactly at people who are still getting it wrong. It issues real certificates from a root nothing trusts, which means everything works end to end and the browser still warns you. That warning is the correct outcome of a successful staging run, not a failure. **Get it working on staging first. Every time. Including the time you are sure you have it right.** Step 4 does staging in full before it goes anywhere near production, and that ordering is not padding. --- ## Step 1 — Check Part 6 is still good **Why:** the same reason as every part. If something drifted, find out now. **Safe to run:** everything here only reads. ```bash lxc info | head -3 # wakes LXD if it's asleep, same trick as before sleep 3 lxc list # expect five RUNNING rows, plus the STOPPED template kubectl get nodes # Expect: five Ready. kubectl get pods -n metallb-system # Expect: one controller, five speakers, all Running. kubectl get svc -n ingress-nginx ingress-nginx-controller # Expect: TYPE LoadBalancer, EXTERNAL-IP around 10.99.99.240. Note the address. ``` Now the Part 6 machinery, which everything in this part goes through: ```bash kubectl get pods -n argocd # Expect: all Running. The server, repo-server, application-controller, # redis and the dex/notifications pods if you kept them. kubectl get applications -n argocd # Expect: your Part 6 Applications, SYNC STATUS Synced, HEALTH STATUS Healthy. # Anything OutOfSync here is a Part 6 problem. Fix it before continuing. ``` And the application you are about to put a certificate on: ```bash curl --max-time 5 -s -o /dev/null -w 'whoami over http: %{http_code}\n' \ http://whoami.k3s.linsnotes.com # Expect: 200. curl --max-time 5 -s -o /dev/null -w 'argocd over http: %{http_code}\n' \ http://argocd.k3s.linsnotes.com # Expect: 200. ``` Both plain HTTP, both working, both about to change. > **One thing to have ready before Step 2.** You will be editing your Git repository, not the cluster. Make sure you have `kuibinlin/k3s-lab-gitops` cloned on the host and that you can push to it. Every change from here on goes `edit file, commit, push, watch Argo CD`. If you find yourself typing `kubectl apply`, stop and check whether Argo CD owns that object. {: .prompt-tip } --- ## Step 2 — Install cert-manager, through Git **What this does.** It adds cert-manager to the cluster. cert-manager is a controller that watches for `Certificate` objects and does everything needed to make them real: generate a key, build a CSR, talk to a CA, answer the challenge, write the result into a Secret, and renew it before it expires. You describe what you want. It does the work, forever. ### Why this cannot be a `kubectl apply` Part 6 turned on automated sync with `selfHeal: true` and `prune: true`. Those two settings have a consequence that this part is the first to run into properly. - **`selfHeal: true`** means Argo CD watches the live cluster and puts back anything that drifts from Git. Change something by hand and it gets reverted, usually within a couple of minutes. - **`prune: true`** means Argo CD deletes objects that exist in the cluster but not in Git, inside the paths it manages. So a bare `kubectl apply -f cert-manager.yaml` would be, at best, an object Argo CD does not know about, and at worst something it deletes. More importantly, it would be invisible: in six months the cluster would have a component nobody can point at a source for, which is the exact problem Part 6 was written to solve. **This is why Part 6 came before Part 7 and not after.** Every component from here on goes into the repository first. The cluster is the output, not the input. ### The CRD problem, honestly cert-manager does not just install a program. It teaches Kubernetes four new object types: `Issuer`, `ClusterIssuer`, `Certificate`, and `CertificateRequest`, plus `Order` and `Challenge` for ACME. Those definitions are **CustomResourceDefinitions**, or CRDs. A CRD has to exist before Kubernetes will accept an object of that type. Write a `ClusterIssuer` before the `ClusterIssuer` CRD is installed and the API server rejects it outright: it has never heard of that kind. That is a plain ordering problem, and with Argo CD there are two ways to handle it. **Install the CRDs as part of the same Application.** The cert-manager Helm chart can install its own CRDs with one setting, and that is the simplest correct answer. It is what this guide does. **Or use sync waves.** A sync wave is a number you put in an annotation, and Argo CD applies everything in wave 0, waits for it to be healthy, then does wave 1, then wave 2. Lower numbers, including negative ones, go first. That is the whole idea: ```yaml metadata: annotations: argocd.argoproj.io/sync-wave: "1" # applied after wave 0 ``` The ordering problem shows up twice here. Inside one Application, the chart's CRDs and its Deployment need ordering, and the chart handles that itself. Across two Applications, `cert-manager-config` cannot be applied until `cert-manager` has installed the CRDs, and that is what the wave annotations below describe. > **Sync waves only order things that Argo CD is applying.** Part 6 applies each Application by hand with `kubectl apply -f apps/.yaml`, so nothing is reading these annotations yet, and the ordering is down to the order you type the two commands. Put them in anyway: they cost nothing, they record the intent, and they start working the day you build the App of Apps that Part 6's Step 6 described. > > There is also a safety net either way. Argo CD retries a failed sync on its own, so `cert-manager-config` applied too early fails once with "no matches for kind ClusterIssuer" and then goes Synced a minute later when the CRDs land. Ugly, but self-correcting. {: .prompt-info } ### Write the two Applications In your repo, add `apps/cert-manager.yaml`: ```yaml apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: cert-manager namespace: argocd annotations: argocd.argoproj.io/sync-wave: "0" # first spec: project: default source: repoURL: https://charts.jetstack.io # the cert-manager Helm repo chart: cert-manager targetRevision: v1.20.1 # PIN IT. See the note below. helm: parameters: - name: crds.enabled # install the CRDs with the chart value: "true" destination: server: https://kubernetes.default.svc namespace: cert-manager syncPolicy: automated: prune: true selfHeal: true syncOptions: - CreateNamespace=true # make the namespace if it's missing - ServerSideApply=true # see the note below ``` Then `apps/cert-manager-config.yaml`, which holds the issuers you write in Steps 3 and 4: ```yaml apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: cert-manager-config namespace: argocd annotations: argocd.argoproj.io/sync-wave: "1" # after cert-manager spec: project: default source: repoURL: https://github.com/kuibinlin/k3s-lab-gitops.git targetRevision: main path: infra/cert-manager-config destination: server: https://kubernetes.default.svc namespace: cert-manager syncPolicy: automated: prune: true selfHeal: true ``` Two of those lines deserve an explanation. > **`targetRevision: v1.20.1` is an example, not advice.** cert-manager releases often, and by the time you read this the current version will be different. Check the [cert-manager releases page](https://cert-manager.io/docs/releases/) and use what is current and supported. Pinning a version is the point; pinning *this* version is not. The same goes for every version number in this series. {: .prompt-warning } > **`ServerSideApply=true` is there because cert-manager's CRDs are enormous.** Argo CD's default way of applying an object stores a copy of the whole thing in an annotation, and Kubernetes caps annotations at about 256 kB. Some cert-manager CRDs are bigger than that, so the apply fails with a message about the annotation being too long. Server-side apply does not use that annotation at all, so the problem disappears. If you ever meet "metadata.annotations: Too long" on a big CRD, this is the fix. {: .prompt-tip } Commit both files, then create the folder the second one points at. Leave the folder empty for now: Step 3 puts the first issuer in it, and `cert-manager-config` does not get applied until then. ```bash cd ~/k3s-lab-gitops # where Part 6 put it mkdir -p infra/cert-manager-config git add apps/cert-manager.yaml apps/cert-manager-config.yaml git commit -m "Add cert-manager and its config Application" git push ``` ### Watch it arrive Application objects still reach the cluster by hand, exactly as in Part 6. Only `cert-manager` for now: ```bash kubectl apply -f apps/cert-manager.yaml # Expect: application.argoproj.io/cert-manager created. # Do NOT apply cert-manager-config yet. Its folder is empty, and it is # Step 3's job. ``` Then watch: ```bash kubectl get applications -n argocd # Expect: cert-manager appears, and because syncPolicy.automated is in the # file, it syncs on its own. Give it a minute to reach Synced and Healthy. # Pulling three container images on first install is most of that minute. kubectl get pods -n cert-manager # Expect: three Running pods. Names vary, but the jobs are always the same: # cert-manager the controller that does the work # cert-manager-webhook validates your Certificate and Issuer YAML # cert-manager-cainjector wires CA data into the webhook's own config ``` Confirm the new object types exist, which is the real test that the CRDs landed: ```bash kubectl api-resources | grep cert-manager.io # Expect: certificates, certificaterequests, issuers, clusterissuers, # and under acme.cert-manager.io, challenges and orders. ``` Six new kinds of object your cluster understands. Nothing has been issued yet, because nothing has been asked for. > **The webhook is a real dependency, not decoration.** It checks every `Certificate` and `Issuer` you submit and rejects bad ones with a readable message. If it is not running, your applies fail with a connection error mentioning `webhook.cert-manager.io`, which looks like a networking problem and is not. Give it thirty seconds after install before you apply anything. {: .prompt-info } --- ## Step 3 — Become your own certificate authority **What this does.** It creates a root CA inside the cluster, uses it to issue a certificate for `whoami.k3s.linsnotes.com`, puts that certificate on the Ingress, and then walks through exactly why the browser complains and what makes the complaint stop. **Why do this at all, when Step 4 gets you real certificates?** Because every moving part is visible and none of them can fail for a reason outside your control. No API tokens, no rate limits, no internet. If something goes wrong here it is your YAML, and that is a much better place to learn. It is also genuinely useful on its own: internal services that no browser will ever visit are often signed exactly this way. ### Three objects, and why it takes three This trips everyone up the first time, so here it is before the YAML. Setting up a CA in cert-manager needs three objects, because there are three genuinely different things happening. | Object | Kind | What it is for | |---|---|---| | `selfsigned` | `ClusterIssuer` with `selfSigned: {}` | An issuer that signs certificates with their own key. It is used exactly once: to sign the CA's own certificate. Nothing else uses it. | | `lab-root-ca` | `Certificate` with `isCA: true` | The CA's own certificate and private key. `isCA: true` is the flag that says "this certificate is allowed to sign others." Without it, it would be an ordinary certificate that cannot sign anything. | | `lab-ca` | `ClusterIssuer` with `ca:` | The issuer you actually use. It signs certificates using the key from `lab-root-ca`. | Read down the table and the chicken-and-egg resolves itself: a root CA has nothing above it to sign it, so it has to sign itself, and `selfSigned` is the thing that performs that one act. After that, `lab-ca` does all the real work. Create `infra/cert-manager-config/selfsigned-ca.yaml`: ```yaml # 1. The bootstrap issuer. Signs one thing: the CA below. apiVersion: cert-manager.io/v1 kind: ClusterIssuer metadata: name: selfsigned spec: selfSigned: {} --- # 2. The CA's own certificate and key, stored in a Secret. apiVersion: cert-manager.io/v1 kind: Certificate metadata: name: lab-root-ca namespace: cert-manager # see the note on namespaces below spec: isCA: true # this is what makes it able to sign commonName: k3s-lab-root-ca secretName: lab-root-ca # the Secret the key pair lands in duration: 87600h # 10 years. It is your CA; you decide. privateKey: algorithm: ECDSA size: 256 issuerRef: name: selfsigned kind: ClusterIssuer group: cert-manager.io --- # 3. The issuer you will actually name in your Ingress rules. apiVersion: cert-manager.io/v1 kind: ClusterIssuer metadata: name: lab-ca spec: ca: secretName: lab-root-ca # signs using the key from step 2 ``` > **A `ClusterIssuer` looks for its Secrets in one fixed namespace, and it is not the one you might expect.** A `ClusterIssuer` has no namespace of its own, so cert-manager reads any Secret it references from what it calls the cluster resource namespace, which defaults to `cert-manager`. That is why `lab-root-ca` is created in `cert-manager` above, and it is why the Cloudflare token in Step 4 goes there too. Put the Secret in `default` and the issuer will report that it cannot find it, while the Secret sits there in plain sight. {: .prompt-warning } Commit it, and now the folder has something in it, so create the Application that watches it: ```bash cd ~/k3s-lab-gitops git add infra/cert-manager-config/selfsigned-ca.yaml git commit -m "Add a self-signed root CA and a CA ClusterIssuer" git push kubectl apply -f apps/cert-manager-config.yaml # Expect: created. This is the last Application object you create by hand in # this part. Everything after it is a commit. kubectl get applications -n argocd cert-manager-config # Expect: Synced and Healthy within a minute. If it says "no matches for kind # ClusterIssuer", cert-manager's CRDs are not in yet; wait and look again. ``` Check both issuers: ```bash kubectl get clusterissuer # Expect: selfsigned and lab-ca, both READY True. # lab-ca goes Ready only after lab-root-ca has issued, so it can lag by a few # seconds. If it stays False, describe it; the message is usually plain. kubectl get certificate -n cert-manager lab-root-ca # Expect: READY True. kubectl get secret -n cert-manager lab-root-ca # Expect: TYPE kubernetes.io/tls, DATA 3 (tls.crt, tls.key and ca.crt). ``` You now have a certificate authority. Look at what it made: ```bash kubectl get secret -n cert-manager lab-root-ca \ -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -text | head -15 # Expect: Issuer and Subject BOTH reading CN=k3s-lab-root-ca. That is what # self-signed means, visible in one line: it signed itself. # Look further down for "CA:TRUE" under X509v3 Basic Constraints. That is # isCA doing its job. ``` ### Issue a certificate and put it on the Ingress You do not have to write a `Certificate` object for this. cert-manager watches Ingress resources, and if it sees the right annotation it creates the `Certificate` for you, using the hostnames from the rule. That shortcut is called the ingress-shim and it is how most people use cert-manager day to day. Two changes to `infra/whoami/ingress.yaml`, the file Part 6 wrote: ```yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: whoami namespace: default annotations: cert-manager.io/cluster-issuer: lab-ca # CHANGE 1: which issuer to use spec: ingressClassName: nginx tls: # CHANGE 2: the whole tls block - hosts: - whoami.k3s.linsnotes.com secretName: whoami-tls # where the result gets stored rules: - host: whoami.k3s.linsnotes.com http: paths: - path: / pathType: Prefix backend: service: name: whoami port: number: 80 ``` Those two changes work together. The `tls` block tells ingress-nginx "serve HTTPS for this host, using the certificate in the Secret `whoami-tls`." The annotation tells cert-manager "that Secret does not exist yet, go and fill it using `lab-ca`." ```bash git add infra/whoami/ingress.yaml git commit -m "Serve whoami over HTTPS with the lab CA" git push ``` Watch it happen: ```bash kubectl get certificate -n default -w # Expect: whoami-tls appears, READY False for a moment, then True. # Ctrl-C when it goes True. With a CA issuer this takes about a second, # because no external service is involved. kubectl get secret -n default whoami-tls # Expect: TYPE kubernetes.io/tls. ``` Nobody created that `Certificate` object. The ingress-shim did, from your annotation: ```bash kubectl get certificate -n default whoami-tls -o yaml | grep -A5 ownerReferences # Expect: an owner reference pointing at the Ingress named whoami. # That is the link: delete the Ingress and the Certificate goes with it. ``` ### Look at it, and understand the warning ```bash curl --max-time 5 -sI https://whoami.k3s.linsnotes.com ``` Expect that to **fail**, with a message about a self-signed certificate in the chain, or an unknown issuer. This is not a bug. This is the whole lesson. `curl` received a perfectly valid certificate for the right hostname, signed by `k3s-lab-root-ca`. Then it went looking for `k3s-lab-root-ca` in the list of authorities your system trusts, did not find it, and refused. **It is not saying the certificate is wrong. It is saying it has no reason to believe you.** Prove that by telling `curl` to skip the check: ```bash curl --max-time 5 -k -sI https://whoami.k3s.linsnotes.com # -k means "do not verify". Expect: HTTP/2 200. The encryption was working # the entire time. Only the trust decision was failing. ``` And look at what it actually served: ```bash echo | openssl s_client -connect 10.99.99.240:443 \ -servername whoami.k3s.linsnotes.com 2>/dev/null \ | openssl x509 -noout -subject -issuer -dates # Expect: subject CN=whoami.k3s.linsnotes.com, issuer CN=k3s-lab-root-ca, # and dates roughly 90 days apart (cert-manager's default for a leaf). # Use your own ingress-nginx address if it is not .240. ``` Open `https://whoami.k3s.linsnotes.com` in the host's browser and you get the full page of warning. Same reason, more red. > **ingress-nginx starts redirecting HTTP to HTTPS the moment an Ingress has a `tls` block.** So `http://whoami.k3s.linsnotes.com` now bounces to `https://`, which you did not ask for and which is almost always what you want. If you need to turn it off for one Ingress, the annotation is `nginx.ingress.kubernetes.io/ssl-redirect: "false"`. {: .prompt-info } ### Now trust your own CA Nothing about the certificate needs to change. You need to add `k3s-lab-root-ca` to the host's list of trusted authorities, which is one file and one command. Pull the CA certificate out of the cluster: ```bash kubectl get secret -n cert-manager lab-root-ca \ -o jsonpath='{.data.tls\.crt}' | base64 -d > /tmp/k3s-lab-root-ca.crt openssl x509 -in /tmp/k3s-lab-root-ca.crt -noout -subject # Expect: subject=CN = k3s-lab-root-ca. Check before you install it. ``` Install it on the host. On Ubuntu, the trust store is a directory plus a command that rebuilds the bundle from it. The `.crt` extension is required; files with other extensions are silently ignored: ```bash sudo cp /tmp/k3s-lab-root-ca.crt /usr/local/share/ca-certificates/k3s-lab-root-ca.crt sudo update-ca-certificates # Expect: a line saying "1 added". If it says 0 added, check the extension. ``` Try again, with no `-k`: ```bash curl --max-time 5 -sI https://whoami.k3s.linsnotes.com # Expect: HTTP/2 200, no error, no flag. The certificate did not change. # Your machine's opinion of who signed it did. ``` That is the entire mechanism of trust on the internet, done by hand in two commands. Every padlock you have ever seen is this, with the copying step done by whoever built your operating system. > **Your browser may not agree, and that is not your mistake.** `curl` and most command line tools read the system trust store you just updated. Firefox ships and uses its own store on every platform, so it will keep warning until you import the certificate through its own settings. Chrome and Edge use the system store on Windows and macOS, and on Linux they read a per-user NSS database that `update-ca-certificates` does not touch. So "curl is happy, the browser is not" is normal and means the system store worked. Adding it to each browser is a per-browser chore, and for a lab it is fine to skip and keep using `-k` in the browser. {: .prompt-warning } To undo this later, delete the file and rebuild: ```bash # Not now. This is the undo, for when you are finished with the lab. # sudo rm /usr/local/share/ca-certificates/k3s-lab-root-ca.crt # sudo update-ca-certificates --fresh ``` --- ## Step 4 — Real certificates from Let's Encrypt, over Cloudflare DNS-01 **What this does.** It replaces your own CA with a public one, and gets a certificate that every browser on earth already trusts, for a service running on a private address that nothing outside your house can reach. That combination is worth pausing on. You are about to get a genuine, publicly trusted certificate for a machine with no public address, no port forwarding, and no inbound path of any kind. That works because of exactly one thing: DNS-01 proves you own the *name*, and never asks about the *machine*. ### 4a. Make a Cloudflare API token cert-manager needs permission to create and delete TXT records in your zone. Cloudflare offers two ways to do that, and only one of them is acceptable. - The **Global API Key** is one secret that can do everything to every zone and every setting on your account, including deleting it. Some old guides still use it. Do not. - An **API token** is scoped: you choose what it may do and which zones it may touch, you can see when it was last used, and you can revoke it on its own without changing anything else. In the Cloudflare dashboard, go to **My Profile**, then **API Tokens**, then **Create Token**. Choose **Create Custom Token** and set it up like this: | Field | Value | |---|---| | Token name | something you will recognise, for example `cert-manager-k3s-lab` | | Permissions, row 1 | **Zone**, **DNS**, **Edit** | | Permissions, row 2 | **Zone**, **Zone**, **Read** | | Zone Resources | **Include**, **Specific zone**, `linsnotes.com` | | TTL | leave it, or set an expiry if you like being reminded | `Zone:DNS:Edit` is the one that writes the challenge record. `Zone:Zone:Read` is there so cert-manager can look up which zone a name belongs to; without it you get an error naming `com.cloudflare.api.account.zone.list`, which reads like a bug and is really a missing permission. Create it, and **copy the token now.** Cloudflare shows it exactly once. ### 4b. Put the token in the cluster, and be honest about where it is not going ```bash kubectl create secret generic cloudflare-api-token \ --namespace cert-manager \ --from-literal=api-token='PASTE_YOUR_TOKEN_HERE' kubectl get secret -n cert-manager cloudflare-api-token # Expect: TYPE Opaque, DATA 1. Never echo the value back. ``` Note the namespace: `cert-manager`, for the cluster resource namespace reason from Step 3. > **This one thing does not go into Git, and that needs saying plainly.** Part 6 put your whole cluster into a public GitHub repository, which was the right call for everything in it. This is the first object that must not follow. A Kubernetes Secret is base64, not encryption; anyone who reads the file reads the token, and a token with `Zone:DNS:Edit` on your domain is enough to take over your DNS. Public repositories are scraped for exactly this within minutes. > > So you created it by hand, outside Git, and your repository now describes 95 percent of your cluster instead of 100. That gap is real and you should feel it, because the grown-up answers to it are a genuine subject: > > - **Sealed Secrets** encrypts the Secret with a key only your cluster holds. The encrypted file is safe to commit; nothing else can read it. > - **External Secrets Operator** leaves the value in a real secret manager (Vault, AWS Secrets Manager, 1Password) and puts only a reference in Git. > - **SOPS**, usually with age or a KMS, encrypts just the values inside a YAML file so the structure stays readable in diffs. > > Any of the three closes the gap. All three are more setup than this part has room for. Doing it by hand and knowing why it is a shortcut beats doing it by hand and not noticing. {: .prompt-warning } If you ever think the token leaked, revoke it in the Cloudflare dashboard. That is instant and breaks nothing else. ### 4c. The staging ClusterIssuer Add `infra/cert-manager-config/letsencrypt-staging.yaml`: ```yaml apiVersion: cert-manager.io/v1 kind: ClusterIssuer metadata: name: letsencrypt-staging spec: acme: # The STAGING address. Note "staging" in the hostname. This one line is # the whole difference between practice and burning a rate limit. server: https://acme-staging-v02.api.letsencrypt.org/directory email: you@example.com # expiry warnings go here. Use a real one. privateKeySecretRef: name: letsencrypt-staging-account-key # cert-manager creates this itself solvers: - dns01: cloudflare: apiTokenSecretRef: name: cloudflare-api-token # the Secret from 4b key: api-token selector: dnsZones: - linsnotes.com # use this solver for this zone ``` Reading the parts that matter: - **`server`** is what makes this staging. Everything else is identical to production. Check this line twice. - **`privateKeySecretRef`** is not your certificate's key. It is the key identifying your *account* with Let's Encrypt. cert-manager generates it on first use and reuses it after. You do not create this Secret. - **`solvers`** is a list, so one issuer can prove different names in different ways. Here there is one, and `selector.dnsZones` restricts it to `linsnotes.com`, which matters the moment you own a second domain. - **`apiTokenSecretRef`** uses the token form. There is an older `apiKeySecretRef` that needs an `email` field alongside it. That is the Global API Key path. Do not use it. ```bash git add infra/cert-manager-config/letsencrypt-staging.yaml git commit -m "Add Let's Encrypt staging ClusterIssuer with Cloudflare DNS-01" git push ``` ```bash kubectl get clusterissuer letsencrypt-staging # Expect: READY True within a few seconds. This means cert-manager registered # an ACME account with staging. It does NOT mean the token works; nothing has # tried to write a DNS record yet. ``` ### 4d. Issue a staging certificate, and watch the whole chain Switch `whoami` to the staging issuer. In `infra/whoami/ingress.yaml`, change one word: ```yaml cert-manager.io/cluster-issuer: letsencrypt-staging # was lab-ca ``` The old certificate will not be replaced while it is still valid, so delete the Secret at the same time to force a fresh issuance. This is a deliberate exception to the "everything through Git" rule, because a Secret's *contents* are not in Git and never were: ```bash git add infra/whoami/ingress.yaml git commit -m "Move whoami to Let's Encrypt staging" git push # Wait for Argo CD to sync the Ingress, then: kubectl delete secret -n default whoami-tls ``` Now watch, because this is where the interesting part is. Six object types are involved and each one has a job: ```bash kubectl get certificate,certificaterequest,order,challenge -n default ``` | Object | What it means | |---|---| | `Certificate` | Your request, as a standing wish. It stays for the life of the service and drives every renewal. | | `CertificateRequest` | One attempt at getting a certificate. A new one appears at every issuance and renewal. | | `Order` | The ACME conversation with Let's Encrypt for that attempt. | | `Challenge` | One name being proved. One `Challenge` per hostname in the certificate. | Watch the `Challenge` in particular, because that is where the Cloudflare token is used for the first time: ```bash kubectl get challenge -n default -w # Expect: one challenge appears, STATE pending, then valid, then it vanishes. # Ctrl-C once it is gone. Typically 30 to 90 seconds, mostly DNS propagation. ``` While it is pending, look at the record from outside: ```bash dig +short TXT _acme-challenge.whoami.k3s.linsnotes.com @1.1.1.1 # Expect: a quoted random string, while the challenge is pending. # Run it again after the challenge disappears and expect nothing: # cert-manager cleans the record up. ``` **That command is the whole idea made visible.** A public DNS resolver, anywhere in the world, can read a record proving you control `linsnotes.com`. Nothing anywhere can reach `whoami.k3s.linsnotes.com`, and nothing needs to. Then: ```bash kubectl get certificate -n default whoami-tls # Expect: READY True. echo | openssl s_client -connect 10.99.99.240:443 \ -servername whoami.k3s.linsnotes.com 2>/dev/null \ | openssl x509 -noout -subject -issuer -dates # Expect: issuer naming Let's Encrypt's STAGING authority. The exact name # changes over time; the giveaway is the word for a fake or test authority # in it. Dates should be about 90 days apart. ``` ### 4e. When a challenge sticks, which it will This is the most useful debugging path in the part, so learn it now while the stakes are a staging certificate. A challenge that stays `pending` for more than a couple of minutes is stuck. Ask it why: ```bash kubectl describe challenge -n default # Read the Events at the bottom. cert-manager writes plain messages here. ``` | Message mentions | What it means | Fix | |---|---|---| | `requires permission ... zone.list` | The token lacks `Zone:Zone:Read` | Add the second permission row in Cloudflare | | `Invalid request headers`, or a 400 or 403 from Cloudflare | Wrong or revoked token, or a stray newline in it | Recreate the Secret. Use `--from-literal`, not a file. | | `could not find the secret`, `secret not found` | The Secret is in the wrong namespace | It must be in `cert-manager` for a `ClusterIssuer` | | `Waiting for DNS-01 challenge propagation`, repeatedly | cert-manager wrote the record but cannot see it yet | Usually just wait. If it never clears, see the nameserver note below. | | `no such host`, or an error naming the zone | cert-manager picked the wrong zone for the name | Check the zone really is `linsnotes.com` in Cloudflare | If `describe` is not enough, read the controller: ```bash kubectl logs -n cert-manager deploy/cert-manager --tail=100 # Follow it live with -f while you retry. Cloudflare's own error text # is passed through here, which is usually the fastest answer. ``` And check the record independently, because that separates "cert-manager could not write it" from "Let's Encrypt could not read it": ```bash dig +short TXT _acme-challenge.whoami.k3s.linsnotes.com @1.1.1.1 dig +short TXT _acme-challenge.whoami.k3s.linsnotes.com @8.8.8.8 ``` - **Nothing from either:** cert-manager did not write the record. Token or permissions. - **Something from both, still stuck:** the write worked. Wait longer, or check what cert-manager itself is querying. > **The self-check can be the thing that is stuck.** Before it tells Let's Encrypt to go ahead, cert-manager checks that it can see the TXT record itself, using the DNS servers from its pod's `/etc/resolv.conf`. In this lab that is CoreDNS, forwarding to whatever your host uses. If your host runs something that answers for your own domain differently (a Pi-hole with local records, a split DNS setup), the self-check can fail forever while the world sees the record perfectly. The fix is to tell cert-manager to use public resolvers instead, by adding these to the cert-manager controller's arguments through the Helm values in `apps/cert-manager.yaml`: > > `--dns01-recursive-nameservers-only` and `--dns01-recursive-nameservers=1.1.1.1:53,8.8.8.8:53` > > Do not add these pre-emptively. Add them only if the two `dig` commands show the record and cert-manager still says it is waiting for propagation. {: .prompt-tip } **Where the rate limits bite.** Every one of those failures with the same set of names counts against the 5-per-week duplicate limit if the order actually reached Let's Encrypt. On staging that limit is far higher, which is precisely why you are here and not in production. ### 4f. It worked, and the browser still warns. Good. ```bash curl --max-time 5 -sI https://whoami.k3s.linsnotes.com # Expect: an error about an unknown or untrusted issuer, exactly like Step 3 # before you installed the CA. ``` **This is the correct result of a successful staging run.** Let's Encrypt staging issues from a root deliberately kept out of every trust store, precisely so that nobody can accidentally run a production site on practice certificates. The certificate is real, the chain is real, the challenge genuinely proved you own the domain. The only thing missing is the one thing staging is designed never to give you. So the checklist for "staging worked" is not "the padlock appeared." It is: ```bash kubectl get certificate -n default whoami-tls # READY True. kubectl get challenge -n default # No resources found. The challenge completed and was cleaned up. curl --max-time 5 -k -sI https://whoami.k3s.linsnotes.com # HTTP/2 200 with -k. Serving fine; only trust is missing. ``` Three greens and an expected warning. That is the whole staging run, and everything after it is one word. ### 4g. Production Add `infra/cert-manager-config/letsencrypt-prod.yaml`. It is the staging file with two names changed: ```yaml apiVersion: cert-manager.io/v1 kind: ClusterIssuer metadata: name: letsencrypt-prod spec: acme: server: https://acme-v02.api.letsencrypt.org/directory # no "staging" email: you@example.com privateKeySecretRef: name: letsencrypt-prod-account-key # a SEPARATE account key solvers: - dns01: cloudflare: apiTokenSecretRef: name: cloudflare-api-token key: api-token selector: dnsZones: - linsnotes.com ``` The account key must have a different name from the staging one. Staging and production are separate services with separate accounts; sharing a key Secret between them is asking for a confusing failure later. Point `whoami` at it, in `infra/whoami/ingress.yaml`: ```yaml cert-manager.io/cluster-issuer: letsencrypt-prod ``` ```bash git add infra/cert-manager-config/letsencrypt-prod.yaml infra/whoami/ingress.yaml git commit -m "Move whoami to Let's Encrypt production" git push # After Argo CD syncs: kubectl delete secret -n default whoami-tls ``` ```bash kubectl get challenge -n default -w # Same shape as staging. Ctrl-C when it clears. kubectl get certificate -n default whoami-tls # Expect: READY True. ``` And now, with no flags: ```bash curl --max-time 5 -sI https://whoami.k3s.linsnotes.com # Expect: HTTP/2 200. No warning. No -k. echo | openssl s_client -connect 10.99.99.240:443 \ -servername whoami.k3s.linsnotes.com 2>/dev/null \ | openssl x509 -noout -issuer -dates # Expect: an issuer naming Let's Encrypt, and about 90 days between the dates. ``` Open it in a browser and you get a padlock, with no exception added, on `10.99.99.240`. **Stop and look at what that address is.** It is a MetalLB address on a bridge that exists only inside one Ubuntu machine. Your router does not route to it from the internet, your ISP would drop it, and no port is forwarded anywhere. And the padlock is genuine, because Let's Encrypt was never asked whether it could reach your server. It was asked whether you own `linsnotes.com`, you proved it with a TXT record, and that was the end of the conversation. > **One certificate for the whole lab, if you want it.** DNS-01 can issue wildcards, which HTTP-01 cannot. One `Certificate` covers every name you will ever add, and it takes one ACME order instead of one per service, which is friendly to the rate limits. The ingress-shim annotation cannot do this, because it builds the name list from the Ingress rules, so you write the `Certificate` yourself and then just reference the Secret from the `tls` block with no annotation: > > ```yaml > apiVersion: cert-manager.io/v1 > kind: Certificate > metadata: > name: lab-wildcard > namespace: default > spec: > secretName: lab-wildcard-tls > issuerRef: > name: letsencrypt-prod > kind: ClusterIssuer > dnsNames: > - "*.k3s.linsnotes.com" > - "k3s.linsnotes.com" # the wildcard does NOT cover the bare name > ``` > > One catch worth knowing: a Secret is namespaced, so an Ingress in another namespace cannot use it. Either issue the wildcard once per namespace, or use trust-manager or a reflector to copy it around. {: .prompt-tip } --- ## Step 5 — Put Argo CD behind HTTPS **What this does.** It closes a loop that Part 6 left open on purpose. ### What `server.insecure` actually did Part 6 set `server.insecure: "true"` in the `argocd-cmd-params-cm` ConfigMap, and described it as a shortcut. Here is what it really does. Argo CD's API server, by default, serves HTTPS itself, using a self-signed certificate it generates on startup. It also redirects any plain HTTP request to HTTPS. Put that behind an Ingress that speaks plain HTTP to its backends, and the two disagree: the Ingress sends HTTP, Argo CD answers "go to HTTPS", the Ingress passes that on, the browser comes back to the same Ingress, and you get a redirect loop or a mess of 307s. `server.insecure: "true"` tells Argo CD to serve plain HTTP and stop redirecting. The Ingress then handles TLS on its own, which is the standard arrangement: **terminate TLS at the edge, speak plain HTTP inside the cluster.** In Part 6 that was a shortcut, because there was no TLS anywhere, so `insecure` meant genuinely insecure end to end. With a real certificate at the Ingress it stops being a shortcut and becomes the ordinary way to run this. Nothing about the setting changed. What changed is what sits in front of it. ### Do it This is the one Ingress in the lab that Argo CD does **not** manage. Part 6 put it in `bootstrap/argocd-server-ingress.yaml` and applied it by hand, on purpose: the thing that lets you reach Argo CD should not depend on Argo CD being healthy. So this edit is a file plus a `kubectl apply`, not a commit and a wait. Rewrite the file with three additions: ```bash cat > ~/k3s-lab-gitops/bootstrap/argocd-server-ingress.yaml <<'EOF' apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: argocd-server namespace: argocd annotations: nginx.ingress.kubernetes.io/backend-protocol: "HTTP" # unchanged cert-manager.io/cluster-issuer: letsencrypt-prod # NEW: get a cert nginx.ingress.kubernetes.io/force-ssl-redirect: "true" # NEW: see below spec: ingressClassName: nginx tls: # NEW: the tls block - hosts: - argocd.k3s.linsnotes.com secretName: argocd-tls rules: - host: argocd.k3s.linsnotes.com http: paths: - path: / pathType: Prefix backend: service: name: argocd-server port: number: 80 # still the HTTP port on the Service EOF kubectl apply -f ~/k3s-lab-gitops/bootstrap/argocd-server-ingress.yaml # Expect: ingress.networking.k8s.io/argocd-server configured. ``` **`force-ssl-redirect` is the annotation Part 6 told you not to add yet.** Back then it would have built a redirect loop, because there was no certificate to redirect *to*. Now there is one, so it does the sensible thing: anyone typing the bare hostname is moved to HTTPS instead of quietly logging in over plain text. This is the promise from Part 6's Step 4, kept. Commit the file too, so the repo still describes the cluster, even though nothing syncs it: ```bash cd ~/k3s-lab-gitops git commit -am "Put the Argo CD Ingress behind TLS" git push ``` > **You are changing Argo CD's own front door, and that is worth a moment of care.** The `argocd-server` pod is untouched; only the Ingress in front of it changed, so the page you are watching may blink or need a reload. But this is the one object where a mistake locks you out of the web interface. Your escape hatch is `kubectl`, which talks to the API server on `10.99.99.11:6443` and has never gone through the Ingress at all. Nothing here can take that away. {: .prompt-warning } ```bash kubectl get certificate -n argocd argocd-tls # Expect: READY True after a challenge cycle, about a minute. curl --max-time 5 -sI https://argocd.k3s.linsnotes.com # Expect: HTTP/2 200. No -k. curl --max-time 5 -sI http://argocd.k3s.linsnotes.com | head -1 # Expect: a 308 redirect. That is force-ssl-redirect working. ``` Log in at `https://argocd.k3s.linsnotes.com` with a padlock. Your GitOps controller's password no longer travels your home network as readable text. ### Where you would do this differently TLS now stops at ingress-nginx. Between ingress-nginx and the `argocd-server` pod, traffic is plain HTTP across the cluster network, which in this lab is a flannel vxlan tunnel between LXD containers on one machine. For a home lab that is fine. In a regulated environment, or on a cluster where you do not trust every workload sharing the network, you would want that last hop encrypted too. Two ways: - **Let Argo CD serve TLS again.** Remove `server.insecure`, and tell ingress-nginx to speak HTTPS to the backend with `nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"`. Argo CD's own self-signed certificate is fine for that hop, because ingress-nginx is not verifying a public name. - **Put a service mesh in.** Istio or Linkerd encrypt every pod to pod connection with certificates they manage, without any application knowing. That is the real answer at scale, and a much bigger commitment than one annotation. Naming the tradeoff is the point. `server.insecure` behind a real certificate is a normal, defensible arrangement, not something to be quietly embarrassed about, as long as you can say where its edge is. --- ## Step 6 — Break it on purpose Every part of this series ends by taking something away. Part 3 killed a master, Part 4 killed a worker, Part 5 killed the node holding the load balancer address, Part 6 deleted something Argo CD was managing. Here you delete a certificate and find out whether anything is watching. ### 6a. Delete a TLS Secret and watch it come back **Do this on a CA-issued certificate, not the Let's Encrypt one, and the reason matters.** Deleting the `whoami-tls` Secret would trigger a real ACME order against production, which counts against the "5 certificates for the same set of names per week" limit. Do that four more times out of curiosity and you are locked out until the window rolls. The `lab-ca` issuer has no such limit, costs nothing, and demonstrates exactly the same mechanism. Add `infra/cert-manager-config/selftest-cert.yaml`, a certificate attached to nothing: ```yaml apiVersion: cert-manager.io/v1 kind: Certificate metadata: name: selftest namespace: cert-manager spec: secretName: selftest-tls commonName: selftest.k3s.linsnotes.com dnsNames: - selftest.k3s.linsnotes.com issuerRef: name: lab-ca # your own CA. Free, instant, no limits. kind: ClusterIssuer ``` ```bash git add infra/cert-manager-config/selftest-cert.yaml git commit -m "Add a throwaway certificate for the renewal experiment" git push ``` ```bash kubectl get certificate -n cert-manager selftest # Expect: READY True. kubectl get secret -n cert-manager selftest-tls -o jsonpath='{.metadata.uid}'; echo # Note this UID. It identifies this exact Secret object. ``` Now destroy it: ```bash kubectl delete secret -n cert-manager selftest-tls kubectl get secret -n cert-manager selftest-tls # Run this immediately. You may catch a "not found", or the Secret may # already be back. Either is correct. sleep 5 kubectl get secret -n cert-manager selftest-tls -o jsonpath='{.metadata.uid}'; echo # Expect: a DIFFERENT UID from before. Not restored. Reissued. ``` **Nobody did anything.** cert-manager watches the Secret named by every `Certificate` it manages. The Secret vanished, so the `Certificate` no longer matched what was asked for, and cert-manager closed the gap by issuing again. Same reconciliation loop as Kubernetes replacing a deleted pod, and the same one Argo CD used in Part 6 to put back the thing you deleted. Note also that Argo CD was not involved. The `Certificate` object is in Git and Argo CD watches it. The Secret is not in Git and never was, and Argo CD does not care about it. **The Secret is output, not configuration**, which is exactly why it is safe to have a public repository describing a cluster full of private keys. Watch the paperwork it generated: ```bash kubectl get certificaterequest -n cert-manager # Expect: at least two for selftest. One per issuance, kept as a record. ``` ### 6b. Find out when it renews, without guessing The renewal question is the one people actually worry about, and the answer is boring in the best way. ```bash kubectl get certificate -A -o custom-columns="NS:.metadata.namespace,\ NAME:.metadata.name,\ EXPIRES:.status.notAfter,\ RENEWS:.status.renewalTime" # Expect: one row per certificate, with two timestamps on each. ``` That prints, for every certificate in the cluster, when it expires and when cert-manager intends to renew it. `-o wide` shows some of this too, but the columns it includes have changed between versions, so the explicit form above is the one that will still work next year. For a Let's Encrypt certificate you should see roughly 90 days to expiry and roughly 60 days to renewal. That gap is the default: **cert-manager renews at two thirds of the certificate's lifetime**, which for a 90 day certificate means about 30 days of margin. Thirty days is a lot of room to notice something is wrong, which is the whole reason the default is not "the day before." You can change it per certificate with `renewBefore`: ```yaml spec: renewBefore: 720h # renew when 30 days remain, stated explicitly ``` But the important thing is what you do not have to do. There is no cron job, no calendar reminder, no script. cert-manager checks continuously and acts when the time comes, and if it fails it retries with a backoff. The reason the 30 day margin exists is that if the Cloudflare token gets revoked in month two, you have a month of failing renewals to notice before anything actually breaks. ### 6c. Actually see a renewal happen, without waiting 60 days You cannot make time pass, and any demonstration that claims to is faking it. But you can issue a certificate that is deliberately short-lived, and then watch a real, time-triggered renewal in about five minutes. Change `infra/cert-manager-config/selftest-cert.yaml` to add two lines: ```yaml spec: secretName: selftest-tls duration: 1h # the shortest cert-manager will accept renewBefore: 55m # so renewal is due 5 minutes after issuance commonName: selftest.k3s.linsnotes.com dnsNames: - selftest.k3s.linsnotes.com issuerRef: name: lab-ca kind: ClusterIssuer ``` ```bash git add infra/cert-manager-config/selftest-cert.yaml git commit -m "Make the selftest certificate short-lived to watch a renewal" git push ``` Once Argo CD syncs it, check the schedule and then leave it: ```bash kubectl get certificate -n cert-manager selftest -o custom-columns=\ 'NAME:.metadata.name,EXPIRES:.status.notAfter,RENEWS:.status.renewalTime' # Expect: EXPIRES about an hour out, RENEWS about five minutes out. kubectl get certificaterequest -n cert-manager -w # Leave this running. Within about five minutes a new CertificateRequest # appears on its own. Ctrl-C once you have seen it. ``` That is a genuine renewal, driven by the clock, with nothing deleted and nothing forced. The only thing that is artificial is the lifetime, and shortening the lifetime is the honest way to compress the experiment rather than fake it. > **The official way to force a renewal on demand is `cmctl renew `**, a small command line tool cert-manager ships alongside the controller. It sets a condition on the `Certificate` that makes the controller reissue immediately, without deleting anything. Install instructions are in the cert-manager documentation, and they change between releases, so check there rather than trusting a URL from a blog post. It is worth having on a real cluster. It is not worth installing here, because deleting the Secret does the same job for a lab and you have already seen that work. {: .prompt-info } Put the certificate back to normal before you finish, or you will have a renewal every hour forever: ```bash git rm infra/cert-manager-config/selftest-cert.yaml git commit -m "Remove the throwaway renewal test certificate" git push ``` ```bash kubectl get certificate -n cert-manager # Expect: only lab-root-ca. Argo CD pruned selftest, because prune: true # means removing a file removes the object. Part 6, still working. kubectl get secret -n cert-manager selftest-tls # Expect: STILL THERE. Read the note below, then remove it by hand: kubectl delete secret -n cert-manager selftest-tls ``` > **Deleting a `Certificate` does not delete its Secret, and that surprises people.** By default cert-manager leaves the Secret behind, so removing a `Certificate` by accident does not instantly take your site down. The Secret is orphaned, not destroyed, and you clean it up yourself. There is a controller flag, `--enable-certificate-owner-ref`, that changes this and makes the Secret go with the Certificate, and it is off by default for exactly the reason above. Worth knowing before you go hunting for where all these leftover TLS Secrets came from. {: .prompt-info } --- ## Step 7 — Clean up and save a restore point Nothing here needs undoing. Every object you created is either infrastructure the next part uses or already pruned by Argo CD. Check the state you are leaving: ```bash kubectl get clusterissuer # Expect four: selfsigned, lab-ca, letsencrypt-staging, letsencrypt-prod. # Keeping staging is deliberate: it is where you test the next new hostname. kubectl get certificate -A # Expect: lab-root-ca in cert-manager, whoami-tls in default, # argocd-tls in argocd. No selftest. kubectl get applications -n argocd # Expect: all Synced and Healthy, including cert-manager and # cert-manager-config. kubectl get challenge,order -A # Expect: No resources found. Nothing in flight. ``` Snapshot all five, stopped and together, exactly as Parts 3 to 6 taught: ```bash for n in worker1 worker2 master1 master2 master3; do lxc stop "$n"; done for n in master1 master2 master3 worker1 worker2; do lxc snapshot "$n" post-tls done for n in master1 master2 master3 worker1 worker2; do lxc start "$n"; done sleep 45 kubectl get nodes # expect: five Ready kubectl get pods -n cert-manager # expect: three Running curl --max-time 5 -sI https://whoami.k3s.linsnotes.com | head -1 # Expect: HTTP/2 200. The certificate survived the restart, because it lives # in a Secret in etcd, which is part of the snapshot. ``` ### What this snapshot does and does not hold This one is worth reading carefully, because it holds something the earlier snapshots did not. **Inside the snapshot**, because it is all in etcd on the masters: - cert-manager, its CRDs, all four `ClusterIssuer` objects. - Every issued certificate and its private key. - **The Cloudflare API token.** The Secret is a cluster object like any other, so it is captured. Your `post-tls` snapshot now contains a live credential for your DNS. That is not a problem, but it means the snapshot is no longer just a machine image, and if you ever copy it off this machine you are copying a working token with it. If you revoke the token in Cloudflare later, restoring this snapshot brings back a token string that no longer works, and you will get authentication errors from a Secret that looks perfectly fine. **Outside the snapshot**, and unchanged by any restore: | Thing | Where it lives | |---|---| | The token's existence at Cloudflare | Cloudflare's side. Revoking it there is not undone by a restore. | | The GitHub repository | GitHub. Rolling nodes back does not roll back your commits, and Argo CD will re-sync forward to whatever Git says. | | `/etc/hosts` on the host and your laptop | Host and network state, same as Parts 5 and 6. | | The CA certificate in `/usr/local/share/ca-certificates/` | The host's trust store. It stays trusted after any node restore, which is what you want. | That last row has an edge worth knowing. If you ever delete the `lab-root-ca` Secret, cert-manager generates a **new** CA with a new key, and the certificate sitting in your host's trust store will no longer match. Everything signed by the new CA will be rejected until you repeat the copy from Step 3. It looks like trust randomly stopped working, and the cause is that the CA quietly became a different CA. --- ## Where you should be | Thing | State | |---|---| | cert-manager | Installed through Argo CD from a pinned Helm chart, CRDs included, three pods Running | | Issuers | `selfsigned`, `lab-ca`, `letsencrypt-staging`, `letsencrypt-prod`, all Ready | | Own CA | Root in `cert-manager/lab-root-ca`, also installed in the host's trust store | | Challenge type | DNS-01 through Cloudflare, using a scoped `Zone:DNS:Edit` plus `Zone:Zone:Read` token | | whoami | `https://whoami.k3s.linsnotes.com`, publicly trusted, no browser warning | | Argo CD | `https://argocd.k3s.linsnotes.com`, TLS terminated at the Ingress | | The Cloudflare token | A hand-created Secret in `cert-manager`, deliberately not in Git | | Renewal | Automatic at two thirds of lifetime, seen happening on a short-lived certificate | | Snapshots | `post-tls` on all five, taken stopped and together | ### Things worth carrying forward - **HTTP-01 needs to be reachable from the internet. DNS-01 does not.** That single difference is why a lab on a private address behind NAT can hold a genuinely trusted certificate. If a guide tells you that you need port 80 open to get a certificate, it is assuming HTTP-01. - **Trust is not a property of a certificate. It is a property of the reader.** The same certificate that `curl` rejected was accepted after you copied one file into `/usr/local/share/ca-certificates/`. Nothing about the certificate changed. - **Use Let's Encrypt staging until it works, then change one line.** The limits are 50 certificates per domain and 5 per identical name set, per week, and there is nobody to appeal to. A staging certificate that a browser rejects is a successful staging run. - **A `ClusterIssuer` reads its Secrets from the `cert-manager` namespace, not from the namespace you are working in.** This is the single most common reason an issuer sits Ready False with a Secret plainly visible somewhere else. - **Certificates are output, not configuration.** The `Certificate` object belongs in Git. The Secret it produces does not, which is what makes a public GitOps repository safe. Delete the Secret and it is reissued, not restored. - **Real credentials need Sealed Secrets, External Secrets or SOPS.** Creating the Cloudflare token by hand leaves a gap between what Git describes and what the cluster runs. Know the gap is there and know the three ways to close it. - **Renewal is automatic at two thirds of lifetime, with about 30 days of margin on a Let's Encrypt certificate.** The margin exists so that a month of failing renewals is visible before anything breaks. It is not something you will have to remember. ### What Part 8 does You now have a cluster that stands up, heals, stores data, routes traffic, deploys itself from Git, and serves real HTTPS. There is one thing it still cannot do: tell you how it is. Right now, every question about the lab is answered by running a command and looking. How much memory is master2 actually using? Which pod restarted at three in the morning? Is that certificate renewal quietly failing? You would only find out by checking, and you only check when you already suspect something. Part 8 fixes that. You install **Prometheus** to collect measurements from every node, every pod, and the control plane itself, and **Grafana** to draw them. Both go in through Argo CD, because that is how everything goes in now. Then, in the series' usual style, you break something and watch a graph notice before you do. cert-manager exports metrics about certificate expiry, so one of the first useful alerts you will build is the one that would have caught a renewal failing while you had 30 days of margin left. Continue with [**Part 8, Monitoring with Prometheus and Grafana**](/posts/building-a-kubernetes-lab-with-k3s-and-lxd-part-8-monitoring-with-prometheus-and-grafana/). ### [Building a Kubernetes Lab with k3s and LXD Part 6 GitOps with Argo CD](https://linsnotes.com/posts/building-a-kubernetes-lab-with-k3s-and-lxd-part-6-gitops-with-argo-cd/) > Put your cluster's configuration in Git, hand it to Argo CD, then delete things on purpose and watch the cluster put them back without you. This is **Part 6** of a series on building a Kubernetes lab with k3s and LXD. If you are just arriving, start with [**Part 1, Preparing the Ubuntu Host**](/posts/building-a-kubernetes-lab-with-k3s-and-lxd-part-1-preparing-the-ubuntu-host/), then [**Part 2, Building the Five Nodes**](/posts/building-a-kubernetes-lab-with-k3s-and-lxd-part-2-building-the-five-nodes/), [**Part 3, Installing k3s**](/posts/building-a-kubernetes-lab-with-k3s-and-lxd-part-3-installing-k3s/), [**Part 4, Adding Persistent Storage**](/posts/building-a-kubernetes-lab-with-k3s-and-lxd-part-4-adding-persistent-storage/), and [**Part 5, Exposing Applications**](/posts/building-a-kubernetes-lab-with-k3s-and-lxd-part-5-exposing-applications/). This part picks up where Part 5 left off. **Where you are.** You have a five node cluster that survives a master dying, keeps data when a worker dies, and serves applications to your browser through one address. It works. And every single piece of it exists because you typed a command into a terminal at some point over five afternoons. Try to answer this question about your own lab, right now, without looking: what exactly is in `/etc/rancher/k3s/config.yaml` on master2, and why? If you had to think about it, you have found the problem this part solves. **What this part does.** It moves the cluster's configuration out of your terminal history and into a Git repository, then installs **Argo CD**, a program that reads that repository and makes the cluster match it. After that, changing the cluster means changing a file and pushing it. Not running a command. **The main point, in one line.** Stop telling the cluster what to do, and start telling it what to be. **What this part does not do.** No TLS, still. The Argo CD web interface will run over plain HTTP, and Step 4 explains exactly why that takes one extra setting and what breaks if you skip it. Certificates are Part 7. **What you need before starting.** The `post-ingress` snapshot from Part 5's Step 9, all five nodes Ready, ingress-nginx holding a MetalLB address, a GitHub account, and Git installed on the host. Step 1 checks the cluster. Step 2 checks Git. --- ## Words you'll need These build on Parts 2 to 5. Here are the new ones. | Word | What it means | |---|---| | **GitOps** | A way of running systems where a Git repository holds the description of what should be running, and a program in the cluster continuously makes reality match it. Git is not a backup of the config. Git **is** the config. | | **declarative vs imperative** | *Imperative* means "do this": `kubectl scale deployment whoami --replicas=5`. *Declarative* means "this is how it should be": a file saying `replicas: 5`. Imperative commands describe a change. Declarative files describe an end state. | | **desired state** | What you have said should be true. In GitOps, this is whatever is in your Git repository at this moment. | | **actual state** | What is really running in the cluster right now. These two are usually the same and occasionally are not, and everything in this part is about that gap. | | **reconciliation** | The act of comparing desired state to actual state and changing actual state to close the gap. Kubernetes already does this internally for Deployments. Argo CD does it for your whole repository. | | **drift** | The gap itself. Something in the cluster no longer matches Git. Usually because a person changed it by hand, at 2am, and forgot. | | **self-heal** | An Argo CD setting that says "when you find drift, fix it." With it on, hand edits get undone automatically. | | **prune** | An Argo CD setting that says "if something is in the cluster but no longer in Git, delete it." This is the setting people are right to think twice about. | | **manifest** | One YAML file describing one or more Kubernetes objects. Everything you wrote with `cat > something.yaml` in Parts 4 and 5 was a manifest. | | **repository** | A Git repository. In this part it is a public GitHub repo holding your manifests. Argo CD reads it and needs no permission to write to it, ever. | | **Argo CD Application** | A Kubernetes object, created by you, that says: "watch *this* folder in *this* repo and apply it to *this* namespace." It is a pointer, not a program. | | **App of Apps** | A single Argo CD Application whose folder contains nothing but more Application objects. One thing to install by hand, and everything else follows. | | **sync** | Making the cluster match Git, once. Can be manual (you press a button) or automated (Argo CD does it on its own). | | **sync wave** | An ordering hint, written as an annotation, for when some objects must be created before others. Lower numbers go first. You will not need one today, but you will meet the word. | | **idempotent** | Safe to do twice. Applying the same manifest ten times leaves the same result as applying it once. This is what makes reconciliation safe to run every three minutes forever. | --- ## Three things to know before you start ### 1. You have spent five parts doing the opposite of this Every install in this series so far has been imperative. You piped a script into a shell to install k3s. You ran `helm install` for the NFS provisioner and for ingress-nginx. You wrote a config file into three containers with a `for` loop. You typed `kubectl apply` on manifests that you then deleted with `rm -f`. All of it worked. None of it is written down anywhere except in this blog and in your shell history. So if the host dies, your only recovery path is to read Parts 1 to 5 again in order. If you want to know whether the MetalLB pool ends at `.249` or `.250`, you have to ask the cluster. And nobody, including you, can look at a change before it happens. This part converts what you already have rather than starting fresh, and that is deliberate. Step 5 takes MetalLB's configuration, the thing you wrote by hand in Part 5, and puts it under Argo CD's control **without deleting or recreating it**. That conversion is the most useful thing in this post, because it is the situation you will actually meet at work. Nobody gets a clean cluster to start GitOps on. They get one with four years of history in it. ### 2. The one thing likely to stop you is the Argo CD web interface Every part of this series has one step that can genuinely block you. Part 3 had the ZFS snapshotter, Part 4 had NFS inside a container, Part 5 had router static routes. This part has a smaller one, but it will absolutely catch you if nobody says it first. **Argo CD's web server speaks HTTPS to itself by default.** It generates its own certificate at startup and, when a request arrives over plain HTTP, it answers with a redirect to HTTPS. You are about to put it behind ingress-nginx over plain HTTP. So the browser asks for `http://argocd.k3s.linsnotes.com`, ingress-nginx forwards it to Argo CD over HTTP, Argo CD says "go to HTTPS", the browser goes back to the same plain HTTP address, and around it goes. The symptom is `ERR_TOO_MANY_REDIRECTS` in the browser, or a `502` if ingress-nginx gives up on the handshake instead. The fix is one line in a ConfigMap called `argocd-cmd-params-cm`, setting `server.insecure` to `"true"`, then restarting the `argocd-server` deployment. Step 4 does it and explains what you are giving up. Part 7 revisits the whole arrangement once real certificates exist. ### 3. GitOps means `kubectl apply` becomes something you stop doing This is the part people underestimate, so read it before Step 7 rather than after. Once an Application has `selfHeal: true`, the cluster will actively undo you. Scale a Deployment by hand and the replica count goes back. Edit an Ingress by hand and your edit disappears. Delete something by hand and it comes back. This is not Argo CD misbehaving. It is Argo CD doing precisely the one job you gave it. That changes your habits, and it should. Reading is always fine: `kubectl get`, `describe` and `logs` are unaffected. But `apply`, `edit`, `scale`, `patch` and `delete` on anything Argo CD manages become **temporary experiments**, not changes. A real change is a commit. The awkward moment comes during an outage, when you want to fix something in ten seconds and Git feels slow. Real teams handle that by turning off automated sync for one application while they work, then putting the fix in Git and turning it back on. Step 7 shows you where that switch is, because knowing beforehand is the difference between a controlled decision and a confusing fight with your own tooling. --- ## Step 1 — Check Part 5 is still good **Why:** the same reason as every part. Argo CD is about to start comparing your cluster to a file. Anything already broken will look like a GitOps problem the moment it surfaces. **Safe to run:** everything here only reads. ```bash lxc info | head -3 # wakes LXD if it's asleep, same trick as before sleep 3 lxc list # expect five RUNNING rows, plus the STOPPED template kubectl get nodes # Expect: five Ready. kubectl get all -n default # Expect ONE line: service/kubernetes. That's Part 5's clean baseline. ``` Now the two pieces of Part 5 that this part depends on. MetalLB first: ```bash kubectl get pods -n metallb-system # Expect: one controller pod and one speaker pod per node, all Running. kubectl get ipaddresspool,l2advertisement -n metallb-system # Expect: ipaddresspool/lab-pool and l2advertisement/lab-l2. # These two objects are the ones you are going to hand to Argo CD in Step 5. ``` Then the front door: ```bash kubectl get svc -n ingress-nginx ingress-nginx-controller # Expect: TYPE LoadBalancer, EXTERNAL-IP holding a MetalLB address # (10.99.99.240 or nearby). If it says , MetalLB is not handing out # addresses and Step 4 will fail confusingly. Fix that first, in Part 5 Step 6. NGINX_IP=$(kubectl get svc -n ingress-nginx ingress-nginx-controller \ -o jsonpath='{.status.loadBalancer.ingress[0].ip}') echo "ingress-nginx is on $NGINX_IP" # Note this down. Step 4 needs it. ``` > **`NGINX_IP` lives only in this terminal.** Same warning as Part 3's `$TOKEN`. Close the window and it is gone. If a later command produces an empty address, re-run the two lines above. {: .prompt-tip } Finally, check the host has what Step 2 needs: ```bash git --version # expect: a version. If not: sudo apt-get install -y git gh --version 2>/dev/null || echo "gh not installed (optional, Step 2 has a plain git path)" ``` --- ## Step 2 — Make the repository that becomes the truth **What this does.** It creates one public GitHub repository and lays out three folders in it. Nothing is applied to the cluster yet. This step is entirely about deciding where things go before you have anything to put there, which is much easier than deciding afterwards. ### Why the layout matters more than it looks An Argo CD Application points at a **folder** in a repository, not a file. So the folder structure is not decoration; it is the unit of control. Every folder you create is a thing you can sync, roll back, or turn off on its own. Three folders is enough for this lab: ``` k3s-lab-gitops/ ├── README.md ├── bootstrap/ # applied by hand, on purpose. See the note below. │ ├── argocd-cmd-params-cm.yaml │ └── argocd-server-ingress.yaml ├── apps/ # Argo CD Application objects: WHAT Argo CD manages │ ├── metallb-config.yaml │ └── whoami.yaml └── infra/ # the Kubernetes manifests each Application points at ├── metallb-config/ │ ├── ipaddresspool.yaml │ └── l2advertisement.yaml └── whoami/ ├── deployment.yaml ├── service.yaml └── ingress.yaml ``` Read it as two questions answered separately. **`apps/` answers "what is Argo CD watching?"**, one small file per thing, and those files are pointers. **`infra/` answers "what does each of those things consist of?"**, one folder per thing, with the real YAML inside. Keeping the pointers apart from the manifests means a single `ls apps/` tells you everything the cluster is managed by, without reading a line of Kubernetes YAML. `bootstrap/` is the honest one. It holds the two files you apply with `kubectl` by hand, because they have to exist before Argo CD can manage anything at all. They live in the repo so that they are written down, not because Argo CD reads them. > **`whoami` is a demo application, not infrastructure, and it is in `infra/` anyway.** A repo that runs real workloads would grow a fourth folder, something like `workloads/`, and split them. Two levels is enough to follow today, and moving a folder later is a `git mv` and a one line edit to the Application. Do not build the structure you might need in a year. {: .prompt-info } ### Create it > **Use your own GitHub username.** Every command and every `repoURL` below says `kuibinlin`. Replace it with yours throughout, including inside the YAML in Steps 5 and 6, or Argo CD will happily sync somebody else's repository into your cluster. {: .prompt-warning } On the host: ```bash mkdir -p ~/k3s-lab-gitops && cd ~/k3s-lab-gitops git init -b main mkdir -p bootstrap apps infra/metallb-config infra/whoami cat > README.md <<'EOF' # k3s lab, managed by Argo CD This repository is the desired state of a five node k3s lab. Anything Argo CD manages is described here. If it is not here, it is not managed. - bootstrap/ : applied by hand with kubectl, before Argo CD exists - apps/ : Argo CD Application objects - infra/ : the manifests each Application points at EOF git add . git commit -m "Empty layout and README" # Expect: one file changed. Git does not track empty folders, so only # README.md is committed right now. The folders fill up in Steps 4 to 6. ``` With the GitHub CLI, the remote and the push are one command: ```bash gh repo create kuibinlin/k3s-lab-gitops --public --source=. --remote=origin --push # Expect: a line confirming the repo was created, then a push. ``` Without `gh`, create the repository on github.com first. Make it **public**, and do not let GitHub add a README, a licence or a `.gitignore`, because those create a commit on the remote that your local `main` does not have and the first push will be refused. Then: ```bash git remote add origin https://github.com/kuibinlin/k3s-lab-gitops.git git push -u origin main ``` Confirm it landed: ```bash git remote -v # expect: origin, pointing at your repo, fetch and push git log --oneline # expect: one commit ``` > **Public, on purpose.** A public repository needs no credentials, so Argo CD can read it with no setup at all, and this part stays about GitOps instead of about secrets. That is only acceptable because nothing in this repo is secret: address ranges, replica counts and hostnames for a lab on your own machine. **Never commit a token, a password or a kubeconfig.** For a private repo you would register a read only deploy key or a token with Argo CD, which is a real and well documented thing, and a distraction today. {: .prompt-warning } --- ## Step 3 — Install Argo CD **What this does.** It installs Argo CD into its own namespace from the project's official manifest, and gets you the password to log in with. **What Argo CD actually is**, before you install it: four programs that do four jobs, plus some extras. | Component | Its job | |---|---| | `argocd-repo-server` | Clones your Git repository and turns whatever is in it into plain Kubernetes YAML. | | `argocd-application-controller` | Compares that YAML to the live cluster, and applies the difference. This is the reconciliation loop. | | `argocd-server` | The web interface and the API. This one does no reconciling at all. | | `argocd-redis` | A cache, so the controller does not recompute everything constantly. | The rest (`dex`, `notifications`, `applicationset`) are optional features you are not using today. They install anyway and sit idle, which is fine. ### Install ```bash kubectl create namespace argocd kubectl apply -n argocd --server-side --force-conflicts \ -f https://raw.githubusercontent.com/argoproj/argo-cd/v3.4.5/manifests/install.yaml # Expect: a long list of "serverside-applied" lines: CRDs, ServiceAccounts, # ConfigMaps, Deployments, a StatefulSet, Services. ``` > **Check the current release rather than trusting this blog.** `v3.4.5` was current when this was written. Argo CD releases a minor version roughly every three months and supports the last three, so a version pinned in a post ages out. Look at [the Argo CD releases page](https://github.com/argoproj/argo-cd/releases) and use the newest stable tag. The URL shape never changes, only the tag. You can also use `stable` in place of the tag, which always points at the newest stable release, but then you cannot tell later which version you installed, which is a strange thing to accept in a post about writing things down. {: .prompt-tip } **Why `--server-side`.** Argo CD's custom resource definitions are large enough that a normal client side `kubectl apply` can exceed the 262144 byte limit on the annotation Kubernetes uses to remember the last applied configuration. Server side apply does not use that annotation, so the limit does not apply, and `--force-conflicts` says you own these fields if anything argues. This is the command the Argo CD documentation gives, and it is worth knowing why rather than copying it. Wait for it to come up. This pulls several images, so give it time: ```bash kubectl wait -n argocd --for=condition=available deployment --all --timeout=300s # Expect: a "condition met" line per deployment. kubectl get pods -n argocd # Expect: roughly seven pods, all Running. The four that matter are # argocd-server, argocd-repo-server, argocd-application-controller # and argocd-redis. Pod name suffixes are random; the prefixes are not. ``` If a pod sits in `ContainerCreating` for a long while, it is almost certainly still pulling images over your connection, not stuck. `kubectl describe pod -n argocd ` says which. ### Get the admin password Argo CD generates a random password for the `admin` user at first install and stores it in a Secret. It is not printed anywhere, so read it out: ```bash kubectl -n argocd get secret argocd-initial-admin-secret \ -o jsonpath='{.data.password}' | base64 -d; echo # Expect: a line of random characters. It is different on every install, # so there is no example to compare yours against. ``` The `; echo` at the end just adds a newline, because the decoded value has none and your prompt would otherwise land on the same line as the password. Keep it somewhere for the next step. Once you have logged in and changed the password, that Secret is no longer needed and you can delete it, which is what the Argo CD documentation suggests and what you would do on anything real: ```bash # Later, after you have changed the password: # kubectl -n argocd delete secret argocd-initial-admin-secret ``` --- ## Step 4 — Put the Argo CD interface behind your Ingress **What this does.** It gives Argo CD a hostname on the front door you built in Part 5, and fixes the HTTPS redirect problem from the intro. Everything stays plain HTTP. ### Why Argo CD fights a plain HTTP Ingress Argo CD's API server is designed to be reachable directly, without anything in front of it, so it does its own TLS. On startup it generates a self signed certificate and serves HTTPS on its container port. When a plain HTTP request arrives, it does the polite web thing and redirects to the HTTPS version of the same URL. Put that behind ingress-nginx with no TLS and you get a loop: 1. Browser asks ingress-nginx for `http://argocd.k3s.linsnotes.com`. 2. ingress-nginx forwards it to `argocd-server` over plain HTTP. 3. `argocd-server` answers with a redirect to HTTPS. 4. ingress-nginx passes the redirect back to the browser. 5. The browser follows it, arrives at the same plain HTTP front door, and you are at step 1 again. The browser counts the round trips, gives up, and shows **`ERR_TOO_MANY_REDIRECTS`**. If instead ingress-nginx tries to speak HTTPS to a backend that is expecting it and the handshake fails, you get a **`502 Bad Gateway`** with a TLS error in the ingress-nginx logs. Two different symptoms, one cause. Notice what is *not* wrong here. Your Ingress rule is fine. MetalLB is fine. DNS is fine. Two programs each have a correct and reasonable opinion about TLS and the opinions disagree. ### The fix, and what it costs Tell Argo CD to stop doing TLS itself and serve plain HTTP, because something in front of it is responsible for that now: ```bash cat > ~/k3s-lab-gitops/bootstrap/argocd-cmd-params-cm.yaml <<'EOF' apiVersion: v1 kind: ConfigMap metadata: name: argocd-cmd-params-cm namespace: argocd data: server.insecure: "true" # argocd-server serves plain HTTP, no self-redirect EOF kubectl -n argocd patch configmap argocd-cmd-params-cm --type merge \ -p '{"data":{"server.insecure":"true"}}' # Expect: configmap/argocd-cmd-params-cm patched. ``` `argocd-cmd-params-cm` is where Argo CD keeps command line style settings for its own components. It ships empty. The `patch` adds one key rather than replacing the whole ConfigMap, which matters because later parts and later versions add other keys to it. **The setting only takes effect at startup**, so restart the one deployment that reads it: ```bash kubectl -n argocd rollout restart deployment argocd-server kubectl -n argocd rollout status deployment argocd-server --timeout=180s # Expect: "deployment ... successfully rolled out". ``` > **"insecure" means exactly what it says, and here it is contained.** Traffic between ingress-nginx and Argo CD is now unencrypted, and so is traffic between your browser and ingress-nginx, since there is no TLS anywhere yet. On a lab network that only exists on your own machine, that is an acceptable trade for one part of a tutorial. It is **not** acceptable on anything real. Part 7 puts a real certificate on ingress-nginx, at which point the browser side is encrypted and this setting becomes the normal, recommended arrangement: TLS terminates once, at the Ingress, and the backend behind it speaks plain HTTP. {: .prompt-warning } ### Write the Ingress rule Same shape as Part 5's, with one annotation: ```bash cat > ~/k3s-lab-gitops/bootstrap/argocd-server-ingress.yaml <<'EOF' apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: argocd-server namespace: argocd annotations: nginx.ingress.kubernetes.io/backend-protocol: "HTTP" spec: ingressClassName: nginx rules: - host: argocd.k3s.linsnotes.com http: paths: - path: / pathType: Prefix backend: service: name: argocd-server port: number: 80 # the HTTP port on the argocd-server Service EOF kubectl apply -f ~/k3s-lab-gitops/bootstrap/argocd-server-ingress.yaml kubectl get ingress -n argocd argocd-server # Expect: CLASS nginx, HOSTS argocd.k3s.linsnotes.com, ADDRESS filling in # with your ingress-nginx address, PORTS 80. ``` Two details worth naming: - **`backend-protocol: "HTTP"` is already the default** for ingress-nginx. It is written out because it is the line you would change to `HTTPS` if you ever left Argo CD doing its own TLS, and because an explicit setting is easier to reason about than an assumed one. - **Do not add `force-ssl-redirect`.** You will find that annotation in most Argo CD Ingress examples, and it is correct in those examples because they have a certificate. Add it now, with no TLS anywhere, and you build a redirect loop of your own at the ingress layer instead of at the Argo CD layer. Part 7 adds it, along with the certificate that makes it make sense. ### Make the name resolve, and log in Part 5 put a line in the host's `/etc/hosts`. Add another for this name: ```bash echo "$NGINX_IP argocd.k3s.linsnotes.com" | sudo tee -a /etc/hosts grep k3s.linsnotes.com /etc/hosts # Expect: two lines now, both pointing at the same ingress-nginx address. ``` If you did Part 5's Step 5 and want the interface from your laptop, add the same line to your laptop's hosts file too. Test from the command line before opening a browser, so a browser cache cannot confuse you: ```bash curl --max-time 5 -s -o /dev/null -w '%{http_code}\n' http://argocd.k3s.linsnotes.com # Expect: 200. # If you get 307 or 308, the server.insecure setting did not take: check the # ConfigMap and confirm the rollout restart actually finished. # If you get 502, same cause, different symptom. Same fix. # If you get 404, the Host header did not match any rule: check the Ingress. ``` Now open `http://argocd.k3s.linsnotes.com` in a browser. Log in with username `admin` and the password from Step 3. You will land on an empty Applications page. That is correct. Argo CD is installed and knows about nothing at all, which is the honest starting point for the next step. > **Who bootstraps the bootstrapper?** You just configured Argo CD with two `kubectl` commands, in a post about not doing that. There is no way around it: something has to create the thing that creates everything else, and that something is a human with `kubectl`. Every GitOps setup has this seam. The usual answer is to keep the bootstrap as small and as written down as possible, which is why both files went into `bootstrap/` in the repo even though nothing reads them from there. A more advanced answer is to have Argo CD manage its own installation, which works and is genuinely useful, and also means a bad commit can take down the thing that would fix it. That is a decision for a real cluster, not a lab. {: .prompt-info } Commit what you have so far: ```bash cd ~/k3s-lab-gitops git add bootstrap/ git commit -m "Record the Argo CD bootstrap: insecure server and its Ingress" git push ``` --- ## Step 5 — Convert what you already built **What this does.** It takes MetalLB's configuration, the two objects you wrote by hand in Part 5, and puts them under Argo CD's control without deleting or recreating anything. This is the most useful step in the post. ### Why start here and not with something new Because adopting a live object is the case everyone actually faces and almost no tutorial covers. Starting a repo with a fresh application proves nothing: of course Argo CD can create something that does not exist. The real question is whether you can hand it something already running, right now, and have it take over without an outage. You can. MetalLB's config is a good first candidate: two small objects, they change almost never, and if something went wrong you would find out immediately, because your lab would stop being reachable. ### Get the real YAML out of the cluster Do not retype it from Part 5. Read what is actually there, because that is the whole point: ```bash kubectl get ipaddresspool lab-pool -n metallb-system -o yaml | head -20 kubectl get l2advertisement lab-l2 -n metallb-system -o yaml | head -20 # Expect: your objects, plus a lot of fields Kubernetes added: creationTimestamp, # resourceVersion, uid, generation, managedFields, status. Those are the # cluster's own bookkeeping and must NOT go into Git. ``` That last point matters. A live object read back from the cluster is not a manifest. It is the manifest plus everything Kubernetes recorded about it. Committing `resourceVersion` and `uid` into Git would give Argo CD a desired state that can never match reality, because those fields are assigned by the cluster and change on their own. So write clean manifests by hand, containing only what you meant to say: ```bash cd ~/k3s-lab-gitops cat > infra/metallb-config/ipaddresspool.yaml <<'EOF' apiVersion: metallb.io/v1beta1 kind: IPAddressPool metadata: name: lab-pool namespace: metallb-system spec: addresses: - 10.99.99.240-10.99.99.250 # outside LXD's DHCP range, from Part 5 Step 6a EOF cat > infra/metallb-config/l2advertisement.yaml <<'EOF' apiVersion: metallb.io/v1beta1 kind: L2Advertisement metadata: name: lab-l2 namespace: metallb-system spec: ipAddressPools: - lab-pool EOF ``` **Check these against the cluster before going further**, because a mismatch here is what turns adoption into an outage: ```bash kubectl get ipaddresspool lab-pool -n metallb-system -o jsonpath='{.spec.addresses}'; echo # Expect: the same range that is in your ipaddresspool.yaml. If your Part 5 # pool was a different range, edit the file to match reality, not the tutorial. ``` ### Write the Application that adopts them ```bash cat > apps/metallb-config.yaml <<'EOF' apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: metallb-config namespace: argocd # Applications live in Argo CD's namespace spec: project: default # the built-in project; no restrictions source: repoURL: https://github.com/kuibinlin/k3s-lab-gitops.git targetRevision: main # which branch to follow path: infra/metallb-config # which FOLDER in that branch destination: server: https://kubernetes.default.svc # "this cluster", from inside it namespace: metallb-system # No syncPolicy yet. Manual sync only, on purpose. Step 7 changes this. EOF git add apps/ infra/ git commit -m "Bring MetalLB's configuration under Argo CD" git push ``` Read those seven meaningful lines once more, because every Application you ever write is this shape: **which repo, which branch, which folder, which cluster, which namespace.** That is all an Application is. Now create it. This is a `kubectl apply`, and it is the last kind you will keep doing: applying a pointer, not a workload. ```bash kubectl apply -f apps/metallb-config.yaml # Expect: application.argoproj.io/metallb-config created ``` ### Watch it adopt, not fight ```bash kubectl get applications -n argocd # Expect, within a few seconds: # NAME SYNC STATUS HEALTH STATUS # metallb-config Synced Healthy ``` **`Synced`, on the first look, with nothing having been applied.** That is the whole point of this step, and it is worth sitting with for a second. Argo CD cloned your repository, rendered the two manifests, compared them field by field with the two objects already running in `metallb-system`, found no difference, and reported the truth: reality already matches Git. Nothing was deleted. Nothing was recreated. No address moved. Check: ```bash kubectl get svc -n ingress-nginx ingress-nginx-controller # Expect: the SAME EXTERNAL-IP as in Step 1. Adoption is not a restart. ``` Open the Argo CD interface and click into `metallb-config`. You get a diagram of the two objects with a green tick on each. That view is the point of the web interface: it is a live answer to "what does Git say should exist, and does it?" > **If yours shows `OutOfSync` at first, that is fine and not a failure.** Argo CD marks the resources it owns, and the exact marking method has changed between major versions: older versions used a label, current versions use an annotation. If your version wants to add a mark the live objects do not have yet, it reports the difference honestly. Press **Sync** once. Argo CD applies the same YAML plus its own tracking mark. Nothing is deleted or recreated, and it goes green. What you must **not** see is Argo CD proposing to change `spec.addresses`. If it does, your file does not match the cluster; fix the file, not the cluster. {: .prompt-info } ### See it notice a difference, without acting on one Change the pool in Git to something wrong, and watch what a manually synced Application does about it: ```bash sed -i 's/10.99.99.240-10.99.99.250/10.99.99.240-10.99.99.249/' infra/metallb-config/ipaddresspool.yaml git commit -am "Deliberate mistake, to see drift reported" git push # Argo CD polls Git about every three minutes. Ask it to look now instead: kubectl -n argocd patch app metallb-config --type merge \ -p '{"metadata":{"annotations":{"argocd.argoproj.io/refresh":"hard"}}}' kubectl get applications -n argocd # Expect: metallb-config now reads OutOfSync. ``` Click the application in the interface, then **App Diff**, and you get a red and green diff of exactly one line. Nothing has been applied. **Argo CD noticed and did not act**, which is what manual sync means and why it is a reasonable default while you are learning. Put it back: ```bash git revert --no-edit HEAD git push kubectl -n argocd patch app metallb-config --type merge \ -p '{"metadata":{"annotations":{"argocd.argoproj.io/refresh":"hard"}}}' kubectl get applications -n argocd # Expect: Synced again. The cluster never changed; only your description of it did. ``` --- ## Step 6 — Add an application through Git only **What this does.** It runs `traefik/whoami` again, the same small program from Part 5, with one rule: you will not type `kubectl apply` for any of its manifests. Not once. The only thing you apply is the pointer. ### Write the manifests ```bash cd ~/k3s-lab-gitops cat > infra/whoami/deployment.yaml <<'EOF' apiVersion: apps/v1 kind: Deployment metadata: name: whoami namespace: default spec: replicas: 2 selector: matchLabels: app: whoami template: metadata: labels: app: whoami spec: containers: - name: whoami image: traefik/whoami ports: - containerPort: 80 EOF cat > infra/whoami/service.yaml <<'EOF' apiVersion: v1 kind: Service metadata: name: whoami namespace: default spec: selector: app: whoami ports: - port: 80 targetPort: 80 EOF cat > infra/whoami/ingress.yaml <<'EOF' apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: whoami namespace: default spec: ingressClassName: nginx # the controller from Part 5 Step 7 rules: - host: whoami.k3s.linsnotes.com http: paths: - path: / pathType: Prefix backend: service: name: whoami port: number: 80 EOF ``` Three files instead of one, split by object. Argo CD applies every YAML file in the folder, so one file with `---` separators would work identically. One object per file is easier to review in a pull request and easier to find in six months, and those are the only two reasons that matter. > **`image: traefik/whoami` has no tag, which means `latest`, which is not a version.** It is fine for a lab where you want the point rather than the practice. In a repository that is supposed to be the truth about what is running, an untagged image is a hole in that truth: two syncs a month apart can pull two different programs from the identical commit. Pin an exact tag on anything you care about, and treat bumping the tag as a commit like any other. That is most of what people mean when they say GitOps gives you an audit trail. {: .prompt-tip } ### Write the pointer and push ```bash cat > apps/whoami.yaml <<'EOF' apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: whoami namespace: argocd spec: project: default source: repoURL: https://github.com/kuibinlin/k3s-lab-gitops.git targetRevision: main path: infra/whoami destination: server: https://kubernetes.default.svc namespace: default EOF git add apps/whoami.yaml infra/whoami/ git commit -m "Run whoami from Git" git push ``` Check the cluster first, so you can see the before clearly: ```bash kubectl get all -n default # Expect: still ONE line, service/kubernetes. Nothing exists yet. ``` Create the pointer: ```bash kubectl apply -f apps/whoami.yaml kubectl get applications -n argocd # Expect: whoami appears, SYNC STATUS OutOfSync, HEALTH STATUS Missing. ``` **`OutOfSync` and `Missing` together mean "Git says these three things should exist, and none of them do."** Which is exactly right. There is no automated sync on this Application yet, so Argo CD has reported the gap and stopped. ### Sync it Three ways, and they are equivalent. Pick one. **In the browser.** Click the `whoami` card, then the **Sync** button, then **Synchronize** in the panel. Watch the tiles turn green in order. This is the most useful one the first time, because you see what is happening. **With `kubectl`**, no extra tools: ```bash kubectl -n argocd patch app whoami --type merge \ -p '{"operation":{"sync":{"revision":"main"}}}' # Writing to the "operation" field is how a sync is requested. The controller # picks it up and runs it. This is what the Sync button does underneath. ``` **With the `argocd` command line tool**, if you would rather have it. Note the port-forward: the CLI speaks gRPC, which needs extra ingress-nginx annotations to survive the trip through your Ingress, and that is a detour you do not need today. ```bash # Check the releases page for the current version before running this. sudo curl -sSL -o /usr/local/bin/argocd \ https://github.com/argoproj/argo-cd/releases/download/v3.4.5/argocd-linux-amd64 sudo chmod +x /usr/local/bin/argocd kubectl port-forward -n argocd svc/argocd-server 8080:80 >/dev/null 2>&1 & argocd login localhost:8080 --username admin --plaintext # asks for the Step 3 password argocd app sync whoami ``` Whichever you used: ```bash kubectl get applications -n argocd # Expect: whoami Synced Healthy kubectl get all -n default # Expect: two whoami pods, one deployment, one replicaset, one service, # plus the permanent service/kubernetes. kubectl get ingress -n default whoami # Expect: CLASS nginx, HOST whoami.k3s.linsnotes.com, your ingress-nginx ADDRESS. curl --max-time 5 http://whoami.k3s.linsnotes.com # Expect: the whoami text block. The hostname already resolves, because # Part 5 put it in /etc/hosts and Step 9 of Part 5 left the line alone. ``` **A Deployment, a Service and an Ingress exist, are serving traffic, and you never applied any of them.** You wrote three files, pushed them, and told Argo CD once that the folder was its problem now. > **You did still run one `kubectl apply`, for the Application itself.** That is the seam again, one level up. The usual way to close it is the **App of Apps** pattern: make one Application whose `path` is `apps/`, so its content is nothing but other Application objects. Apply that one, and every application in the folder appears on its own. Adding a new application then becomes a single new file in `apps/` and a push, with no `kubectl` at all. It is four lines different from the Application above, and it is the natural next thing to build once you have more than two of them. {: .prompt-info } --- ## Step 7 — Turn on automated sync **What this does.** It changes both Applications from "tell me about differences" to "close them". Two settings, and they do genuinely different things. ### The two settings, precisely | Setting | What it does | What it does NOT do | |---|---|---| | `automated: {}` on its own | When Git changes, apply the change. New objects get created, changed objects get updated. | It does not react to changes made *in the cluster*, and it does not delete anything. | | `selfHeal: true` | When the **cluster** drifts from Git, put it back. A hand edit, a `kubectl scale`, a deleted object: all reverted. | It does not delete objects that Git no longer mentions. Those are a different problem. | | `prune: true` | When something is removed from **Git**, delete it from the cluster. | It does not touch anything Argo CD never created, and it does not touch other applications' objects. | The distinction that trips people up is between `selfHeal` and `prune`, so say it in one line each: - **`selfHeal` fixes the cluster when the cluster is wrong.** - **`prune` fixes the cluster when Git has become shorter.** Without `prune`, deleting a file from Git does nothing at all. The object stays in the cluster forever, unmanaged and unmentioned, which over a year is how a cluster fills up with things nobody can explain. With `prune`, deleting a file deletes the thing. That is the correct behaviour and it is also the reason people are nervous, and they are right to be: **`prune` means a bad `git rm`, or a merge that drops a folder, or a mistyped `path`, removes real objects from a real cluster.** The mitigations are ordinary software ones, not Kubernetes ones: require review on `main`, keep `path` values boring, and know that Argo CD's diff view shows you what a sync would delete before you let it. In a lab, turn it on and go and see what it does, which is Step 8. ### Turn it on Edit both Application files in Git. This is a change to how the cluster is managed, so of course it is a commit: ```bash cd ~/k3s-lab-gitops cat >> apps/whoami.yaml <<'EOF' syncPolicy: automated: prune: true # delete things removed from Git selfHeal: true # undo changes made in the cluster EOF cat >> apps/metallb-config.yaml <<'EOF' syncPolicy: automated: prune: false # deliberately NOT pruning the network config. See below. selfHeal: true EOF git commit -am "Automate sync for both applications" git push ``` Careful with those `>>` appends: `syncPolicy` must line up under `spec`, at two spaces of indentation, and the block above assumes the files end exactly as Steps 5 and 6 wrote them. Check before applying: ```bash tail -6 apps/whoami.yaml # Expect: syncPolicy indented two spaces, automated four, prune and selfHeal six. ``` Apply the updated pointers by hand, which is still how Application objects themselves reach the cluster: ```bash kubectl apply -f apps/whoami.yaml -f apps/metallb-config.yaml # Expect: two "configured" lines. kubectl get applications -n argocd # Expect: both Synced and Healthy, and now they will stay that way by themselves. ``` > **`prune: false` on `metallb-config` is a deliberate choice, not an oversight.** Think through what pruning that folder would mean. Delete `ipaddresspool.yaml`, push, and MetalLB has no addresses to hand out. The ingress-nginx Service loses its address, every hostname in your lab stops resolving to anything that answers, and the Argo CD interface you would use to fix it goes down with everything else. That is not a reason to fear `prune` in general. It is a reason to think about **blast radius per application**, which is one of the real arguments for splitting a repository into several small Applications rather than one big one. Fast moving workloads get `prune: true`. The network's foundations do not. {: .prompt-warning } ### Where the off switch is You will want this at some point, so find it now while nothing is on fire. To take one application out of automated control temporarily: ```bash # Suspend automation on whoami: kubectl -n argocd patch app whoami --type merge -p '{"spec":{"syncPolicy":null}}' # Put it back: kubectl apply -f ~/k3s-lab-gitops/apps/whoami.yaml ``` The first command is drift you are creating on purpose, and the second is the fix you push through Git afterwards. Doing it in that order, deliberately, is fine. Doing it by accident and forgetting is how a cluster ends up half managed. --- ## Step 8 — Break it on purpose Every part of this series ends by taking something away to see what the design was for. Part 3 killed a master to watch quorum protect itself. Part 4 killed a worker to watch data survive. Part 5 killed the node holding the load balancer address. Here you attack the deployment process itself, and the interesting result is that it fights back. ### First, understand the two different clocks This is the thing that will confuse you if nobody says it, and it explains why some of these experiments finish in seconds and one takes minutes. **Argo CD watches the cluster continuously.** The application controller has open watches on the objects it manages, so when something in the cluster changes or disappears, it usually knows within seconds. `selfHeal` acts on that. Experiments (a) and (b) below are therefore fast. **Argo CD polls Git on a timer.** The default is `timeout.reconciliation: 180s` in the `argocd-cm` ConfigMap, so about three minutes. A commit you push right now might sit unnoticed for up to three minutes. Experiment (c) is therefore slow, and if you do not expect that you will conclude something is broken when it is merely waiting. Confirm the setting on your own install rather than trusting the number: ```bash kubectl -n argocd get configmap argocd-cm -o jsonpath='{.data.timeout\.reconciliation}'; echo # Expect: 180s, or blank. Blank means "unset", which means the built-in # default, which is also 180s. ``` To stop waiting at any point, ask for an immediate look: ```bash kubectl -n argocd patch app whoami --type merge \ -p '{"metadata":{"annotations":{"argocd.argoproj.io/refresh":"hard"}}}' ``` Or press **Refresh** in the web interface, which is the same request. > **The real answer to the three minute delay is a webhook, and you cannot have one here.** In a normal setup GitHub sends Argo CD a request the instant you push, and the sync happens in a second or two. That needs GitHub to be able to reach your Argo CD server, and yours lives on `10.99.99.x` behind your home router, which is exactly the kind of address the whole internet is built not to reach. Polling is the honest fallback and it is what this lab uses. If you ever put this cluster behind a real name, the webhook is a five minute job and worth doing. {: .prompt-info } Open a second terminal and leave this running through all three experiments, so you see the states change rather than catch them afterwards: ```bash kubectl get applications -n argocd -w # Ctrl-C when you are done. ``` ### (a) Delete an object and watch it come back ```bash kubectl get ingress -n default whoami # Expect: it exists. Note the ADDRESS. kubectl delete ingress -n default whoami # Expect: ingress.networking.k8s.io "whoami" deleted curl --max-time 5 -s -o /dev/null -w '%{http_code}\n' http://whoami.k3s.linsnotes.com # Expect: 404. Run this quickly. The rule is gone, so ingress-nginx has no # reason to route that hostname anywhere. ``` Now watch the other terminal, and check back: ```bash kubectl get ingress -n default whoami # Expect, usually within a few seconds: it exists again. You did not do that. curl --max-time 5 http://whoami.k3s.linsnotes.com # Expect: the whoami text block. Back to normal, with no action from you. ``` In the watch terminal you will have seen `whoami` flick from `Synced` to `OutOfSync` and straight back to `Synced`. That is the entire mechanism in four seconds: the controller saw the object vanish, compared with Git, found something missing, and applied it. If nothing has happened after a minute, force a refresh with the patch command above and look at the application in the browser. The most common cause is that the `syncPolicy` append in Step 7 was indented wrong, in which case `kubectl get app whoami -n argocd -o jsonpath='{.spec.syncPolicy}'` prints nothing. ### (b) Change something and watch it get changed back Deleting is dramatic. Editing is the one that actually happens to people, so do that too: ```bash kubectl scale deployment -n default whoami --replicas=5 kubectl get deployment -n default whoami # Expect, immediately: READY climbing toward 5/5. Kubernetes obeyed you. ``` For a few seconds you have five copies running, exactly as you asked. Then: ```bash kubectl get deployment -n default whoami # Expect, shortly after: back to 2/2. Argo CD reverted your change, because # infra/whoami/deployment.yaml says replicas: 2 and that file is the truth. ``` The same happens to a `kubectl edit` of any field the manifest names. See who did it: ```bash kubectl get events -n argocd --sort-by=.lastTimestamp | tail -15 # Expect: events on the whoami Application, with reasons like OperationStarted # and OperationCompleted, timed to when your change disappeared. kubectl describe application -n argocd whoami | grep -A20 'Status:' # Expect: sync status Synced, and a history entry with the Git revision that # was applied. ``` The Argo CD interface tells the same story more clearly. Click `whoami`, then the **History and Rollback** tab: every sync is listed with the commit it applied. That list, not your shell history, is now the record of what happened to this application. **This is the moment to feel the trade rather than just read about it.** `kubectl scale` did nothing lasting. If that annoys you, the answer is not to turn `selfHeal` off. It is to notice that you were about to make an undocumented change to a running system, and that something just stopped you. ### (c) Delete something in Git, and watch prune do the same thing in reverse The two experiments above make Argo CD look purely protective. It is not. It is symmetric, and here is the direction that cuts the other way: ```bash cd ~/k3s-lab-gitops git rm infra/whoami/ingress.yaml git commit -m "Remove the whoami Ingress" git push ``` Now wait. This is the slow clock: up to three minutes before Argo CD looks at Git again. Watch the terminal running `kubectl get applications -n argocd -w`, or force it: ```bash kubectl -n argocd patch app whoami --type merge \ -p '{"metadata":{"annotations":{"argocd.argoproj.io/refresh":"hard"}}}' kubectl get ingress -n default whoami # Expect: "not found". Argo CD deleted it, because Git no longer asks for it. curl --max-time 5 -s -o /dev/null -w '%{http_code}\n' http://whoami.k3s.linsnotes.com # Expect: 404. The application is down, and it is down entirely correctly. ``` **Nothing went wrong.** One `git rm` and a push removed a live routing rule from a running cluster, in under three minutes, with no `kubectl delete` anywhere. That is the same mechanism that put the Ingress back in experiment (a), pointed the other way, and it is exactly what `prune: true` promises. Sit with that for a second before undoing it. The mechanism has no opinion about whether a deletion was intended. It only knows what the repository says. Everything that makes this safe in practice happens **before** the push: review, a protected branch, someone reading the diff. Argo CD is not the safety net. The pull request is. Put it back the way you now put everything back: ```bash git revert --no-edit HEAD git push kubectl -n argocd patch app whoami --type merge \ -p '{"metadata":{"annotations":{"argocd.argoproj.io/refresh":"hard"}}}' sleep 20 kubectl get ingress -n default whoami # Expect: it exists again. curl --max-time 5 http://whoami.k3s.linsnotes.com # Expect: the whoami text block. ``` `git revert` rather than editing the file back by hand, on purpose. It records that the removal happened and then was undone, which is a more honest history than a repository that quietly forgets. --- ## Step 9 — Clean up and save a restore point Less to remove than usual, and that is itself the result. Almost everything this part created is either infrastructure you are keeping or a file in a repository that does not live in the cluster at all. ### Tidy up ```bash # The port-forward from Step 6, if you started one and it is still running: pkill -f "port-forward.*argocd-server" 2>/dev/null || true # Confirm the repository is clean and pushed: cd ~/k3s-lab-gitops git status --short # expect: no output git log --oneline | head -8 # expect: your commits, including the revert ``` Check what is actually in the cluster: ```bash kubectl get applications -n argocd # Expect: metallb-config and whoami, both Synced and Healthy. kubectl get all -n default # Expect: the whoami deployment, replicaset, two pods, its service, and # service/kubernetes. NOT the usual one line baseline, and that is on purpose. ``` **`whoami` stays**, unlike Parts 3, 4 and 5 where the demo application always went. Two reasons. It costs nothing to keep now, because it is fully described in Git rather than being residue nobody can account for. And Part 7 needs a hostname to put a certificate on, so it is a head start rather than a mess. If you would rather end this part with an empty `default` namespace, do it the new way, which is also the last exercise: ```bash # Optional. The GitOps way to delete an application: delete the pointer. # kubectl delete application whoami -n argocd # Deleting an Application does NOT delete its objects by default. To have # them go too, the Application needs a deletion finalizer, which is a # separate setting worth reading about before you use it. ``` Leave `metallb-config`, Argo CD, MetalLB and ingress-nginx exactly as they are. ### Snapshot Same careful pattern as Parts 3, 4 and 5. All five stopped together, so the three etcd members are captured at one consistent instant: ```bash for n in worker1 worker2 master1 master2 master3; do lxc stop "$n"; done for n in master1 master2 master3 worker1 worker2; do lxc snapshot "$n" post-gitops done for n in master1 master2 master3 worker1 worker2; do lxc start "$n"; done sleep 45 kubectl get nodes # Expect: five Ready. Give this one a little longer than usual; Argo CD's # pods take a moment to settle after a cold start. kubectl get applications -n argocd # Expect: both Synced and Healthy again, with nobody having done anything. # Argo CD reconciled on startup, exactly as it does every three minutes. curl --max-time 5 http://whoami.k3s.linsnotes.com curl --max-time 5 -s -o /dev/null -w '%{http_code}\n' http://argocd.k3s.linsnotes.com # Expect: the whoami block, then 200. ``` ### What this snapshot does not contain Three things, and each for a good reason: | Not in the snapshot | Where it lives | What that means | |---|---|---| | The GitHub repository | github.com | Rolling nodes back does not roll Git back. Read the warning below. | | `/etc/hosts` on the host and your laptop | Those machines | Same as Part 5. Roll back and your names still resolve, which is what you want. | | The router's static route | Your router | Also unchanged by any rollback, also on purpose. | > **Rolling back to `post-gitops` does not roll back your repository, and Argo CD will notice.** This is new, and it is the one genuinely surprising consequence of running GitOps in a lab with snapshots. Restore all five nodes to `post-gitops` a month from now, after twenty commits, and within about three minutes Argo CD will have pulled the cluster **forward** to whatever `main` says today. The snapshot restores the machines; Argo CD restores the intent, and the intent has moved on. That is correct behaviour and it is also the opposite of what "restore a snapshot" usually feels like. If you genuinely want the cluster as it was on this date, roll Git back too, by checking out the commit you were on and pushing, or by pointing `targetRevision` at a tag. That the two now have to be rolled back together is the price of Git being the truth. It is a fair price, and it is better to meet it here than during an incident. {: .prompt-warning } --- ## Where you should be | Thing | State | |---|---| | Argo CD | Installed in namespace `argocd`, version pinned, admin password changed | | Argo CD interface | `http://argocd.k3s.linsnotes.com`, HTTP only, `server.insecure: "true"` | | Repository | Public GitHub repo, folders `bootstrap/`, `apps/`, `infra/` | | Applications | `metallb-config` (adopted from Part 5) and `whoami` (created from Git) | | Sync policy | Automated on both; `selfHeal` on both; `prune` on `whoami` only | | Sync interval | About three minutes for Git, seconds for cluster drift | | Adoption | Proven: live MetalLB objects taken over with no restart and no outage | | Failure test | Object deleted and restored; a hand edit reverted; a Git deletion pruned | | Snapshots | `post-gitops` on all five, taken stopped and together | | Still no TLS | Everything is plain HTTP. That is Part 7. | ### Things worth carrying forward - **GitOps is not "keep your YAML in Git". It is "the cluster follows Git".** Storing manifests in a repository and then applying them by hand gives you a nice archive and none of the benefits. The difference is entirely in whether a program is continuously closing the gap. - **Adopting live objects is normal and undramatic.** Argo CD compares, it does not recreate. If the YAML in Git matches what is running, the application goes `Synced` immediately and nothing restarts. That is what makes it usable on a cluster that already exists, which is every real cluster. - **Never commit `kubectl get -o yaml` output straight into Git.** It carries `uid`, `resourceVersion`, `creationTimestamp` and `managedFields`, which the cluster assigns and changes on its own. A desired state containing them can never match reality. Write clean manifests and check them against the cluster. - **`selfHeal` and `prune` are opposite directions of the same idea.** `selfHeal` fixes the cluster when the cluster is wrong. `prune` fixes the cluster when Git is shorter than it was. Turning on the first and not the second is a perfectly reasonable place to stop. - **Think about blast radius per application, not per cluster.** `prune: true` on a demo application is nothing. On the object that gives your load balancer its addresses, it can take out the interface you would use to fix it. Split applications along the lines of what you can afford to lose. - **Argo CD's server does its own TLS and will fight a plain HTTP Ingress.** `ERR_TOO_MANY_REDIRECTS` or a `502`, fixed by `server.insecure: "true"` in `argocd-cmd-params-cm` and a restart of `argocd-server`. This is not a lab quirk. It is the standard arrangement whenever something else terminates TLS. - **Two clocks: seconds for cluster drift, about three minutes for Git.** Knowing which one you are waiting on is the difference between patience and debugging. - **Something always bootstraps the bootstrapper.** Argo CD itself, its ConfigMap and its Ingress went in with `kubectl`. Keep that seam small, write it down, and stop pretending it does not exist. ### What Part 7 does Everything you have built is reachable and none of it is private. Your Argo CD password crosses the network in the clear. Your browser shows "Not secure" on every lab hostname. And the one setting that made the Argo CD interface work at all, `server.insecure`, is currently doing exactly what its name warns about, because there is no TLS anywhere for it to hand off to. Part 7 fixes all of that at once. You install **cert-manager**, which is a program that obtains and renews certificates for you, then point it at Let's Encrypt and issue real certificates for your lab hostnames. Along the way it deals with the awkward part of certificates on a private network: proving you own a name that only exists inside your house. Then `https://argocd.k3s.linsnotes.com` gets a padlock, `server.insecure` becomes the correct setting rather than a shortcut, and the break it on purpose step is a certificate expiring, on purpose, to watch renewal happen. There is a pleasing detail waiting for you. Once cert-manager is installed, adding a certificate to a hostname is a few lines in a manifest, in a folder Argo CD already watches. Which means Part 7's real work will be a commit. Continue with [**Part 7, Securing with TLS and cert-manager**](/posts/building-a-kubernetes-lab-with-k3s-and-lxd-part-7-securing-with-tls-and-cert-manager/). ### [Building a Kubernetes Lab with k3s and LXD Part 5 Exposing Applications](https://linsnotes.com/posts/building-a-kubernetes-lab-with-k3s-and-lxd-part-5-exposing-applications/) > Get traffic from your browser into a pod. Walk the three Service types, route by hostname with Ingress, then swap in MetalLB and ingress-nginx and see what actually changed. This is **Part 5** of a series on building a Kubernetes lab with k3s and LXD. If you are just arriving, start with [**Part 1, Preparing the Ubuntu Host**](/posts/building-a-kubernetes-lab-with-k3s-and-lxd-part-1-preparing-the-ubuntu-host/), then [**Part 2, Building the Five Nodes**](/posts/building-a-kubernetes-lab-with-k3s-and-lxd-part-2-building-the-five-nodes/), [**Part 3, Installing k3s**](/posts/building-a-kubernetes-lab-with-k3s-and-lxd-part-3-installing-k3s/), and [**Part 4, Adding Persistent Storage**](/posts/building-a-kubernetes-lab-with-k3s-and-lxd-part-4-adding-persistent-storage/). This part picks up where Part 4 left off. **Where you are.** You have five nodes running one Kubernetes cluster. Three masters share a database and vote. Two workers run programs. The cluster survives a master dying (Part 3) and keeps data when a worker dies (Part 4). Everything you have tested so far, though, you tested from *inside*. Every check ran through `kubectl`, or from a pod, or from the host over the lab network. Nothing you have built is reachable from a browser. **What this part does.** It closes that gap. You will get traffic from your own laptop, through your home network, into a pod running on a node, and back again. Then you will do it properly, so that ten applications can share one address instead of needing ten. **The main point, in one line.** Getting *out* of the cluster is easy. The interesting question is how many applications can share one way out, and the answer to that is Ingress. **What this part does not do.** No HTTPS yet. Everything here is plain HTTP on purpose, because certificates are a separate idea and mixing them in would hide what the networking is doing. Certificates are Part 7. **What you need before starting.** The `post-storage` snapshot from Part 4's Step 10, all five nodes Ready, and a router you can log into. Step 1 checks the cluster. Step 5 is the router part, and it is optional if you are happy browsing from the host itself. --- ## Words you'll need These build on Parts 2, 3 and 4. Here are the new ones. | Word | What it means | |---|---| | **Service** | A stable name and address for a group of pods. Pods come and go and their addresses change. A Service does not, so everything talks to the Service instead of to pods. | | **ClusterIP** | The default kind of Service. It gets an address that only works *inside* the cluster. Nothing outside can reach it. | | **NodePort** | A Service that also opens one high-numbered port, the same port, on **every** node. Reach any node on that port and you reach the Service. | | **LoadBalancer** | A Service that asks the cluster for a real, outside address of its own. What creates that address depends on what you installed. | | **ServiceLB** | The thing k3s ships that answers `LoadBalancer` requests. Also called Klipper. It uses the nodes' own addresses rather than handing out new ones. | | **MetalLB** | A replacement for ServiceLB. It owns a pool of spare addresses and hands out a real, separate one per Service. | | **Ingress** | A rule that says "traffic asking for *this hostname* goes to *that Service*." One address can serve many hostnames. | | **Ingress controller** | The program that reads your Ingress rules and actually does the routing. k3s ships **Traefik**. The most common alternative is **ingress-nginx**. | | **IngressClass** | Which controller should handle a given Ingress rule. You name it in the rule. This is what makes swapping controllers a one word change. | | **reverse proxy** | A program that takes a request meant for somewhere else and forwards it on. An Ingress controller is a reverse proxy with Kubernetes rules. | | **Host header** | A line every browser sends with a request, saying which hostname it asked for. Ingress reads it to decide where the request goes. This is the whole trick. | | **ARP** | How a machine on a local network asks "who owns this address?" MetalLB works by answering that question on behalf of an address nobody really owns. | | **static route** | A rule you add to a router telling it "traffic for *this* network goes to *that* machine." This is how your laptop learns to reach the lab. | --- ## Three things to know before you start ### 1. You already have both pieces, and nobody told you Most guides about exposing applications start by telling you that bare metal Kubernetes has no load balancer, so a `LoadBalancer` Service sits forever showing `EXTERNAL-IP `, and that you must install MetalLB to fix it. **That is not true on k3s.** k3s ships a load balancer called ServiceLB and it is switched on by default. It also ships an Ingress controller, Traefik, and that is on by default too. You have already seen the proof, back in Part 3. When you first ran `kubectl get pods -A`, the expected output included `traefik` and `svclb-traefik`. Those `svclb` pods are the load balancer. The `traefik` pod is the Ingress controller. Both have been running since your first hour of Part 3, doing nothing, because nothing has asked them for anything yet. So Steps 2 to 4 cost you no installation at all. You will learn Services and Ingress with what is already there. Only later, in Steps 6 and 7, do you install anything, and when you do it is a **swap**, not a filling of a hole. Knowing that changes what those steps mean. You are not fixing something broken. You are choosing a different tool and finding out what the choice buys you. ### 2. The one thing likely to stop you is not Kubernetes Every part of this series has had one step that can genuinely block you. Part 3 had the snapshotter on ZFS. Part 4 had NFS mounting inside a container. This part has routing, and it is worth naming now. Your nodes live on `10.99.99.0/24`, a private network that exists only on your host (Part 1, Step 7). Your laptop lives on your home network, something like `192.168.1.0/24`. Those two networks have never heard of each other. Your laptop has no idea `10.99.99.0/24` exists, and if you ask it to reach `10.99.99.240` it will hand the request to your router, which will also have no idea, and send it out to the internet where it dies. **This is not a Kubernetes problem and no amount of Kubernetes will fix it.** It is one line of configuration on your router. Step 5 does it. If your router will not let you, you can still do everything in this part from the host's own browser, because the host is *on* both networks. Nothing is lost, and I say so where it matters. ### 3. A load balancer and an Ingress are not the same thing, and the difference is the point These two words get used as if they are interchangeable. They are not, and the whole shape of this part depends on the difference. - A **load balancer** gets you an address. That is all. One address, one Service. Ten applications, ten addresses. - An **Ingress** sits behind *one* of those addresses and splits traffic by hostname. Ten applications, still one address. So they stack. You use a load balancer once, to get a front door, then you use Ingress to put ten rooms behind that one door. Steps 2 and 3 build the door and show you why one door per room does not scale. Step 4 puts the rooms behind it. --- ## Step 1 — Check Part 4 is still good **Why:** the same reason as every part. If something drifted, find out now, while a fault still looks like itself. **Safe to run:** everything here only reads. ```bash lxc info | head -3 # wakes LXD if it's asleep, same trick as before sleep 3 lxc list # expect five RUNNING rows, plus the STOPPED template kubectl get nodes # Expect: five Ready. kubectl get all -n default # Expect ONE line: service/kubernetes. That's the clean baseline from Part 4. kubectl get storageclass # Expect two: local-path (default) and nfs-client. Part 4's work is still here. ``` Now look at the two things this part is about, both of which have been running since Part 3 and which you have probably never looked at: ```bash kubectl get svc -n kube-system traefik ``` Read that line carefully, because it contradicts what most guides told you to expect: ``` NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) traefik LoadBalancer 10.43.x.x 10.99.99.11,10.99.99.12,10.99.99.13,10.99.99.21,10.99.99.22 80:3xxxx/TCP,443:3xxxx/TCP ``` **`EXTERNAL-IP` is not ``.** It lists your five node addresses. Something answered the request for an outside address, and that something is ServiceLB. Look at it directly: ```bash kubectl get pods -n kube-system -l svccontroller.k3s.cattle.io/svcname=traefik -o wide # Expect: one svclb-traefik pod per node, all Running. ``` One pod per node, and each one holds port 80 and port 443 on the node it runs on. That is the whole mechanism. There is no clever virtual address. ServiceLB just parks a tiny pod on every node that grabs the port and forwards inward. Keep that in mind, because it is exactly why ServiceLB runs out of room in Step 3. --- ## Step 2 — The three ways out, walked one at a time **What this does.** It deploys one small application and then changes only its Service type, three times, so you can see what each type gives you and what it costs. Same pods throughout. Only the way in changes. The application is `traefik/whoami`. It is about 4 MB, it does one thing, and that one thing is perfect here: it answers every request by printing the request back at you, including which pod answered and which hostname you asked for. When you get to Ingress in Step 4, that last detail is what makes the routing visible instead of something you take on faith. ```bash kubectl create deployment whoami --image=traefik/whoami --replicas=2 kubectl get pods -l app=whoami -o wide # Expect: two Running pods. Note which nodes they landed on. ``` Two copies, so you can also see requests being spread between them. ### 2a. ClusterIP, the inside only address ```bash kubectl expose deployment whoami --port=80 # No --type flag, so you get the default: ClusterIP. kubectl get svc whoami # Expect: TYPE ClusterIP, an address in 10.43.x.x, EXTERNAL-IP . ``` That `10.43.x.x` is from the service range Part 1 warned you not to collide with. Now try to reach it from the host: ```bash curl --max-time 5 http://$(kubectl get svc whoami -o jsonpath='{.spec.clusterIP}') # Expect: it times out. This address does not exist outside the cluster. ``` Now reach it from *inside*, the way one pod talks to another: ```bash kubectl run probe --image=busybox --rm -it --restart=Never -- \ wget -qO- --timeout=5 http://whoami # Expect: a block of text starting with "Hostname: whoami-xxxxx". # Run it two or three times and the Hostname changes between your two pods. ``` **That is a ClusterIP doing its job.** Inside the cluster it is a name that always works. Outside, it does not exist at all. This is the right default: most Services in a real cluster are one program talking to another, and those have no business being reachable from the internet. ### 2b. NodePort, a door on every node ```bash kubectl patch svc whoami -p '{"spec":{"type":"NodePort"}}' kubectl get svc whoami # Expect: PORT(S) now reads something like 80:31234/TCP. ``` That second number is the NodePort. Kubernetes picked it from the range 30000 to 32767. It is now open on **all five nodes**, whether or not a `whoami` pod is running there: ```bash NODEPORT=$(kubectl get svc whoami -o jsonpath='{.spec.ports[0].nodePort}') echo "NodePort is $NODEPORT" curl --max-time 5 http://10.99.99.21:$NODEPORT # a worker curl --max-time 5 http://10.99.99.11:$NODEPORT # a master, with no whoami pod on it # Expect: both work, and both answer with a whoami pod's details. ``` The master works even though no `whoami` pod runs there. The node took the request and forwarded it across the cluster network, the same vxlan path you proved in Part 3. **So NodePort works. Here is why nobody uses it directly.** - The port is a random high number, and `http://10.99.99.21:31234` is not an address you give anyone. - You have to know a node address, and you have to pick one. Pick the wrong one and you are down when that node is. - Every application needs its own port, and you have to keep track of which is which. NodePort is a building block, not an answer. Almost every load balancer, ServiceLB included, is quietly built on top of it. ### 2c. LoadBalancer, an address of its own ```bash kubectl patch svc whoami -p '{"spec":{"type":"LoadBalancer","ports":[{"port":8080,"targetPort":80}]}}' kubectl get svc whoami ``` Note the port. The Service now listens on **8080**, not 80, and that is deliberate. Traefik already holds port 80 on every node, and Step 3 is about what happens when two things want the same one. For now, stay out of its way. Expect `EXTERNAL-IP` to fill in with your five node addresses, the same as Traefik's. Then: ```bash curl --max-time 5 http://10.99.99.21:8080 # Expect: whoami answers. ``` A sensible port, and an address list you did not have to invent. Check what appeared to serve it: ```bash kubectl get pods -n kube-system -l svccontroller.k3s.cattle.io/svcname=whoami -o wide # Expect: a new svclb-whoami pod on every node, Running. ``` **ServiceLB made a second set of pods, one per node, holding port 8080.** That is the same trick as Traefik's, on a different port. It is honest and it works, and its limit is now obvious enough to walk into on purpose. --- ## Step 3 — Where one address per application stops working **What this does.** It adds a second application that wants the same port, and watches the wheels come off. This is not a bug. It is arithmetic, and it is the reason Ingress exists. Add a second application, and ask for a `LoadBalancer` on port 8080 as well: ```bash kubectl create deployment web --image=nginx --replicas=2 kubectl expose deployment web --port=8080 --target-port=80 --type=LoadBalancer kubectl get svc web # Expect: EXTERNAL-IP stays . This is the famous pending, and now # you know it is not "bare metal has no load balancer". It is a port clash. ``` Find out why, from the thing that could not start: ```bash kubectl get pods -n kube-system -l svccontroller.k3s.cattle.io/svcname=web # Expect: svclb-web pods Pending, not Running. kubectl describe pod -n kube-system -l svccontroller.k3s.cattle.io/svcname=web | grep -A5 Events # Expect: a message about not having free ports for the requested pod ports. ``` **There it is, in plain words.** ServiceLB works by parking a pod on each node that grabs a port on that node. `whoami` already took 8080 on all five. There is no sixth node to put `web` on, and there is only one port 8080 per machine, so `web` waits forever. You have three bad options and one good one: | Option | Why it is bad | |---|---| | Give `web` a different port | Now you are back to remembering which application is on which port. This is NodePort with extra steps. | | Add more nodes | The clash is per node, so every node still has one port 8080. More nodes does not help at all. | | Install MetalLB so each Service gets its **own** address | Better, and Step 6 does it. But ten applications still means ten addresses to remember, and each still needs a DNS name eventually. | | Put **one** thing on port 80 and let it split traffic by hostname | This is Ingress. One address, one port, as many applications as you like. | Clean up the failed attempt before moving on. Both applications stay, but they go back to being plain ClusterIP Services, which is what Ingress expects behind it: ```bash kubectl patch svc web -p '{"spec":{"type":"ClusterIP","ports":[{"port":80,"targetPort":80}]}}' kubectl patch svc whoami -p '{"spec":{"type":"ClusterIP","ports":[{"port":80,"targetPort":80}]}}' kubectl get svc whoami web # Expect: both ClusterIP, both on port 80, EXTERNAL-IP for both. kubectl get pods -n kube-system | grep svclb # Expect: only svclb-traefik pods left. The svclb-whoami and svclb-web # pods are gone, because ServiceLB cleans up when the Service stops # asking for an address. ``` **Notice what you just did.** You took the outside address away from both applications. They are now unreachable from outside again, exactly like Step 2a. That is correct, and it is the setup for the next step: from here on, only *one* thing in the whole cluster has an outside address, and everything else hides behind it. --- ## Step 4 — Route by hostname with the Ingress you already have **What this does.** It puts both applications behind Traefik, which already holds port 80 on every node, and tells Traefik which one to use based on the hostname the browser asked for. No installation. Traefik has been sitting there since Part 3 waiting to be given a rule. ### The idea, before the YAML When your browser fetches `http://whoami.k3s.linsnotes.com`, it does two separate things that are easy to blur together: 1. It looks up the **name** to get an address, and connects to that address. 2. It sends the **name itself** along in the request, in a line called the `Host` header. Step 1 is DNS, and every one of your lab hostnames will resolve to the *same* address. Step 2 is what Ingress reads. So all your applications share one address, and the controller behind that address opens each request, reads the `Host` line, and decides where to send it. That is the whole mechanism. It is a receptionist at one front desk reading the name on the envelope. ### Write the rule One Ingress object, two rules, one per hostname: ```bash cat > lab-ingress.yaml <<'EOF' apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: lab spec: ingressClassName: traefik # which controller handles this. See Step 7. rules: - host: whoami.k3s.linsnotes.com http: paths: - path: / pathType: Prefix backend: service: name: whoami # the ClusterIP Service from Step 2 port: number: 80 - host: web.k3s.linsnotes.com http: paths: - path: / pathType: Prefix backend: service: name: web port: number: 80 EOF kubectl apply -f lab-ingress.yaml ``` Check it registered: ```bash kubectl get ingress lab # Expect: CLASS traefik, HOSTS listing both names, ADDRESS filling in with # your node addresses, PORTS 80. ``` The `ADDRESS` column is Traefik's own `LoadBalancer` address from Step 1, borrowed. Your Ingress does not get an address of its own. It rides on the controller's. ### Test it without touching DNS You do not need any DNS set up to test this, and you should test it before you do, so that a DNS mistake later cannot be confused with a routing mistake. `curl` will send whatever `Host` header you tell it to: ```bash curl --max-time 5 -H "Host: whoami.k3s.linsnotes.com" http://10.99.99.11 # Expect: the whoami text block. curl --max-time 5 -H "Host: web.k3s.linsnotes.com" http://10.99.99.11 # Expect: the nginx welcome page HTML. curl --max-time 5 -H "Host: nothing.k3s.linsnotes.com" http://10.99.99.11 # Expect: 404 page not found. No rule matches, so Traefik refuses. ``` **Read those three results together, because they are the point of this whole part.** Same address. Same port. Three different answers, decided entirely by one line of text in the request. That third one matters as much as the first two: a hostname with no rule gets nothing, which is what makes it safe to point many names at one address. Look at the `whoami` output once more. Near the top it prints the `Host:` line it received. That is not `curl` echoing back at you, it is the application reporting what actually arrived, having passed through Traefik untouched. ### Now make the names work in a browser Nothing in DNS knows about `whoami.k3s.linsnotes.com` yet, so a browser cannot find it. The quickest honest fix is your host's own hosts file, which every machine checks before it asks DNS: ```bash echo '10.99.99.11 whoami.k3s.linsnotes.com web.k3s.linsnotes.com' | sudo tee -a /etc/hosts ``` Then, on the host: ```bash curl --max-time 5 http://whoami.k3s.linsnotes.com # Expect: whoami answers. No -H flag this time; the browser-style path works. ``` Open `http://whoami.k3s.linsnotes.com` in the host's browser and you will see the whoami page. Refresh a few times and the `Hostname` line alternates between your two pods, which is the Service spreading requests without being asked. > **Why `10.99.99.11` and not something nicer?** Because right now, with ServiceLB, Traefik's address really is "any of the five nodes", so you have to pick one, and if you pick `master1` and stop `master1`, your name stops working. That is a genuine weakness and it is what MetalLB fixes in Step 6. Do not fix it now. Feel it first. {: .prompt-info } --- ## Step 5 — Reach the lab from the rest of your network **What this does.** It teaches your home network that `10.99.99.0/24` exists and lives behind your Ubuntu host. After this, your laptop and your phone can reach the lab. **This step is optional.** Everything in Parts 5 to 8 works from the host's own browser. Skip it if your router will not cooperate, and go to Step 6. ### Why nothing works right now Your host sits on two networks at once. You saw both back in Part 1: ```bash ip -brief addr # Expect two addresses that matter: your real LAN one on something like # enp5s0, and 10.99.99.1 on lxdbr0. ``` Your laptop only knows about the first one. When it wants `10.99.99.240`, it checks its own network, does not find it, and hands the packet to the router, which is the standard "I do not know, you deal with it" move. The router does the same thing and sends it to your internet provider, where it is dropped, because `10.99.99.0/24` is private and nobody outside your house should ever route it. Nothing is broken. Nobody has ever been told where that network lives. ### Tell the router Find your host's address on the real network: ```bash ip -brief addr | grep -v lxdbr0 | grep -v LOOPBACK # Note the address on your physical interface, for example 192.168.1.20. ``` Now log into your router and add a **static route**. Every router calls this something slightly different, usually under Advanced, Routing, or Static Routes. The three values are always the same: | Field | Value | |---|---| | Destination network | `10.99.99.0` | | Subnet mask | `255.255.255.0` (this is what `/24` means) | | Gateway, or Next hop | your host's LAN address, for example `192.168.1.20` | In plain words, you are telling the router: *anything for `10.99.99.x`, hand it to that Ubuntu machine, it knows what to do.* Test from your laptop: ```bash ping -c3 10.99.99.11 curl --max-time 5 -H "Host: whoami.k3s.linsnotes.com" http://10.99.99.11 ``` Then add the same hosts line on your laptop and browse to it. On macOS and Linux the file is `/etc/hosts`. On Windows it is `C:\Windows\System32\drivers\etc\hosts`, and you need to open the editor as Administrator. ### If your router cannot do static routes Plenty of provider supplied routers cannot. You can add the route on each client instead. It only helps that one machine, but that is often all you need: ```bash # macOS sudo route -n add 10.99.99.0/24 192.168.1.20 # Linux sudo ip route add 10.99.99.0/24 via 192.168.1.20 # Windows, in an Administrator command prompt route add 10.99.99.0 mask 255.255.255.0 192.168.1.20 ``` Those are temporary and disappear at the next reboot, which is fine for a lab and easy to undo if you get it wrong. ### Two things that make this work, which you already did This step is one line on a router, but it only works because of two decisions made much earlier, and they are worth connecting up. **Your host is willing to forward.** Part 2's Step 3 set `net.ipv4.ip_forward = 1`. Without it, the host would receive your laptop's packet, see it was addressed to someone else, and drop it. It was set for the containers' own internet access, and it happens to be exactly what is needed here too. **The reply does not get rewritten on the way back, and it easily could have.** Part 1's Step 7 set up NAT, so traffic leaving `10.99.99.0/24` for anywhere else gets its sender address rewritten to the host's. Read that rule on its own and you would expect this step to fail: a reply from `10.99.99.240` heading to `192.168.1.50` matches the description, and if it were rewritten, your laptop would receive an answer from an address it never asked, and drop it. It works because Linux only decides about address rewriting on the **first** packet of a connection. After that the connection is remembered, and every later packet is handled to match. Your laptop started this connection, and no rewriting was done on that first packet, so nothing is rewritten on the replies either. The NAT rule is for connections started from *inside*, and it stays out of the way of connections started from outside. You do not need to memorise that. It is here because "reaching in worked, and I half expected NAT to eat it" is a reasonable thing to wonder, and the answer is a real mechanism rather than luck. --- ## Step 6 — Swap ServiceLB for MetalLB **What this does.** It replaces the load balancer k3s gave you with the one you would meet on real bare metal, and gives Traefik a single stable address instead of "any of five nodes". **Be clear about why.** Everything works right now. This is not a fix. You are trading a simple thing that works for a more capable thing that is also what almost every bare metal cluster in the world actually runs. Three things you get: - **One address, not five.** Your hosts file stops naming a specific node, so stopping that node stops nothing. - **Addresses that are not node addresses.** Ten Services can each have their own port 80, because each has its own address. - **A pool you control.** In Step 7 you install a second Ingress controller, and it needs its own address. With ServiceLB it would clash on port 80. With MetalLB it just takes the next free address. ### 6a. Keep MetalLB's addresses away from LXD's MetalLB is about to start handing out addresses from `10.99.99.0/24`. LXD's own DHCP server is also handing out addresses from `10.99.99.0/24` to any container without a fixed one, which is how `template` ended up on `10.99.99.105` back in Part 2. Nothing stops those two from picking the same address, and if they ever do, the result is two machines answering to one address and traffic going to whichever replied last. It would be intermittent, it would look like a Kubernetes fault, and it would be miserable to trace. So do the ten seconds of work that makes it impossible: ```bash lxc network set lxdbr0 ipv4.dhcp.ranges 10.99.99.100-10.99.99.200 lxc network get lxdbr0 ipv4.dhcp.ranges # Expect: 10.99.99.100-10.99.99.200 ``` LXD now only gives out `.100` to `.200`. Your fixed node addresses (`.11` to `.13`, `.21`, `.22`) are outside that range and are unaffected, because they are assigned by name, not from the pool. MetalLB will use `.240` to `.250`, which LXD can no longer touch. ### 6b. Turn ServiceLB off Two load balancers both trying to answer `LoadBalancer` Services is a fight you do not want to referee. Switch ServiceLB off first. The tidy way to change k3s settings is `/etc/rancher/k3s/config.yaml`, which k3s reads at startup. It does the same job as adding flags to the install command, but you can edit it later without reinstalling anything: ```bash for n in master1 master2 master3; do lxc exec "$n" -- bash -c 'mkdir -p /etc/rancher/k3s && cat > /etc/rancher/k3s/config.yaml < **Write the whole file, do not append to it.** If you add a second `disable:` block later instead of editing the existing one, the file has the same key twice, which is invalid YAML, and k3s will refuse to start. Step 7 edits this file again, and it rewrites it whole for exactly this reason. {: .prompt-warning } Restart the masters one at a time, waiting for each, so quorum is never at risk. This is the same care Part 3 asked for when joining them: ```bash for n in master1 master2 master3; do echo "== restarting k3s on $n" lxc exec "$n" -- systemctl restart k3s sleep 30 kubectl get nodes done ``` Confirm ServiceLB is gone: ```bash kubectl get pods -n kube-system | grep svclb # Expect: no output. The svclb-traefik pods are gone. kubectl get svc -n kube-system traefik # Expect: EXTERNAL-IP is now . NOW you have the pending everyone # warned you about, and it is because you asked for it. ``` Your lab is unreachable from outside at this moment. That is expected, and MetalLB is about to fix it. ### 6c. Install MetalLB MetalLB installs from one manifest. The URL is version pinned, so check the current release on the MetalLB site rather than trusting a version number in a blog post, including this one: ```bash kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.14.9/config/manifests/metallb-native.yaml kubectl wait --namespace metallb-system \ --for=condition=ready pod \ --selector=app=metallb \ --timeout=180s # Expect: pods condition met. This waits rather than making you poll. kubectl get pods -n metallb-system # Expect: one controller pod, and one speaker pod per node (five). ``` Two kinds of pod, doing two different jobs: - The **controller** hands out addresses. One per cluster. - The **speaker** claims an address on the network. One per node, because the claim has to be made from a machine actually on that network. Nothing has been given an address yet. MetalLB will not invent a pool for you, and that is deliberate, because guessing at a range on someone else's network is how you take out a printer. ### 6d. Give it a pool ```bash cat > metallb-pool.yaml <<'EOF' apiVersion: metallb.io/v1beta1 kind: IPAddressPool metadata: name: lab-pool namespace: metallb-system spec: addresses: - 10.99.99.240-10.99.99.250 # outside LXD's DHCP range from 6a --- apiVersion: metallb.io/v1beta1 kind: L2Advertisement metadata: name: lab-l2 namespace: metallb-system spec: ipAddressPools: - lab-pool EOF kubectl apply -f metallb-pool.yaml ``` Two objects, and both are needed: - **`IPAddressPool`** is the list of addresses MetalLB is allowed to give out. On its own it does nothing except reserve them on paper. - **`L2Advertisement`** is what makes an address real. It tells the speakers to answer ARP for these addresses, which is how a machine on a local network claims one. Without it, a Service gets an address that nothing on the network has ever heard of, and every request for it goes nowhere. This is the most common MetalLB mistake, and the symptom is an `EXTERNAL-IP` that looks perfectly healthy and does not answer. Now watch Traefik pick up an address on its own: ```bash kubectl get svc -n kube-system traefik # Expect: EXTERNAL-IP is now a single address, 10.99.99.240. ``` **One address, not five.** Nobody edited the Traefik Service. It has been asking for a `LoadBalancer` address since Part 3, ServiceLB used to answer with node addresses, and now MetalLB answers with a real one from the pool. Point your names at it and test: ```bash sudo sed -i 's/^10\.99\.99\.11 whoami/10.99.99.240 whoami/' /etc/hosts grep k3s.linsnotes.com /etc/hosts # Expect: 10.99.99.240 whoami.k3s.linsnotes.com web.k3s.linsnotes.com curl --max-time 5 http://whoami.k3s.linsnotes.com curl --max-time 5 http://web.k3s.linsnotes.com # Expect: both answer, same as before, through a single address now. ``` Update the same line on your laptop if you did Step 5. --- ## Step 7 — Swap Traefik for ingress-nginx **What this does.** It replaces the Ingress controller with the one you are most likely to meet at work, and shows that your Ingress rules barely change. **Why bother, when Traefik works?** Two honest reasons and one that is not. - **ingress-nginx is what most workplaces and most documentation assume.** If a job description says Ingress, this is usually the one behind it. Knowing both, and knowing they are interchangeable, is worth more than knowing either. - **It proves the Ingress object is portable.** You will change one word and everything keeps working. That is the actual lesson here, and you cannot learn it with only one controller installed. - It is **not** because Traefik is worse. It is not. This is a swap, not an upgrade. ### 7a. Turn Traefik off Rewrite the config file whole, with both entries in one `disable:` list: ```bash for n in master1 master2 master3; do lxc exec "$n" -- bash -c 'mkdir -p /etc/rancher/k3s && cat > /etc/rancher/k3s/config.yaml < **`ingressClassName` is the whole reason this was easy.** An Ingress does not name a program, it names a class, and a controller claims the classes it owns. That indirection is why a rule written for Traefik works on ingress-nginx untouched. Where they differ is the extra features each one adds through annotations, and those are not portable. Keep the plain rules plain and you keep the choice. {: .prompt-tip } --- ## Step 8 — Break it on purpose Every part of this series ends by taking something away to see what the design was for. Part 3 killed a master to watch quorum protect itself. Part 4 killed a worker to watch data survive. Here you kill the node holding your one address, and find out something about MetalLB that its name actively hides. ### What "layer 2 mode" really does You configured MetalLB in L2 mode, and it is easy to assume that spreads traffic across all five nodes. It does not. **In L2 mode, exactly one node holds a given address at a time.** One speaker answers ARP for `10.99.99.240` and all traffic for it arrives at that one node, which then forwards it into the cluster like any other. The other four speakers stay quiet and watch. So MetalLB in this mode gives you **failover**, not load spreading. If the holder dies, another speaker takes over the address in a few seconds. That is a real and useful guarantee, and it is a different one from what most people assume when they hear "load balancer". Worth knowing before you build a mental model on the name. Find out which node holds it: ```bash kubectl logs -n metallb-system -l component=speaker --tail=200 | grep -i announc | tail -20 # Look for a line mentioning your address and a node name. That node is # the current holder. ``` ### Take that node away ```bash # Replace with the node name from above. lxc stop ``` Watch what happens from the host, in a loop, so you can see the gap rather than guess at it: ```bash for i in $(seq 1 30); do printf '%s ' "$i" curl --max-time 2 -s -o /dev/null -w '%{http_code}\n' http://whoami.k3s.linsnotes.com || echo "FAILED" sleep 2 done ``` Expect a short run of failures and then `200` again. The gap is the other speakers noticing and one of them claiming the address. It is usually a few seconds, not minutes. ```bash kubectl logs -n metallb-system -l component=speaker --tail=100 | grep -i announc | tail -5 # Expect: a different node announcing your address now. ``` **Nothing was reconfigured and nothing was restarted.** The address moved, and your hostname kept working, because the name points at the address and not at a machine. Compare that to Step 4, where your hosts file named `10.99.99.11` and stopping `master1` would have taken your lab down with it. That is the difference this step bought, made visible. Bring the node back: ```bash lxc start sleep 30 kubectl get nodes # expect: five Ready curl --max-time 5 http://whoami.k3s.linsnotes.com ``` The address usually stays where it moved to. It has no reason to move back, and MetalLB does not shuffle things without cause. --- ## Step 9 — Clean up and save a restore point Remove the two demo applications, and keep the plumbing. Same rule as Part 4's NFS provisioner: the test applications go, the infrastructure stays, because the next parts build on it. ```bash kubectl delete ingress lab kubectl delete deployment whoami web kubectl delete service whoami web kubectl delete pod probe --ignore-not-found rm -f lab-ingress.yaml metallb-pool.yaml kubectl get all -n default # Expect ONE line: service/kubernetes. The familiar clean baseline. ``` What stays, on purpose: ```bash kubectl get pods -n metallb-system # MetalLB, still running kubectl get pods -n ingress-nginx # ingress-nginx, still running kubectl get svc -n ingress-nginx ingress-nginx-controller # Expect: still holding its address. This is the front door Parts 6, 7 # and 8 will put things behind. ``` Leave the hosts file entry too. You will add more names to it in the parts ahead. Snapshot all five, stopped and together, exactly as Parts 3 and 4 taught: ```bash for n in worker1 worker2 master1 master2 master3; do lxc stop "$n"; done for n in master1 master2 master3 worker1 worker2; do lxc snapshot "$n" post-ingress done for n in master1 master2 master3 worker1 worker2; do lxc start "$n"; done sleep 30 kubectl get nodes # expect: five Ready kubectl get svc -n ingress-nginx ingress-nginx-controller # Expect: same address as before the restart. MetalLB re-claims it on startup. ``` One thing this snapshot does not hold, for the same reason Part 4's did not hold the NFS server: **your hosts file and your router's static route are host and network state, not cluster state.** Roll the nodes back and both are still there, which is what you want. --- ## Where you should be | Thing | State | |---|---| | Load balancer | MetalLB, L2 mode, pool `10.99.99.240-250`. ServiceLB disabled. | | LXD DHCP | Narrowed to `.100` to `.200`, so it can never collide with the pool | | Ingress controller | ingress-nginx, on a MetalLB address. Traefik disabled. | | k3s config | `/etc/rancher/k3s/config.yaml` on all three masters, disabling `servicelb` and `traefik` | | Name resolution | `/etc/hosts` on the host, and on your laptop if you did Step 5 | | Reachability | Host always. Rest of your network, if you added the static route. | | Failover | Seen: the address moved to another node in seconds when its holder died | | Snapshots | `post-ingress` on all five, taken stopped and together | ### Things worth carrying forward - **k3s already gave you a load balancer and an Ingress controller.** ServiceLB and Traefik have been running since Part 3. If a guide tells you bare metal Kubernetes leaves `EXTERNAL-IP` pending, it is not describing k3s. Check `kubectl get svc -n kube-system` before you install anything. - **A load balancer gets you an address. Ingress splits one address between many applications.** They stack, and you need both. Ten applications is one load balancer address and ten Ingress rules, not ten addresses. - **Ingress routes on the `Host` header, which means DNS and routing are separate problems.** You can test every rule with `curl -H "Host: ..."` before any name resolves anywhere. Do that first, and a DNS mistake can never look like a routing mistake. - **`ingressClassName` is what makes controllers swappable.** Plain Ingress rules move between Traefik and ingress-nginx untouched. Controller specific annotations do not. Prefer plain rules. - **MetalLB in L2 mode is failover, not load spreading.** One node holds each address. Losing it costs you a few seconds, not the service. - **Reaching a lab network from your laptop is a routing problem, not a Kubernetes one.** One static route on the router, and `ip_forward` on the host, which Part 2 already set. ### What Part 6 does Everything you installed in this part, you installed by hand: a manifest here, a Helm command there, a config file edited on three masters. It works, and in six months you will not remember any of it. Part 6 fixes that. You put all of it in a Git repository, hand the repository to **Argo CD**, and let the cluster keep itself matching what the repository says. Then you delete something important on purpose and watch it come back on its own, which is the same break it and see trick this series always ends on, aimed at the deployment process itself. Argo CD also has a web interface, and you already have everything needed to put it behind a hostname. Continue with [**Part 6, GitOps with Argo CD**](/posts/building-a-kubernetes-lab-with-k3s-and-lxd-part-6-gitops-with-argo-cd/). ### [Building a Kubernetes Lab with k3s and LXD Part 4 Adding Persistent Storage](https://linsnotes.com/posts/building-a-kubernetes-lab-with-k3s-and-lxd-part-4-adding-persistent-storage/) > Give the k3s cluster persistent storage with local-path and NFS, then kill a worker to prove a stateful app keeps its data when it moves to another node. This is **Part 4** of a series on building a Kubernetes lab with k3s and LXD. If you're just arriving, start with [**Part 1, Preparing the Ubuntu Host**](/posts/building-a-kubernetes-lab-with-k3s-and-lxd-part-1-preparing-the-ubuntu-host/), [**Part 2, Building the Five Nodes**](/posts/building-a-kubernetes-lab-with-k3s-and-lxd-part-2-building-the-five-nodes/), and [**Part 3, Installing k3s**](/posts/building-a-kubernetes-lab-with-k3s-and-lxd-part-3-installing-k3s/). This part picks up right where Part 3 left off. **Where you are.** Part 3 turned five machines into one healthy Kubernetes cluster: three masters that share a database and vote, and two workers that run programs. You proved the cluster comes up, that pods on different nodes can talk to each other, and that quorum protects itself when a master dies. But the cluster is empty. Every test in Part 3 cleaned up after itself and confirmed `kubectl get pvc -A` was empty; nothing was ever stored. **What this part does.** It gives your programs a place to keep data. First with the storage k3s already ships with (local disk), then with network storage (NFS) that any node can reach. Along the way it shows you the one fact that makes cluster storage different from storage on a single machine: **where the data actually lives decides whether it survives a pod moving to another node.** Then it kills a worker on purpose to prove the point. **What this part does NOT do.** It doesn't set up production-grade replicated block storage in full. That's mentioned at the end (it's what the `open-iscsi` package was for), but the lab's lesson lands with NFS, which is easier to see through. **What you need before starting.** The `post-install` snapshot from Part 3's Step 6, all five nodes Ready, and the `nfs-common` package that Part 2 put in the template. Step 1 checks all of this. --- ## Words you'll need These build on Parts 2 and 3. Here are the new ones. | Word | What it means | |---|---| | **stateful** | A program that keeps data it cares about: a database, a wiki, anything with a file that must outlive the program. The opposite is **stateless**: nginx and busybox from Part 3 kept nothing, so killing them lost nothing. | | **volume** | A directory that Kubernetes hands to a pod for storing files. The question that matters all through Part 4 is: *where does that directory physically live?* | | **PersistentVolume (PV)** | A piece of storage that exists in the cluster on its own, independent of any pod. Think of it as "a disk the cluster knows about." | | **PersistentVolumeClaim (PVC)** | A pod's *request* for storage: "I need 1 GB that survives restarts." Kubernetes matches the claim to a PV. Pods ask for PVCs; they don't touch PVs directly. | | **StorageClass** | A named recipe for making storage automatically. When a PVC names a StorageClass, the cluster creates a PV to satisfy it, so no one has to make the disk by hand. k3s ships with one called `local-path`. | | **provisioner** | The program behind a StorageClass that actually creates the storage when a PVC asks. `local-path` has one; NFS will need one you add. | | **dynamic vs static** | *Dynamic*: a PVC arrives, and the provisioner makes a PV on the spot. *Static*: you create the PV by hand first, and the PVC binds to it. Dynamic is the normal way; static is useful for seeing the wiring. | | **local-path** | k3s's built-in StorageClass. It stores data on the local disk of whichever node the pod runs on. It's fast and needs zero setup, but it's node-locked, which is the whole lesson of Steps 2 and 3. | | **NFS** | Network File System. One machine (the **server**) shares a directory; other machines (the **clients**) mount it over the network and use it as if it were local. The data lives on the server, so every node reaches the same files. | | **export** | The directory an NFS server chooses to share, plus the rules for who may use it. It's listed in `/etc/exports` on the server. | | **ReadWriteOnce (RWO)** | A volume one node can mount at a time. `local-path` is RWO: its data is on one node, so only that node can use it. | | **ReadWriteMany (RWX)** | A volume many nodes can mount at once. NFS is RWX, because the files live on a server every node can reach. | | **node-locked / node affinity** | A PV that's tied to one specific node. A pod using it can only run on that node. This is what makes `local-path` unable to follow a pod that moves. | | **eviction / reschedule** | When a node goes away, Kubernetes waits a grace period, gives up on the pods there, and tries to start fresh copies elsewhere. Whether the *data* comes with them depends entirely on where it lived. | --- ## Three things to know before you start ### 1. The one question this whole part turns on: *where does the data live?* On a single machine, "storage" is simple: there's one disk, and everything's on it. A cluster breaks that assumption, because a pod can be moved to a different machine at any time. So there are really two kinds of storage here, and the difference isn't speed or size; it's **location**: - **Local storage** (`local-path`) puts the data on the node the pod happens to run on. If the pod moves, the data doesn't come with it. - **Network storage** (NFS, and later iSCSI) puts the data on a machine both nodes can reach. The pod can move freely, because the data was never on the node in the first place. Everything in Part 4 is built to make that difference visible rather than take it on faith. You'll write data with local storage and watch it get stranded, then write data with network storage and watch it survive a worker being killed. ### 2. One thing here can fail loudly, and it's NFS mounting inside a container This is Part 4's version of Part 3's snapshotter: the single most likely thing to stop you, and (mercifully) it fails visibly. Your nodes are containers, and mounting a network filesystem *inside* a container is more restricted than doing it on a normal machine. Two things have to be true on the host for it to work: - the host kernel has NFS support loaded (a module thing, like Part 2's six modules), and - the container is allowed to perform an NFS mount (an LXD permission thing). Because your nodes are privileged containers (Part 2's `uid_map` reading `0 0 4294967295` is the sign of that), NFS mounts often *just work*. But if they don't, the failure is loud and specific: a pod stuck with a mount error naming `nfs`, or an `apparmor="DENIED"` line. Step 5 tries the mount, and if it's refused, tells you the one setting to add. Same shape as Step 1 of Part 3: try the default, check, and only reach for the fix if the check fails. ### 3. "The pod is Running" does not mean "the data is safe" A stateless pod that dies is replaced and nobody notices. A stateful pod that dies is only fine if its data outlived it. Those are separate claims, and Kubernetes reports the first one loudly (`Running`, `Ready`) while saying nothing about the second. Part 4 makes you check the second one directly: write a value, kill the thing holding it, and read the value back, because that's the only honest test of storage. --- ## Step 1 — Restore the clean cluster and re-check the basics **Why:** Part 3 ended by breaking quorum on purpose. It healed, but the tidiest known-good starting point for Part 4 is the `post-install` snapshot, which was taken while everything was stopped and consistent. Starting from it means any trouble in Part 4 belongs to Part 4, not to leftovers from the last experiment. If your cluster is already healthy and you'd rather not roll back, skip the restore and just run the health checks below. If you want a guaranteed-clean slate, restore first: ```bash # Optional but recommended: return to Part 3's post-install snapshot. for n in master1 master2 master3 worker1 worker2; do lxc stop "$n"; done for n in master1 master2 master3 worker1 worker2; do lxc restore "$n" post-install; done for n in master1 master2 master3 worker1 worker2; do lxc start "$n"; done sleep 30 ``` Now confirm the cluster is whole. These commands only read: ```bash lxc info | head -3 # wakes LXD if it's asleep, same trick as before sleep 3 lxc list # expect five RUNNING rows, plus the STOPPED template kubectl get nodes # Expect: five Ready. Masters read control-plane,etcd,master; workers . kubectl get nodes -l node-role.kubernetes.io/etcd=true # Expect: three etcd members; quorum intact. kubectl get all -n default # Expect ONE line: service/kubernetes. That's the clean baseline. Anything # else here is leftover from an earlier experiment; delete it by name. kubectl get pvc -A # Expect: "No resources found." You're about to change that on purpose. ``` Now check the one Part 2 package this part depends on. `nfs-common` is the NFS **client**, and every node needs it to mount an NFS share. Part 2 put it in the template, so all five nodes inherited it. Confirm on one: ```bash lxc exec worker1 -- dpkg -l nfs-common | grep -q '^ii' \ && echo "ok nfs-common present" || echo "MISSING nfs-common" # Expect: ok. If MISSING, install it inside each node: # lxc exec -- apt-get install -y nfs-common ``` If all five nodes are Ready, etcd shows three members, `default` shows only `service/kubernetes`, and `nfs-common` is present, you're ready. --- ## Step 2 — Use the storage k3s already gave you (local-path) **What this does.** k3s ships with a working StorageClass called `local-path`, so you already have dynamic storage with no setup at all. This step proves it works: you write a value to disk, delete the pod, then read the value back from a fresh pod. That's the core "data survives a restart" test in its simplest form. First, confirm the StorageClass is there and is the default: ```bash kubectl get storageclass # Expect: one class, "local-path", marked (default). "default" means a PVC # that names no class gets this one automatically. ``` Now for the real test. First, here's why saving data matters at all. When Kubernetes runs a container, it gives it a fresh, empty filesystem. Anything the program writes stays inside that container and nowhere else. That's fine until the pod goes away, and pods go away often: they restart when the program crashes, when you roll out a new version, or when the node reboots. Each restart builds a brand-new container from the image, with an empty filesystem again, so whatever the old one wrote is gone. For a stateless app, like a web server that only serves pages, losing that is no loss at all. But most real programs keep something they can't afford to lose: a database file, uploaded photos, a user's settings. If that file disappears on every restart, the program is useless. A **PersistentVolumeClaim (PVC)** fixes this by giving the program a folder that isn't part of the container. The program writes to it normally, but the bytes land on a disk that outlives the pod. Delete the pod, start a new one pointing at the same claim, and the file is still there. You'll do it in four short steps: create the storage and a pod to use it, write one line into the volume once the pod is up, delete the pod, then read the same line back from a completely different pod. If the second pod can read what the first wrote, the data survived. That is persistence. **Stage 1. Create the claim and a pod to hold the data.** The manifest below holds two objects, joined by the `---` line: - The **PersistentVolumeClaim** named `local-data` is the request for storage: 100Mi that outlives any single pod. Applying it makes the `local-path` provisioner create a real **PersistentVolume** for it (a folder on one node's disk) and bind the claim to that volume. - The **Pod** named `writer` is a plain busybox container that just stays running (its command is `sleep 3600`). It declares a volume called `store` that points at the claim (`claimName: local-data`) and mounts it at `/data` inside the container. It doesn't write anything on its own. You'll write the file yourself in a moment, once the pod is up, so the write is a step you can see rather than something hidden in the pod's startup. Because `/data` is the mounted volume, whatever you write there lands on the claimed disk, not inside the throwaway container. First write both objects into a file called `writer.yaml`: ```bash cat > writer.yaml <<'EOF' apiVersion: v1 kind: PersistentVolumeClaim metadata: name: local-data spec: accessModes: ["ReadWriteOnce"] # local-path is RWO, one node at a time resources: requests: storage: 100Mi --- apiVersion: v1 kind: Pod metadata: name: writer spec: containers: - name: writer image: busybox command: ["sh", "-c", "sleep 3600"] volumeMounts: - name: store mountPath: /data # the volume appears here inside the pod volumes: - name: store persistentVolumeClaim: claimName: local-data # the pod uses the claim above EOF ``` Then apply that file to the cluster: ```bash kubectl apply -f writer.yaml ``` **Stage 2. Check that the storage bound, and see where the pod landed.** Applying that manifest set off a chain: Kubernetes read the claim, the `local-path` provisioner made a volume for it, bound the two, then scheduled the `writer` pod onto a node and mounted the volume. These two commands confirm each half worked: ```bash kubectl get pvc local-data # Expect: STATUS Bound. A PV was created automatically to satisfy the claim. kubectl get pod writer -o wide # Expect: Running, on one of the workers. Note WHICH node; call it node A. # local-path put the data on node A's local disk. ``` `Bound` means the claim now has a real volume behind it, and `writer` is running with that volume mounted at `/data`. Take note of which worker it landed on. With `local-path`, the volume is a folder on that node's own disk, so the data physically sits on **node A** and nowhere else. That detail looks minor right now, but it's the whole point of Step 3. **Stage 3. Now that the pod is up, write a line into the volume.** The container is running, but nothing is stored yet. Write one line into `/data` (the mounted claim) by running a command inside the pod with `kubectl exec`: ```bash kubectl exec writer -- sh -c "echo 'hello from the first pod' > /data/note.txt" ``` Read it straight back to confirm it landed: ```bash kubectl exec writer -- cat /data/note.txt # Expect: hello from the first pod ``` The write went to `/data/note.txt` inside the pod, but `/data` is the volume, so the line is really stored on the `local-data` claim, not in the container. The next step proves that by throwing the pod away. **Stage 4. Delete the writer, then read the file back from a new pod.** This is the real test. Deleting the pod leaves the claim and its data untouched, so a fresh `reader` pod that mounts the same claim should still find the line the writer saved. Notice the `reader` pod is a different pod with a different name, but it points at the same `claimName: local-data`. First delete the writer pod. The claim and its data stay behind: ```bash kubectl delete pod writer # the pod is gone; the PVC and its data remain ``` Write the reader pod into `reader.yaml`: ```bash cat > reader.yaml <<'EOF' apiVersion: v1 kind: Pod metadata: name: reader spec: containers: - name: reader image: busybox command: ["sh", "-c", "cat /data/note.txt && sleep 3600"] volumeMounts: - name: store mountPath: /data volumes: - name: store persistentVolumeClaim: claimName: local-data EOF ``` Apply it: ```bash kubectl apply -f reader.yaml ``` Give it a moment to start, then read the file back: ```bash sleep 5 kubectl logs reader # Expect: "hello from the first pod". A different pod read a file the first # pod wrote. The data outlived the pod; that's persistence. ``` That's storage working. But notice something the test quietly relied on: the reader landed on **node A**, the same node as the writer. It had to, and the next step shows why that's a limit, not a coincidence. --- ## Step 3 — See where local-path stops: the data can't leave its node **What this does.** It shows the one weakness of local storage. The data from Step 2 lives on node A's disk. So any pod that wants it *must* run on node A. If node A is busy or down, the pod can't move somewhere else and bring its data along, because the data isn't stored anywhere else. Look at the PV (the PersistentVolume) that Kubernetes created, and you'll see it's pinned to one node: ```bash kubectl get pv # Find the PV bound to local-data, then describe it: kubectl describe pv $(kubectl get pv -o jsonpath='{.items[0].metadata.name}') | grep -A3 'Node Affinity' # Expect: a Node Affinity rule naming ONE node (node A). This is the pin: # "this volume only exists on node A, so only node A may use it." The literal # line reads: Term 0: kubernetes.io/hostname in [] (e.g. [worker2]). ``` Now force the issue. Tell Kubernetes it may not run new pods on node A (`cordon`), then try to start a pod that needs the data. First cordon node A and remove the running reader, so the next pod has to schedule fresh: ```bash # Replace with the node name from Step 2. kubectl cordon # "no new pods here", and doesn't touch running ones kubectl delete pod reader ``` Write a new pod, `reader2`, that wants the same data: ```bash cat > reader2.yaml <<'EOF' apiVersion: v1 kind: Pod metadata: name: reader2 spec: containers: - name: reader2 image: busybox command: ["sh", "-c", "cat /data/note.txt && sleep 3600"] volumeMounts: - name: store mountPath: /data volumes: - name: store persistentVolumeClaim: claimName: local-data EOF ``` Apply it: ```bash kubectl apply -f reader2.yaml ``` Now watch it fail to schedule: ```bash sleep 5 kubectl get pod reader2 # Expect: Pending, NOT Running. The pod needs node A (that's where its data # is), but node A is cordoned. It can't run anywhere else, so it waits. kubectl describe pod reader2 | grep -A5 Events # Expect: a message about node affinity / volume node conflict. In plain terms: # "the only node that can supply this volume is off-limits, and no other node # has the data." The pod is stuck, and no amount of free workers helps. ``` This is the whole point. **Local storage ties a pod to a place.** For a stateless web server that's fine: it has no data, so any node will do. For a database it's a real constraint. Lose that node and the data is unreachable until the node comes back. Now clear the local-path example so the lab is clean before the NFS steps. Undo the cordon, delete the leftover pod, and delete the claim. Deleting the claim is what removes the data: local-path sees the claim go and deletes the volume and its files on node A along with it. ```bash kubectl uncordon kubectl delete pod reader2 --ignore-not-found kubectl delete pvc local-data # deletes the claim; local-path deletes the data kubectl get all -n default # expect: only service/kubernetes again kubectl get pvc,pv # Expect: "No resources found." Deleting the claim made local-path remove the volume # and its data directory on node A. (The PV may take a few seconds to disappear.) rm -f writer.yaml reader.yaml reader2.yaml # remove the local manifest files ``` The fix for a pod that must be free to move is storage that doesn't live on any single node. That's NFS, and it's the next four steps. --- ## Step 4 — Stand up an NFS server on the host **What this does.** It turns your LXD host into a small file server. It shares one directory over the lab network, and every node will mount it. Because the files live on the host, not on any node, a pod carrying that storage can run anywhere. Why the host? It's already on the lab network, it's always up, and it keeps the lab self-contained, with no sixth machine to build. (In production the NFS server would be a dedicated, backed-up box, never a node of the cluster it serves. For a lab, the host is the honest, simple choice.) First, find the host's address on the lab network; the nodes will mount from it. The host is the gateway the containers already use, almost certainly `10.99.99.1`, but confirm rather than assume: ```bash # Ask a container what its gateway is; that's the host's IP on the lab bridge. lxc exec worker1 -- ip route | grep default # Expect something like: default via 10.99.99.1 dev eth0 # Use whatever IP that shows as HOST_IP below. This guide assumes 10.99.99.1. ``` Now install the server and share a directory. Run this **on the host**: ```bash sudo apt-get update sudo apt-get install -y nfs-kernel-server sudo mkdir -p /srv/nfs/k8s sudo chown nobody:nogroup /srv/nfs/k8s sudo chmod 0777 /srv/nfs/k8s # lab-simple: any node may write. See note. ``` Declare who may use the share by adding one line to `/etc/exports`: ```bash echo '/srv/nfs/k8s 10.99.99.0/24(rw,sync,no_subtree_check,no_root_squash)' \ | sudo tee -a /etc/exports sudo exportfs -ra # re-read the exports file sudo systemctl enable --now nfs-kernel-server ``` What each export option means: | Option | Why it's there | |---|---| | `10.99.99.0/24` | Only machines on the lab network may mount. Your five nodes qualify; nothing outside does. | | `rw` | Nodes may read *and* write. | | `sync` | Writes are committed to disk before the server says "done". Safer, and a good default. | | `no_subtree_check` | Turns off a fragile old check; standard on modern shares. | | `no_root_squash` | **Lab convenience.** By default NFS turns a client's `root` into a powerless `nobody` (that's "root squash"). Kubernetes mounts as root, so squashing causes permission-denied errors. Turning it off lets it work. On a real server you would *not* do this: it lets a client's root write as root on the server. Here it's a deliberate, contained shortcut, in the same spirit as Part 3's `--write-kubeconfig-mode=644`. | Confirm the share is live: ```bash sudo exportfs -v # Expect: a line showing /srv/nfs/k8s exported to 10.99.99.0/24 with your options. ``` The server is up. Nothing is using it yet; that's the next two steps. --- ## Step 5 — Prove a node can actually mount the share (the loud-failure step) **What this does.** Before you trust Kubernetes to mount NFS on its own, mount it by hand from one node. If a plain manual mount works, Kubernetes will too. If it's refused, you find out right here, cleanly, in one command, instead of debugging a stuck pod later. This is the step most likely to need a fix, and the fix is small. Try the mount from `worker1`: ```bash lxc exec worker1 -- mkdir -p /mnt/nfstest lxc exec worker1 -- mount -t nfs 10.99.99.1:/srv/nfs/k8s /mnt/nfstest # (use your HOST_IP if it wasn't 10.99.99.1) ``` **If that command returns silently**, it worked. First confirm it really is an NFS mount, not just an empty directory, with `findmnt`: ```bash lxc exec worker1 -- findmnt /mnt/nfstest # Expect: TARGET /mnt/nfstest SOURCE 10.99.99.1:/srv/nfs/k8s (fstype nfs4). # Nothing printed means nothing is mounted there, and /mnt/nfstest is just a # plain local folder on the container's own disk. ``` Now prove the share is really shared, both ways, so a local folder can't fool you: write from the node and see it on the host, then delete on the host and watch it vanish in the node. ```bash lxc exec worker1 -- sh -c 'echo "written from worker1" > /mnt/nfstest/proof.txt' sudo cat /srv/nfs/k8s/proof.txt # on the host, expect the same line sudo rm -f /srv/nfs/k8s/proof.txt # now delete it on the host lxc exec worker1 -- ls /mnt/nfstest # expect: empty. The host delete reached # the node, so it really is one shared folder. lxc exec worker1 -- umount /mnt/nfstest # clean up the manual test ``` If that worked, skip the rest of this step and go to Step 6. > **A hand mount does not survive a restart, and a bare mountpoint is a trap.** A `mount -t nfs` done by hand lasts only until the container stops. Restart it (the AppArmor fix below does exactly that) and the mount is gone, but `/mnt/nfstest` stays behind as an ordinary local folder. Write to it then and the file lands on the container's own disk, not the share, so it can look like NFS still works when nothing is mounted. When unsure, run `findmnt /mnt/nfstest`: a real mount names `10.99.99.1:/srv/nfs/k8s` as its source, and a bare folder shows nothing. This only bites the by-hand test; k3s and the provisioner mount NFS themselves and re-establish it after a restart. **If the mount was refused**, read the error. It tells you which of two problems you have: - **`access denied by server while mounting`** means the *export* is wrong (the node isn't in `10.99.99.0/24`, or `exportfs -ra` wasn't run). Fix it back in Step 4. This is a server-side problem. - **`permission denied`, or an `apparmor="DENIED"` line in the host's `dmesg`** means the *container* isn't allowed to do NFS mounts. This is the LXD restriction from the intro. Here's the fix: ```bash # Allow NFS mounts inside the node containers. Run on the host, per node. for n in master1 master2 master3 worker1 worker2; do lxc config set "$n" raw.apparmor "mount fstype=nfs, mount fstype=nfs4, mount fstype=rpc_pipefs," lxc restart "$n" done sleep 30 kubectl get nodes # expect all five Ready again after restart ``` Also make sure the host kernel has NFS client support loaded and that it survives reboots (same pattern as Part 2's modules): ```bash # On the host: sudo modprobe nfs nfsd echo -e "nfs\nnfsd" | sudo tee /etc/modules-load.d/nfs.conf ``` Finally, if the mount prints no error but just **hangs or times out**, and `dmesg` shows no `apparmor="DENIED"` line, suspect the host firewall. This one only affects you if you kept a custom firewall from Part 1. Most readers skipped that, because a fresh Ubuntu install allows everything and LXD's own table already lets the bridge reach the host, so there is nothing here to do. If you did keep a firewall, here is why NFS trips it. Your NFS server runs on the host at `10.99.99.1`, so a node mounting from it sends traffic *to the host*, which hits the host's `input` chain, not the forward one (the input versus forward point from Part 1). If your firewall's input policy is `drop`, it blocks the NFS ports unless you allow them, whatever LXD's table does, because in nftables a packet has to clear every input chain. Confirm it by asking the server what it shares, from a node: ```bash lxc exec worker1 -- showmount -e 10.99.99.1 # Lists /srv/nfs/k8s: the firewall is fine, look elsewhere. # Hangs or errors (an RPC or timeout message) while `ping 10.99.99.1` still # works: the host is dropping the NFS ports. ``` NFS needs two ports open coming in on `lxdbr0`: `2049` (the NFS data port) and `111` (rpcbind, which `showmount` and older NFSv3 clients use). Allow both, TCP and UDP, on the host's input chain. In nftables that is: ```nft table inet filter { chain input { iifname "lxdbr0" tcp dport { 111, 2049 } accept iifname "lxdbr0" udp dport { 111, 2049 } accept } } ``` Add that to your ruleset the way you manage your other rules. (This guide names the bridge `lxdbr0` throughout; use your own variable if your config defines one.) Then check the syntax and load it: ```bash sudo nft -c -f /etc/nftables.conf # syntax check only, changes nothing sudo systemctl reload nftables # apply it ``` One caution carried over from Part 1: if your `/etc/nftables.conf` begins with `flush ruleset`, reloading wipes every table, LXD's `table inet lxd` included, and LXD will not rebuild it until it restarts. If yours flushes, reboot or run `sudo systemctl restart snap.lxd.daemon` after loading, so LXD's table comes back. If you only ever use NFSv4 (what the Kubernetes NFS provisioner uses), port `2049` alone is enough; `111` is only there for `rpcbind` and `showmount`. Opening both is fine for a lab and keeps `showmount` working as a troubleshooting tool, so this guide keeps both. Then run the manual mount test above again. Once a hand mount succeeds from a node, whatever was blocking it is solved for good, and Kubernetes will mount without trouble. > Why bother with a manual mount at all? Because when a Kubernetes pod can't mount storage, the symptom is an unhelpful `ContainerCreating` that never finishes. The manual mount turns that silent stall into a plain error message you can act on. It's the "suspect the check before the machine" reflex again: test the smallest piece directly. --- ## Step 6 — Wire NFS into the cluster as a StorageClass **What this does.** So far, NFS is just a share you can mount by hand. This step teaches the cluster about it, so that a PVC asking for NFS storage gets a volume automatically, exactly like `local-path` did, but backed by the server instead of a node's disk. You install a small **provisioner**, a program that watches for NFS claims and carves out a subdirectory on the share for each one. Install Helm on the host if you don't have it (it's the easiest way to add the provisioner): ```bash # On the host: sudo snap install helm --classic ``` Add and install the NFS provisioner, pointing it at your server and share: ```bash helm repo add nfs-subdir-external-provisioner \ https://kubernetes-sigs.github.io/nfs-subdir-external-provisioner/ helm repo update helm install nfs-provisioner \ nfs-subdir-external-provisioner/nfs-subdir-external-provisioner \ --set nfs.server=10.99.99.1 \ --set nfs.path=/srv/nfs/k8s \ --set storageClass.name=nfs-client # (use your HOST_IP for nfs.server) ``` Confirm the new StorageClass and the provisioner pod: ```bash kubectl get storageclass # Expect two now: local-path (default) and nfs-client. kubectl get pods -l app=nfs-subdir-external-provisioner # Expect: one pod, Running. This is the program that makes NFS volumes on demand. # If it's stuck ContainerCreating, that's the mount problem from Step 5; go back. ``` You now have two kinds of storage the cluster can hand out: `local-path` (on a node) and `nfs-client` (on the server). Both use the same PVC syntax; only the `storageClassName` differs. That symmetry is what makes the next step a fair comparison. --- ## Step 7 — Run a stateful app on NFS and prove the data can move **What this does.** Step 2 proved data survives a pod restart, but only just: the reader had to land back on **node A**, because that's where local-path pinned the data. Real clusters don't respect that. Pods get moved to other nodes all the time, when a node fills up, gets drained for maintenance, or dies. So the sharper question is: can a new pod pick up its data on a *different* machine? With local-path the answer was no (Step 3). This step asks NFS the same question, and this time it forces the reader onto a different node to make the answer unmistakable. The answer is yes, and the reason is the whole point of NFS: the storage doesn't live on any node. It lives on the server you built in Step 4, which every node can reach over the network. The data was never tied to a place, so a pod carrying it can run anywhere. You'll run the same kind of test as Step 2, with one twist at the end: write a line on one node, confirm it's really sitting on the server, then read it back from a pod pinned to a *different* node. **Stage 1. Create an NFS-backed claim and a pod that writes to it.** This is Step 2's manifest with two changes, both on the claim. `storageClassName: nfs-client` sends the request to the NFS provisioner instead of local-path, so the volume is carved out on the server rather than on a node's disk. And `accessModes: ["ReadWriteMany"]` (RWX) lets many nodes mount the volume at once, which is exactly what network storage allows and local-path could not. The `nfs-writer` pod is otherwise the same as before: it mounts the claim at `/data` and writes one line into `/data/note.txt`. Write it to a file, apply it, then confirm it bound and see which node the writer landed on: ```bash cat > nfs-writer.yaml <<'EOF' apiVersion: v1 kind: PersistentVolumeClaim metadata: name: nfs-data spec: accessModes: ["ReadWriteMany"] # NFS is RWX, many nodes at once storageClassName: nfs-client resources: requests: storage: 100Mi --- apiVersion: v1 kind: Pod metadata: name: nfs-writer spec: containers: - name: nfs-writer image: busybox command: ["sh", "-c", "echo 'written on the NFS share' > /data/note.txt && sleep 3600"] volumeMounts: - name: store mountPath: /data volumes: - name: store persistentVolumeClaim: claimName: nfs-data EOF ``` Apply it: ```bash kubectl apply -f nfs-writer.yaml ``` Then confirm the claim bound and see which node the writer landed on: ```bash kubectl get pvc nfs-data # expect: Bound kubectl get pod nfs-writer -o wide # expect: Running. Note which node, node A. ``` **Stage 2. Confirm the data is really on the server, not on a node.** With local-path you couldn't check this easily, because the data sat inside one node's disk. With NFS the file is an ordinary file in the host's shared folder, so you can read it straight from the host: ```bash sudo find /srv/nfs/k8s -name note.txt -exec cat {} \; # Expect: "written on the NFS share". The file is on the server, not on a node. ``` **Stage 3. Delete the writer, then read the file back from a pod on a *different* node.** This is the twist that local-path failed in Step 3. You delete the writer, then start an `nfs-reader` pod with its `nodeName` set to a worker that isn't node A (the manifest uses `worker2`; if node A was already `worker2`, change it to `worker1`). Setting `nodeName` forces Kubernetes to run the pod on that exact node, so there's no chance it quietly lands back where the data was written. If it still reads the line, the data genuinely moved with the pod: ```bash kubectl delete pod nfs-writer ``` Now write the reader into `nfs-reader.yaml`. Pick a worker that isn't node A: the file uses `worker2`, so if node A was already `worker2`, change it to `worker1`: ```bash cat > nfs-reader.yaml <<'EOF' apiVersion: v1 kind: Pod metadata: name: nfs-reader spec: nodeName: worker2 # force it onto a specific, different node containers: - name: nfs-reader image: busybox command: ["sh", "-c", "cat /data/note.txt && sleep 3600"] volumeMounts: - name: store mountPath: /data volumes: - name: store persistentVolumeClaim: claimName: nfs-data EOF ``` Apply it: ```bash kubectl apply -f nfs-reader.yaml ``` Give it a moment, then check where it ran and what it read: ```bash sleep 5 kubectl get pod nfs-reader -o wide # expect: Running, on worker2 kubectl logs nfs-reader # Expect: "written on the NFS share", read on a DIFFERENT node than it was # written on. This is the exact thing local-path could not do in Step 3. ``` That contrast is the heart of Part 4. Same test, one word changed (`local-path` to `nfs-client`), and the Pending pod of Step 3 becomes a Running pod here, because the storage stopped being tied to a place. --- ## Step 8 — Break it on purpose: kill the worker, watch the data survive **What this does.** This is Part 4's version of Part 3's "break it on purpose." There you killed a master and watched the *control plane* survive. Here you kill the worker running a stateful app and watch its *data* survive, because the data was on the NFS server, not on the worker. Then the app comes back on another node, still holding everything. First, remember the same "kubectl lags reality" warning from Part 3: after you stop a node, `kubectl get nodes` may show it `Ready` for up to a minute before it flips to `NotReady`. `lxc list` shows the truth right away. When they disagree for a minute, believe `lxc list`. Set up a small app that keeps *counting*, so you can prove not just that a file survived, but that its exact contents did. It appends a line every few seconds to the NFS share. Write the deployment to `counter.yaml`: ```bash cat > counter.yaml <<'EOF' apiVersion: apps/v1 kind: Deployment metadata: name: counter spec: replicas: 1 selector: matchLabels: { app: counter } template: metadata: labels: { app: counter } spec: containers: - name: counter image: busybox command: ["sh", "-c", "while true; do echo \"tick $(date +%T)\" >> /data/log.txt; sleep 3; done"] volumeMounts: - name: store mountPath: /data volumes: - name: store persistentVolumeClaim: claimName: nfs-data EOF ``` Apply it: ```bash kubectl apply -f counter.yaml ``` Give it a few seconds to start, then check it's running and already writing: ```bash sleep 10 kubectl get pod -l app=counter -o wide # Expect: one Running pod. Note its node; call it node X. kubectl exec deploy/counter -- tail -3 /data/log.txt # Expect: three recent "tick HH:MM:SS" lines. Remember the last timestamp. ``` Now kill node X, the worker running the counter: ```bash # Replace with the worker name above. lxc stop ``` Watch what happens. Kubernetes notices the node's heartbeats stop, waits out its grace period (about 40 to 60 seconds, plus a pod-eviction timeout of a few minutes by default), then gives up on the pod there and starts a fresh copy on the surviving worker: ```bash lxc list # is STOPPED immediately (the truth) kubectl get nodes # flips to NotReady after ~1 min # Keep checking; the pod gets recreated on the other worker. This can take a # few minutes (that's the eviction timeout, not a hang): kubectl get pod -l app=counter -o wide -w # Expect: eventually a new counter pod, Running, on the OTHER worker. # Ctrl-C to stop watching once it's Running. ``` While you wait, `kubectl get pod` keeps showing the old pod as `Running` on the stopped node, often for a few minutes. Don't trust that line. The node is off, but its kubelet is no longer alive to report the truth, so the control plane just shows the last status it heard until the eviction timeout expires. It's the same "believe `lxc list`, not `kubectl`" lag from the start of this step. Once the timeout passes, the old pod goes `Terminating` and a fresh one appears `Running` on the other worker. Only then has the app actually moved. Here's the payoff. **Wait until that new pod shows `Running` on the other worker**, then check the log it's writing to. If you run the next command too early, while the pod on the stopped node is still the target, `kubectl exec` fails with `502 Bad Gateway` or `error dialing backend`. That just means exec tried to reach the pod through its dead node's kubelet; it's not a fault in your setup, only too early, so wait and retry. ```bash kubectl exec deploy/counter -- tail -5 /data/log.txt # Expect: the OLD ticks from before you killed node X, followed by NEW ticks # with a gap in the timestamps. The gap is the outage. The old lines proving # the data survived is the point: a different pod, on a different node, picked # up the exact same file. ``` Nothing was lost. The worker died, the app moved, and the data was waiting for it on the server. That move is Kubernetes reconciling **desired state** (the Deployment asks for one running replica) with **actual state** (the node holding it was gone): once it gave up on the dead node, it made a fresh pod on a healthy one to close the gap. **Had this been `local-path` storage, the new pod would be stuck `Pending`**, because its data was on the stopped worker, exactly like Step 3. That's the difference network storage buys you, shown under a real failure instead of a cordon. Bring the worker back and confirm the cluster returns to five: ```bash lxc start sleep 30 kubectl get nodes # expect: five Ready again ``` ### Clean up and return to the storage-free baseline ```bash kubectl delete deployment counter --ignore-not-found kubectl delete pod nfs-reader --ignore-not-found kubectl delete pvc nfs-data # removes the claim; provisioner archives its subdir (see below) rm -f nfs-writer.yaml nfs-reader.yaml counter.yaml # remove the manifest files kubectl get all -n default # expect: only service/kubernetes kubectl get pvc -A # expect: No resources found # The NFS provisioner (Step 6) is infrastructure; leave it. But confirm the # share is empty of test data: sudo ls /srv/nfs/k8s # expect: empty, or only archived subdirs ``` **About that `archived-...` directory.** If `ls` shows something like `archived-default-nfs-data-pvc-`, that's expected. The NFS provisioner defaults to *archiving* a claim's folder instead of deleting it: when you removed the PVC, it renamed the folder with an `archived-` prefix rather than erasing it, as a safety net against accidental data loss. Nothing uses it now, so for the lab it's safe to remove: ```bash sudo rm -rf /srv/nfs/k8s/archived-* sudo ls -A /srv/nfs/k8s # expect: no output, a truly empty share ``` You've now proven storage the same way Part 3 proved quorum: by taking the machine underneath it away and watching the thing above stay up. --- ## Step 9 (optional) — The production answer, and why `open-iscsi` is installed NFS taught the lesson, but it has one weakness worth naming. The NFS server is a **single point of failure**. Every node depends on that one host. If it dies, all the network storage goes with it. That's fine for a lab, but not for production. The production answer is **replicated block storage**: storage that keeps copies of your data on several nodes at once, so losing any one node loses nothing. The common k3s-friendly option is **Longhorn**. It's built on iSCSI, which is exactly why Part 2 installed `open-iscsi` in the template. That package has sat unused until now for the same reason `nfs-common` did. It's the client half of a storage system you hadn't set up yet. Setting up Longhorn in full is beyond this lab, but here's the outline. You install it into the cluster. It turns each node's spare disk into a pool, then hands out volumes that are automatically mirrored across nodes. A pod gets a volume that behaves like a fast local disk but survives any single node dying. It combines local-path's speed with NFS's mobility, and adds redundancy that neither has. If you want to explore it later, Longhorn installs as a single manifest and gives you its own StorageClass, which you'd use in a PVC just like `nfs-client`. Check the current version on the Longhorn site before installing, since the manifest URL is version-pinned. The concepts you'd carry in are the ones this part built: PVs, PVCs, StorageClasses, and the one question that never changes: *where does the data live, and does it survive the node?* --- ## Step 10 — Save a restore point You have a cluster that now knows how to store data. Save the state the same careful way as Part 3, with all nodes stopped together so the etcd members stay consistent: ```bash for n in worker1 worker2 master1 master2 master3; do lxc stop "$n"; done for n in master1 master2 master3 worker1 worker2; do lxc snapshot "$n" post-storage done for n in master1 master2 master3 worker1 worker2; do lxc start "$n"; done sleep 30 kubectl get nodes # expect: five Ready ``` Two things worth remembering about *this* snapshot versus Part 3's: - The **NFS server lives on the host**, not in a node, so an LXD node snapshot does *not* capture it. The share directory (`/srv/nfs/k8s`) and the export config are host state. If you ever roll the nodes back, the server is still there, which is fine, because it's separate by design. - The provisioner and StorageClass *are* in the cluster, so they're in this snapshot. Rolling back to `post-storage` gives you a cluster that already knows about NFS. --- ## Where you should be | Thing | State | |---|---| | Storage classes | Two: `local-path` (default, on-node) and `nfs-client` (network) | | NFS server | Running on the host, sharing `/srv/nfs/k8s` to the lab network | | NFS mounting | Proven by hand from a node, then used by the cluster automatically | | Stateful test | Data written, pod deleted, data read back; persistence proven | | Node-locking | Seen: local-path strands a pod's data on one node; NFS does not | | Failure test | Worker killed with a live app; app rescheduled elsewhere, data intact | | Snapshots | `post-storage` on all five, taken stopped and together | --- ## Things worth carrying forward - **The only question that matters in cluster storage is where the data lives.** On a node (`local-path`) it's fast but stranded when the node goes away. On the network (NFS, iSCSI) it's reachable from anywhere, so a pod can move and keep its data. Everything else is detail. - **local-path is RWO and node-locked; NFS is RWX and node-free.** A pod on local-path can only run where its data is. That's fine for stateless apps, but a real limit for databases. - **NFS mounting inside a container is Part 4's snapshotter**, the one loud failure. If a storage pod hangs in `ContainerCreating`, mount the share by hand from the node to turn the silent stall into a readable error, then fix modules or the apparmor rule. - **"Running" says nothing about your data.** Test persistence directly: write a value, destroy the thing holding it, read the value back. That's the only honest check. - **NFS is a single point of failure; replicated block storage (Longhorn, on iSCSI) is the production fix.** That's what `open-iscsi` was installed for, and the concepts transfer unchanged. - **Snapshot the nodes stopped and together, and remember the NFS server is host state.** It's outside the node snapshots on purpose. --- ## Where this leaves the series You now have a highly-available cluster (Part 3) that can also keep data safely across node failures (Part 4). That's a genuinely working small Kubernetes platform: it stands up, it heals when a master dies, it runs programs, and it holds onto their data when a worker dies. `post-storage` is the clean cluster you'd return to before running anything real on it. ### [Building a Kubernetes Lab with k3s and LXD Part 3 Installing k3s](https://linsnotes.com/posts/building-a-kubernetes-lab-with-k3s-and-lxd-part-3-installing-k3s/) > Install k3s across five LXD nodes to form a three master HA cluster with two workers, then break quorum on purpose to see how etcd protects itself. This is **Part 3** of a series on building a Kubernetes lab with k3s and LXD. If you are just arriving, start with [**Part 1, Preparing the Ubuntu Host**](/posts/building-a-kubernetes-lab-with-k3s-and-lxd-part-1-preparing-the-ubuntu-host/) and then [**Part 2, Building the Five Nodes**](/posts/building-a-kubernetes-lab-with-k3s-and-lxd-part-2-building-the-five-nodes/). This part picks up exactly where Part 2 left off. **Where you are.** Part 2 built five containers: `master1` to `master3` on `10.99.99.11` to `.13`, and `worker1` to `worker2` on `.21` to `.22`. They can already reach each other and the internet, every kernel module, kernel setting, and permission k3s will ask for is in place, and there is a `pre-k3s` snapshot on all five. **What this part does.** It turns those five machines into one Kubernetes cluster: three masters that share a database and vote on decisions, and two workers that run programs. Then it deliberately breaks the cluster to show you what the voting was for. **What this part does not do.** It does not run any real applications. That is Part 4. This part gets the cluster standing and proves it is healthy. --- ## Words you'll need Building on Part 2's list. These are the new ones. | Word | What it means | |---|---| | **server (k3s sense)** | A node that runs the control plane *and* holds a copy of the database. Your three masters. Confusingly, a k3s "server" also runs your programs unless you tell it not to. | | **agent (k3s sense)** | A node that only runs programs and takes orders. Your two workers. | | **control plane** | The part of Kubernetes that makes decisions: what runs where, what's healthy, what to do when something dies. Lives on the masters. | | **API server** | The single front door to the cluster. Every command, every component, every `kubectl` goes through it. Listens on port `6443`. | | **token** | A shared secret. A node proves it's allowed to join by presenting it. You pick one string and reuse it on all five. | | **`--cluster-init`** | The flag that tells the *first* master "start a brand-new cluster and its database." Used exactly once, ever. | | **quorum** | The minimum number of database members that must agree before the cluster will accept a change. For three members, that's two. This is the whole reason there are three masters. | | **etcd member** | One master's copy of the database. Three masters means three members of one etcd. | | **leader** | The one etcd member that coordinates all writes; the other two are followers that acknowledge its changes. etcd elects it automatically. Quorum is *how many* must agree; the leader is *who* they agree with. | | **leader election** | What the surviving members do when the leader vanishes: pick a new one, automatically and in about a second, as long as a quorum is still present. No quorum, no election. | | **kubeconfig** | A small file holding the address of the API server and the key to talk to it. `kubectl` reads it to know where to connect. | | **`kubectl`** | The command you type to talk to the cluster. Runs anywhere that has a kubeconfig. | | **snapshotter** | The part of containerd that stacks image layers into a running filesystem. It has a few implementations. **Which one works depends on your storage, and this is the one thing in Part 3 that can genuinely stop you.** | | **taint** | A mark on a node that says "don't run ordinary programs here." Real clusters taint masters so the control plane isn't fighting your apps for memory. | | **Ready / NotReady** | Kubernetes' word for whether a node is answering. A stopped node goes `NotReady`, not "gone." | --- ## Three things to know before you start ### 1. Part 3 is where Part 2 gets paid back Everything you did in Part 2 was so that this part would be boring. If it was done right, k3s installs in about a minute per node and just works. If a Part 2 step was skipped, **this is where it surfaces**, but usually not pointing at itself. A missing module shows up as a node that never goes Ready. A missing sysctl shows up as the *third* master failing while the first two are fine. The inotify limit from Part 2's Step 2 is the classic: masters 1 and 2 join, master 3 dies with "too many open files," and it looks like master 3 is broken hardware. So Step 1 below re-checks Part 2 before touching k3s. Don't skip it to save ninety seconds. ### 2. One thing here *does* fail loudly, and that's a relief Part 2 warned you that nothing fails loudly. Part 3 has one exception, and it's worth knowing in advance because it's the single most likely thing to stop you: **the containerd snapshotter, on ZFS.** If your Part 1 storage was ZFS, k3s's built-in containerd may refuse to start with an error naming `overlayfs`. Unlike everything in Part 2, this one turns red and stops. The node never goes Ready and the log says so plainly. The relief is that it's visible. The trap is that the error blames `overlayfs` when the real cause is "ZFS underneath, inside a container." Step 2 handles it head-on. If your storage was `dir`, this can't happen to you. ### 3. "server" and "agent" are the two words to keep straight k3s calls a master a **server** and a worker an **agent**, and the join commands are shaped differently for each. Servers join with a `--server` *flag*. Agents join with a `K3S_URL` *environment variable*. Mix them up and you'll either turn a worker into a database member by accident, or watch an agent refuse to start. Each command below says which it is at the top. --- ## Step 1 — Check Part 2 is still good **Why:** if a container drifted, or the host lost a module across a reboot you did last week, you want to know before k3s is involved, not after, when the same fault wears a Kubernetes costume. **Safe to run:** everything here only reads. First, **wake LXD**. This is the socket thing from Part 2's Step 0, and it will still fool you after a host reboot: ```bash lxc info | head -3 # THIS starts LXD if it's asleep. Don't skip it. sleep 3 lxc list # Expect: five RUNNING rows on .11-.13 and .21-.22, plus a STOPPED "template". ``` Now the host still holds what Part 2 gave it: ```bash # The six modules, checked the way k3s checks them. for m in overlay nf_conntrack br_netfilter iptable_nat iptable_filter vxlan; do [ -d "/sys/module/$m" ] && echo "ok $m" || echo "MISSING $m" done # Expect: six "ok". Any MISSING means Part 2 Step 1 didn't survive; fix # it there before continuing, or the matching node will never go Ready. sysctl fs.inotify.max_user_instances kernel.keys.maxkeys # Expect: 1024 and 2000. If these read 128 and 200, Part 2 Step 2 is gone, # and master3 will be the one that pays for it. swapon --show # expect: nothing ``` And each node still has its permissions. Check one; they're all copies of it: ```bash lxc exec master1 -- cat /proc/self/uid_map # expect: 0 0 4294967295 lxc exec master1 -- ls /dev/kmsg # expect: /dev/kmsg lxc exec master1 -- stat -fc %T /sys/fs/cgroup # expect: cgroup2fs lxc exec master1 -- free -h | grep -i mem # expect ~4.0Gi, NOT the host's RAM ``` **If `free` shows the host's full memory instead of ~4 GB, stop and fix it in Part 2's Step 5.** This is the one that comes back to bite hardest: kubelet reports whatever `free` says, so an unlimited node tells Kubernetes it has 32 GB, and the scheduler will cheerfully overbook a machine that doesn't have the room. Better to catch it now as a wrong number than in Part 4 as a dead host. ### Pick your token now, before any install command You'll reuse it on all five nodes. Choose one string, and **run this line first, in the terminal you'll be installing from:** ```bash export TOKEN="lab-shared-token" # your choice; the same on all five nodes ``` `export` puts it in the environment so the install commands below can substitute it in. This is a lab on your own machine, so a memorable token is fine. On anything real it would be a long random secret. It's the password to your cluster: anything holding it can add a node. **This lasts only as long as this terminal.** If you close the window, drop the SSH connection, or come back tomorrow and open a fresh shell, `$TOKEN` will be empty and the next join will fail with a blank token. Before each install command, or any time you're unsure, confirm it's still set: ```bash echo "$TOKEN" # should print your token. If it's blank, re-run the export line above. ``` Re-running the `export` line is harmless, since it's the same string every time. You only need it during the install steps (2, 4, and 5); once all five nodes have joined, the token isn't used again. --- ## Step 2 — Start the cluster on master1 ### What this command does One node becomes the first master: it starts the control plane, creates the database, and makes itself the first of what will be three voting members. `--cluster-init` is the flag that means "begin a new cluster." **You run it exactly once, on master1, ever.** The other four nodes *join* what this one creates. ### Settle the snapshotter question before you install This is the ZFS thing from the intro. Rather than install and hope, decide up front. **If your Part 1 storage was `dir`:** skip this box. The default snapshotter works. Go to "Install." **If it was ZFS:** k3s's containerd will try to use `overlayfs` on top of your ZFS-backed container filesystem. Whether that works depends on your ZFS version. Ubuntu 24.04 ships ZFS 2.2, which added the support that makes it *usually* fine. But "usually" is not "always," and the failure is total: the node never goes Ready and the log repeats a line about `overlayfs` "cannot be enabled ... try using fuse-overlayfs or native." You have two honest options: - **Try the default first.** It may just work on 24.04. If it does, you've spent nothing. The install below does this, and the check after it tells you within a minute whether containerd came up. - **Force `native` up front** and never think about it. Add `--snapshotter=native` to the install line. It's slower and uses more disk (it copies layers instead of stacking them), but it does not care what's underneath it, so it cannot hit this wall. For a five-node lab the slowness is not something you'll notice. This guide tries the default and checks. If the check fails, it tells you the one flag to add and re-run. **Don't reach for the flag pre-emptively unless you'd rather not think about it.** On 24.04 you may not need it at all. ### Install ```bash lxc exec master1 -- bash -c " curl -sfL https://get.k3s.io | \ INSTALL_K3S_CHANNEL=stable \ K3S_TOKEN='$TOKEN' \ sh -s - server \ --cluster-init \ --node-ip 10.99.99.11 \ --write-kubeconfig-mode=644 " ``` Line by line: | Part | Why | |---|---| | `INSTALL_K3S_CHANNEL=stable` | Pins to the current stable release train instead of "whatever's newest today." All five nodes should be on the same channel. | | `K3S_TOKEN='$TOKEN'` | The shared secret the other four will present to join. | | `server` | This node is a master (a k3s "server"). | | `--cluster-init` | Start a new cluster and its database. **master1 only.** | | `--node-ip 10.99.99.11` | Pin the address k3s advertises. The node has one interface, but saying so removes any guessing; flannel and kubelet both use this. | | `--write-kubeconfig-mode=644` | Makes the cluster's key file readable so you can pull it out easily. A **lab convenience**; on a shared machine you'd leave it locked to root. | **Notice what's *not* here.** No `--flannel-backend`: k3s defaults to vxlan, which is exactly the module you loaded in Part 2 for cross-node traffic. No `--tls-san`: k3s automatically puts each node's own IP into its certificate, and you'll join and connect using those IPs, so there's nothing to add. You'd only need `--tls-san` later if you put a name or a shared address in front of the cluster. ### Check it came up (the snapshotter verdict lands here) ```bash lxc exec master1 -- systemctl is-active k3s # Expect: active. If it says "activating" for more than a minute or # "failed", the next command tells you why. lxc exec master1 -- k3s kubectl get nodes # Expect: one node, master1, STATUS Ready, ROLES control-plane,etcd,master. ``` If `get nodes` shows `master1` as `Ready`, the snapshotter is fine and you're done with Step 2. If the node never reaches `Ready`, read the log: ```bash lxc exec master1 -- journalctl -u k3s --no-pager | grep -i 'snapshotter' | tail -20 ``` **The one line that means trouble is `"overlayfs" snapshotter cannot be enabled`.** That, and only that, is the ZFS wall. Fix it by reinstalling master1 with the `native` snapshotter (the uninstall is clean and there's nothing to lose yet): ```bash lxc exec master1 -- /usr/local/bin/k3s-uninstall.sh lxc exec master1 -- bash -c " curl -sfL https://get.k3s.io | \ INSTALL_K3S_CHANNEL=stable \ K3S_TOKEN='$TOKEN' \ sh -s - server \ --cluster-init \ --node-ip 10.99.99.11 \ --write-kubeconfig-mode=644 \ --snapshotter=native " ``` **If you switch to `native` here, use it on every other node too.** A cluster where one node stacks layers and another copies them isn't broken, but it's an inconsistency you'll forget and then trip over. Whatever master1 uses, all five use. ### The scary log line that isn't: `modprobe: FATAL: Module ... not found` If you go looking through the full k3s log, you will almost certainly see lines like these near the start: ``` modprobe: FATAL: Module br_netfilter not found in directory /lib/modules/6.8.0-... modprobe: FATAL: Module overlay not found in directory /lib/modules/6.8.0-... ``` **This is the exact thing Part 2's Step 1 predicted, and it is not a fault.** It's the container's own `modprobe` failing, which, as Part 2 explained at length, it *always* does, because a container is not allowed to load kernel modules. k3s only reaches for `modprobe` when it didn't already find `/sys/module/`, so seeing it try just means its early check ran a moment before the module was visible. It logs a warning and carries on, exactly as designed. **The log line is not the ground truth. `/sys/module/` is.** That's the folder k3s actually checks, and it's the host's kernel, shared into the container. So don't trust or distrust the FATAL line either way. Check the real thing, on the **host**: ```bash for m in overlay nf_conntrack br_netfilter iptable_nat iptable_filter vxlan; do [ -d "/sys/module/$m" ] && echo "ok $m" || echo "MISSING $m" done ``` - **Six `ok`** → the modules are present, the FATAL lines were cosmetic, you're done. This is the normal outcome, and it's why master1 is `Ready` despite the scary log. - **`overlay` or `br_netfilter` MISSING** → your host actually lost Part 2's Step 1 (usually a reboot where the module list didn't reload). Fix it on the host, then restart k3s so the parts that check early re-run against a host that now has them: ```bash sudo systemctl restart systemd-modules-load.service # reloads /etc/modules-load.d/k3s.conf # re-run the six-module check above; expect all ok lxc exec master1 -- systemctl restart k3s ``` This is the whole "suspect the check before the machine" reflex from Part 2, in its most common Part 3 form: a red-looking log line, a healthy machine underneath, and one host-side command that tells you which you've got. ### One more look before moving on ```bash lxc exec master1 -- k3s kubectl get pods -A ``` Expect a handful of pods in `kube-system`: `coredns`, `local-path-provisioner`, `metrics-server`, and (unless you disabled them) `traefik` and a `svclb`. They should be `Running` or moving toward it. This is the control plane's own machinery, running as the first programs on your cluster. If they're stuck `Pending` or `ContainerCreating` for more than a couple of minutes, that's the snapshotter or a Part 2 module talking; check the log as above. **Two of those pods will read `Completed`, not `Running`, and that's success, not a problem:** ``` helm-install-traefik-crd-xxxxx 0/1 Completed 0 helm-install-traefik-xxxxx 0/1 Completed 2 (Nm ago) ``` These are **one-time setup Jobs**, not services. k3s installs its default ingress controller (Traefik) by running two Helm jobs: the first (`-crd`) registers Traefik's custom object types so Kubernetes knows what an `IngressRoute` is, and the second installs Traefik itself, which is why you also see a running `traefik` pod and the `svclb-traefik` load-balancer pods. A Job does its work and exits, so: - **`Completed` is the finished state**, the opposite of a crash. It's a receipt that the install ran. - **`0/1` next to it is correct**, not a warning. `READY` counts *live* containers; a finished Job has none. Read `Completed` as the verdict and `0/1` as its natural consequence. - **A couple of `RESTARTS` on the second one is normal.** It retries while the API server and CRDs are still settling in the first minute, then succeeds. A *completed* Job with early restarts is a non-event. Only a Job stuck in `Error` or `CrashLoopBackOff`, never reaching `Completed`, is worth chasing. They'll sit there as `Completed` records for the life of the cluster. Ignore them from here on. --- ## Step 3 — Talk to the cluster from the host ### Why bother, when `k3s kubectl` already works You *can* run every command as `lxc exec master1 -- k3s kubectl ...`, and for a quick look that's fine. But you'll be running a lot of these, and threading each one through `lxc exec` gets old fast. Pulling the kubeconfig out to the host lets you type plain `kubectl`. **This needs `kubectl` on the host.** If you don't have it, the `lxc exec master1 -- k3s kubectl` form keeps working and you can skip this whole step without losing anything. If you want it: ```bash # On the host, if kubectl isn't already installed: sudo snap install kubectl --classic ``` ### Pull the key and point it at master1 The kubeconfig k3s wrote assumes you're on master1 itself, so it points at `127.0.0.1`. From the host, that's the wrong address; you need master1's real one. ```bash mkdir -p ~/.kube lxc file pull master1/etc/rancher/k3s/k3s.yaml ~/.kube/config # Pulls the cluster's key file out of the container onto the host. sed -i 's/127.0.0.1/10.99.99.11/' ~/.kube/config # The file points at 127.0.0.1 (fine on master1, wrong from the host). # Repoint it at master1's real address. kubectl get nodes # Expect: master1, Ready. Now from the host, no lxc exec. ``` That `sed` is the one thing people forget, and the symptom is a `kubectl` that hangs or refuses the connection, because it's trying to reach a Kubernetes API on the host's own `127.0.0.1`, where nothing is listening. **If `kubectl` can't connect, check that line ran.** `~/.kube/config` is the default place `kubectl` looks, so you don't need to export anything. From here on, this guide writes plain `kubectl`. If you skipped this step, put `lxc exec master1 -- k3s ` in front of each one. --- ## Step 4 — Join master2 and master3 ### What "joining a server" means master2 and master3 install the same way as master1, with one difference each: instead of `--cluster-init` (start a new cluster), they get `--server https://10.99.99.11:6443` (join the one master1 started). Presenting the same token, each one adds itself as a second and third **etcd member**: a third of the shared database, and a third of the vote. **Do these one at a time and let each finish.** They're joining a database, and a database would rather add members in an orderly line than all at once. It costs you thirty seconds. ### master2 This is a **server** join. Note the `--server` *flag* (not the `K3S_URL` variable that workers use): ```bash lxc exec master2 -- bash -c " curl -sfL https://get.k3s.io | \ INSTALL_K3S_CHANNEL=stable \ K3S_TOKEN='$TOKEN' \ sh -s - server \ --server https://10.99.99.11:6443 \ --node-ip 10.99.99.12 \ --write-kubeconfig-mode=644 " ``` Wait for it to settle, then confirm from the host: ```bash kubectl get nodes # Expect: master1 AND master2, both Ready, both control-plane,etcd,master. ``` **Give master2 up to a minute to reach `Ready`.** It has to download k3s, join etcd, and start its own control plane. `NotReady` for thirty seconds is normal; `NotReady` for five minutes is not; check `journalctl -u k3s` on master2, and remember the snapshotter and inotify from Steps 1 and 2. ### master3, and watch this one master3 is identical except for its address: ```bash lxc exec master3 -- bash -c " curl -sfL https://get.k3s.io | \ INSTALL_K3S_CHANNEL=stable \ K3S_TOKEN='$TOKEN' \ sh -s - server \ --server https://10.99.99.11:6443 \ --node-ip 10.99.99.13 \ --write-kubeconfig-mode=644 " ``` Wait for it to settle, then confirm from the host: ```bash kubectl get nodes # Expect: three masters, all Ready. ``` **master3 is the node Part 2 kept warning you about.** If you skipped the inotify or keys sysctls in Part 2's Step 2, this is very often where it shows: masters 1 and 2 came up on a fresh pool of a shared limit, and master3 is the one that runs out. The error will say "too many open files" or a program will fail to start, and it will look like master3 is the problem. It isn't. It's the host setting, and the fix is in Part 2, not here. If all three are `Ready`: **you now have a highly available control plane.** Three members, a quorum of two. What that's worth is Step 8. Confirm the database sees three members: ```bash kubectl get nodes -l node-role.kubernetes.io/etcd=true # Expect: exactly master1, master2, master3. ``` --- ## Step 5 — Join the workers ### What "joining an agent" means Workers install differently from masters, and this is the distinction to keep straight. An **agent** joins with `K3S_URL` as an *environment variable*, not a `--server` flag, and its type is `agent`, not `server`. It gets no copy of the database and no vote; it shows up, presents the token, and waits for work. Getting this wrong is a real mistake with a quiet result: use the `server` form on a worker and you've accidentally added a fourth database member, which breaks the "odd number for voting" arithmetic Step 8 depends on. So read the type in each command. ### worker1 **Agent** join, using the `K3S_URL` variable and type `agent`: ```bash lxc exec worker1 -- bash -c " curl -sfL https://get.k3s.io | \ INSTALL_K3S_CHANNEL=stable \ K3S_URL='https://10.99.99.11:6443' \ K3S_TOKEN='$TOKEN' \ sh -s - agent \ --node-ip 10.99.99.21 " ``` ### worker2 ```bash lxc exec worker2 -- bash -c " curl -sfL https://get.k3s.io | \ INSTALL_K3S_CHANNEL=stable \ K3S_URL='https://10.99.99.11:6443' \ K3S_TOKEN='$TOKEN' \ sh -s - agent \ --node-ip 10.99.99.22 " ``` ### Check ```bash kubectl get nodes -o wide ``` Expect all five, all `Ready`. The three masters read `control-plane,etcd,master`; the two workers read `` under ROLES. **`` is correct, not an error.** It just means "an ordinary worker with no special jobs." The `-o wide` view also shows each node's INTERNAL-IP, which should be exactly the `.11` to `.13` and `.21` to `.22` you assigned. A wrong IP here means a node advertised something other than what you pinned, which is worth chasing now. If you'd like the workers to *say* "worker" instead of ``, it's cosmetic, but you can label them: ```bash kubectl label node worker1 node-role.kubernetes.io/worker=worker kubectl label node worker2 node-role.kubernetes.io/worker=worker ``` That changes the ROLES column and nothing else. Kubernetes doesn't use it; it's for your eyes. --- ## Step 6 — Prove the cluster actually works (and leave no trace) `Ready` means "the node is answering." It does not mean "a program can run here and reach a program over there." Those are different claims, and this step checks the second one, cheaply, before Part 4 leans on it. ### What this test does and does not leave behind Read this first if you're wary of it downloading things or making a mess. It's a fair worry, so here's the whole footprint up front: - **It creates** a deployment (`web`, two nginx copies), a service (`web`), and one throwaway probe pod (`busybox`). That's four short-lived objects, all in the `default` namespace, all named by you. - **It downloads** two small public images, `nginx` (~70 MB) and `busybox` (~4 MB), onto whichever nodes the pods land on. They get cached in each node's containerd image store. This is the only thing that lingers, and the cleanup below removes it. - **It creates no storage.** Neither image asks for a volume, so no PersistentVolumeClaim, no data written to disk, nothing to clean in that department. You can confirm at the end that `kubectl get pvc -A` is still empty. - **Nothing touches the control plane or `kube-system`.** If a step fails, the worst case is a leftover `web` deployment you delete by name, and it cannot damage the cluster. The plan is four phases: **record the clean state → run the test → remove everything → confirm you're back where you started.** After phase 4 the cluster is byte-for-byte the tidy thing Part 4 expects. ### Phase 1: Record the clean state, so you can prove you got back to it ```bash kubectl get nodes kubectl get pods -A -o wide # Every kube-system pod should be Running (the two helm-install Jobs read # Completed, that's fine, see Step 2). The -o wide column shows which node # each landed on: control-plane pods across the masters, svclb on all five. kubectl get all -n default # Expect ONE line: service/kubernetes (ClusterIP, port 443). That's the # built-in API-server endpoint; it lives in "default" permanently, in every # cluster, and is not test residue. Your baseline is "only service/kubernetes, # nothing else." Cleanup should return you to exactly this one line. kubectl get pvc -A # Expect: "No resources found." Nothing stored now; nothing should be after. ``` ### Phase 2: Run one real thing across two nodes This is the test that matters: schedule a program twice, make the copies land on different nodes, then confirm they can talk. It exercises flannel and the vxlan tunnel from Part 2 for real. ```bash kubectl create deployment web --image=nginx --replicas=2 kubectl expose deployment web --port=80 # Wait for both copies to be Running, and see where they landed: kubectl get pods -l app=web -o wide # Ideally one on worker1 and one on worker2. Kubernetes chooses; with two # replicas and two workers it usually spreads them. # Now reach the service from a third pod, by name: kubectl run probe --image=busybox --rm -it --restart=Never -- \ wget -qO- --timeout=5 http://web # Expect: the nginx welcome HTML. This proves a pod resolved a service # NAME, found a pod on ANOTHER node, and got bytes back, which is # exactly the cross-node path vxlan carries. # The --rm flag deletes the probe pod automatically when it exits. ``` **If the `wget` returns HTML, the hard part works.** DNS inside the cluster, service routing, and cross-node networking all just closed in one line. **If it hangs or times out** and the two `web` pods are on different nodes, that's the flannel/vxlan path; check that `vxlan` is still loaded on the host (Part 2 Step 1) and that node-to-node traffic passes (Part 2 Step 6). This is the same networking you already proved between containers; k3s is just using it now. ### Phase 3: Remove everything the test created ```bash # 1. The Kubernetes objects. Deleting the deployment cascades to its pods; # deleting the service removes it. This empties the default namespace. kubectl delete deployment web kubectl delete service web # 2. The probe pod is usually already gone (that's what --rm does), but if the # wget was interrupted it can linger. This removes it if present, and is a # no-op if it isn't: kubectl delete pod probe --ignore-not-found # 3. The cached images. This is the "extra stuff downloaded" part, the only # thing kubectl's deletes don't touch. Remove nginx and busybox from every # node's containerd store. (They're not in use once the pods are gone, so # this succeeds; it's a no-op on nodes that never ran them.) for n in master1 master2 master3 worker1 worker2; do echo "== $n" lxc exec "$n" -- k3s crictl rmi docker.io/library/nginx 2>/dev/null || true lxc exec "$n" -- k3s crictl rmi docker.io/library/busybox 2>/dev/null || true done ``` **Removing the images is optional, not required.** Kubernetes garbage-collects unused images on its own once a node's disk fills past a threshold, so leaving them costs you ~74 MB per affected node and nothing else. Remove them if you want a pristine slate to compare against; skip step 3 if you'd rather not bother. Part 4 works either way. ### Phase 4: Confirm you're back to the baseline ```bash kubectl get all -n default # Expect ONE line again: service/kubernetes, same baseline as Phase 1. # If web or probe still show here, their delete didn't finish; re-run it. kubectl get pvc -A # Expect: "No resources found." Nothing was ever stored. # If you did the image cleanup, confirm they're gone (blank output = gone): for n in master1 master2 master3 worker1 worker2; do echo -n "$n: "; lxc exec "$n" -- k3s crictl images | grep -E 'nginx|busybox' || echo "clean" done kubectl get nodes # Expect: five Ready. The cluster itself never changed. ``` If `default` shows only `service/kubernetes` and the nodes are all `Ready`, you've proven the cluster works *and* returned it to exactly the state Part 4 wants. The only permanent thing this step added to your cluster is confidence. ### A note on the masters running your programs By default, k3s lets the masters run your programs alongside the control plane. Deploy something and Kubernetes may place it on any of the five nodes. On a five-node lab that's usually what you want: more nodes to schedule onto, and the control plane here isn't under enough load to mind the company. Real clusters do the opposite. The masters run the machinery everything else depends on (the API server, scheduler, and the etcd database), and you don't want a busy application starving those of CPU or memory. So production clusters put a **taint** on the masters: a "don't schedule ordinary programs here" mark. A program can only land on a tainted node if it carries a matching **toleration**, an explicit "I'm allowed here" pass. The control plane's own pods carry that pass, so they keep running on the masters; your ordinary deployments don't, so they get pushed to the workers. If you'd rather run the lab the production way, taint the three masters: ```bash kubectl taint nodes master1 master2 master3 \ node-role.kubernetes.io/control-plane=:NoSchedule ``` `NoSchedule` affects **new** pods only. It won't evict anything already running on a master; it just stops fresh ones landing there. Undo it by repeating the command with a trailing `-` instead of `=`: ```bash kubectl taint nodes master1 master2 master3 \ node-role.kubernetes.io/control-plane:NoSchedule- ``` That's a genuine choice, not a fix; the default is fine for this lab. It's here so you understand what the default *is*, and so tainting doesn't look like magic if you meet it in a real cluster later. If you do taint, remember you've cut your schedulable nodes from five to two; a couple of small deployments won't notice, but it's why the guide leaves the masters open. --- ## Step 7 — Save a restore point (and understand why it's trickier now) You have a working cluster. That's worth being able to get back to. But snapshotting a cluster is **not** the same as snapshotting the five idle machines you saved in Part 2, and the difference is the whole reason Part 2 kept harping on stopped snapshots. ### Why you can't just snapshot the running masters Each master holds a live etcd member, and the three are constantly agreeing with each other about the cluster's exact state. If you snapshot them while they're running: - Each snapshot catches a database mid-write, at a slightly different instant. - Worse, if you ever **restore just one master** from such a snapshot, you're dropping a member with a stale, divergent history back into a cluster that has moved on. etcd may reject it, or, more confusing, accept it and disagree with itself. Part 2 said "a mid-write snapshot of a running database member is far less trustworthy than a stopped one." This is the situation it was warning you about. So the safe pattern is: **stop all three masters together, snapshot all five at the same quiet moment, start them back up.** All-or-nothing. ### Do it ```bash # Stop everything. Masters last is tidy but order barely matters when all go. for n in worker1 worker2 master1 master2 master3; do lxc stop "$n"; done # Snapshot all five at the same stopped instant. for n in master1 master2 master3 worker1 worker2; do lxc snapshot "$n" post-install done # Bring them back. for n in master1 master2 master3 worker1 worker2; do lxc start "$n"; done sleep 20 kubectl get nodes # expect: all five Ready again within a minute sudo zpool list default # ALLOC barely moves; the snapshots are near-free ``` `lxc snapshot` refuses to overwrite an existing name, so re-running this is safe; worst case, an error. The twenty-second sleep is because etcd needs a moment to re-elect and agree after a cold start; `NotReady` for a few seconds after is normal. ### The right tool for cluster *state* is different An LXD snapshot rolls back a whole *machine*, good for "I broke this node, give me the node back." For rolling back the *cluster's data*, the "I deleted something I shouldn't have" case, k3s has its own etcd snapshots, which are consistent by design because etcd takes them itself: ```bash lxc exec master1 -- k3s etcd-snapshot save # Writes a consistent database backup inside master1. This is what you'd # actually restore cluster contents from. Different job from lxc snapshot. ``` **When do you actually run that?** Two things to know, and they answer it together: First, **you're already covered for routine protection.** k3s takes etcd snapshots *automatically* out of the box: every 12 hours by default, keeping the last 5, saved under `/var/lib/rancher/k3s/server/db/snapshots`. You didn't set anything up; it's on. So you don't run `save` manually just to "have a backup"; the schedule handles that. Confirm it any time with: ```bash lxc exec master1 -- k3s etcd-snapshot list # Lists the snapshots k3s has taken; scheduled ones appear on their own. ``` Second, **you run `save` manually right before you deliberately do something risky to cluster data.** The scheduled snapshot might be eleven hours old; a manual one is a fresh, named point taken *at this exact instant*, just before the scary thing. Concretely, save first when you're about to: - upgrade k3s to a new version, - apply a large or unfamiliar manifest that creates/changes a lot of objects, - bulk-delete things, or do any "cluster surgery" you're unsure about, - run an experiment that mutates cluster state and you want a clean point to return to. The pattern is `save → do the risky thing → if it went wrong, restore; if it went fine, carry on`. It's the same reflex as Part 2's `lxc snapshot before-i-wreck-it`, just aimed at the database instead of the machine. **One caveat with manual saves:** unlike the scheduled ones, on-demand snapshots are **not** auto-pruned; they pile up until you remove them yourself with `k3s etcd-snapshot delete ` or `k3s etcd-snapshot prune`. Not a concern on this lab yet; worth knowing before you get in the habit. The two aren't interchangeable, and the difference is which layer you broke: the *machine*, or the *data on it*. Concretely: | What went wrong | Which snapshot | Why | |---|---|---| | "I ran a bad command inside master2 and its OS is a mess." | **LXD** (`lxc restore master2 ...`) | The node itself is broken. You want the whole machine back, k3s and all. | | "A host reboot left a node wedged / won't rejoin." | **LXD** | Same; restore the machine to a known-good moment. | | "I `kubectl delete`d the wrong deployment / namespace." | **etcd** | The machines are fine; the *cluster's records* are wrong. Only an etcd restore rewinds what Kubernetes remembers. | | "An upgrade or a bad manifest corrupted cluster objects." | **etcd** | The data is the casualty, not the OS. | | "I want to experiment and be able to undo *everything*." | **LXD, all five together** (Step 7's `post-install`) | Rolling every node back to the same stopped instant rewinds machines *and* their etcd data at once, the blunt, reliable reset. | The rule of thumb: **broke a box → LXD; broke what's *in* the cluster → etcd.** And when in doubt on this lab, the all-five-together LXD restore from Step 7 covers both, because it takes the whole cluster back to one consistent moment. You don't need an etcd snapshot right now; there's nothing in the cluster to lose yet. It's here so that when Part 4 puts real data in, you already know which of the two "snapshots" saves it. **LXD snapshots restore machines, etcd snapshots restore data.** Part 2's whole snapshot habit was the first kind; this is the second. --- ## Step 8 — Break it on purpose This is the point of building three masters. You're about to watch quorum work, then watch it fail, then bring it back, and it's safe because Step 7 gave you a way back. > **Read this before you start, or the first command will confuse you.** `kubectl get nodes` is **not a live ping.** It shows the control plane's *last recorded belief* about each node, and it lags reality by up to a minute. So when you stop a master, it can keep showing `Ready` for ~40 to 60 seconds before flipping to `NotReady`, because the control plane waits out a grace period after the node's last heartbeat before declaring it down. And while quorum is lost, the control plane can't write *any* status updates, so a node's status can be frozen at whatever it was when the outage began. **The honest, real-time view is `lxc list`** (the container's actual state); `kubectl` is answering "what did I last hear," not "is it alive right now." When the two disagree for a minute, believe `lxc list`. This is Part 2's "suspect the check before the machine" in yet another costume. ### What "voting" actually is: quorum has a leader The guide has said the three masters "vote on decisions," which is true enough to build on but hides one piece worth knowing before you watch it break: **the three etcd members are not equal peers taking a fresh vote each time. One of them is the *leader*.** etcd uses a consensus algorithm called Raft, and it works like this: - **One member is elected leader; the other two are followers.** All writes go through the leader. It proposes each change to the followers, and the change is *committed* once a majority, the quorum, has acknowledged it. So "the masters vote" really means "the leader proposes, and a change counts the moment a majority has stored it." - **Quorum is the count; the leader is the coordinator.** They're two halves of the same mechanism. Quorum is *how many must agree* (two of three); the leader is *who they're agreeing with*. - **If the leader disappears, the survivors hold an election.** They notice the missing heartbeats and, *provided a quorum still exists*, pick a new leader in about a second, and then writes resume. This is automatic; you don't do anything. - **No quorum means no leader can be elected.** You can't crown a leader without a majority agreeing to it. That's the deeper reason the cluster freezes when two masters are down: not just "can't write," but "can't even agree who's in charge." This reframes the experiment you're about to run. Killing a master does one of two things depending on which one it was: | You stop... | What happens | Why | |---|---|---| | a **follower** | barely a blip | the leader is still there; nothing needs electing | | the **leader** | a ~1-second pause, then normal | the two survivors elect a new leader, then writes resume | | a **second** master | writes stop entirely | one member left can't form a majority, so no leader can hold or be elected | You won't know in advance which master is the leader, so when you kill one you might see a brief write pause (you killed the leader) or nothing at all (you killed a follower). Both are correct. **Want to see who the leader is?** k3s embeds etcd inside its own process rather than running it as a pod, and, unlike some Kubernetes distributions, it doesn't ship the `etcdctl` tool, so there's no one-liner built in. Two ways to look. **Option A, read the log (no install).** The trap here is that the word "leader" appears in the log for *three unrelated things*: etcd's database leader (what you want), Kubernetes' controller-manager leader-election (noise), and a command-line flag (noise). The clean filter is to keep only the **raft** lines, because raft is etcd's consensus engine, so those lines are exactly the database-leader events and nothing else: ```bash # Past elections; returns immediately. Note: no -f, and --no-pager. lxc exec master1 -- bash -c "journalctl -u k3s --no-pager | grep raft | grep -iE 'leader|term'" ``` Reading the output: each line ends in a `msg` describing one event: | Line says (roughly) | What happened | |---|---| | `became leader at term N` / `elected leader X at term N` | An election completed. Member `X` is now the leader. | | `changed leader from X to Y` / `leadership transfer` | The leader moved from `X` to `Y`; this is what a *graceful* `lxc stop` of the leader produces. | | `lost leader X` then `no leader at term N; dropping...` | The brief gap with **no** leader; this is the ~1-second window before a new one is elected. Seeing it is seeing quorum re-forming. | Two things to know reading it: the members are named by **hex IDs** like `887ca2eda3b97d6e`, not `master1` (map them to node names with the `member list` command below), and the **term** number rises by one at every election, so a jump in "term" is your count of how many elections have happened. To *watch an election live*, use `-f` (follow), but know it will **sit blank until you actually trigger one**, because it only prints new events as they occur: ```bash # Terminal 1: starts blank on purpose; leave it running. lxc exec master1 -- bash -c "journalctl -u k3s -f | grep --line-buffered raft | grep --line-buffered -iE 'leader|term'" # Terminal 2: stop the current leader (find it with Option B first), # and within a second or two Terminal 1 prints the election. Ctrl-C to stop. ``` (The `--line-buffered` flags matter for the live version: without them, `grep` holds output in a buffer and you'd see nothing until a lot had piled up. For the past-events version above they're unnecessary.) **Option B, the instant "who is leader right now" table (installs a small tool).** This is the more reliable way to answer "which node is the leader this second," and it returns immediately with an `IS LEADER` column that's `true` for exactly one member: ```bash lxc exec master1 -- apt-get install -y etcd-client # Map hex member IDs to node names: lxc exec master1 -- bash -c ' ETCDCTL_API=3 \ ETCDCTL_ENDPOINTS="https://127.0.0.1:2379" \ ETCDCTL_CACERT="/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt" \ ETCDCTL_CERT="/var/lib/rancher/k3s/server/tls/etcd/server-client.crt" \ ETCDCTL_KEY="/var/lib/rancher/k3s/server/tls/etcd/server-client.key" \ etcdctl member list -w table ' # Show status per member, including which one IS LEADER: lxc exec master1 -- bash -c ' ETCDCTL_API=3 \ ETCDCTL_ENDPOINTS="https://127.0.0.1:2379" \ ETCDCTL_CACERT="/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt" \ ETCDCTL_CERT="/var/lib/rancher/k3s/server/tls/etcd/server-client.crt" \ ETCDCTL_KEY="/var/lib/rancher/k3s/server/tls/etcd/server-client.key" \ etcdctl endpoint status --cluster -w table ' ``` The `etcd-client` install is the only thing in Step 8 that adds a package to a node. It's harmless and small; remove it later with `lxc exec master1 -- apt-get remove -y etcd-client` if you want the node pristine, or just leave it; it does nothing unless you run it. **Two things to keep straight about these `etcdctl` commands:** *They only work because your cluster uses embedded etcd.* That's not a given; it's a consequence of the `--cluster-init` flag from Step 2, which is exactly what chose etcd over the default datastore. A k3s server started *without* `--cluster-init` (and without an external-datastore flag) uses **SQLite** instead, which has no members, no leader, and no `2379` port, so none of these commands apply to it. Your three-master HA setup is the etcd case, so they do. *Installing the tool isn't enough; it needs the certs and the right endpoint.* `etcdctl` talks to etcd over TLS, so every command carries the four `ETCDCTL_CACERT/CERT/KEY/ENDPOINTS` variables. Drop them and you don't get "command not found"; you get a confusing `context deadline exceeded` as it tries the wrong address with no credentials. That's why the block above looks verbose; the env vars are the point, not decoration. Quick reference for the three (plus one) commands: | Command | Purpose | |---|---| | `apt-get install -y etcd-client` | Installs `etcdctl`, the tool for inspecting the etcd datastore. | | `etcdctl member list -w table` | Lists the members (your HA masters) and maps each **hex member ID → node name**; this is what turns `887ca2eda3b97d6e` into `master1`. | | `etcdctl endpoint status --cluster -w table` | Shows each member's status: DB size, raft term, and the `IS LEADER` flag that marks the current leader. | | `etcdctl endpoint health --cluster -w table` | *(optional)* The actual health check; reports whether each member is responding. | To find the current leader you run two of these together: `endpoint status` tells you *which ID* is leader, and `member list` tells you *which node* that ID is. The leader it shows is the same member that most recently logged `became leader at term N` in Option A; the table just answers "who, right now" without waiting for an event. ### First, kill one master, and the cluster shrugs Three members, quorum of two. Take one away and two remain, which is still a quorum, so nothing should break. ```bash lxc stop master3 sleep 15 kubectl get nodes # master3 may still show Ready for up to ~40-60s (the lag from the note # above) before flipping to NotReady. lxc list shows it STOPPED immediately. # Either way the cluster answers; the other two masters are a quorum. kubectl create deployment survive --image=nginx --replicas=1 kubectl get pods -l app=survive -o wide # Expect: it schedules and runs. The cluster is fully operational on two # masters, because two of three is still a majority. ``` **master3 is `NotReady` (or about to be), not gone.** Kubernetes is holding its place, waiting for it to come back. Writes still work because the two surviving members can still agree. This is high availability doing exactly its job: a master died and you didn't notice from the outside. (If you want to *see* the status flip rather than wait, `sleep 60` then `kubectl get nodes`, or just check `lxc list` for the truth now.) ### Now kill a second master, and the cluster stops agreeing Take away a second member and only one remains. One of three is not a majority. The remaining master can't get a second vote, so it refuses to accept changes; it would rather stop than risk disagreeing with members it can't reach. ```bash lxc stop master2 sleep 15 kubectl get nodes # Likely hangs, or errors. The API server on master1 can't write to a # database that has lost quorum, so it stops answering normally. kubectl create deployment fail --image=nginx --replicas=1 # Expect: an error or a hang. No quorum, no writes. ``` **This is not your cluster breaking. It's your cluster protecting itself.** Faced with "keep going alone and maybe corrupt the shared truth" versus "stop until a majority is back," etcd stops. That refusal is the feature. A database that kept accepting writes with no way to agree on them is how you get two masters that each think they're right. ### Bring quorum back Start one of the stopped masters. Two of three is a majority again, and the cluster heals itself: ```bash lxc start master2 sleep 30 kubectl get nodes # Expect: master1 and master2 Ready, master3 still NotReady (still stopped). # The cluster ACCEPTS WRITES AGAIN, because two members can agree. lxc start master3 sleep 30 kubectl get nodes # all five Ready again ``` Give it a little longer than before; the members have to find each other and re-agree on everything that's true. ### Clean up: put the cluster back exactly as Step 7 left it Same idea as Step 6's cleanup: this experiment created a couple of `nginx` deployments and re-pulled the image onto whichever nodes ran them, so remove all of it and confirm you're back to the `post-install` baseline. ```bash # 1. The deployments. --ignore-not-found means "fail" is fine to name even if # it never got created (it may not have, since quorum was down when you tried). # These created no services and no volumes, so deployments are all there is. kubectl delete deployment survive fail --ignore-not-found # 2. Confirm the default namespace is back to just the built-in API service. kubectl get all -n default # Expect ONE line: service/kubernetes. If survive/fail still show, their # delete didn't finish; re-run step 1. kubectl get pvc -A # Expect: "No resources found." Nothing was stored, same as before. # 3. Optional: drop the nginx image from every node's containerd, same as # Step 6. Skip if you already pruned it there and it just re-pulled. for n in master1 master2 master3 worker1 worker2; do echo "== $n" lxc exec "$n" -- k3s crictl rmi docker.io/library/nginx 2>/dev/null || true done # 4. Final health check; the whole point is that the cluster survived. kubectl get nodes # Expect: five Ready. kubectl get nodes -l node-role.kubernetes.io/etcd=true # Expect: three etcd members again; quorum fully restored. ``` If `default` shows only `service/kubernetes`, all five nodes are `Ready`, and etcd lists three members, you're back to exactly the `post-install` state: you broke quorum on purpose, watched it protect itself, and healed it, leaving no trace. **Re-snapshotting isn't needed**; `post-install` from Step 7 is still an accurate picture of this clean state, so it remains your fallback for Part 4. ### If it doesn't come back cleanly You have `post-install` from Step 7. Because those snapshots were taken stopped and together, restoring them is the trustworthy move; all three etcd members roll back to the same consistent instant: ```bash for n in master1 master2 master3 worker1 worker2; do lxc stop "$n"; done for n in master1 master2 master3 worker1 worker2; do lxc restore "$n" post-install; done for n in master1 master2 master3 worker1 worker2; do lxc start "$n"; done sleep 30 kubectl get nodes ``` This is the payoff of doing Step 7 as all-or-nothing. A pile of snapshots taken at different moments would be worse than none here; taken together, they're an exact way back. --- ## Where you should be | Thing | State | |---|---| | k3s | Installed on all five, pinned to the `stable` channel | | Masters | `master1` to `master3`, `control-plane,etcd,master`, three-member etcd, quorum of two | | Workers | `worker1` to `worker2`, agents, running your programs | | kubectl | Working from the host against `10.99.99.11`, or via `k3s kubectl` on master1 | | Networking | Cross-node service reached by name, over vxlan; proven, not assumed | | Snapshotter | Settled; default on `dir`, default-or-`native` on ZFS, same choice on all five | | Snapshots | `post-install` on all five, taken stopped and together | | Quorum | Seen surviving one loss and refusing on two, and recovered | ### Things worth carrying forward - **"server" is a master, "agent" is a worker, and they join differently.** Servers use the `--server` flag; agents use the `K3S_URL` variable. The mistake is silent: a worker joined as a server becomes an unwanted fourth vote. - **The snapshotter is the one thing that fails loudly, and only on ZFS.** If a node won't go Ready, read `journalctl -u k3s` for `overlayfs` before suspecting anything else. `--snapshotter=native` is the escape hatch, but use it on all five or none. - **LXD snapshots and etcd snapshots are different tools.** LXD snapshots restore *machines*; `k3s etcd-snapshot` restores *cluster data*. Reach for the right one. - **Snapshot the masters stopped and together, always.** A running etcd member snapshotted mid-write is a trap, and restoring one stale member into a live cluster is worse than having no snapshot at all. - **master3 is still the canary for Part 2's Step 2.** If a third node ever misbehaves where the first two were fine, suspect a shared host limit (inotify or keys) before you suspect the node. - **Quorum is `floor(n/2)+1`, and it has a leader.** Three members tolerate one loss. Writes flow through one elected leader; lose the leader and the survivors elect a new one in about a second, but only while a quorum survives to do the electing. That's why the cluster has three masters and not two or four. ### What Part 4 does 1. Give the cluster real storage; the `nfs-common` and `open-iscsi` you installed back in Part 2's template were for exactly this. 2. Run stateful programs that keep data across restarts. 3. Watch what happens to that data when a worker dies and comes back. Step 7's `post-install` snapshot is the clean cluster you'll return to at the start of Part 4. Continue with [**Part 4, Adding Persistent Storage**](/posts/building-a-kubernetes-lab-with-k3s-and-lxd-part-4-adding-persistent-storage/). ### [Building a Kubernetes Lab with k3s and LXD Part 2 Building the Five Nodes](https://linsnotes.com/posts/building-a-kubernetes-lab-with-k3s-and-lxd-part-2-building-the-five-nodes/) > Get five LXD containers ready to run k3s. Load kernel modules, set sysctls, create a privileged LXD profile, and build five nodes from one template. This is **Part 2** of a series on building a Kubernetes lab with k3s and LXD. If you have not done Part 1 yet, start there: [**Part 1, Preparing the Ubuntu Host**](/posts/building-a-kubernetes-lab-with-k3s-and-lxd-part-1-preparing-the-ubuntu-host/). It gets the machine ready for everything below. **Where you are.** By the end of Part 1, one Ubuntu machine had become a host ready to run containers. Docker was removed, swap was off, LXD was installed, a virtual network called `lxdbr0` was set up on the `10.99.99.0/24` range, and a ZFS storage pool was ready. There are no containers yet. **What this part does.** It builds the five machines that k3s will run on, three masters and two workers, and proves each one works properly *before* Kubernetes is anywhere in the picture. **What this part does not do.** It does not install k3s. That happens in [**Part 3, Installing k3s**](/posts/building-a-kubernetes-lab-with-k3s-and-lxd-part-3-installing-k3s/). --- ## Words you'll need This part leans on a fair number of Linux and Kubernetes terms, and the steps read much more smoothly if those words already mean something to you. So rather than stop and explain each one mid-step, here they all are up front, in plain language. You do not need to memorise them. Skim the list now to build a rough picture, and come back to it whenever a word trips you up. | Word | What it means | |---|---| | **kernel** | The core of Linux. It talks tothe hardware and decides whatevery program is allowed to do. | | **kernel module** | An optional piece of the kernelthat can be switched on whenneeded, like a plug-in. Example:the code for handling a certaintype of network traffic. | | **`lsmod`** | Command that lists which kernelmodules are currently switched on. | | **`modprobe`** | Command that switches akernel module on. | | **sysctl** | A kernel setting you can changewhile the machine is running.Example: "how many files can oneuser watch at once." | | **container** | A program (or whole operatingsystem) that runs in an isolatedbox on your machine. Faster andsmaller than a virtual machine. | | **LXD** | The tool from Part 1 that createsand manages containers. `lxc` isthe command you type; LXD is thebackground service that does thework. | | **host** | Your actual Ubuntu server,the machine everything runs on. | | **node** | One of the five containers you'reabout to build. Kubernetes callsits machines "nodes." | | **profile** | A saved set of container settingsin LXD. Apply a profile to acontainer and it gets all thosesettings at once. | | **k3s** | A small, complete version ofKubernetes. What you're buildingthis lab for. | | **kubelet** | The part of Kubernetes that runson each node and starts and stopsthe actual programs. | | **kube-proxy** | The part of Kubernetes that setsup network rules so programs canfind each other. | | **containerd** | The program k3s uses to actuallyrun containers. So: containers,inside your container. | | **etcd** | The database where Kubernetesstores everything. The threemasters will share one. | | **flannel** | The default networking system ink3s. It connects programs runningon different nodes. | | **`lxdbr0`** | The virtual network switch LXDcreated in Part 1. Your five nodesplug into it. | | **snapshot** | A saved copy of a container atone moment. You can jump backto it later. | --- ## Three things to know before you start ### 1. Nothing here fails loudly This is the most important thing on the page. If you skip a step in this guide, k3s will **not** stop with an error. It will start, report that it is healthy, and then something that looks unrelated will break later on. Here are some real examples of what you would actually see: | What you skipped | What you would see instead | |---|---| | One kernel module | Programs on different nodes can't talk to each other | | One kernel setting | The *third* node fails to start, while nodes 1 and 2 are fine | | One container permission | "Permission denied" errors in five different places | None of those point back at the real cause. That is why this part exists, and why it checks everything as it goes. ### 2. Most of this is needed because of LXD, not Kubernetes If you installed k3s straight onto an Ubuntu server, with no containers involved, you could skip almost this entire guide. k3s sets itself up. The catch is that k3s cannot do that from *inside* a container, because a container is not allowed to change the kernel. So you do that groundwork for it, on the host, before k3s ever runs. Wherever a step is genuinely required rather than just tidy, this guide says so. ### 3. Some checks will look alarming for no reason A few times in this guide, a perfectly healthy system gives you an answer that looks like a disaster. For example, a command may report that LXD is not running when LXD is actually fine. This happens because a check sometimes answers a slightly different question than the one you had in mind. It is not your mistake, and it is not a fault. Each case is explained right where it happens, so there is nothing to memorise now. Just carry one habit with you: **when a check surprises you, suspect the check before you suspect your machine.** --- ## Step 1 — Check Part 1 is still good **Why:** everything below assumes Part 1's setup. If something drifted, find out now, not in three hours. **Safe to run:** every command here only *reads*. Nothing changes. ```bash # --- The host itself --- swapon --show # Lists swap space. Swap = disk used as pretend RAM. # Kubernetes refuses to start if swap is on. Expect: nothing printed. free -h | grep -i swap # Second opinion on the same thing. Expect: Swap: 0B which docker || echo "docker: gone" # Part 1 removed Docker so there's only one thing managing container # networking. Expect: "docker: gone" # --- LXD --- snap list lxd # Confirms LXD is installed and shows its version. systemctl is-active snap.lxd.daemon.unix.socket # See the note below. Check the SOCKET, not the daemon. Expect: active lxc info | head -3 # This wakes LXD up. Don't skip it; the next commands need it awake. lxc list # Expect: an empty table. No containers yet. lxc network list # Expect: lxdbr0, MANAGED = YES, IPv4 = 10.99.99.1/24 lxc storage list # Expect: a pool named "default", type zfs (or dir) # --- The virtual network --- ip -brief addr show lxdbr0 # Shows lxdbr0's address. Expect: 10.99.99.1/24 ip route | grep lxdbr0 # Shows that traffic for 10.99.99.0/24 goes to lxdbr0. # --- Free space --- sudo zpool list # Your ZFS storage pool. Expect: ~199G, HEALTH ONLINE, almost empty. df -h / # Free space on the main disk, separate from the ZFS pool. ``` ### About that LXD check **Don't run `systemctl is-active snap.lxd.daemon` here.** It will say `inactive`, and that is **normal**. Here's why. LXD doesn't sit running all the time when it has nothing to do. Instead: 1. A **socket** listens quietly. (A socket is a doorway other programs knock on.) 2. When something knocks (say, your `lxc list`), LXD starts up and answers. 3. When it's idle again with no containers, it stops. So with zero containers, "the daemon isn't running" is a healthy state, not a fault: ``` snap.lxd.daemon.unix.socket active ← always, if LXD is installed ↓ "lxc info" knocks on the door snap.lxd.daemon active ← now it's awake ``` Once your five nodes are running, LXD stays awake permanently and this stops being confusing. **Write down your network range now.** This guide uses `10.99.99.0/24` from Part 1's example. If you chose something different, swap it in everywhere below. --- ## Step 2 — Switch on the kernel modules k3s needs This is the first of two small config files you create in this part. Both follow the same shape: you look at what is already there, you write one file, and then you check that it took effect. This step writes `/etc/modules-load.d/k3s.conf`, which tells Linux to switch on a handful of kernel modules every time the machine boots. ### Why this file has to exist A kernel module is an optional piece of Linux you switch on when you need it, a bit like a plug-in for the core of the system. A lot of networking and container features live in these modules. On a normal server you would never think about this. When k3s starts, it switches on the modules it needs by itself. Your nodes are not normal servers, though. They are containers, and that is what changes things: > All five containers share the host's single kernel. There is only one set of modules for the whole machine, and a container is not allowed to change them. An unprivileged container that tries to switch a module on is refused, because changing the kernel would affect the host and every other container too. So when k3s starts inside a node and reaches for a module, it cannot switch one on. The fix is to switch the modules on yourself, once, on the host. Because all five containers share that same kernel, a module you load on the host is instantly available inside every node. ### Why k3s will not warn you Here is the part that makes this step easy to skip and painful to debug: k3s does not stop when a module is missing. It just carries on. When k3s starts, for each module it needs it does two things, in order: 1. It looks for a folder named `/sys/module/`. That folder exists only when the module is already switched on. If k3s finds it, it says "already loaded" and moves on happily. 2. If the folder is not there, k3s tries to switch the module on itself, with a command called `modprobe`. Inside a container that attempt fails, but k3s only writes a warning to its log and keeps going. So a node with a missing module still starts, still reports "Ready", and is quietly broken. Here is what that looks like in a real log, with one module missing and one already present: ``` level=warning msg="Failed to load kernel module nf_conntrack with modprobe" level=info msg="Module br_netfilter was already loaded" ``` Nothing crashed. Nothing turned red. One module simply is not there. The good news is hidden in point 1 above. If you switch the module on from the host first, that `/sys/module/` folder already exists by the time k3s looks. Your container can see it, because it is the host's kernel. k3s finds it, says "already loaded", and never runs the `modprobe` that would have failed. You are not fighting k3s. You are handing it the answer it checks for first. ### Which modules to switch on For a typical k3s setup with its default Flannel networking on Ubuntu 24.04, these are the modules that matter. The exact set can shift a little with your kernel, your k3s version, and your networking choices, but this list covers the common case, and loading a couple of extras does no harm. | Module | What it is for | If it is missing | |---|---|---| | `overlay` | Lets containerd stack image layers into a container's filesystem | containerd usually cannot unpack images, so nothing runs | | `nf_conntrack` | Tracks live network connections so replies find their way back | Replies get dropped | | `br_netfilter` | Makes bridge traffic visible to the firewall rules Kubernetes writes | Kubernetes' internal service addresses do not work | | `iptable_nat` | Lets the older iptables path redirect traffic | Traffic can go nowhere on that path | | `iptable_filter` | The table the older iptables path writes rules into | Nothing to write rules into on that path | And one more that k3s does not name, but Flannel needs: | Module | What it is for | If it is missing | |---|---|---| | `vxlan` | Builds the tunnel Flannel uses to carry traffic between different nodes | Programs on the same node still talk. Across nodes, nothing gets through. | A note on `vxlan`: Flannel expects Linux to load it on demand when it builds the tunnel, and on modern kernels that often happens on its own. Loading it up front just removes the "did it or did not it" question. A note on `iptable_nat` and `iptable_filter`: Ubuntu 24.04 leans on the newer nftables system, so these older iptables modules are often not strictly needed. They are tiny and harmless, so this guide loads them anyway, as insurance in case something reaches for the older path. You will also see longer lists online with names like `xt_conntrack`, `ip_vs`, or `dm_thin_pool`. The `xt_*` ones get pulled in automatically as dependencies, and the rest are for features you are not using here. A short list you understand beats a long one you copied. ### First, check what is already loaded Before writing anything, see which of these are switched on already: ```bash lsmod | grep -E '^(overlay|br_netfilter|nf_conntrack|vxlan|iptable_nat|iptable_filter)\s' # lsmod lists the modules that are currently switched on. # The grep narrows it to just the ones we care about. ``` On the reference machine (Ubuntu 24.04, right after Part 1), the whole output was a single line: ``` nf_conntrack 200704 3 nf_nat,nft_ct,nft_masq ``` Only one was present. Here is why: | Module | Present? | Why | |---|---|---| | `nf_conntrack` | **yes** | LXD switched it on itself, so containers can reach the internet. | | `overlay` | **no** | Docker usually loads this, and Part 1 removed Docker. Nothing else needed it. | | `br_netfilter` | **no** | A plain Ubuntu server has no reason to switch it on. | | `vxlan` | **no** | Same. | | `iptable_nat`, `iptable_filter` | **no** | Ubuntu 24.04 uses nftables by default, so the old iptables modules were never loaded. | `overlay` being missing is the one that earns this whole step. Without it, containerd usually cannot unpack an image, and the error you get back tends to look like a download or disk problem rather than a missing module. It is the most important line in the file you are about to write. ### Write the file This writes the module list to `/etc/modules-load.d/k3s.conf`, which Linux reads at every boot, and then loads the modules right now so you do not have to reboot: ```bash sudo tee /etc/modules-load.d/k3s.conf >/dev/null <<'EOF' # Linux reads this file at every boot and switches on the modules listed. # # They are here because k3s cannot switch them on from inside a container. # k3s checks /sys/module/ first and skips its own attempt if the # module is already on, which is exactly what this file arranges. # --- The ones k3s asks for --- overlay nf_conntrack br_netfilter iptable_nat iptable_filter # --- Not named by k3s, but Flannel needs it --- # Without this, programs on different nodes cannot reach each other. vxlan EOF ``` Then switch the modules on right now, so you do not have to reboot: ```bash sudo systemctl restart systemd-modules-load.service # Reads the file above and switches on the listed modules immediately. ``` ### Verify it worked ```bash systemctl status systemd-modules-load.service --no-pager | head -5 # Did the service run cleanly? Expect: "active (exited)". # "exited" is correct here. It does its job once and stops. lsmod | grep -E '^(overlay|br_netfilter|nf_conntrack|vxlan)\s' # Expect: all four now listed. # The check that matters most. This is exactly what k3s will look for: for m in overlay nf_conntrack br_netfilter iptable_nat iptable_filter vxlan; do [ -d "/sys/module/$m" ] && echo "ok $m" || echo "MISSING $m" done ``` That last loop is worth more than trusting `lsmod`, because `/sys/module/` is the exact folder k3s checks. Six `ok` lines mean k3s will say "already loaded" six times and never reach for the `modprobe` it cannot run. Any `MISSING` is a warning line waiting for you in Part 3. You may see `refcnt 0` next to the new modules. That just means nothing is using them yet, which is correct. Nothing will, until k3s starts. If the service reports a failure: ```bash journalctl -u systemd-modules-load.service -b --no-pager # Shows why. Usually a typo in a module name. ``` ### Worth knowing - **LXD has a setting that looks like it does the same job** (`linux.kernel_modules`). It does work, but it only runs when a container starts, it fails without telling you, and it hides a fact about your host inside a container's config. Keep this file as the real answer. You will add the LXD setting in Step 4 as a backup, not a replacement. - **The order is handled for you.** Step 3 changes kernel settings, and some of those settings do not exist until `br_netfilter` is switched on. Linux runs modules-load before sysctl automatically, so this works out. You just cannot do it the other way round. - **If you ever rebuild this lab on virtual machines, delete this step.** Each VM has its own kernel and k3s handles the modules itself. This file exists only because of containers. --- ## Step 3 — Change the kernel settings k3s needs This is the second of the two config files. It follows the same shape as Step 2: check what is there, write one file, then verify. This step writes `/etc/sysctl.d/99-k3s-lxd.conf`, a few kernel options that k3s needs but cannot set for itself from inside a container. ### Why this file is needed A **sysctl** is just a kernel setting you can change while the machine is running. An example is "how many things one user is allowed to watch for changes at once." k3s does set its own sysctls when it starts, so it is fair to ask why this step exists at all. The answer is that these settings come in two kinds, and k3s can only handle one of them from inside a container. **Per-container settings.** Each container gets its own private copy. k3s changes these inside each node and it works fine. Setting them on the host would do nothing for your nodes, so you leave them alone. **Whole-machine settings.** There is a single copy shared by the entire machine, and a container is allowed to read it but not change it. When k3s tries, it gets "permission denied", writes a line to a log nobody reads, and carries on with whatever value was already there. That second kind is the whole reason for this file. There are only four of them, and they are the ones to get right. ### The settings that actually matter These are the whole-machine settings a container cannot change for itself. If you do nothing else in this step, do these four. | Setting | Default | Change to | Why | |---|---|---|---| | `fs.inotify.max_user_instances` | 128 | `1024` | "inotify" is how programs watch files for changes, and Kubernetes leans on it heavily. All five nodes run as the same user and share this one pool. With the default of 128 and enough pods, a node can run out and refuse to start with "too many open files". Because the pool is shared, it tends to be a later node that hits the wall while the earlier ones are fine, which makes it look like that one node is broken when it is not. | | `fs.inotify.max_user_watches` | 65536 | `1048576` | The same shared-pool problem, just slower to appear. When it runs out, changes to config files quietly stop being noticed. | | `kernel.keys.maxkeys` | 200 | `2000` | containerd uses the kernel's keyring to hold secrets and identifiers, and that allowance is shared across all your nodes. Set too low, it can show up as containers randomly failing to start. | | `kernel.keys.maxbytes` | 20000 | `2000000` | The same keyring, measured in bytes rather than number of keys. Raise it for the same reason. | The pattern is the same every time: a single shared pool that all five nodes draw from. That is the shared-kernel trade-off again. On five real servers, each would have its own pool and none of this would come up. ### Settings k3s handles, that you pin anyway These next four are the per-container kind, so k3s already sets them inside each node. You set them on the host too, not because the nodes need it, but so that if something goes wrong later the host is not one more place you have to check. | Setting | Change to | Why bother | |---|---|---| | `net.netfilter.nf_conntrack_max` | `393216` | k3s sets this per node, but whether it fully takes effect from inside a container is not guaranteed, and if it fails it fails silently. A sensible value on the host gives you a sensible fallback. | | `net.bridge.bridge-nf-call-iptables` | `1` | This one does not reach your nodes; each has its own copy that k3s sets. On the host it only affects `lxdbr0`, and it is usually already `1` once `br_netfilter` is on. You are writing down a value, not fixing one. | | `net.bridge.bridge-nf-call-ip6tables` | `1` | The same thing for IPv6. Harmless. | | `net.ipv4.ip_forward` | `1` | LXD already turned this on so containers can reach the internet. You are pinning it so a later change cannot quietly undo it. | If you set only the first four, the lab will work. This second group is insurance, and it is worth knowing which half is which. ### Two quiet traps before you write the file Neither of these gives you an error. Both quietly produce the wrong result and report success, so they are worth knowing first. **Trap 1: a comment only works at the very start of a line.** If you write: ``` net.ipv4.ip_forward = 1 # for lxdbr0 ``` Linux tries to set the value to the whole string `1 # for lxdbr0`. You get away with it here by luck, because Linux reads the `1`, hits a space, and stops, but there is no warning and next time you may not be so lucky. Keep every comment on its own line. **Trap 2: your file runs last, so it can undo something Ubuntu set on purpose.** Files in `/etc/sysctl.d/` are applied in number order, and your file is numbered `99-`, so it runs after Ubuntu's own files and wins any disagreement. Suppose you copied `vm.max_map_count = 262144` from some guide. Ubuntu 24.04 has already set that to `1048576` in its own file. Because yours runs later, you would quietly cut it to a quarter of what Ubuntu chose, and `sysctl --system` would report that as a success. So before adding any setting, check that nobody has already set it: ```bash grep -rn "vm.max_map_count\|inotify\|conntrack_max" \ /etc/sysctl.d/ /usr/lib/sysctl.d/ /etc/sysctl.conf 2>/dev/null # Searches every sysctl file on the system for these settings. # If Ubuntu already sets one, leave it alone. ``` The numbering system exists so you can override things. That cuts both ways. ### Write the file ```bash sudo tee /etc/sysctl.d/99-k3s-lxd.conf >/dev/null <<'EOF' # Kernel settings for running five k3s nodes as LXD containers. # Comments must be on their own line. Linux does not strip a comment # from the end of a value line; it tries to use it as part of the value. # --------------------------------------------------------------- # The four that matter: a single shared copy for the whole machine, # and no container can change them. All five nodes run as the same # user and draw from these shared pools. THIS IS WHY THIS FILE EXISTS. # --------------------------------------------------------------- fs.inotify.max_user_instances = 1024 fs.inotify.max_user_watches = 1048576 kernel.keys.maxkeys = 2000 kernel.keys.maxbytes = 2000000 # --------------------------------------------------------------- # The rest: k3s sets these per node itself. These host values are # just so the host is not a second thing to check when debugging. # --------------------------------------------------------------- net.netfilter.nf_conntrack_max = 393216 # Each node has its own copy, so this only affects lxdbr0 on the host. # Usually already 1 once br_netfilter is on. Pinned, not fixed. net.bridge.bridge-nf-call-iptables = 1 net.bridge.bridge-nf-call-ip6tables = 1 # For lxdbr0's internet access. LXD already set this. net.ipv4.ip_forward = 1 # --------------------------------------------------------------- # LEFT OUT ON PURPOSE: vm.max_map_count # Ubuntu 24.04 already sets it to 1048576 in 10-map-count.conf. # This file runs later (99-), so setting it here would only risk # lowering Ubuntu's value. Leave Ubuntu's alone. # --------------------------------------------------------------- EOF ``` Then apply the settings now, without rebooting: ```bash # --system is very noisy; it prints every file and every setting. # The grep hides the noise and shows only real problems. sudo sysctl --system 2>&1 | grep -iE 'error|cannot|denied|invalid' ``` That grep should print nothing at all. Silence is success here. ### Verify it worked ```bash sysctl fs.inotify.max_user_instances \ fs.inotify.max_user_watches \ kernel.keys.maxkeys \ kernel.keys.maxbytes \ net.netfilter.nf_conntrack_max \ net.bridge.bridge-nf-call-iptables \ net.ipv4.ip_forward \ vm.max_map_count ``` - The four that matter should read back exactly what you put in the file. - `vm.max_map_count` should be **1048576**, Ubuntu's value, which you left alone. Anything lower means something else in `/etc/sysctl.d/` is overriding it, so find it with the `grep` above. - `No such file or directory` on the `net.bridge.*` lines means `br_netfilter` is not switched on. Go back to Step 2. This is the ordering point from Step 2 showing up as a failure. ### Two lines in the output worth reading Scroll back through what `sysctl --system` printed. Two lines are worth a look. **`net.ipv4.conf.all.rp_filter = 2` is good news.** This setting decides how strictly Linux checks that a reply comes back the way the request went out. Ubuntu ships mode `2`, the relaxed one. Mode `1`, the strict one, breaks Kubernetes networking, because replies take a different path and get thrown away. You already have the right value and did nothing to get it. If you ever inherit a server set to `1`, this is where that problem hides. **`/run/sysctl.d/zz-lxd.conf` means LXD wrote a settings file of its own.** The next section is about that. ### The Ubuntu 24.04 AppArmor setting, which LXD already handles Ubuntu 24.04 added a security restriction that stops ordinary programs from creating certain kinds of isolation. The programs k3s runs inside your container need exactly that kind of isolation. This restriction did not exist on 22.04, and left on, it causes confusing failures. You do not need to do anything about it, but it is worth understanding rather than skipping. Ubuntu's own file turns the restriction on: ``` kernel.apparmor_restrict_unprivileged_userns = 1 ``` Yet at the very end of `sysctl --system`, you see this: ``` * Applying /run/sysctl.d/zz-lxd.conf ... kernel.apparmor_restrict_unprivileged_userns = 0 kernel.apparmor_restrict_unprivileged_unconfined = 0 ``` LXD wrote that file itself when you installed it in Part 1. The `zz-` prefix is deliberate: files apply in alphabetical order, so `zz-` runs last and wins, over Ubuntu's `10-` and over your `99-` too. LXD knows its containers need this and turns the restriction off on their behalf. (The exact settings LXD writes have shifted a little across LXD versions, so treat this as how current Ubuntu 24.04 with the LXD snap behaves, and check yours below rather than trusting the text.) Three things follow from that: - **Do not add this setting to your own file.** LXD handles it, and `zz-` beats `99-` anyway, so your line would do nothing while looking important. - **It lives in `/run`, not `/etc`.** `/run` is wiped and rebuilt at every boot. That is correct: the change lasts exactly as long as LXD is installed and disappears cleanly if you ever remove it. Do not "fix" this by copying it into `/etc`. - **You have accepted a trade-off without being asked.** A security protection is off so your lab can run. It is the same bargain as Part 1's, and LXD simply made the call at install time. Check yours rather than trusting the text: ```bash sysctl kernel.apparmor_restrict_unprivileged_userns # Expect: 0 cat /run/sysctl.d/zz-lxd.conf 2>/dev/null || echo "not present, investigate before Step 4" # Expect: a small file that explains itself. ``` If you get `1`, LXD did not write it. Find out why before you build five nodes on top. ### Reboot now, while it is free The whole point of these two files is that they survive a reboot. The only honest way to know is to reboot and look. Do it here. You have no containers yet, so there is nothing to lose, and this is the last moment that stays true. ```bash sudo reboot ``` When it comes back, check the host first, and only the host. None of this involves LXD yet: ```bash # Did Step 2 survive? for m in overlay nf_conntrack br_netfilter iptable_nat iptable_filter vxlan; do [ -d "/sys/module/$m" ] && echo "ok $m" || echo "MISSING $m" done # Did Step 3 survive? sysctl fs.inotify.max_user_instances kernel.keys.maxkeys vm.max_map_count # Does Part 1 still hold? swapon --show # expect: nothing ``` Expect six `ok`, your four settings, `vm.max_map_count = 1048576`, and no swap. Now wake LXD up before you check anything LXD owns. This is the socket thing from Step 1 again, and it will fool you if you forget: ```bash lxc info | head -3 # THIS is what starts LXD. Don't skip it. sleep 3 ip -brief addr show lxdbr0 ls -l /run/sysctl.d/zz-lxd.conf sysctl kernel.apparmor_restrict_unprivileged_userns ``` If you check these before waking LXD, you get two alarming answers that are both wrong: - `Device "lxdbr0" does not exist`, because LXD creates the network when it starts, and with no containers nothing has woken it yet. - `apparmor_restrict_unprivileged_userns = 1`, because `/run` was wiped at boot and LXD has not rewritten its file yet. Neither is a fault, though both look exactly like one. Two more things in that output are worth understanding: **`lxdbr0 DOWN` is correct.** A virtual switch with nothing plugged into it has no signal, so Linux marks it down. The address is there and the setup is fine. It comes UP the moment the first container plugs in, in Step 5. A switch still DOWN once five containers are running would be a real problem. **The AppArmor setting reads `0`, and the file's timestamp is only seconds old, not from boot time.** That answers a question the file's location raises. Ubuntu applies sysctl files very early in boot, long before LXD starts, so it could never have read `zz-lxd.conf`, because the file did not exist yet. LXD does not rely on the file at all. It applies the setting itself when it starts, and writes the file only so anything that reads it later sees the right value. So the restriction really is `1` early in boot and drops when LXD wakes. That is harmless, because no container can start before LXD does anyway. --- ## Step 4 — Create the k3s profile ### Why the default settings aren't enough Part 1's `lxd init` created a `default` profile giving every container a network connection and a disk. That's enough to boot Ubuntu and get an address. For most purposes, that *is* a machine. k3s is not most purposes. **k3s wants to be an operating system that runs containers, inside a container.** Specifically: - It runs **its own containerd**, which needs to create isolated environments and mount filesystems, things containers are normally blocked from doing. - It runs **kube-proxy**, which rewrites firewall rules and writes to system folders that are normally read-only. - It runs **kubelet**, which reads the kernel's log device, which containers don't have. Each of those hits a wall. The profile below takes the walls down one at a time. **Each one costs you something.** ### What each setting does | Setting | What it does | What breaks without it | |---|---|---| | `security.nesting: true` | Allows containers **inside** this container | containerd won't start at all. k3s never gets going. | | `security.privileged: true` | Turns off the usual translation that makes "root" inside the container a harmless nobody outside it | A scattering of "permission denied" errors that each look like a different problem | | `lxc.apparmor.profile=unconfined` | Removes the security profile wrapped around the container | containerd errors when starting programs, because it wants to apply its own profiles and can't | | `lxc.mount.auto=proc:rw sys:rw` | Makes two system folders writable | kube-proxy can't write its network settings and quietly gives up | | `/dev/kmsg` device | Passes the kernel's log device into the container | kubelet fails to start. **Very common, rarely explained.** | | `boot.autostart: true` | Nodes come back after you reboot the host | You reboot and wonder where your cluster went | | `linux.kernel_modules` | Backup for Step 2. **Must list the same six modules.** | Nothing, if Step 2 is right | ### The cost, said plainly `security.privileged` + no AppArmor + writable system folders means: > **Root inside one of these containers is root on your host.** These nodes are **not** a security boundary. Anything that escapes one owns your machine. That's fine on a computer you own and use for learning. It is **not** fine on a shared server, and it is not something to copy into anything real. Part 1 named this trade-off, 2-second boots in exchange for weaker isolation. This is where you sign it. ### Create it ```bash lxc profile create k3s # Makes an empty profile named "k3s". ``` Now fill it in. The command below replaces the whole profile with the YAML, which is why `name` and `description` are in it. It replaces the profile rather than merging into it: ```bash cat <<'EOF' | lxc profile edit k3s name: k3s description: Privileged node profile for running k3s inside LXD. NOT a security boundary. config: # Come back automatically after a host reboot. boot.autostart: "true" # Allow containers inside this container. k3s runs its own. security.nesting: "true" # Container root = host root. Needed for the mounts and settings # kubelet and kube-proxy make. THIS IS THE ONE THAT REMOVES ISOLATION. security.privileged: "true" # Backup for Step 2's /etc/modules-load.d/k3s.conf. # Must be the SAME SIX modules. If you change one, change both. linux.kernel_modules: overlay,nf_conntrack,br_netfilter,iptable_nat,iptable_filter,vxlan raw.lxc: | # Remove the security profile; containerd wants to apply its own. lxc.apparmor.profile=unconfined # Make /proc and /sys writable; kube-proxy writes to them. lxc.mount.auto=proc:rw sys:rw devices: # Pass in the kernel's log device. kubelet reads it at startup # and fails without it. kmsg: source: /dev/kmsg path: /dev/kmsg type: unix-char EOF ``` ### Check it ```bash lxc profile show k3s lxc profile list ``` Both `default` and `k3s` should be there, both with `USED BY 0`. The `k3s` profile has no network and no disk in it, and **that's correct.** **LXD will not show you back exactly what you typed, and that's fine.** It sorts the settings alphabetically and adds `used_by: []` and `project: default`. What matters is that every setting you wrote is present with the value you gave it. **A setting that's *missing* means LXD rejected your text, and it does that quietly.** That's the failure to look for. **Profiles stack.** You'll launch containers with `-p default -p k3s`: ``` default profile → network connection + disk k3s profile → the permissions above → container gets both ``` Keeping them separate means `lxc profile show k3s` shows you exactly what you gave away, on one screen, with nothing else mixed in. **Keep the module list matching Step 2:** ```bash lxc profile get k3s linux.kernel_modules grep -v '^#\|^$' /etc/modules-load.d/k3s.conf | tr '\n' ',' | sed 's/,$//' # Prints both lists so you can compare them. ``` Different order is fine. Different **contents** is a question to settle now. Two lists that disagree are worse than no list at all. Nothing will break, because this setting is only a backup and Step 2's file does the real work, but in six months, neither list tells you which one you meant. --- ## Step 5 — Build one machine, not five ### Why one first You're about to install the same packages and make the same checks five times. Don't. Build **one** container, get it right, save it, and copy it. Three reasons, in order of importance: **1. A saved template is an undo button for this whole step.** Get something wrong and you fix one container, not five that have already drifted apart. **2. It's a comparison tool later.** In Part 3, when one node misbehaves, `lxc copy template/base test1` gives you a known-good node in one second to compare against. That's worth more than the setup time it saves. **3. It's nearly free on ZFS.** This is Part 1's storage choice paying off. ZFS uses **copy-on-write**: making a copy doesn't duplicate anything, it just points the copy at the same data and only writes something when the copy changes. Five nodes from one 700 MB template cost about 700 MB, not 3.5 GB. ### Launch it ```bash lxc launch ubuntu:24.04 template -p default -p k3s # ubuntu:24.04 = Canonical's official image # template = the container's name # -p default = network + disk (from Part 1) # -p k3s = the permissions (from Step 4) lxc list ``` **About the image name.** Use `ubuntu:24.04`. Older guides say `images:ubuntu/24.04`, but that source was removed from LXD's defaults and the command now fails on a fresh install. There's no reason to use it here. Give it a few seconds, then check it can reach the outside world: ```bash lxc exec template -- ping -c2 1.1.1.1 # Can it reach the internet by address? Tests LXD's address translation. lxc exec template -- curl -sI https://archive.ubuntu.com | head -1 # Can it reach a website by name? Tests DNS, routing, and encryption # all at once. Expect: HTTP/1.1 200 OK ``` **If either fails, stop.** Don't continue into k3s with a node that can't reach the internet, or you'll spend the afternoon reading installer errors instead. This is Part 1's firewall setup coming back. Fix it there. ### Three separate things just got proven They're separate, and passing one tells you nothing about the others: | What you saw | What it proves | |---|---| | The container has an address at all (`10.99.99.105` or similar) | It asked the host for one and got an answer. Requests to `10.99.99.1` go **to the host**, so a closed firewall would silently starve it of an address, and nothing would point at the firewall. | | `ping 1.1.1.1` works | LXD's address translation is working. This is a completely different mechanism from the one above. | | `curl` works | Name lookup, address choice, routing, and encryption all work, which is what `apt` and the k3s installer actually need. | The address will be some number from LXD's pool. Ignore the specific value; Step 6 replaces it with a fixed one per node. ### Why `curl` and not the obvious DNS command The obvious way to test name lookup is `getent hosts archive.ubuntu.com`. **Don't.** On a setup with IPv6 turned off (which is what Part 1 told you to do), it returns **nine IPv6 addresses and zero IPv4 addresses.** It looks like a disaster. It's nothing at all. `getent hosts` asks for IPv6 first and reports what the *name service* knows, ignoring what your container can actually *reach*. (`getent ahosts` shows the truth: only IPv4, because Linux notices the container has no IPv6 address and filters the IPv6 results out before you see them. Your Part 1 choice is being respected all the way down.) But **`curl -sI` is the better test regardless**, because it tests what you actually depend on: DNS, address choice, route, and encryption, in one line. `HTTP/1.1 200 OK` closes all of it at once. ### Install what the nodes need ```bash lxc exec template -- bash -c ' set -e # set -e = stop immediately if any command fails. apt-get update apt-get -y upgrade apt-get -y install curl ca-certificates iptables nfs-common open-iscsi jq netcat-openbsd systemctl enable --now iscsid ' ``` What each package is for: | Package | Why | |---|---| | `curl` | The k3s installer is downloaded with it | | `iptables` | k3s brings its own, but this lets you *look at* the firewall rules from inside a node. You'll want to, constantly. | | `nfs-common`, `open-iscsi` | Storage exercises in Part 4. One apt run now instead of five later. | | `jq` | Reads JSON output nicely. You'll thank yourself. | | `netcat-openbsd` | Tests whether a specific network port is reachable. Used in Step 7 to check the ports k3s needs. | **Expect apt to tell you it had almost nothing to do.** On Ubuntu's official image, `curl`, `ca-certificates`, `iptables`, `open-iscsi` and `jq` are already there. Only `nfs-common` is genuinely new. That's fine; the point is that the requirement is now written down and verified rather than assumed. **Expect the upgrade to install some odd things.** You'll watch a container with no screen install `plymouth` (a boot animation), a container with no hardware install `fwupd` (a firmware updater), and Linux defer building a boot image for a kernel this container doesn't have and can't boot. All of it inert. It's the shared-kernel trade-off from the other side: the image is a complete operating system, and some of it is for hardware that isn't there. On ZFS your five copies share those files and it costs nothing. On the `dir` storage option you'd have paid for the boot animation five times. **The last line is the one that matters.** `systemctl enable --now iscsid` succeeding proves that: - **systemd is running as the container's main program**, and - **services can be enabled and started normally.** That's Part 1's whole "LXD not Docker" argument, demonstrated instead of asserted. In a Docker container, that command has nothing to talk to. ### Check the firewall inside the container Ubuntu's image includes `ufw`, a firewall. It should be off, because an active firewall **inside** a node would fight Kubernetes' own rules in a way that's genuinely nasty to trace. ```bash lxc exec template -- ufw status # Expect: Status: inactive ``` **Ask ufw, not systemd.** `systemctl is-enabled ufw` says `enabled` on every Ubuntu image and tells you nothing useful. The *service* is enabled, so it runs at boot, reads its config, sees it's switched off, and exits without doing anything. Enabled service, inactive firewall. That's the shipped default. If it says `inactive`, **do nothing.** Don't disable the service, because you'd be removing something you might want in Part 5, to solve a problem you don't have. ### Check the permissions actually applied ```bash lxc exec template -- cat /proc/self/uid_map # Expect exactly: 0 0 4294967295 lxc exec template -- ls -l /dev/kmsg # Expect: a device file. kubelet needs this. lxc exec template -- stat -fc %T /sys/fs/cgroup # Expect: cgroup2fs (the modern resource-control system) lxc exec template -- ls /sys/module/br_netfilter # Expect: a folder listing. See below; this one is the point of Step 2. lxc exec template -- free -h | grep -i swap # Expect: 0B. Inherited from the host. Kubernetes checks this. ``` **Reading `uid_map`:** it says which user IDs inside map to which outside. `0 0 4294967295` means "user 0 inside = user 0 outside, for all 4 billion of them." **No translation. Root inside is root outside.** That's `security.privileged` working, and it's the bill for the 2-second boots. If you see something like `0 1000000 1000000000` instead, your profile didn't apply: check you used `-p k3s`, and remember `security.privileged` needs a **restart** to take effect if you set it after launch. **Reading `ls /sys/module/br_netfilter`:** this is Step 2 paying off, and it's worth a moment. You are looking at the **host's** kernel information, from inside the container, because they share one kernel. When k3s runs here it will check this exact folder, find it, and log "Module br_netfilter was already loaded", never attempting the `modprobe` that would have failed. Step 2 explained that from k3s's source code. This is the same thing seen from the node's side. ### Freeze it ```bash sudo zpool list default # note the ALLOC column lxc stop template # Stop it first. See below. This matters. lxc snapshot template base # Save the current state under the name "base". lxc list template # The SNAPSHOTS column should now read 1. lxc info template | sed -n '/Snapshots:/,$p' # Show the snapshot's details. sudo zpool list default # compare ALLOC ``` **Watch `ALLOC`. It should not move.** On the reference machine it read `738M` before and `738M` after. **The snapshot cost zero measurable bytes**, because ZFS copied nothing. It simply stopped throwing away the data that's already there. That's Part 1's storage argument, checked instead of claimed. And it matters for a reason that isn't about disk space: **because saving is free, you'll actually do it** before every risky thing in Parts 3 to 5. On the `dir` option this would have been ~700 MB and a real pause, which is exactly how that habit dies. **`STATEFUL NO` is correct**, and it's why you stopped it first. A "stateful" snapshot also captures the container's live memory, which only applies to a running container. You want disk state here. A frozen memory image of a half-started k3s node is a much more fragile thing to restore. **A note on the command:** use `sed -n '/Snapshots:/,$p'`, not `grep -A3`. `lxc info` prints a heading, a border, a column header, and another border, four lines in all, *before* the data. `grep -A3` shows you an empty box and hides the snapshot you just made. `template` now stays stopped forever. It is your known-good starting point. --- ## Step 6 — Make five copies ### The address plan From Part 1. Repeated here because you're about to type it: | Node | Address | Job | |---|---|---| | the host | `10.99.99.1` | not a node, this is your server | | master1 | `10.99.99.11` | starts the cluster | | master2 | `10.99.99.12` | joins it | | master3 | `10.99.99.13` | joins it | | worker1 | `10.99.99.21` | runs your programs | | worker2 | `10.99.99.22` | runs your programs | Three masters, because Kubernetes' database needs an odd number to vote on decisions. Two workers, because that's enough to see things move between them. ### The order, and why Everything here happens **while the containers are stopped.** `lxc copy` creates them stopped and they stay that way until 5.4. That's not incidental; it's what makes 5.2 and 5.3 free to get wrong and redo. | # | Do this | When | If you get it wrong | |---|---|---|---| | **5.1** | Copy the five | first | none | | **5.2** | Fixed addresses | **before first start** | Node grabs a random address and keeps it; needs a restart | | **5.3** | Memory/CPU limits | any time before start | Just run `lxc config set` again; it overwrites | | **5.4** | Start them | after 5.2 and 5.3 | none | | **5.5** | Check | after starting | none | **Only 5.2 is order-sensitive**, and only mildly. A node that starts without a fixed address asks the host for one, gets a random one from the pool, and holds onto it. Setting the fixed address afterwards doesn't move it, so you'd restart anyway. Doing it first means the node's **first** request gets the right answer. ### What needs a restart, and what doesn't This decides whether changing your mind costs 2 seconds or 20: | Change | Takes effect | |---|---| | `limits.memory`, `limits.cpu` | **Immediately.** LXD applies it live. No restart. | | Fixed address | At the container's next start | | `security.privileged`, `security.nesting` | **Restart required.** Set on a running container, it *appears* to work and doesn't. | | Profile changes | Restart required, for every container using it | ### 5.1 Copy ```bash sudo zpool list default # note ALLOC, should be ~738M for n in master1 master2 master3 worker1 worker2; do lxc copy template/base "$n" done # Copies the SNAPSHOT (template/base), not the live container. # Creates each one STOPPED. lxc list # Expect: five STOPPED rows with no addresses yet. That's correct. sudo zpool list default # compare ``` **Watch `ALLOC` again.** Five nodes from a 700 MB template should cost almost nothing, because ZFS points all five at the same data rather than duplicating it. On the reference machine: **738M → 792M. Fifty-four megabytes, for five machines.** Each node has drifted by about 11 MB (its own ID, its own logs) and shares everything else. If `ALLOC` instead jumps by ~3.5 GB, ZFS made real copies rather than pointers. Worth knowing before you build a habit on a promise that isn't holding. **This closes Part 1's storage argument.** Step 5 proved snapshots are free. This proves copies are. ### 5.2 Fixed addresses (do this before starting them) ```bash lxc config device override master1 eth0 ipv4.address=10.99.99.11 lxc config device override master2 eth0 ipv4.address=10.99.99.12 lxc config device override master3 eth0 ipv4.address=10.99.99.13 lxc config device override worker1 eth0 ipv4.address=10.99.99.21 lxc config device override worker2 eth0 ipv4.address=10.99.99.22 ``` **What `override` means here.** The network connection (`eth0`) came from the `default` profile, which all five share. `override` makes a **private copy** of that setting for one container, which you can then change. The profile itself stays untouched, which is why this is per-node config and not five separate profiles. **To change an address later, use `set`, not `override`.** The `override` command is only for the first time, because its whole job is to copy the inherited `eth0` into the container. Once that private copy exists, running `override` on it again just fails with an error that the device already exists. From then on, `lxc config device set` is what edits the copy you already made, for example `lxc config device set master1 eth0 ipv4.address=10.99.99.14`. Check before moving on: ```bash for n in master1 master2 master3 worker1 worker2; do echo -n "$n: " lxc config device get "$n" eth0 ipv4.address 2>/dev/null || echo "(none)" done ``` Expect `.11`, `.12`, `.13`, `.21`, `.22`. Any `(none)` means it didn't apply, so fix it now, not after starting. **This doesn't configure the container's network card.** It tells the **host's address service** to always give that container the same answer. The node still asks for an address at boot like normal; it just always gets the same one. That's the right place for it: nothing inside the node knows or cares, so a rebuilt node keeps its address for free. It's also why the `template`, which had no override, got a random `10.99.99.105`. ### 5.3 Memory and CPU limits (set them, but not for the obvious reason) **The obvious reason is wrong.** You'd think limits protect the host from a runaway node, and conclude you can skip them on a big machine. That conclusion is wrong, for an interesting reason. **LXD makes each container see its own limits as if they were the whole machine.** When you set `limits.memory=4GiB`, the container's own "how much RAM do I have?" answer becomes 4 GB. That's a feature called lxcfs, and it exists precisely for this. **Set no limits and every node sees the entire host.** On a 32 GB / 12-core server: ``` Node CPU Memory master1 12 32Gi master2 12 32Gi master3 12 32Gi ← every node believes it has the whole machine worker1 12 32Gi worker2 12 32Gi Kubernetes now thinks it has 60 cores and 160 GB. ``` kubelet reads exactly those numbers and reports them to Kubernetes. Kubernetes then places programs based on capacity **that does not exist**. The first real deployment runs the host out of memory, and **Linux picks what to kill**, not Kubernetes, not you. Nothing in `kubectl` will point at the cause. So limits aren't protection here. **They're how each node learns its share of one machine.** That's the shared-kernel trade-off again: on real servers or virtual machines you'd have assigned RAM at creation and never thought about it. **For a 32 GB / 12-core host:** ```bash for n in master1 master2 master3; do lxc config set "$n" limits.memory=4GiB limits.cpu=2 done for n in worker1 worker2; do lxc config set "$n" limits.memory=6GiB limits.cpu=4 done lxc config get master1 limits.memory # check one ``` That's 24 GB committed, 8 GB left for the host. - **Masters get 4 GB.** The database uses a few hundred MB at this size, so there's real room for Part 3's exercises. **Don't go below 4 GB.** You'll see 2 GB suggested elsewhere. It's tight enough that a master can be killed **by your own limit** in the middle of an experiment, and separating "the failure I was studying" from "the failure I configured" is a genuinely annoying hour. - **Workers get 6 GB** because they run your actual programs. - Scale both down together if you're near Part 1's 8 GB floor. The ratio matters more than the numbers. - **CPU deliberately over-allocates** (3×2 + 2×4 = 14 of 12). `limits.cpu` shapes what each node *reports*; containers still share the real cores. **Note the `=` signs.** `lxc config set ` works for a *single* setting, but with more arguments it switches to `key=value` form. The space-separated version fails with `Invalid key=value configuration: limits.memory`, which is the command complaining, not your container. Nothing was changed. **One caveat.** ZFS uses spare memory as a cache, up to about half your RAM by default. So 24 GB of nodes plus that cache is technically more than you have. It's fine; the cache gives memory back when something needs it. Cap `zfs_arc_max` if you'd rather it be exact. **Changed your mind?** These are settings on stopped containers; nothing is committed until they start: ```bash for n in master1 master2 master3 worker1 worker2; do lxc config unset "$n" limits.memory lxc config unset "$n" limits.cpu done ``` ### 5.4 Start them ```bash for n in master1 master2 master3 worker1 worker2; do lxc start "$n"; done sleep 10 lxc list ``` The `sleep 10` matters. `lxc list` straight after `lxc start` can show `RUNNING` with no address, because the container is up but hasn't finished asking for one. ### 5.5 Check Three separate things, and none implies the others. ```bash # 1. Running, with the addresses you chose lxc list # 2. Each node sees its own limit, not the host's lxc exec master1 -- free -h | grep -i mem # expect ~4.0Gi, NOT 32Gi lxc exec worker1 -- free -h | grep -i mem # expect ~6.0Gi lxc exec master1 -- nproc # expect 2 # 3. Each node has its own name for n in master1 master2 master3 worker1 worker2; do echo -n "$n -> "; lxc exec "$n" -- hostname done # And what the copies cost sudo zpool list default ``` **1. Addresses.** Five `RUNNING` rows on `.11` to `.13` and `.21` to `.22`. A *different* address means that node started before its override and is holding a random one: `lxc restart ` and it'll ask again. **2. Capacity.** If `free` shows 32 GB, the limit isn't being reflected and **kubelet will report your host's size as the node's size.** Fix it here. In Part 3 this shows up as Kubernetes claiming 32 GB per node and cheerfully overbooking five times over, with Linux, not Kubernetes, choosing what dies. **3. Names.** LXD updates the hostname when copying, so this usually just works. Check anyway: **k3s registers nodes by hostname**, and five nodes all called `template` is a mess you don't want to untangle from inside the database. ```bash lxc exec master1 -- hostnamectl set-hostname master1 lxc restart master1 ``` **`Error: Instance is not running`** from any of these just means you're at 5.4, not 5.5. `lxc copy` leaves containers stopped. --- ## Step 7 — Check everything, before k3s exists **Why this is the step not to skip:** everything here is cheap now and expensive in three hours. Right now, a networking problem is a networking problem. After Part 3, the exact same problem arrives disguised as a Kubernetes error, and you'll be looking in the wrong place. ```bash # --- 1. All five up, right addresses --- lxc list # --- 2. Node to node, by address --- lxc exec master1 -- ping -c2 10.99.99.12 lxc exec worker1 -- ping -c2 10.99.99.11 # Proves the virtual switch carries traffic between containers. # --- 3. Node to node, by name --- lxc exec master1 -- getent ahosts master2.lxd lxc exec master1 -- ping -c2 master3.lxd # Proves the host's name service knows your new nodes. # (Names end in .lxd, which LXD adds automatically.) # --- 4. Node to internet --- lxc exec worker2 -- ping -c2 1.1.1.1 lxc exec worker2 -- curl -sI https://get.k3s.io | head -1 # That URL is literally what the Part 3 installer downloads. # Expect: HTTP/2 200 # --- 5. Permissions, on every node, not just the template --- for n in master1 master2 master3 worker1 worker2; do echo "== $n" lxc exec "$n" -- cat /proc/self/uid_map # expect: 0 0 4294967295 lxc exec "$n" -- ls /dev/kmsg # expect: /dev/kmsg lxc exec "$n" -- stat -fc %T /sys/fs/cgroup # expect: cgroup2fs done # You've been ASSUMING the copies inherited the profile. This checks it. # --- 6. The host still has its modules and settings --- lsmod | grep -E '^(overlay|br_netfilter|vxlan|nf_conntrack)\s' sysctl fs.inotify.max_user_instances net.netfilter.nf_conntrack_max # --- 7. Disk, you're about to add five k3s installs --- df -h / sudo zpool list ``` ### Reading the results | What you see | Where the cause is | |---|---| | Node→node ping fails | LXD's firewall rules, or your own firewall wiped them. Part 1's firewall setup. **Reboot the host and retest before touching anything.** | | By address works, by name fails | The host's name service. Those requests go **to the host**, which is a different firewall path from traffic passing *through* it. Part 1's firewall setup again. | | Internet fails but node→node works | LXD's address translation is gone. Classic sign a firewall script ran while LXD was live. Reboot. | | `uid_map` shows an offset | The profile didn't apply, or the container needs a restart. | | `/dev/kmsg` missing | Check the container was launched **with** `-p k3s`, not just that the profile exists. | | `/sys/fs/cgroup` isn't `cgroup2fs` | Unusual on 24.04. Investigate before installing k3s. | **Notice what none of those mention: Kubernetes.** That's the entire value of doing this now. ### Two numbers worth reading **Node-to-node is ~0.08 ms. Internet is ~3.5 ms.** That 40× gap is what you want to see; it means your nodes really are talking directly across the virtual switch, not being sent the long way round. The database in Part 3 will be happy at those speeds. **`HTTP/2 200` from `get.k3s.io`** isn't a stand-in for the real thing. That **is** the address the Part 3 installer downloads from, tested from a real worker node. ### One thing ping does *not* prove The pings above show the nodes can reach each other. But `ping` is a different kind of traffic from what k3s actually uses. A firewall can allow ping and still block the ports k3s needs, and that failure shows up in Part 3 disguised as "master2 won't join the cluster," which is a much worse place to discover it. So here are the exact ports k3s will need, from the official requirements. **If you have no custom firewall, these are all open already and you can just read this as a preview.** If you have your own firewall rules (Part 1's `nftables.conf`), check them against this list *now*. | Port | Type | Between | What it's for | Testable now? | |---|---|---|---|---| | 6443 | TCP | workers → masters | The main Kubernetes control channel | Yes | | 2379, 2380 | TCP | master ↔ master | The shared database (etcd). **Only the three masters.** | Yes | | 10250 | TCP | all nodes | Node statistics | Yes | | 8472 | UDP | all nodes | flannel's tunnel, carries all traffic between programs on different nodes. This is what `vxlan` is for. | No, UDP, see below | **You can test this now**, before k3s exists. The trick: you don't even need something listening. An open path and a blocked path behave *differently*, and this command reads that difference for you and prints a plain verdict: ```bash lxc exec master1 -- bash -c ' nc -zv -w3 10.99.99.12 2379 2>&1 | grep -q -e succeeded -e refused \ && echo "PASS: path to master2:2379 is open" \ || echo "FAIL: path blocked (timed out), check firewall"' ``` Expect **PASS**. **Why "refused" counts as a pass.** Left to itself, `nc` prints `Connection refused` here and calls it a "failure", which is misleading. Nothing is listening on 2379 yet, because k3s isn't installed. But the packet still **reached master2 and got an instant answer back**, which is the only thing this test needs to prove: the network carries traffic between masters on the etcd port. The wrapper above treats both "succeeded" (something listening) and "refused" (nothing listening, but reachable) as PASS, because both mean the path is open. **The one real failure is a timeout.** If master1's packet leaves and nothing *ever* comes back, `nc` waits the full 3 seconds (`-w3`) and reports a timeout, and *that's* the firewall symptom. Only that prints FAIL. If you have a custom firewall (Part 1's `nftables.conf`), this is where it would show up. Fix it before Part 3. The distinction is entirely about **speed**: refused comes back instantly (open), timed-out takes 3 seconds (blocked). (`nc` is netcat, a network-testing tool. `-z` = test only, `-v` = explain, `-w3` = give up after 3 seconds.) To check every master-pair k3s needs, in one go: ```bash for pair in "master1 10.99.99.12" "master1 10.99.99.13" "master2 10.99.99.13"; do set -- $pair lxc exec "$1" -- bash -c " nc -zv -w3 $2 2379 2>&1 | grep -q -e succeeded -e refused \ && echo 'PASS: $1 -> $2:2379 (etcd)' \ || echo 'FAIL: $1 -> $2:2379 blocked'" done ``` Three PASS lines means the etcd network is ready for Part 3. **The other ports k3s needs.** The etcd test above is the meaningful one; every port below crosses the *same* virtual switch with the *same* firewall treatment, so if 2379 is open, these almost certainly are too. But the docs list them, so here they are tested rather than assumed: ```bash # Port 6443 (TCP), workers reach the masters' control channel: lxc exec worker1 -- bash -c ' nc -zv -w3 10.99.99.11 6443 2>&1 | grep -q -e succeeded -e refused \ && echo "PASS: worker1 -> master1:6443 (API) open" \ || echo "FAIL: blocked, check firewall"' # Port 10250 (TCP), node metrics, between all nodes: lxc exec worker1 -- bash -c ' nc -zv -w3 10.99.99.11 10250 2>&1 | grep -q -e succeeded -e refused \ && echo "PASS: worker1 -> master1:10250 (metrics) open" \ || echo "FAIL: blocked, check firewall"' ``` Both should PASS instantly, for the same reason as the etcd test: nothing's listening yet, so you get a fast "refused," which means the path is open. **Port 8472 (UDP), flannel's tunnel, can't be tested honestly this way, and it's worth understanding why.** TCP either completes a handshake or gets refused, a clear yes or no. UDP is "send and forget": nothing sends a reply, so a UDP test reports success just because the packet *left*, whether or not anything received it. So this tells you almost nothing: ```bash # Informational only. A PASS here does NOT prove the packet arrived: lxc exec worker1 -- nc -zvu -w3 10.99.99.11 8472 ``` The honest test for 8472 is Part 3 itself: **if programs on different nodes can talk to each other, flannel's tunnel is working.** There's nothing listening on 8472 before k3s exists to confirm against, so don't chase this one now. Just know it's the port that carries all cross-node traffic, and that Part 3's first cross-node test is what actually verifies it. **Two important addresses k3s claims for itself**, which is why Part 1 chose `10.99.99.0/24` and not something in these ranges: - `10.42.0.0/16`, where k3s puts your running programs - `10.43.0.0/16`, where k3s puts its internal service addresses Your nodes are on `10.99.99.x`, well clear of both. If you picked your own range in Part 1, confirm it doesn't overlap these two, because an overlap causes networking failures that look nothing like their cause. **On memory, for reference:** k3s's official minimum is 2 GB for a master and 512 MB for a worker. Your 4 GB / 6 GB from Step 6 sits comfortably above that. If you're tight on RAM, the real floor is Part 1's 8 GB total, not k3s's per-node numbers, so you can shrink the limits toward the official minimums, keeping masters larger than workers. --- ## Step 8 — Save a restore point You now have five clean, prepared, connected machines and no k3s. **That is a state worth getting back to in one second.** ### Stop them first A snapshot of a **running** container captures the disk mid-write. It is exactly like pulling the power cord; restoring it gives you a machine that boots as though it crashed. Right now nothing important is happening, so you'd probably get away with it. **"Probably" is worthless in the thing you fall back to when you're already confused about something else.** Stopped snapshots are exact. It costs twenty seconds. ```bash # Check what already exists (should be nothing on the five) for n in master1 master2 master3 worker1 worker2 template; do echo "== $n"; lxc info "$n" | sed -n '/Snapshots:/,$p' done # Blank output for a node = it has no snapshots. lxc info leaves the # section out entirely when there are none. That's the answer, not an error. # Stop, save, start for n in master1 master2 master3 worker1 worker2; do lxc stop "$n"; done for n in master1 master2 master3 worker1 worker2; do lxc snapshot "$n" pre-k3s; done for n in master1 master2 master3 worker1 worker2; do lxc start "$n"; done sleep 10 lxc list # SNAPSHOTS column: 1 on all five sudo zpool list default # ALLOC barely moves ``` `lxc snapshot` **refuses** to overwrite an existing name rather than silently replacing it, so re-running this is safe. Worst case, an error. This is the same distinction as `template/base` in Step 5, which read `STATEFUL NO` because you stopped it first. It matters more as you go: a mid-write snapshot of a **running database member** is far less trustworthy than a stopped one, and Part 3 is entirely about database members. ### The habit this is for ```bash lxc snapshot master1 before-i-wreck-it # ...break something on purpose... lxc restore master1 before-i-wreck-it # Back to normal in about a second. ``` **That reflex is the whole reason Part 1 chose ZFS.** Saving costs nothing and takes no time, so you'll actually do it, which means you'll experiment freely, because undoing a mistake is trivial. If you used `dir` or a loop file instead, snapshots are slower and bigger. **Take `pre-k3s` anyway.** It's the one that matters most. --- ## Where you should be | Thing | State | |---|---| | Kernel modules | Six switched on, saved in `/etc/modules-load.d/k3s.conf`, survive reboot | | Kernel settings | Set, saved in `/etc/sysctl.d/99-k3s-lxd.conf`, checked after a reboot | | `k3s` profile | Exists, stacks on `default`, permissions understood and accepted | | `template` | Stopped, saved as `base`, ready to rebuild from | | Nodes | Five running: `master1-3` on `.11-.13`, `worker1-2` on `.21-.22` | | Networking | Node↔node by address and by name, node→internet, all proven | | Capacity | Each node reports its own limit, not the host's | | Snapshots | `pre-k3s` on all five | | k3s | Not installed. That's next. | ### Things worth carrying forward - **The template isn't just a shortcut. It's a comparison tool.** When a node misbehaves in Part 3, one command gives you a known-good node to compare against. - **`security.privileged` and profile changes need a restart.** Set on a running container, they look like they worked. - **The inotify limit will catch you eventually if you skipped Step 3.** Not at install. At node three, or the fourth deployment, looking like anything except a host setting. - **Your firewall and LXD are still in tension.** Part 1's rule stands: never run a firewall script that flushes rules on a host with LXD running, without rebooting afterwards. It's now five nodes that go quiet instead of zero. - **Nothing in this guide made your nodes secure.** They're deliberately not a security boundary. That's the right call for a lab on your own machine, and the wrong call everywhere else. ### What you've built, and what's next That is Part 2 done. You now have five containers that look and behave like five separate servers: the kernel modules are loaded, the kernel settings are pinned, the privileged `k3s` profile is applied, each node has a fixed address and its own memory and CPU limits, and every node has a `pre-k3s` snapshot to fall back to. None of it is Kubernetes yet, but everything Kubernetes will reach for is already in place. Part 3 turns these machines into one cluster. You start it on `master1`, then join `master2` and `master3` so that all three masters share the database and vote on decisions, and join `worker1` and `worker2` to run your programs. After that you deliberately stop a master and watch what that voting actually protects you from. The `pre-k3s` snapshots from Step 8 are what let you do that over and over, instead of just once. Continue with [**Part 3, Installing k3s**](/posts/building-a-kubernetes-lab-with-k3s-and-lxd-part-3-installing-k3s/). ### [Building a Kubernetes Lab with k3s and LXD Part 1 Preparing the Ubuntu Host](https://linsnotes.com/posts/building-a-kubernetes-lab-with-k3s-and-lxd-part-1-preparing-the-ubuntu-host/) > Prepare a single Ubuntu server to run k3s inside LXD containers. Remove Docker, disable swap, handle the nftables firewall, and install LXD with ZFS storage. Most Kubernetes tutorials hand you a cluster that someone else already set up. You learn how to *use* it, which is useful, but you never build one from scratch, and you never get to break it and watch it recover. That last part is where the real learning is. Building a cluster, breaking it, and seeing how it picks itself back up teaches you far more than reading about it ever will. So why doesn't everyone learn this way? Usually it comes down to hardware. A proper multi-node cluster needs several servers, and that means either buying machines or renting cloud instances that charge you for every hour they're running. **This lab gets around all of that.** All you need is one computer with Ubuntu on it, and that's easy to set up on an old PC or a spare laptop. That single machine becomes your *host*. On top of it you run five small LXD containers, and each one plays the part of a separate server: three for the control plane and two as workers. Because all five share the host's kernel, they barely use any resources and start in about two seconds, so you can happily run all five at once, and tear any of them down and rebuild it whenever you feel like it. And it genuinely behaves like the real thing. Every node has its own hostname and IP, the three control-plane nodes form a real etcd quorum, and when one goes down, it goes down for real. All of that from a single machine, with no second computer and no cloud bill. > **Goal of this part:** get one Ubuntu machine ready to run five LXD containers as k3s nodes (3 masters, 2 workers). > > **Scope:** we don't install k3s yet. This part only gets the host ready, so that container networking and Kubernetes both work later. Skip it, and you'll spend a later afternoon chasing Kubernetes errors that really come from a host setting you changed months ago. {: .prompt-info } **Before we go further, a quick word about the firewall.** In this lab it matters more than you might expect, so it's worth a moment now. Your containers talk to each other, and out to the internet, by passing through the host. If the host's firewall blocks that traffic, nothing points at the firewall as the cause. Instead your nodes can't find each other, the cluster never comes together, and downloads inside a node just hang. It all looks like Kubernetes is broken, when the real culprit is a single firewall setting. That's why it's worth getting right before k3s is even in the picture. > **Good news:** if you've never touched your firewall, there's nothing to do here. A fresh Ubuntu install lets everything through by default, so your containers just work. The firewall steps later in this guide are only for people who've deliberately locked their firewall down. {: .prompt-tip } This guide uses **nftables**, the firewall built into Ubuntu 22.04 and 24.04. One thing to know: the old `iptables` command still exists, but these days it just writes to nftables behind the scenes. So `iptables` and `nft` aren't two firewalls, they're the same one with two names. To keep things clear, everything here uses `nft`. ## Hardware **Reference machine** — what this guide was written against: | | | |---|---| | OS | Ubuntu 24.04 LTS | | RAM | 32 GB | | CPU | 12 cores | | Disk | 1 TB, LVM, with unallocated space in the volume group | **Requirements for the 5-node lab** (3 masters + 2 workers): | | Requirement | Why | |---|---|---| | OS | **Ubuntu 24.04 LTS** | What this guide is built and tested on | | RAM | **8 GB** | ~1.5 GB per node | | CPU | **4 cores** | 3 etcd members plus workloads | | Disk | **At least 50 GB** of unallocated spacein your LVM volume group(this becomes the LXD storage pool),plus ~20 GB free for the host | Images, k3s, snapshots, monitoring | **This guide assumes you have at least 50 GB of unallocated space in your volume group** and uses ZFS on a dedicated logical volume. Step 1 checks that. If your volume group is full, see the note at the end of Step 6. --- ## Step 1 — Survey the host Every command here is read-only. Nothing changes yet. The point is to find out what you're working with before making decisions that depend on it. ```bash # Capacity lsb_release -a free -h nproc df -h / # Existing network layout ip -brief addr ip route # Existing container stack docker ps 2>/dev/null docker system df 2>/dev/null # The entire packet filter, in one place sudo nft list ruleset # Swap swapon --show grep -i swap /etc/fstab # Real disk capacity (not just what's mounted) sudo vgs sudo lvs ``` ### What you're looking for | Command | The question | Why it matters | |---|---|---| | `free -h` | At least 8 GB, for fivenodes at ~1.5 GB each? | 8 GB is therequired minimum | | `df -h /` | At least ~20 GB freeon the host? | 20 GB is the host minimum;later parts want more(images, monitoring) | | `ip route` | Which subnets arealready claimed? | Your bridge must not collidewith any. This guide uses`10.99.99.0/24` | | `docker ps` | Anything runningyou must not break? | Docker is removed in Step 2,so on a fresh Ubuntu installthere's nothing to do here | | `nft list ruleset` | Is forwardedtraffic allowed? | Forwarded traffic must beallowed; bridged containertraffic hits the forward hook | | `swapon --show` | Will kubeletrefuse to start? | It will, if swap is on | | `vgs` / `lvs` | At least 50 GB unallocatedin the volume group? | 50 GB is the required minimum;installers often carve outfar less than the disk holds | **Reading `nft list ruleset`.** Empty output means nothing is being filtered, which is Ubuntu's default and exactly what you want. Nothing to do here. If the output is *not* empty, you've set up a firewall at some point. The easy path is to reset nftables back to its default of allowing everything. If you'd rather keep your firewall, that's fine, but make sure it allows the traffic this lab depends on: - **All traffic in and out of the LXD bridge `lxdbr0`** (it gets created later). This is what carries traffic between the nodes and out to the internet. - **DHCP and DNS from the nodes to the host.** Without these, a node can't get an IP address or resolve names. Steps 4 and 8 come back to the firewall once the bridge exists. --- ## Step 2 — Remove Docker (if present) ### Why **Removing Docker isn't about a conflict with k3s.** The two actually get along fine. k3s brings its own containerd on its own socket and ignores whatever else is installed, so they run side by side without any trouble. The real reasons are more specific, and there are three of them. The first is about networking. Every time the Docker daemon starts, it sets the firewall's forward policy to *drop*. Traffic between containers on a bridge is forwarded traffic, so it runs straight into that policy. Docker then adds its own allow rules to get around the block it just created, and LXD adds its own on top. Most of the time this works. When it doesn't, you get a networking failure that is genuinely hard to trace. The second reason matters more, and it's about keeping problems easy to diagnose. Things will break during this lab. That's the whole point. When two nodes can't reach each other, you want only one thing that could be at fault. With Docker gone, the cause is either LXD or something you did yourself. Leave Docker running, even when it's idle, and every networking issue turns into a three-way guessing game that costs you an hour you didn't need to spend. The third reason is specific to nftables. When LXD starts, it decides which firewall system to use by looking for existing iptables rules, and Docker's rules look exactly like those. So if Docker is still installed, LXD can quietly fall back to the older firewall backend, which is the one this guide is trying to avoid. Step 8 checks which backend you ended up with. So remove it. You can always put it back once the lab is stable, and by then you might well prefer to move those workloads into k3s anyway. ### Check whether Docker is installed A couple of quick checks tell you whether Docker is on the machine at all: ```bash which docker && docker --version snap list docker 2>/dev/null ``` If both come back empty, Docker isn't installed and you can skip the rest of this step. If either one shows something, Docker is here and needs to go. ### Remove it completely If you're running Docker, you already know your way around it, so this part is light on step-by-step. Do three things, in this order: 1. **Back up any container data you want to keep.** Once Docker is removed, its volumes go with it and there is no undo. Save anything that matters before you start. 2. **Remove Docker entirely.** Not just the packages, but its images, volumes, config, the apt repository it added, and any leftover network bridges. A half-removed Docker is worse than none, because the leftovers are exactly what send you chasing ghosts later. 3. **Clear the firewall rules it left behind.** Docker rewrites its own rules on every start, including the forward policy set to `drop`. The cleanest way to wipe them is to reboot once the packages are gone, since nothing puts them back. ### Final state When Docker is truly gone, all of these should hold. This is the clean starting point the rest of the guide assumes: ```bash which docker || echo "docker: gone" dpkg -l | grep -i -E 'docker|containerd' || echo "packages: gone" sudo nft list ruleset ip -brief addr df -h / ``` - The `docker` command is gone, and no `docker` or `containerd` packages remain. - `nft list ruleset` shows no Docker tables or chains, and nothing setting the forward policy to `drop`. - There is no `docker0`, and no leftover `br-*` or `veth*` interfaces. - `df -h /` shows noticeably more free space than before. --- ## Step 3 — Turn off swap ### Why **Kubernetes won't start while swap is on.** The kubelet checks for it and refuses to run. This is a hard error by default, not a warning you can wave away. Here's the reasoning. Kubernetes assumes that memory limits mean something real. When you tell it a pod may use 512 MB, the scheduler packs pods onto nodes based on that number, and the kubelet steps in to evict pods when a node runs low on memory. Swap quietly breaks all three of these. A pod that goes over its limit should be killed, but with swap on it just spills to disk and keeps crawling along about a hundred times slower. The node reports free memory that isn't really there. And eviction never kicks in, because the node never looks like it is under pressure. Nothing crashes. The cluster just gets slower and slower for no reason you can see, and that silent, confusing failure is exactly why Kubernetes refuses to start instead. **You only turn swap off once, on the host, not inside each container.** LXD containers share the host's kernel, and swap is a kernel-level facility, so there is no separate swap inside a container. When the kubelet in a node checks whether swap is on, it is really reading the host's setting. Turn swap off on the host and all five nodes see it as off. (Kubernetes 1.28 and later has a beta feature called NodeSwap that allows swap in certain cases. It is off by default, needs cgroup v2, and adds complexity for no real gain in a lab, so we leave it alone.) ### Turn it off First, find out what your swap actually is, a file or a partition: ```bash swapon --show grep -i swap /etc/fstab ``` On Ubuntu it is almost always a swap **file**. If so: ```bash sudo swapoff /swap.img sudo sed -i '/swap.img/ s/^/#/' /etc/fstab sudo rm /swap.img ``` If yours is a swap **partition** instead, use its device name and skip the `rm`: ```bash sudo swapoff /dev/ sudo sed -i '/\sswap\s/ s/^/#/' /etc/fstab ``` Here is what those commands do: - `swapoff` turns swap off in the running system right away. On its own, though, it only lasts until the next reboot. - The `sed` line is what makes it stick. It finds the swap line in `/etc/fstab` and puts a `#` in front of it, so the line is commented out and swap does not come back at boot. Commenting instead of deleting means you can undo it later by removing that one `#`. - `rm` deletes the swap file itself, which frees the disk space it was using (often several gigabytes). Swap stays off for this lab, so there is no reason to keep the file around. ### Check it worked, before you reboot ```bash grep -i swap /etc/fstab # the swap line should now start with # swapon --show # should print nothing free -h # Swap should read 0B ``` Do this now, while a mistake is easy to fix. If the `sed` edit went wrong, this is where you catch it. Miss it, and swap quietly comes back at the next boot, and then k3s fails for a reason you were sure you had already dealt with. If you ever want swap back, recreate the file and turn it on with `sudo fallocate -l 8G /swap.img && sudo mkswap /swap.img && sudo swapon /swap.img`, then uncomment the line in `/etc/fstab` so it survives a reboot. --- ## Step 4 — Review your firewall This step is only for people who have deliberately set up a firewall on the host. If you've never touched yours, skip ahead to Step 5. A fresh Ubuntu install allows everything by default, so LXD will just work. If you did set one up, deal with it now, before LXD goes in. For a lab, the simplest and safest choice is to reset your firewall back to its default of allowing everything. This opens every port again, SSH on port 22 included, so you won't lock yourself out of the machine. With nothing filtering, nothing can quietly block your nodes. If you'd rather keep your firewall, that's fine, but it needs to allow the traffic this lab depends on: - All traffic in and out of the LXD bridge `lxdbr0` (it gets created in Step 7). This carries traffic between the nodes and out to the internet. - DHCP and DNS from the nodes to the host, so each node can get an IP address and resolve names. You set the firewall up, so you already know how to open those. This guide won't walk through your specific ruleset. Either way, check where you're starting from and save a copy for later: ```bash sudo nft list ruleset | tee ~/nft-before-lxd.txt ``` Empty output means nothing is filtering, which is the clean state you want. If it shows rules, sort them out now, then run this again so the saved copy reflects your final state. Step 8 uses this file to show you exactly what LXD adds once the bridge exists. --- ## Step 5 — Install LXD ### Why LXD and not something else k3s needs to run on something that looks like a real machine. It expects `systemd` running as its first process, its own network, its own filesystem, and its own hostname. That rules out some options straight away and turns the rest into a tradeoff. | | What it gives you | Boot | RAM per node | Verdict for this lab | |---|---|---|---|---| | **Docker** | one process, no init | instant | tiny | **No.** The first process is your app, not systemd, so k3s can't run here. | | **VirtualBox** | full VM, own kernel | 30s+ | 2 GB+ | Works, but heavy and awkward to script. | | **Multipass** | full VM, own kernel | 20s+ | 2 GB+ | Works, but still a full kernel per node. | | **LXD** | full OS, shared kernel | ~2s | ~1.5 GB | **Yes.** Machine-like, cheap, and instant. | Docker is out because a container there runs a single process with no init system, which is not the machine-like environment k3s needs. VirtualBox and Multipass both work, but each node is a full virtual machine with its own kernel, so you are looking at 2 GB or more per node and 20 to 30 seconds just to start one. LXD is the sweet spot. Because all the containers share the host's kernel, a node uses about 1.5 GB and starts in around two seconds. That speed changes how you use the lab. When you kill a master to watch etcd lose its quorum, `lxc stop master1` is instant and `lxc start master1` brings it back before you have finished reading the error. You will do that fifty times without thinking about it. With real VMs you would do it five times and give up. There is one real downside, and it is worth being honest about. Sharing the kernel means k3s needs extra privileges that a normal container does not get, so these containers are **not** a security boundary. That is fine on your own machine, and it is exactly why you would never do this on shared or production infrastructure. A VM would give you real isolation; LXD gives you speed, and for a lab that is the better trade. ### Install LXD On Ubuntu, LXD comes as a snap, and one command installs it: ```bash sudo snap install lxd ``` `sudo` is required, because installing a snap changes the system. On Ubuntu 24.04 this gives you the current, supported version of LXD, which is exactly what you want. There is nothing else to pick. One thing you might notice: `which lxd` can point at `/usr/sbin/lxd`. That is just a small helper script Ubuntu ships, not LXD itself. The real LXD is the snap you just installed, so you can ignore the helper. ### Check it installed ```bash snap list lxd lxc list ip -brief addr ``` - `snap list lxd` should show the snap and its version. - `lxc list` should print an **empty table**. That alone proves the LXD daemon is running and you can reach it. If you get a permission error on the socket instead, add yourself to the `lxd` group with `sudo usermod -aG lxd $USER`, then run `newgrp lxd`. You can also just put `sudo` in front of your `lxc` commands. - `ip -brief addr` should show **no `lxdbr0` yet**. Installed is not the same as set up, and you configure the network in Step 7. If a bridge is already there, something set LXD up before you did, so look into that before moving on. --- ## Step 6 — Decide, and prepare storage ### What `lxd init` is for Installing LXD gave you the software, but the daemon still has nothing set up. It has no storage to put containers on and no network to attach them to. `lxd init` is a one-time setup wizard that fills both in, a lot like `git init`. You run it once, answer a short list of questions, and LXD saves the result. Until you do, `lxc launch` has nowhere to put a container, so it just fails. You will actually run the wizard in Step 7. It asks fifteen questions, and nearly all of them have a sensible default you can accept. Only two are worth deciding up front: the **network** your containers sit on, and the **storage** they run from. This step settles both. ### Decision 1: the container subnet Your five nodes need a small private network of their own. You pick the range, and the one real rule is that it must not overlap a network you already use. Steer clear of two things: - **Your own LAN.** If the container range overlaps your real home or office network, traffic meant for those addresses goes to the containers instead and quietly vanishes, and the cause looks nothing like a container problem. Check what you already use with `ip route` and `ip -brief addr`, and check your router too, since a subnet you reach through the gateway won't show up in `ip route`. - **k3s's own ranges**, which are `10.42.0.0/16` for pods and `10.43.0.0/16` for services. `10.99.99.0/24` sits well clear of all of that, so that's what this guide uses. Give each node a fixed address from it: | Node | Address | |---|---| | bridge / gateway | `10.99.99.1` | | master1 | `10.99.99.11` | | master2 | `10.99.99.12` | | master3 | `10.99.99.13` | | worker1 | `10.99.99.21` | | worker2 | `10.99.99.22` | ### Decision 2: ZFS storage For storage, use **ZFS**. The whole reason is snapshots. On ZFS, saving a container and restoring it later takes about a second and almost no disk, because ZFS points at the existing data instead of copying it. That matters here, because later parts of this lab are all about breaking things on purpose and rolling them back: ```bash # 1. Save master1's current state. The snapshot name is yours to choose; # "before-i-wreck-it" just describes what it is for. lxc snapshot master1 before-i-wreck-it # 2. Now break something on purpose: kill etcd, corrupt a config, anything. # 3. Roll master1 back to exactly how it was when you took the snapshot. lxc restore master1 before-i-wreck-it ``` When that costs a second, you take a snapshot before every risky step and never think about it. So use ZFS if you possibly can. ZFS wants its own space to manage, and the cleanest way to give it that is a dedicated logical volume. All it needs is **50 GB of unallocated space in your LVM volume group**, which on any normal disk in 2026 is easy to find. Ubuntu's installer usually leaves most of the disk unallocated in the volume group anyway, which is exactly what the `vgs` check back in Step 1 was for. Create the volume now, before `lxd init`, so the wizard can point straight at it. > **Your volume group is almost certainly named `ubuntu-vg`.** When you install Ubuntu with its default LVM layout, the installer creates one volume group and calls it `ubuntu-vg`, and every command below assumes that name. Confirm it with `sudo vgs` and look at the `VG` column. If yours shows a different name, use that in place of `ubuntu-vg` from here on. {: .prompt-tip } ```bash sudo vgs # confirm the volume group name and free space sudo lvcreate -L 50G -n lxd-zfs ubuntu-vg # carve a 50 GB volume for ZFS out of it ``` That gives you a fresh, empty block device at `/dev/ubuntu-vg/lxd-zfs`. Nothing is formatted or resized, and your system disk is untouched. You are only claiming unused space, and you can hand it back any time with `sudo lvremove /dev/ubuntu-vg/lxd-zfs`. Step 7 installs the ZFS tools and runs `lxd init`, which formats and manages this device for you. 50 GB is enough for the whole series. If you have plenty of room, feel free to give it more, say 100 or 200 GB, which leaves the storage and monitoring parts in later chapters a bit more breathing space. > **If you can't spare the space for ZFS:** LXD's `dir` backend stores each container as a plain folder and needs no dedicated volume, so the lab still runs. The only thing you give up is fast, near-free snapshots, since with `dir` each one is a full copy that takes real time and disk. If you go this way, choose `dir` instead of `zfs` when `lxd init` asks, and skip the volume above. {: .prompt-info } --- ## Step 7 — Run `lxd init` ### Run it > **These answers assume you chose ZFS in Step 6**, which is the path this guide uses. If you picked `dir` instead, it is simpler: skip the `zfsutils-linux` install just below, answer `dir` at the storage-backend question, and the three ZFS-only questions that follow it (create a ZFS pool, use an existing block device, and the path to it) never appear. The network answers are identical either way. {: .prompt-info } On ZFS, install the ZFS tools first, or `lxd init` won't offer the backend: ```bash sudo apt -y install zfsutils-linux ``` Then: ```bash sudo lxd init ``` Answers matching the decisions above: ``` Would you like to use LXD clustering? no Do you want to configure a new storage pool? yes Name of the new storage pool default Name of the storage backend to use zfs (or dir) Create a new ZFS pool? yes Would you like to use an existing empty block device? yes (if you made an LV) Path to the existing block device /dev/ubuntu-vg/lxd-zfs Would you like to connect to a MAAS server? no Would you like to create a new local network bridge? yes What should the new bridge be called? lxdbr0 What IPv4 address should be used? 10.99.99.1/24 Would you like LXD to NAT IPv4 traffic? yes What IPv6 address should be used? none Would you like the LXD server to be available over network? no Would you like stale cached images to be updated? yes ``` If you have no spare block device, answer `no` to the block-device question and give a size for the loop file instead (50GiB is comfortable). ### What those answers actually configured **The bridge** — `lxdbr0` is a software Ethernet switch. Everything plugged into it reaches everything else at layer 2, as if patched into the same physical switch. **The IPv4 address** does two things at once: 1. Gives the bridge **the host's address on that network** (`10.99.99.1`). Your host is a participant, not just a switch. 2. Defines the subnet. It also implicitly starts **dnsmasq** on the host, bound to the bridge, serving DHCP and DNS. That's why `master2.lxd` will resolve later. **This is the address at the centre of the firewall question.** `10.99.99.1` is *on the host*, so DHCP and DNS requests from your nodes are addressed to the host — they traverse the **input** hook, not forward. The forward hook only sees traffic passing *through* the host to somewhere else. If your input policy is `drop` with no rule for the bridge, containers get no IP and no DNS, and nothing points at the firewall. **NAT** — LXD adds a masquerade rule rewriting `10.99.99.x` to your host's real address on the way out. Same trick your router does for your LAN. Without it: no `apt`, no k3s installer, no images. **The storage pool** — where container root disks live. **The default profile** — `lxd init` also edits the `default` profile so every container automatically gets `eth0` on the bridge and a root disk from the pool. That's why `lxc launch ubuntu:24.04 master1` will work later with no extra flags. ### Verify ```bash lxc network list lxc storage list ip -brief addr show lxdbr0 ip route ``` `lxdbr0` shows your chosen address. `ip route` shows `10.99.99.0/24 dev lxdbr0` and **no collision** with anything you use. Confirm you can still reach whatever you were reaching before. ### What exists now ``` your host ├── enp5s0 ← your real network └── lxdbr0 10.99.99.1 ← new: virtual switch + host's address on it ← dnsmasq here for DHCP/DNS ← NAT: 10.99.99.0/24 → out via enp5s0 ``` Plus an empty storage pool and a default profile. **No containers yet** — this built the room, not the machines. --- ## Step 8 — Confirm the firewall isn't in the way `lxdbr0` now exists, so two quick checks confirm the firewall will not quietly block your nodes. For most people this is a formality, but do it now anyway: a firewall problem is obvious here, while the exact same problem three hours into Kubernetes just looks like Kubernetes is broken. ### LXD is using nftables ```bash lxc info | grep -i -A2 firewall ``` You want to see `firewall: nftables`. On Ubuntu 24.04 with Docker removed (Step 2), that is what you get: LXD found no old iptables rules and picked the modern backend on its own. If it says `firewall: xtables` instead, something is still leaving iptables-style rules around, usually an active `ufw` or a Docker leftover that Step 2 missed. Clear it, then run `sudo systemctl restart snap.lxd.daemon` so LXD chooses again. ### What LXD set up for you ```bash sudo nft list table inet lxd ``` LXD creates its own `table inet lxd` that lets your nodes pull an address and DNS from the host, allows traffic in and out of the bridge, and NATs it on the way to the internet. In other words, it opens exactly what the nodes need without you writing a single rule. (If you saved a baseline in Step 4, `sudo diff ~/nft-before-lxd.txt <(sudo nft list ruleset)` shows the same thing as a clean before-and-after.) ### If you kept your own firewall If you never set up a firewall, you are finished: LXD's table is the only thing filtering the bridge, and it already allows everything the nodes need. If you did keep a firewall, the rules you added back in Step 4 are what let the bridge traffic through, so there is nothing new to add here. Just watch out for one trap: > **Do not reload a `flush`-based nftables config while LXD is running.** Most `/etc/nftables.conf` files begin with `flush ruleset`, which clears *every* table, LXD's `table inet lxd` included, and LXD will not rebuild it until it restarts. After editing that file, reboot (or run `sudo systemctl restart snap.lxd.daemon`) instead of just reloading, so LXD's table comes back. {: .prompt-warning } --- ## Where you should be | | | |---|---| | Docker | removed entirely, including its nftables tables | | Swap | off, and won't return at boot | | Firewall | nftables only; dead rules closed; `lxc info` reports the nftables backend; `table inet lxd` present alongside your own rules for `lxdbr0` | | LXD | installed, `lxc list` returns an empty table | | `lxdbr0` | exists, on a subnet that collides with nothing | | Storage pool | exists | | Containers | none yet — that's next | **Next:** kernel modules and sysctls for k3s-in-LXD, the privileged profile, and launching the five nodes. Continue with [**Part 2, Building the Five Nodes**](/posts/building-a-kubernetes-lab-with-k3s-and-lxd-part-2-building-the-five-nodes/). ### [Welcome to My Blog](https://linsnotes.com/posts/welcome-to-my-blog/) > A short introduction to who I am and what you'll find here — notes on AI, self-hosting, and whatever I'm learning next. Hi, and welcome. I'm Kuibin Lin — an AI engineer who spent seven years teaching Chinese before finding my way into technology, mostly through self-study. ## What this blog is I originally started this site to learn Jekyll and GitHub Pages. It has grown into a place where I document what I'm learning, write up the projects I build, and explain the things I wish someone had spelled out for me more simply. You'll find posts on: - **AI & machine learning** — the concepts I'm working through, explained plainly - **Self-hosting & homelab** — Raspberry Pi, Docker, Proxmox, VPNs, and running my own services - **Web & code** — Python, Jekyll, and small tools I've built for this site - **Notes to my future self** — setup guides I keep coming back to ## How to get around - Browse by [**Categories**](/categories/) or [**Tags**](/tags/) to find a topic - Skim the [**Archives**](/archives/) for everything in date order - Try the [**Apps**](/projects/) — small web tools I've built - The [**About**](/about/) page has the longer version of my story ## Say hi Links to my GitHub, LinkedIn, and email are in the sidebar. If something here helps you, or you're tinkering with something similar, I'd love to hear from you. Thanks for stopping by. 👋 ### [How to Add Newsletter Subscribe Feature to Jekyll Chirpy with Brevo](https://linsnotes.com/posts/how-to-add-newsletter-subscribe-feature-to-jekyll-chirpy-with-brevo/) > A complete guide to adding a newsletter subscribe feature to a Jekyll Chirpy blog, with a sidebar button, a dedicated subscribe page, and a Brevo email backend. If you publish regularly, a newsletter gives you something that search engines and social platforms cannot: a direct line to readers who actually want to hear from you. Search rankings shift. Social feeds prune. Algorithms decide who sees what, and they change without warning. Email is the one channel where, if someone subscribes, you reach them. The list compounds over time. A subscriber from a year ago still receives your next post. Three reasons make the setup work worth doing: - **Algorithm independence.** The audience is yours, not rented from a platform. If a search update changes your traffic next quarter, the newsletter list is unaffected. - **Reach the readers who care most.** Most visitors arrive via search, read one post, and leave. The few who subscribe are the ones who want the next post too. Sending it to them costs a few minutes per issue. - **A real engagement signal.** Page views are noisy. Open rates and click rates tell you what content is landing, which is useful for deciding what to write next. The setup below adds this channel to your Chirpy site without locking you into any one provider. Brevo handles the email side. The front end stays under your control. ## What we are building Two pieces working together: 1. A **SUBSCRIBE button** in the sidebar nav. It sits as the last item, after all your tabs, with an icon and a coloured label that matches Chirpy's font and alignment. Clicking it routes to a dedicated page. 2. A **`/subscribe/` page** that hosts the full Brevo signup form. The form includes the email field, GDPR consent checkbox, captcha widget, and submit button. When a visitor signs up, Brevo handles double opt in, contact storage, and any campaign sends you trigger later. Brevo is the email service backend. It is not the focus of this post. You could swap in Mailchimp or ConvertKit and the structural pattern below would be identical. ## Why this layout Two design goals drove the shape of this feature. **Conversion.** Newsletter signups are an action people skip easily. A tiny envelope icon hidden behind a hamburger gets ignored. A visible SUBSCRIBE entry in the main nav is hard to miss without being intrusive. **Transferability.** The Brevo HTML embed code is long, full of inline styles, and account specific. If a reader wants to copy this setup to their own site, they should be able to paste their own embed verbatim into one file and have it work. No editing required. ## Architecture in one sentence The feature is built from five pieces: one config block, two includes, one page, and a small SCSS section. The Chirpy sidebar receives a single line override that conditionally renders the button include. | File | Role | | ---- | ---- | | `_config.yml` | Toggle, button label, colour preset | | `_includes/sidebar.html` | Chirpy override, one line addition to render the button | | `_includes/sidebar-subscribe.html` | The button list item with icon and label | | `_includes/brevo-form.html` | Verbatim Brevo embed code, paste and go | | `_pages/subscribe.html` | Thin page shell that wraps the Brevo include | | `assets/css/jekyll-theme-chirpy.scss` | Button styling, colour presets, font neutralisers | ## Step 1: Get your Brevo embed code Sign in to Brevo, then go to **Contacts**, **Forms**, and create a new subscription form. Brevo lets you customise the heading, fields, GDPR consent text, captcha provider, and styling. Once the form looks right, click **Share and Embed** and copy the **Simple HTML form** code. This snippet contains everything you need: the form action URL, hidden fields, captcha widget, scripts, and Brevo's reset stylesheet. Keep this snippet handy. You will paste it into one file later. A few Brevo settings worth double checking before you copy: - **Captcha hostnames.** If Brevo's auto generated captcha is enabled, the captcha key is locked to the production domain you registered with Brevo. Submissions from `localhost` will fail with an "Invalid site key" error during local development. There are workarounds further down. - **Double opt in.** This is on by default. New signups receive a confirmation email and only become "Confirmed" after they click the link. Recommended for compliance and list quality. - **Confirmation email content.** Customise the confirmation email and the success page link. Visitors land here after confirming. ## Step 2: Add the subscribe block to `_config.yml` The feature gates on a `subscribe.enabled` flag, so you can ship it disabled and turn it on when ready. Two more knobs (`text` and `color`) let you adjust the button without touching HTML. ```yaml # Newsletter subscribe (Brevo) — sidebar button linking to /subscribe/ subscribe: enabled: true text: "Subscribe" # button label # Color presets: blue (default) | green | purple | orange | red | dark color: blue ``` Setting `enabled: false` removes the button entirely. The label text is uppercased automatically at render time using Liquid's `| upcase` filter (covered in Step 4), so writing `"Subscribe"` here yields `SUBSCRIBE` in the rendered nav, matching Chirpy's tab convention. ## Step 3: Override the Chirpy sidebar Chirpy ships with `_includes/sidebar.html` inside the gem. Jekyll's theme system lets you override any gem file by creating a file at the same path in your project. The override approach has two strengths: - The override file stays a near verbatim copy of Chirpy's original. When Chirpy releases a new version, you can re diff and re apply your single line addition. - Reverting is one delete away. Remove your project's `_includes/sidebar.html` and Jekyll falls back to the gem. Find Chirpy's sidebar at `vendor/bundle/ruby//gems/jekyll-theme-chirpy-/_includes/sidebar.html`. Copy the file verbatim to `_includes/sidebar.html` in your project. Then add **one line** inside the `` block, right after the `{% endfor %}` that closes the tabs loop. This makes the subscribe button render as the last item in the nav, after every tab, regardless of tab order. ```liquid {% for tab in site.tabs %} {% endfor %} {% if site.subscribe.enabled %}{% include sidebar-subscribe.html %}{% endif %} ``` The conditional is what gates the feature on the `_config.yml` flag. With `enabled: false`, the line is a no op and nothing renders. ## Step 4: Create the sidebar button include Create `_includes/sidebar-subscribe.html` with the markup for the button itself. It is one `` containing an `` with an icon and a span. The structure matches Chirpy's tab markup so the button inherits Chirpy's nav alignment and font sizing automatically. ```html {{ site.subscribe.text | default: 'Get Updates' | upcase }} ``` A few details that earn their place: - **`fa-fw`** is Font Awesome's fixed width modifier. Combined with Chirpy's existing icon styles, the bell icon takes the same column width as every tab icon, so labels line up vertically. - **`fas fa-bell`** is the solid bell icon, the same shape that has trained users to associate "subscribe" with this glyph thanks to YouTube. Swap to `far fa-bell` for the lighter outline version, or `fas fa-envelope` for a more email forward look. - **`data-color`** is read by the SCSS file in Step 7. The CSS variable cascade lets you change the icon and text colour by editing one line in `_config.yml`, no markup changes. - **`| upcase`** mirrors how Chirpy uppercases tab labels. Liquid does the transformation at render time so the HTML actually contains uppercase letters, not just CSS visual transforms. ## Step 5: Create the verbatim Brevo embed include Create `_includes/brevo-form.html` and paste the **entire Brevo embed code from Step 1** into it. Do not strip or modify anything. The verbatim approach is what makes this feature transferable. The file should look like this at the top: ```liquid {%- comment -%} Brevo signup form embed — paste-and-go. To replace with your own Brevo form: 1. Brevo dashboard → Contacts → Forms → [your form] → Share & Embed 2. Copy the "Simple HTML form" embed code 3. Replace EVERYTHING below this comment with your pasted code No need to edit anything else. The page wrapper at _pages/subscribe.html and the SCSS overrides in assets/css/jekyll-theme-chirpy.scss handle visual integration with the Chirpy theme automatically. {%- endcomment -%} ... (your Brevo embed code) ... ``` The Liquid comment at the top documents the workflow. Anyone cloning your site can paste their own Brevo embed below it without touching anything else. The SCSS file in Step 7 contains font and link colour overrides that visually integrate the form with Chirpy without modifying the embed itself. ## Step 6: Create the subscribe page Create `_pages/subscribe.html` as a thin shell that wraps the Brevo include. This is what visitors see at `linsnotes.com/subscribe/` (substitute your domain). ```html --- layout: page title: Subscribe permalink: /subscribe/ compress_html: false --- {% include brevo-form.html %} ``` Three details: - **`layout: page`** uses Chirpy's standard page layout, so the subscribe page inherits your site header, footer, sidebar, and theme colours. - **`compress_html: false`** is important. Brevo's embed contains `` blocks with global variables that get assigned before its `main.js` runs. HTML compression can mangle these in subtle ways. Disabling compression on this page only is the safe call. - **`
`** is a hook for the SCSS in the next step. Wrapping the include with a single class lets all visual overrides scope to this page only without leaking elsewhere. ## Step 7: Add the SCSS Append the following to `assets/css/jekyll-theme-chirpy.scss`. Three blocks: the button styling with colour presets, the page wrapper styling with font neutralisers, and a link colour override. ```scss /* ------------------------------------------------------------------ * Sidebar SUBSCRIBE button. Renders as the last item in