Part 2 ended with six Talos VMs, an etcd quorum, and a kubeconfig — and nodes that were all stubbornly NotReady. That’s expected: Talos ships with cni.name: none, so the cluster has no pod networking until you give it some. This post is the rest of the journey — CNI, storage, secrets, certificates, ingress, monitoring, and GitOps — and, more honestly, the handful of things that didn’t go to plan.
(All addresses below are illustrative doc-range IPs. Kubernetes-internal CIDRs are the real, conventional defaults.)
CNI: Cilium, and deleting kube-proxy on purpose
The first thing the NotReady nodes need is a CNI. I run Cilium in eBPF mode, and I let it replace kube-proxy entirely. On Talos that’s a clean pairing, because Talos exposes a local, HA-load-balanced API server endpoint on every node called KubePrism at localhost:7445. Cilium points its kube-proxy-replacement there:
kubeProxyReplacement: true
k8sServiceHost: localhost
k8sServicePort: "7445"
A few minutes after helm install cilium, all six nodes flip to Ready and cilium status reports KubeProxyReplacement: True. No kube-proxy DaemonSet, no iptables sprawl — service routing is handled in eBPF.
The storage pivot: Longhorn → NFS CSI
Part 2 assumed Longhorn (replicated local block storage). When I actually got here, I changed my mind.
Longhorn replicates a local disk on each worker three ways across nodes. But my persistent data was always going to live on the NAS anyway, which already does RAID + snapshots. Backing Longhorn with NAS storage would mean replicating three times on top of storage that’s already redundant — and funnelling every “local” replica back to one box over one network. So I dropped Longhorn and went with csi-driver-nfs: control-plane/etcd disks stay on fast local storage, and persistent volumes come straight off the NAS.
This is where I lost an hour, and it’s worth writing down. The CSI driver mounted the export and the server answered — but with:
mount.nfs: ... failed, reason given by server: No such file or directory
The server was reachable (so routing and the export allow-list were fine), it just couldn’t resolve the path. The cause: I was mounting NFSv4.1, and this NAS presents NFSv4 paths relative to a pseudo-root that doesn’t match the v3-style export path. The fix was to pin NFSv3 — except the CSI node container has no rpc.statd, so v3 also needs nolock:
mountOptions:
- nfsvers=3
- nolock
- hard
Each volume is single-writer (one StatefulSet replica owns its PVC), so local-only locks are fine. One more NAS-ism: the export squashes root, so chown fails — set mountPermissions: "0777" on the StorageClass and let workloads write without it.
To keep storage traffic off the application VLAN, the workers get a second NIC on the storage network with a pinned MAC, a static address, and a route to just the NAS subnet — so the cluster’s node IPs stay on the app VLAN and only NFS rides the storage net.
GitOps: Argo CD owns everything past the foundation
Only four things are installed imperatively: Talos, Cilium, MetalLB, and Argo CD itself. After that, Argo CD owns the cluster from a private Git repo via an app-of-apps. Argo reads the repo with a read-only deploy key — it never needs write access to anything.
The children are ordered with sync-waves so dependencies come up first:
wave 0 storageclass
wave 1 cert-manager · csi-driver-nfs · vault
wave 2 external-secrets · ingress-nginx · cluster glue
wave 3 monitoring · opentelemetry
The wave numbering matters more than it looks. My first attempt put the StorageClass in wave 2, but Vault (wave 1) needs a StorageClass to bind its PVCs — and Argo won’t advance to wave 2 until wave 1 is healthy. Instant deadlock. Moving the StorageClass to wave 0 fixed it: it can exist before its provisioner does, since PVCs just wait.
Secrets and certificates, the self-hosted way
No cloud secret manager. Vault runs in-cluster as a 3-node HA Raft cluster, and the External Secrets Operator bridges Vault into Kubernetes Secrets using Vault’s Kubernetes auth. cert-manager issues a wildcard *.k8s.homelabnj.tech from Let’s Encrypt over DNS-01, so nothing has to be exposed on port 80.
The neat part is the chain: the Cloudflare API token lives in Vault → ESO pulls it into a Kubernetes Secret → cert-manager’s ClusterIssuer consumes that Secret for the DNS-01 challenge. The token never touches Git or a manifest.
The honest caveat: Vault comes up sealed after every restart, and unsealing is a manual step that can’t be GitOps’d. That’s the tradeoff for self-hosting your own root of trust — I keep the unseal keys offline and accept the manual unseal.
LoadBalancer without a cloud: MetalLB
On bare metal there’s no cloud LB, so type: LoadBalancer would hang forever. MetalLB in L2 mode hands out addresses from a reserved range on the app VLAN (192.0.2.200–192.0.2.230). ingress-nginx grabs one as its external IP, and that’s the front door for everything with an ingress.
The gotchas that actually cost time
A few real ones, because the polished walkthroughs never mention these:
- A worker config that wouldn’t apply.
talosctl gen config --config-patch @cp.yamlquietly applies that patch to both control-plane and worker configs — so the control-plane VIP leaked into the worker config, and Talos rejected it (“virtual IP is not allowed on non-controlplane nodes”). The fix is--config-patch-control-plane, which scopes it. - Workers advertising the wrong IP. With two NICs, the kubelet picked the storage network as the node’s
InternalIP. That quietly breaks cross-node networking andkubectl logs/exec, because the control plane can’t reach that subnet. Pin it withmachine.kubelet.nodeIP.validSubnets. - Monitoring that was broken for hours without anyone noticing. Talos enforces the
baselinePod Security standard by default. node-exporter needs hostNetwork, hostPID, host paths, and a host port — allbaselineviolations — so its DaemonSet was silently0/6, and that hung an Argo sync waiting on it. The namespace needsenforce: privileged(keptaudit/warnatbaselineso escalations elsewhere are still flagged). qmis node-local. My VM-creation script assumedqm create --node Xwould place VMs cluster-wide. It won’t —qmonly acts on the host it runs on. The script now detects its own hostname and creates only that node’s VMs, run once per host.
Where it landed
The end state: six Ready nodes, kube-proxy-less networking, GitOps reconciling ten applications to green, NFS-backed persistent volumes, a sealed-by-default secrets store feeding a working wildcard-cert pipeline, and a monitoring stack actually collecting metrics. Every fix above lives in the Git repo, so the whole thing is reproducible — which, for a homelab you’ll inevitably rebuild, is the entire point.
If Part 1 was why Talos and Part 2 was how to bootstrap it, this was everything between a kubeconfig and a platform you’d actually run things on. Next up: putting real workloads on it.