The Shoemaker's Children Aren't Barefoot

This article covers the architecture and the design decisions behind Home Automated Infrastructure, an open source project that runs a smart home with GitOps and Infrastructure as Code. Everything sits on a Kubernetes cluster of four Raspberry Pi 4B boards. Talos Linux is the operating system, ArgoCD keeps the manifests in sync, Terraform provisions the cloud resources, and Home Assistant runs the automation. The point I want to make is simple. The DevSecOps practices we use at work hold up fine in a house, and they let me stay off proprietary home automation platforms. How the house got here We bought the house in 2021. I wanted the freedom to make it as self-sufficient as I could, and I forget things constantly, so anything I can automate, I automate. At the back there was an edícula, an outbuilding nobody used. It held leftover construction material and nothing else. And yes, that thing hanging from the ceiling was the server rack. I turned the outbuilding into the hub. Every floor got Cat5e cabling and one UniFi AP, and the PoE switch in the outbuilding feeds them data and power. Next to the switch sits a Cloud Gateway Ultra that shapes traffic and balances the internet connections. I keep two separate ISPs for work and fail over between them at the router. To power all of that, plus the rest of the house, I put in solar panels and an inverter. What I ended up with was a house made of separate subsystems: network, solar, IoT devices. Each one had its own dashboard and its own logic. Checking three interfaces to answer one question gets old fast, so I needed something in the middle. That something is Home Assistant. Solution architecture The architecture has three layers: edge hardware, container orchestration, and cloud infrastructure. They talk to each other declaratively, and the Git repository holds the single source of truth for everything above the network. The network gear is the exception, and it is worth naming up front rather than letting you find it later. The UniFi controller, the VLANs and the ISP failover are configured by hand, in a UI. Everything else in this post is checkable against the repository; that part isn't. Separation of responsibility layers The cluster is called skynet. It has four Raspberry Pi 4B boards, three control plane nodes and one worker. The nodes are named after the Three Musketeers, so athos, porthos and aramis run the control plane and dartagnan does the work. The Pi 4B is a compromise between price, power draw and CPU. For what runs here, automation, monitoring and ingress, it is enough. None of it is critical enough to justify more. The Talos config pins the generic ARM64 image rather than a board revision, so if you try this on a Pi 5 expect the install disk path to be the first thing that breaks. Talos Linux Talos Linux is an immutable operating system built only for Kubernetes. It has no interactive shell and no SSH. You configure it through an API, and I keep that configuration in Terraform. That buys me two things. There is almost nothing installed on the box for an attacker to use, and updates either apply completely or roll back. Anything that drifts from the declared config gets corrected on the next deploy, so manual tweaks can't pile up over months the way they do on a normal server. general_patch = yamlencode({ cluster = { proxy = { extraArgs = { "metrics-bind-address" = "0.0.0.0:10249" } } } machine = { features = { kubePrism = { enabled = true port : 7445 } } install = { image = "factory.talos.dev/nocloud-installer/${local.schematic_id}:${var.talos_version}" disk = "/dev/mmcblk0" } network = { interfaces = [{ interface = "eth0" dhcp = true }] } } }) That block is the whole common configuration, shared by the control plane nodes and the worker. Since the code defines the entire machine, replacing a board is a rebuild, not an investigation. Home Assistant on Kubernetes The normal way to run Home Assistant is Home Assistant OS, on a mini PC or on its own Raspberry Pi. That is the supported install, with the Supervisor, the add-on store and one-click backups. Running it inside Kubernetes is more work by any measure, so let me explain why I did it anyway. Home Assistant is not the system. It is one application inside a system. The house already ran on a managed network, solar generation, DNS, certificates and monitoring, and all of that already lived in Git. Leaving automation outside that flow would give me two ways to change the house: one versioned and reviewed, and one where I click around a UI and hope I remember what I did. Containers are not the win here. The win is that Home Assistant gets everything the cluster already does. ArgoCD deploys it, the External Secrets Operator pulls its secrets from SSM, Traefik gives it valid TLS, Prometheus scrapes it, and anything that drifts gets put back. The price for that is real. No Supervisor, no add-ons. The container image is not Home Assistant OS. MQTT, ESPHome and Zigbee2MQTT stop being plugins and become applications in the cluster, each with its own chart, its own resources and its own lifecycle. Under GitOps that is an improvement, since every piece becomes code someone can review instead of settings buried in a UI. It is also work that Home Assistant OS would have done for me. Discovery does not cross the pod network. Plenty of integrations depend on mDNS, SSDP or DHCP broadcast, and those packets never reach a pod on the overlay network. Most of my devices have static IPs, including the FoxESS inverter, the UniFi APs and the gateway, so I don't lean on automatic discovery much. State is the hard part. Home Assistant keeps its configuration and history in /config, so it needs a persistent volume. With local-storage the pod lands on whichever node holds that disk. The PV pins by node role rather than by name, so it resolves to dartagnan only because dartagnan is the only worker, but the effect is the same: the cluster's high availability stops there. Three control plane nodes do not make Home Assistant highly available. And the other half of that sentence deserves saying too, because I nearly wrote the comfortable version. There is no backup. No Velero, no restic, no cron job copying /config anywhere. If that SD card dies I can rebuild the node in minutes and Home Assistant's entire configuration and history are gone with the card. Rebuild is not restore. It is the largest hole in this setup and the only item on the list that would actually cost me something. The house is production. A bad deploy in a lab costs me an evening. A bad deploy here turns off someone's lights, and that someone lives with me. It forces a discipline a homelab usually doesn't, which is the main reason the project has taught me so much. Distribution of responsibilities Two systems keep the repository and the running cluster in agreement. ArgoCD watches the /k8s/ directory and syncs any difference to the cluster. The ApplicationSet generator reads two levels deep, k8s//, and that directory convention is what does the work — drop a chart in the right place and it deploys itself. Self heal and prune are on, so what is in Git is what is running. Terrateam picks up pull requests that touch /terraform/, runs terraform plan and posts it as a PR comment. It also runs Infracost against the plan, so every pull request tells me what the change costs in BRL before I merge it, which is a strange and useful thing to have on a homelab. I never run Terraform by hand, and no credentials sit in an external CI pipeline, because the actions that run the plan execute inside the cluster. Example of an action performed by Terrateam. Why Terrateam and not Atlantis? The permissions for infrastructure secrets go into the GitHub Action itself instead of collecting in a long-running service, which is what happens with Atlantis. I get tighter separation, and I can run it on my own hardware with the Actions Runner Controller. The free tier covers me, and only the workers run in my environment. Their drift detection works, which keeps stray manual changes from surviving in my account. GitOps and the development flow GitOps means the Git repository holds the desired state of the system. Every change to infrastructure or applications goes in as a commit, so the history of the house is the history of the repository, and I can go back to any point in it. Here that comes down to one rule: no manual changes with kubectl, none in the AWS console. If a setting is not in Git it does not exist, and the next sync should remove it. The rule covers the cluster and the AWS account, which is to say everything except the network gear I flagged earlier. Kubernetes applications Modify Helm charts and values under /k8s/ Commit and push to the main branch ArgoCD notices the difference between Git and the cluster within 3 minutes ArgoCD syncs the changes The cluster converges on the state declared in Git Terraform infrastructure Create a branch and modify files under /terraform/ Open a pull request on GitHub Terrateam runs terraform plan and posts the result as a comment A reviewer reads the plan and approves it or asks for changes Apply is not automatic — I trigger it once the plan looks right, and Terrateam merges the pull request and deletes the branch after it succeeds Initial bootstrap Cluster initialization follows a dependency order. The modules in terraform/bootstrap/ run in this sequence: state creates the S3 bucket and the DynamoDB table that hold Terraform state remotely, with locking cluster provisions the Talos Linux cluster across the four boards machines applies the init-data and boots the nodes argocd installs ArgoCD argocd-applicationset configures the ApplicationSet that watches /k8s/ and syncs every application Security and DevSecOps Least privilege Every AWS IAM role gets the smallest set of permissions that function needs. The self-hosted runners assume roles through their ServiceAccount, and Terrateam authenticates with an OIDC pre-hook, so no AWS keys sit in GitHub secrets. One place still uses a static key pair, and it is the one that matters most. The External Secrets Operator authenticates to SSM with an access key held in a Kubernetes Secret, not through IRSA. Every secret in the cluster comes through that door, so it is the piece I would most like to be wrong about, and moving it to IRSA is the next thing on this list. Secrets management Secrets take one path: AWS SSM Parameter Store, then the External Secrets Operator, then a Kubernetes Secret, then the pod. Nothing sits in plain text in the repository. SSM encrypts everything with KMS. The External Secrets Operator is a controller that watches ExternalSecret objects and copies values from SSM into Kubernetes Secrets on a refresh interval I set. The rule I don't break: no credentials, tokens, passwords or private keys in the repository. It is not a spotless repo. An early terraform.tfstate from the bootstrap module made it into git and carries an AWS account ID and the state bucket name. Those are identifiers rather than secrets, and it is out now, but it went in against an instruction written in that module's own README. Which is roughly the argument for the secret-scanning hook I don't have. Kubernetes RBAC Each application has its own ServiceAccount and a Role with only the permissions it needs. No pod uses the default ServiceAccount, and no Role hands out cluster-admin. Network policies, and why there aren't any This is the control I expected to have and don't. Home Assistant runs with hostNetwork: true, because it has to reach devices on the LAN that a pod on the overlay network cannot see. That is also the case where NetworkPolicy gives you the least. The pod sits on the host's network stack, so a policy scoped to cluster namespaces protects almost nothing that matters, and any egress rule tight enough to be worth writing would cut the application off from the devices it exists to control. So the segmentation that actually contains Home Assistant is not in Kubernetes at all. It is the VLAN layout on the Ubiquiti gear, further down this post. That is a worse answer in the sense that it lives outside Git, and a better one in the sense that it works. CI/CD without external credentials The GitHub Actions runners run inside the cluster through the Actions Runner Controller. The pipelines never need AWS credentials in GitHub secrets, because they authenticate with the IAM role attached to the runner's ServiceAccount. That comes with an exception to the least-privilege paragraph above. The runner scale set carries a Docker-in-Docker sidecar running privileged. Docker needs it and I have not found an alternative I trust more, but it is worth stating plainly rather than leaving in a values file: it is the highest-privilege workload in the cluster, and it is the one that executes code arriving from pull requests. Pre-commit hooks Pre-commit hooks check every commit before it lands: terraform fmt formats Terraform code terraform validate checks syntax tflint catches bad practice and likely errors trivy scans manifests and configuration for vulnerabilities What is not in that list is secret scanning. I found that out fact-checking this post, having believed otherwise for months because my own documentation said so in six places. It is twenty minutes of work and it is now the top of the backlog. Observability Metrics Prometheus collects metrics from the infrastructure, so nodes, CPU, memory, disk and network, and from the applications on their /metrics endpoints. Its configuration is a Helm chart in the repository like everything else, so config changes get versioned and reviewed. Request volume, phone battery levels, all of it ends up in Prometheus. Grafana Grafana reads those metrics and draws the dashboards. The dashboards ship as Kubernetes ConfigMaps, so the visualization config is versioned too. Monitoring ArgoCD through Grafana. Next I want to pull the consumption data out of Home Assistant and rebuild its energy view in Grafana. Home Assistant's default monitoring of energy consumption and generation. Alerts and notifications Prometheus Alertmanager talks to Home Assistant, so infrastructure alerts come out as things happening in the house. A node running low on memory can make a speaker chime or a light change color. Network and connectivity Traefik as ingress controller Traefik is the ingress controller and reverse proxy, and it routes outside traffic to the right service in the cluster. Every service opens over HTTPS with a real certificate and no browser warning, even on the internal network. The cluster also runs cert-manager, so there are two things here capable of issuing a certificate; the config below is Traefik's own ACME resolver. Let's Encrypt issues them over ACME, and the Cloudflare API answers the DNS-01 challenge. The DNS and the token are both on the free tier. The configuration lives in the controller: traefik: env: - name: CF_DNS_API_TOKEN valueFrom: secretKeyRef: name: cloudflare-api-token key: token service: enabled: true type: ClusterIP ports: web: port: 80 # changes --entrypoints.web.address=:8000 to :80 hostPort: 80 # binds host node port 80 to container port 80 exposedPort: 80 # updates the K8s Service object protocol: TCP websecure: port: 443 # changes --entrypoints.websecure.address=:8443 to :443 hostPort: 443 # binds host node port 443 to container port 443 exposedPort: 443 # updates the K8s Service object protocol: TCP metrics: port: 9500 exposedPort: 9500 hostPort: 9500 protocol: TCP hostNetwork: true ingressRoute: dashboard: enabled: true matchRule: Host(`traefik.cavaleiro.in`) entryPoints: - web securityContext: capabilities: add: - NET_BIND_SERVICE drop: - ALL readOnlyRootFilesystem: true providers: kubernetesCRD: allowExternalNameServices: true allowCrossNamespace: true kubernetesIngress: allowExternalNameServices: true kubernetesGateway: enabled: false persistence: enabled: true name: data storageClass: "local-storage" accessMode: ReadWriteOnce size: 128Mi path: /data additionalArguments: - "--certificatesresolvers.cloudflare.acme.dnschallenge.provider=cloudflare" - "--certificatesresolvers.cloudflare.acme.storage=/data/acme.json" - "--certificatesresolvers.cloudflare.acme.dnschallenge.resolvers=1.1.1.1:53,8.8.8.8:53" DNS with Cloudflare Cloudflare handles DNS, and Terraform provisions the records in the terraform/global/ module. A new service exposed through Traefik gets its DNS record in the same deploy. Cloudflare does one more thing, and it belongs in the security section rather than this one. A Cloudflare Tunnel runs in the cluster, and it is the only inbound path from the public internet — there are no ports forwarded on the router. A tunnel is a good way to do this: the exposure is one authenticated outbound connection instead of an open port, and its credentials arrive through the same SSM to External Secrets path as everything else. It is still real inbound exposure from the internet, and listing controls in this much detail while omitting the front door would be the wrong kind of thorough. Mesh Wi-Fi with Ubiquiti The Ubiquiti gear carries both the cluster and the smart home devices. IoT sits on its own VLAN, the cluster on another, and the main network on a third. If a cheap IoT device gets compromised, it lands in a segment where there is nothing worth reaching. This is the segmentation I pointed at when I said there are no NetworkPolicies, and it is also the layer I admitted at the start is not in Git. The thing actually containing my smart home devices is configured by hand in a UI. I am not thrilled about that either. A capture of the router's WAN management view An excerpt of the network topology drawn by the router Conclusion Call it a homelab if you like. What it showed me is that the gap between platform engineering at work and home automation is mostly a question of scale. GitOps, IaC, immutability, secrets management and RBAC work the same on four Raspberry Pis as they do on a production account. The other thing I got out of it is a place to practice that behaves like production. Anyone who contributes to the repository works with Kubernetes, Terraform, ArgoCD, Helm, Prometheus and Grafana on a system where mistakes have consequences. Misconfigure the cluster and the house stops working. Nobody gets paged at 3am over it, and nobody loses money. I fact-checked this post against the repository before publishing it, claim by claim, and it is the reason several paragraphs above read the way they do. Three things I believed were running were not, and one of them had been sitting in my own documentation as fact for months. So the list I leave with is shorter than the one I started with and more useful: back up /config, move the External Secrets Operator off its static key pair and onto IRSA, and add the secret scanning I had already told six files I had. That gap between what the documentation says and what the cluster does is the real lesson here, and it is not a homelab problem. It is the same drift that shows up in any system where the docs and the deploy are edited by different hands on different days. Mine just happened to be the same hands. The repository is at https://github.com/catdevsecops/home-automated-infrastructure if you want to copy any of it — including the parts I have just told you are missing. References Talos Linux: https://www.talos.dev/ ArgoCD best practices: https://argo-cd.readthedocs.io/en/stable/user-guide/best-practices/ External Secrets Operator: https://external-secrets.io/ Terrateam: https://www.terrateam.io/ Terraform state: https://developer.hashicorp.com/terraform/language/state Kubernetes security: https://kubernetes.io/docs/concepts/security/ Home Assistant: https://www.home-assistant.io/ Project repository: https://github.com/catdevsecops/home-automated-infrastructure