I run my homelab like a small company: the architecture, the org chart, and the rules I only learned by breaking things

Most homelab tours are a hardware list with the price crossed out. This isn't that. I want to show you how the pieces talk to each other, because after a couple of years the interesting thing about my setup isn't what's in it — it's that it's organised less like a rack of services and more like a small organisation, with a front desk, a records office, a security guard, a chief of staff, and a set of standing rules that every one of them was written down only after something went wrong. Here's the org chart, then how a request, a secret, and a scheduled job each actually move through it — and the handful of principles that fell out of two years of getting it wrong. The org chart Physically it's boring: a couple of small servers running a hypervisor, a NAS, a low-power box that does nothing but host local language models, and a router. The roles are where it gets interesting, because each layer has one job and knows almost nothing about the others. The front desk — one door for everything. Every service, without exception, is reached through a single reverse proxy that terminates TLS with real certificates, sitting behind a CDN. In front of the apps is one identity provider doing single sign-on and 2FA. You authenticate once, to one thing, and every service inherits that. (I wrote up the security side of this separately — including the time I ran it with the example config's placeholder secrets still in place. Securing one door is only simple until you realise it's also one point of failure.) The records office — secrets live in one place. No service holds its own credentials on disk. There's a dedicated secrets store, and everything — the proxy's DNS token, database passwords, API keys, the identity provider's signing keys — is pulled from it at runtime via a machine identity, never written into a config file that could end up in a backup or a git history. A secret has exactly one home and one system of record. Security — detect in one place, block in another. The identity provider writes a log line for every auth attempt. A separate intrusion-detection service tails those logs, and when it sees a run of failures it pushes the offending address up to the CDN's edge to be blocked before it reaches the network at all. Detection and enforcement are deliberately different systems that share nothing but a log file and an API. Loose coupling is why it's never taken anything else down with it. The chief of staff — one brain, several hats. This is the part people find odd, so it's worth explaining. The estate is orchestrated by a self-hosted AI agent that runs as a set of profiles, each a different role: one that manages infrastructure, one that writes, one that researches, one that watches security. They share a task board and hand work between them. It sends my morning briefing, drafts documentation, watches for drift, and answers questions about the estate — but, crucially, it is not trusted to gather its own facts (more on that rule below). The workforce — the actual services. Behind all that: the password vault, a git server, a monitoring stack, home automation, a media library with offsite backup, a couple of tools I've built, and the local-model host that the AI layer runs against so that most of its work costs nothing and leaves no data with a vendor. How a request moves Say I open my dashboard from a laptop. DNS resolves the hostname. Internally it points at the one machine running the proxy; externally the same name is CDN-fronted. (Split-horizon, and yes, that split has bitten me — a client using the wrong resolver got a page that looked down when the service was fine.) The CDN edge takes the request first. If your address is on the block list the intrusion-detector pushed up there, this is as far as you get. The reverse proxy terminates TLS and checks: is this person already authenticated? If not, it bounces them to the identity provider. The identity provider takes the login and 2FA, and either hands back a session (for the forward-auth apps) or completes an OpenID Connect exchange (for the apps that speak it properly and want to know who you are, not just that you're allowed). Only now does the request reach the actual service — which has, this whole time, never seen an unauthenticated packet. Five layers, and each one can say no on its own terms. The important property isn't any single wall; it's that the app at the end gets to be simple because everything in front of it already did the hard part. How a secret moves This one's shorter and it's the habit I'd most want a new self-hoster to steal. Nothing has its password baked in. A service boots, authenticates to the secrets store with a machine identity that was itself provisioned once and lives in an environment file with tight permissions, pulls exactly the secrets it needs into memory, and runs. Rotate a credential in one place and every consumer picks up the new one on restart. When I audited the estate, the single best thing I could say about it was that I could rotate any password without hunting through a dozen docker-compose.yml files — because none of them held one. The corollary, learned the hard way: verify a secret by fingerprint, never by printing it. The moment you echo a key to check it matches, it's in a scrollback buffer and a terminal log. I compare SHA-256 hashes across the three places a secret lives and never look at the value itself. How a scheduled job moves — and the one rule that governs all of them This is the principle the whole estate is built on, and it came from a specific failure. I used to have the AI layer run health checks and "find me opportunities" scans — let the model go and look at the system and report back. It hallucinated. It reported backups that hadn't run as successful. It cited demand for a product with no source. It appended a "[SILENT]" tag to a report while violating its own instruction not to. A model asked to gather facts will, under the slightest ambiguity, invent them and present them with total confidence. So every scheduled job on the estate now follows one rule: Scripts gather facts. Models never do. A monitoring job is deterministic Python under a system timer. It queries the real state — the actual backup timestamp, the real disk figure, the true service status — and writes those numbers down. Then, if a human-readable summary is wanted, the model is handed the already-gathered facts to phrase, and is forbidden from adding any of its own. The model is a writer, never a researcher. The crons are boring, they don't drift when the model does, and when one fails it says so loudly rather than reporting a cheerful ok. That last part is its own rule, and it's the through-line of everything I build now: a healthy system reports ok under both the correct schedule and a broken one, so a quiet week proves nothing. Every monitor I run has been deliberately broken once, to watch the alert actually arrive. The one that mattered most had a 24-hour blind spot I'd never have found any other way. The rules, collected None of these were in the plan. Each is a scar. One door, and know it's also one point of failure. Centralising auth is the right call and it centralises your blast radius in the same move. Run it knowing that, with monitoring that can tell "the door is down" from "the house is down." One home per secret, pulled at runtime, never on disk. And fingerprinted, never printed. Scripts gather facts; models never do. Let the clever, non-deterministic layer phrase a decision or a report — never make one you can't reproduce or find a fact you can't verify. Fail loudly, and prove it. Alert on absence, not just errors, and break each alarm on purpose to confirm it fires. Silence is not success. Loose coupling between the security layers. Detection and blocking, proxy and identity, share as little as possible — a log file, an API — so one can't drag the others down. Deterministic where it matters, model-driven only where it's safe. The AI runs the estate the way a good chief of staff runs an office: it drafts, summarises, reminds, and routes — and it is never the thing that quietly decides something important on its own. The result is a homelab that is genuinely less impressive to look at than the sum of its parts, and that's the point. It's organised so that each piece can be simple, each failure is loud, and the clever component — the one most likely to confidently tell me something untrue — is kept firmly in the role of writer, not witness. Everything I got right, I got right by first getting it wrong in public, which is more or less the whole methodology. Notes from running a small self-hosted estate as if it had an org chart. Specifics are kept deliberately vague — the useful part is the shape, not my IP addresses. 🤖 Drafted with AI assistance from my own homelab notes, logs and repos, then reviewed and edited before publishing.