Sentinel: The Core Runtime

Run two Sentinel cores against the same message stream and nothing fails. The durable consumer they both bind to splits the observations between them, so each builds its picture of the estate from half the data, and both report healthy. The timer wheel in Watchdog & Derived Points, the single arbitration resolver in The Command Plane and the interlocks in The Rules Engine all assume this never happens, and none of them says what guarantees it. This part does. Most of the rest is what having only one core costs. While it is down the message stream has to hold everything, so the buffers are sized for a four-hour outage, and how long the backlog takes to drain afterwards depends on how busy the core normally is: an hour at 20% utilisation, sixteen hours at 80%. When it falls behind there is no second core to share the load. A core that is down protects nothing, so protection that has to survive it belongs in a local device and not in a rule. Nearly every figure here is a design target rather than a measurement, and the document says so. The one the rest of the arithmetic depends on, how many observations a second the core can sustain, is still unmeasured, and the exit criteria end by asking for it, so that the four-hour figure can be checked instead of assumed. What this document owns: single writer as a requirement and what breaks when two cores run; the advisory lock and the startup order that takes it first; failing fast when the lock is lost; the core epoch and fencing the bus; the scale targets and the observation rates behind them; the difference between a planned restart and a tolerable unplanned outage; buffer sizing and the drain arithmetic; how the system behaves during a drain, including the pause on warmup deadlines; backpressure, meaning lag read from the consumer's pending count, a full stream that rejects rather than discards, and the poll schedule as the only valve; what stops while the core is down; restart time; the rule against interlocks at a site that can be partitioned from the core; and multiple cores as a recorded non-goal. It does not own the staleness suppression mechanism and its constraints, the startup grace, the recovery rule for late observations, or the strategies for bounding rehydration, which belong to Watchdog & Derived Points; the driver's epoch tracking and its local disk queues, which belong to Driver Contract; what the core does with a stale_epoch rejection, which belongs to Command Plane; the input-loss policies and warmup themselves, which belong to Rules Engine; or retention and compression, which belong to Retention & Compaction. The core is a single process, and exactly one instance of it runs at a time. Single writer is a requirement, not an assumption Five mechanisms already assume it: the in-memory projection is authoritative, the timer wheel holds freshness deadlines, the JetStream consumer is durable and single, verification timers are core-side, and the arbitration resolver is the only writer of point_desired. Two cores would not degrade gracefully. They would: Split the observation stream rather than duplicate it. Two consumers bound to the same durable consumer each receive a share of the messages. Each core would build a projection from half the data and believe it complete. This is the worst of the failure modes because nothing errors — both cores look healthy and both are wrong. Double-fire commands. Two arbitration resolvers reaching the same conclusion issue two writes; reaching different conclusions issue contradictory ones. Duplicate timer firing. Two watchdogs expiring the same deadline, two verification timers on one command. Because the first failure is silent, the requirement is enforced mechanically. A convention that only one runs is not enforcement, and the day it is violated is the day someone starts a second instance to test something. Enforcement: an advisory lock, not a lease A Postgres session-level advisory lock, held on a dedicated connection for the process lifetime, taken before anything else happens. This is preferred over a lease in NATS KV for one reason: there is no timeout to tune. A lease requires choosing a renewal interval and an expiry, and getting that wrong produces either a slow failover or a window where two cores both believe they hold it. An advisory lock is released by the database when the connection dies, so liveness is delegated to TCP and to Postgres rather than to a chosen number. Acquisition order at startup: Connect to Postgres, acquire the advisory lock, or exit Increment and read the core epoch (below) Load vocabularies and registry Load and restore the projection Bind the JetStream durable consumer Start the watchdog, derived points and verification timers Start the command plane and the API Nothing may bind the consumer before the lock is held. Binding the consumer (step 5) only after taking the lock (step 1) is the whole reason for writing this list down. A core that binds first and locks second has already stolen messages from the running instance before it discovers it should not be there. A core that cannot acquire the lock exits non-zero immediately. It does not wait, retry in a loop, or start in a read-only mode. Under a supervisor this produces a restart loop, which is visible; a silently idle second core is not. Losing the lock while running Fail fast. On detecting that the lock is gone — connection death, or a periodic ownership check — the core stops publishing commands, stops consuming, and exits non-zero. No draining, no finishing the batch in flight, no degraded read-only mode. A core that has lost the lock has, by definition, possibly been replaced. Anything it does from that moment is a second writer. An uncommitted batch is not a loss: the messages are unacked and will be redelivered, and the dedup key makes redelivery harmless. Fencing the bus The advisory lock protects Postgres. It does not protect NATS, and the dangerous window is exactly the one it cannot cover: core A is partitioned from Postgres, its lock is released, core B acquires it and starts, and core A has not yet noticed because its TCP socket has not timed out. For that interval, core A can still publish on cmd.>. So commands carry a core epoch: a monotonically increasing integer stored in Postgres and incremented on each successful lock acquisition. Drivers track the highest epoch they have seen and reject any command carrying a lower one, with outcome rejected and reason stale_epoch. This is one integer in the command envelope and one comparison in the driver, and it is the only thing standing between a partitioned core and a relay pulsing on the instructions of a process that has already been replaced. The epoch is also stamped on every control_event, which makes "which core did this" answerable after the fact. Drivers do not need to know what an epoch means, only that it never goes backwards. Scale targets Every number below is a design target rather than a measurement. They exist so that the sizing arithmetic has terms, and so that a future measurement has something to disagree with. quantity target devices 200+ network segments, and therefore driver hosts 16+ device timeseries points 2,000+ derived points 1,000+ self-points ~700 total points ~4,000 Self-points are the points the system publishes about itself. Their figure is the one most easily forgotten, and it is not small: roughly two per device, seven per driver host, one convergence point per writable point, and the core's own signals. They are points in every respect, so they consume registry entries, timer-wheel slots, storage and ingest capacity like any other. Observation rate. Three tiers, and the middle one dominates: a handful of points at 1 Hz — liveness, air quality, heart rate the bulk at roughly 10 s a few hundred hourly That works out at ~220 observations/second steady state, of which the 1 Hz points are around 5%. The design figure is 500/s, leaving room for the fleet growing and for the 10 s tier tightening, which is the parameter the whole figure is sensitive to: at 5 s the rate doubles. Storage figures, all following from the above: ~19M raw rows/day, ~40 GB rolling under a 90-day retention with compression from day seven ~5.3M rows/day in the minutely continuous aggregate, which is the long-term storage driver rather than the raw table ~4,000 timer-wheel entries, well inside a single wheel's design ~1,000 rehydration queries at cold start, which is the restart time almost in its entirety Sustainable ingest throughput is the one term still unmeasured, and it is the term utilisation depends on. Per-observation validation, registry lookup and unit conversion in application code is where the ceiling sits, not in Postgres. It is measured against the 500/s design figure before the stream is sized, rather than assuming the batch insert path is the limit. Two different outage numbers These are not the same quantity, and conflating them sizes the buffer against the wrong case. Planned restart is lock release plus cold start: minutes, dominated by derived-point rehydration. This is what an upgrade costs, and Restart time below covers it. Tolerable unplanned outage is how long the core can be down without losing data — hardware failure with nobody around, or a fault in the middle of the night. The working figure is four hours, with the expectation that the real case is usually far shorter. Only the second sizes the buffer. Sizing the stream against restart time would leave the system losing data in exactly the scenario it cannot control. Buffer sizing While the core is down, drivers publish normally and the observation stream absorbs everything. The stream is the first buffer; the drivers' local disk queues are the second, engaged only once the stream is full and rejecting. The stream is bounded by bytes and messages only, never by age. A max_age limit deletes the oldest messages silently once they pass the age, regardless of whether anything has consumed them, which is precisely the middle-of-the-chain loss that rejecting on full was chosen to avoid (see Where the loss goes below). The stream size is derived from observation rate, not expressed as a duration limit. Recovery is set by headroom, not by outage length. After an outage of length O, the core drains the backlog while new observations keep arriving. With C the sustainable core throughput, and steady-state utilisation u the ingest rate divided by C, the drain takes O × u/(1-u) and the peak backlog reaches O/(1-u) of steady-state volume. For a four-hour outage: utilisation drain time buffer needed 20% 1 hour 5 hours 50% 4 hours 8 hours 80% 16 hours 20 hours At 80% the system takes four times the outage to recover and needs five times the outage in buffer, which is how a system that looked fine in steady state turns a short fault into most of a day. Recovery within an hour of a four-hour outage requires running at roughly 20%. At the scale targets above, this is comfortable, and the stream is sized generously rather than carefully. Provided sustainable throughput is anywhere above about 1,000/s, utilisation sits under 25%, a four-hour outage drains inside 80 minutes, and the buffer requirement is roughly 5.3 hours of steady state — 8M messages and 4 GB, which is cheap enough that rounding up costs nothing and buys margin against the throughput figure being worse than assumed. Two refinements: Drain throughput is usually better than steady-state throughput, because batches fill completely rather than flushing on the 100 ms timer. So treating C as the steady-state figure is conservative in the right direction. The total buffer is the stream plus the sum of the driver queues, so the requirement can be split between them. The stream is the better place for it, though, because a driver queue filling means that host is also at risk of losing data on its own bound. Draining The system behaves correctly during a drain, and this is worth stating because it looks alarming. Replayed observations carry their original observed_at, which is older than the current deadline, so under the recovery rule in Watchdog & Derived Points they do not clear staleness. Points do not reach live until the backlog is worked through and genuinely current data arrives. Rules stay in warmup and the system declines to automate until it has caught up, rather than making four hours of retrospective decisions in ninety seconds. That is the design working rather than a fault. Two things must hold for it to keep working: Staleness evaluation is suppressed while lag is high, as described below, so the drain does not also produce a fleet-wide staleness cascade. Warmup deadlines pause under the same condition. warmup: deadline(60s) exists to stop a rule waiting forever for an input that is never coming. During a drain the input is coming, just not yet, and letting the deadline expire makes rules evaluate under their input-loss policy while the data they need sits in the queue. Safe, but premature, and for the wrong reason. The pause is the same principle as the suppression: a core-internal condition must not drive device-level decisions. Backpressure Single writer determines the shape of this before any policy is chosen: the usual answer is unavailable. When a consumer cannot keep up, the standard response is to add consumers, and there is exactly one core by construction. Backpressure here is solved by shedding load or by going faster, never by scaling out. Lag is already measured, for free. Because ingest acks after commit, the durable consumer's pending count is the lag, and the age of its oldest unacked message is the second half of the picture. No separate instrumentation is needed and no watermark table exists to disagree with. A backlog survives a restart. Resuming from the ack position is what makes restart correct, and it also means a backlog cannot be cleared by bouncing the core. Draining or discarding is a deliberate operation, never a side effect of a restart. A slow core must not look like a dead fleet This is the failure that makes backpressure a safety concern rather than a throughput concern. Freshness is computed from observed_at. If the core is ten minutes behind, every point in the system passes its deadline and goes stale while its data sits queued and perfectly healthy on the bus. Rules with on_input_loss: release drop their claims. Fail-safe interlocks assert. The system takes protective action across the entire estate because ingest is slow, which is close to the worst available response and is entirely self-inflicted. The watchdog therefore suppresses staleness while ingest lag is high, in the same shape as the startup grace it already implements. In both cases the absence of data is a fact about the core rather than about the device, and reporting it as device staleness is a lie. The mechanism and its three constraints are in Watchdog & Derived Points. Where the loss goes when it has to go somewhere The observation stream rejects new publishes when full, rather than discarding old messages. That choice puts the loss at one end of the chain instead of silently in the middle. A full stream means drivers fail to publish, fall back to their local disk queues, and those queues are bounded by size and age with a drop count published as a self-point. So there is exactly one place in the whole path where data is dropped, it is bounded by chosen numbers, and it announces itself. Discarding old messages instead would delete observations that were already accepted, between components that both believe the handoff succeeded. The valve is the poll schedule There is no load-shedding mechanism inside the core, deliberately. The deadband does not help, because every poll publishes regardless of whether the value changed — the deadband decides what counts as an edge, not what reaches the bus. The lever that does exist is the assignment: poll intervals are pushed over the bus and applied without restarting, so raising intervals across a fleet is a config change that takes effect in seconds. That is the response to sustained lag, alongside making ingest faster. It stays manual. An automatic feedback loop between ingest lag and collection rate would be a control system nobody specified, tuned by nobody, changing what the data means during the exact period when it is being diagnosed. While the core is down This is worth stating plainly, because it defines what the availability target has to be: Collection continues. Driver hosts keep polling, the stream absorbs the backlog, and the local queues take over if it fills. Nothing is lost that the bounds above do not lose. Devices hold their last commanded state. Nothing reverts, because nothing is driving them. No automation. No rules evaluate, no derived points compute, no freshness deadlines fire, no commands are issued or verified. No interlock protection. A fail-safe interlock is a claim held by a rule inside the core. If the core is not running, the rule is not evaluating and the protection is not being applied. That last point is the honest cost of a single-writer design. Protection that must survive the core being down cannot be implemented in the core. Restart time Cold start, the larger part of a planned restart, is almost entirely derived-point rehydration: at the scale targets above, roughly a thousand queries against the raw tables, some of them reaching into compressed chunks. The strategies for keeping that bounded are in Watchdog & Derived Points. Restart time is to be measured rather than estimated, and it is the number to keep small for two reasons: it is what an upgrade costs, and a long cold start makes the four-hour figure harder to meet, since recovery is cold start plus drain. Zero-downtime upgrade is not available under this design and is not pursued. Achieving it means two cores running simultaneously during handover, which is precisely what the epoch and the lock exist to prevent. The realistic improvement is a shorter cold start, not an overlapping one. Geographic split A second site that occasionally loses contact with the first is a driver-plane concern, not a reason for a second core. During a partition the remote site's driver hosts keep collecting and buffering exactly as they would through any other outage, and replay when the link returns. The observations are not lost. What is lost is control: rules and the command plane are at the core, so the remote site has no automation for the duration, and any claim on its points sits undeliverable until the link returns. This is tolerable for most things and unacceptable for one: interlock-band protection must not be placed on points at a site that can be partitioned from the core. An interlock that is unenforced whenever the link drops is not protection, and worse, it reads as protection in the rule set. A remote site needing protection that survives partition gets a local device that does not depend on the network — a mechanical thermostat, a hardware cutout, a relay with its own logic — reported into this system as a point so its state is visible, but not driven by it. That is cheaper, more reliable and easier to reason about than a distributed control plane, and it fails in the right direction. Non-goal: multiple cores This is recorded so the absence reads as a decision, with the conditions that would reopen it: Zero-downtime upgrades. Would require overlapping instances. Revisit only if measured cold-start time becomes operationally intolerable, and prefer reducing it first. Geographic redundancy. Handled above by buffering plus local protective devices. Revisit if a remote site acquires a control requirement that genuinely cannot be met locally. Both are real, neither is pressing, and a multi-core design would need distributed arbitration, distributed timers and a story for a split-brain claim set. That is a different system, and it should be entered deliberately rather than arrived at by adding a second instance one afternoon. Exit criteria Single writer is enforced rather than assumed, which means it has to be demonstrated rather than asserted in prose. A second core cannot start, and cannot steal messages before it discovers that. Start one core, start another, and assert not only that the second exits non-zero but that it never bound the JetStream consumer. The exit is the visible part; the ordering is the part that matters, and a test that only checks the exit code would pass an implementation that locks after binding. Lock loss mid-run exits the process. Drop the advisory-lock connection and assert the core stops publishing commands, stops consuming, and exits non-zero, with any uncommitted batch redelivered on the next start rather than half-applied. The epoch never goes backwards, increments only on lock acquisition, and appears on every control_event written. Together with the driver-side rejection test in Driver Contract, that closes the fencing path end to end. Drain behaves. Build a backlog, restart, and assert that points do not reach live from replayed observations, that rules stay in warmup, and that no commands are issued from data that is hours old. This is the behaviour that looks alarming and is correct, so it needs a test that says so. Two figures to measure and record rather than assert: Sustainable ingest throughput against the 500/s design rate. Everything in the buffer and recovery arithmetic waits on this number. Cold-start duration, per startup step, with rehydration broken out since it is the term that grows with the rule set. Once both exist, the stream size and the recovery time stop being estimates and the four-hour figure becomes checkable rather than aspirational. On Monday - the Dev Diary. Build step six, read against The Rules Engine, and this part's startup order read against what was built. The next part - The Security Model. Why the bus rather than the API is the primary control surface, which makes subject-scoped bus permissions the real authorisation model, what the design defends against in descending order of likelihood, and why mTLS is deliberately absent. Start of the series: An Introduction. The map, the two decisions every later document is downstream of, and why a specification at this scale is being published in public while the system it describes gets built.