Gemma couldn't tell my robot was upside down, so I gave it a head camera. Now I sleep while it trains.

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 3: Slow and Steady At step 100,000, my robot was lying on its back with its legs in the air, kicking. The reward curve said it was having its best run yet. step 0 step 100,000 I'd written a reward term to keep its body level and got the sign wrong, so it was paid for tilting. Over 600,000 training steps the reward climbed from 908 to 3,525. Every alert I'd normally set (reward dropped, KL divergence spiked, loss exploded) stayed quiet, because nothing went down. TL;DR: Night Watch watches a robot-learning run so you can sleep through it. A local Gemma 4 looks through a camera bolted to the robot, TabPFN reads the training metrics, plain rules catch crashes, and a Temporal workflow keeps the watch alive even when the watcher itself dies. It wakes you at most 3 times a night; everything else waits for a one-page morning report. Open models only, on a laptop with no GPU card, $0 a night. What I Built I'm a robotics master's student. Robot learning in simulation is slow and steady by nature: a policy trains for hours, so you start it, go to bed, and wake up at 2 a.m. to check on it, because runs die quietly (a hung loader, a NaN, a crash) or go wrong while looking great, like the one above. Night Watch does the 2 a.m. check for me: Gemma 4 (E4B, through Ollama) looks through a head camera on the robot and says where the floor and the sky are; TabPFN-3.5 reads the training metrics and asks whether they look like past runs that broke; plain rules catch NaNs, a silent log and a dead process; a Temporal workflow runs the checks, and survives its own worker dying. It sends a notification (ntfy or Telegram) only when it's worth waking up for: at most 3 times a night, and never twice for the same problem within an hour. Everything else goes into a morning report with a time-lapse of the run, the reward curve with every alert marked, and a three-sentence summary. Demo The video is a second live run (25 Oct, same flip bug), and it went less cleanly. TabPFN raised the first alert at step 98,304 (p=0.98), but it pointed at training speed and value loss while the laptop was also recording and running out of memory, so I don't count it as a clean catch. Gemma's first looks failed for lack of memory; it woke me at step 401,408. The time-lapse from the first run's morning report, one frame per test episode across 600,000 steps: What woke me: 00:38 step 100,000 [vision/robot_looks_wrong] The robot looks wrong in the last 2 test runs: posture=upside_down. Gemma: "head camera, floor/sky: top/bottom, top/bottom, top/bottom, top/bottom" 00:38 step 106,496 [tabpfn/metrics_anomaly] Training metrics look like past runs that broke (p=0.86). Code su0as / night-watch Sleep while your robot learns. TabPFN reads the training metrics, Gemma 4 watches the robot, Temporal keeps the watch alive. Hacktoberfest 2026. Night Watch Sleep while your robot learns. Two open models watch an overnight reinforcement-learning run, and you only get woken when it is worth it. TabPFN reads the training metrics and compares them with past runs that broke. Gemma 4, running locally through Ollama, looks through a camera bolted to the robot's head. A few plain rules catch NaNs, stalls and crashes. A Temporal workflow keeps the watch going even if the watcher process dies at 3 a.m. In the morning you get a one-page report: a time-lapse of the night, the reward curve, and what (if anything) went wrong. Built for the DEV x Hacktoberfest 2026 Open-Source AI Challenge, Week 3: Slow and Steady. The repository was started and completed during the challenge window (19 to 25 October 2026). The failure this is built for A real 600k-step run on a Windows laptop (i7-1165G7, no NVIDIA GPU)… View on GitHub MIT licensed, built during the challenge. 32 tests, including a full 8-hour night through Temporal's time-skipping test server in a few seconds. GitHub Actions runs them on Linux and Windows on every push, plus short real training runs with injected faults. How I Built It trainer (PPO + MuJoCo HalfCheetah) watcher (a Temporal workflow, a check every few minutes) every PPO update -> metrics.jsonl ------> rules NaN? log gone silent? process dead? every 20k steps -> a short test run ----> TabPFN do these metrics look like past runs that broke? 4 head-camera ------> Gemma 4 where are the floor and the sky? (code decides the posture) snapshots heartbeat.json -> alive? policy wake me (max 3 a night) or note it for the morning labels/ (hidden) -> ground truth report time-lapse, reward chart, summary with checked numbers A way to know when it's actually right The trainer writes the files a watcher may read (metrics, snapshots, a heartbeat) and a labels/ folder it may not: the fault I injected and when, and the robot's true body angle from the physics engine. Only the evaluation reads labels/, so "Gemma caught the flip" is checked against the simulator, not against Gemma. Then I broke things on purpose: a flip bug and a spin bug (shaping terms with the sign flipped), a learning rate x200 mid-run, a physics blow-up (timestep x4, motor gear x3), a hung training loop and a silent crash. Gemma couldn't see it. So the robot got a head camera. My first version showed Gemma the obvious picture: the side view, four snapshots in a 2x2 grid. Before trusting it with a night, I tested its eyesight with nightwatch eyetest: drop the robot into a known pose (the right way up, on its back, or crouching), let the physics settle it, and ask the same judge that runs at night. The simulator knows the answer. It called every robot "upright", including the six lying on their backs. Bigger pictures didn't help (Ollama gives Gemma 4 about one image token per 48x48 pixels, so my grid snapshots were a few dozen tokens each; at 1024x768, 336 tokens, still "upright"). Neither did other questions ("what is touching the floor?"), or thinking mode: six minutes per look, still wrong. The 12B model doesn't fit my laptop's GPU. Then I treated it as a robotics problem instead of a prompt problem. A real robot doesn't work out which way up it is from a photographer standing beside it; it has its own sensors. So I bolted a camera to the robot's torso, in a render-only copy of the model built with MuJoCo's MjSpec, so the physics the policy learns on is untouched. The camera draws only the world, not the robot, like an artificial horizon. When the robot flips, its camera flips with it, and the question becomes one a 4B model can answer: where is the floor in this picture? 18 test episodes (6 upright, 6 upside down, 6 crouching or rearing, which count as upright), timed on my laptop's Intel Iris Xe: what Gemma is shown right calls upside-down robots caught seconds per test episode side camera, 2x2 sheet (first version, 12 episodes) 6 / 12 0 / 6 10 side camera, full-size snapshots 12 / 18 0 / 6 22 head camera, E4B 18 / 18 6 / 6 25 head camera, the smaller E2B 12 / 18 0 / 6 18 Gemma answers one question per snapshot in a fixed JSON schema (floor and sky: top half, bottom half, whole picture, a side, not visible). Plain code turns the four answers into a posture: sky below floor means upside down, the picture flipping between snapshots means spinning. Gemma never decides to wake me, and the same problem has to show up in two test episodes in a row. TabPFN reads the metrics Every window of 8 training-log rows becomes one table row of 22 numbers: how far the reward is below its best, KL divergence and value loss relative to their usual level, the learning-rate ratio and so on, labelled 1 if the run was already broken at that point. TabPFN-3.5 learns in context from that table, so there's no extra training run to babysit (which would have been ironic). It runs locally on the laptop CPU. The first version was too slow for a live watcher: a 7-second fit, but about 96 seconds per prediction, because TabPFN re-processed the whole table every time. Reading TabPFN's source turned up fit_mode="fit_with_cache": one 92-second fit at the start of the night, then under a second per check. Temporal keeps the watch alive The watch is a Temporal workflow: check, decide, maybe alert, wait on a durable timer, repeat until morning. Checks are activities with retries, so Ollama loading a model is just a retry. Every 150 checks it continues as new, carrying who woke me and when, and snooze, stop and status work from the command line. The test that matters is killing the watcher mid-watch. In the first run, with a check every 60 seconds: 00:38 woke me for the flip; 00:42:00 I killed the worker. Training kept going, and the workflow stayed "Running" on the Temporal server with nobody executing it; 00:42:15 its timer fired on schedule anyway, with no worker to hand the next check to; 00:46:15 I restarted the worker. It picked up from its last completed check and still knew it had already woken me, so the same problem at step 500,000 went into the morning notes instead of waking me twice; 00:49 the morning report went out. The second run on video did the same: worker killed at 11:12:21, timer fired at 11:12:58 with no worker, the new worker picked it up at 11:16:06. A plain while True: sleep(300) loop would have died silently at 00:42, which is the exact failure Night Watch exists to catch. A morning summary that can't make things up Gemma writes the summary, but only from facts the code computed. A check pulls every number out of its paragraph, and if any number isn't in the facts, the paragraph is replaced by a plain template. The report says which one you got. The first run's summary is Gemma's, and it passed: The killtest run in HalfCheetah-v5 finished with 598016 steps done. The reward increased from 908 to 3525, matching the best reward. A problem was noted at 00:38 where the robot looked upside down in the last two test runs. Results On 4 short runs with every watcher (healthy, flip bug, LR blow-up, crash), Night Watch caught all 3 failures; the thresholds and Gemma caught 2 each, TabPFN 1. The rules caught the crash, the thresholds the LR blow-up, and only Gemma the robot on its back. Per test episode, Gemma matched the simulator on all 23 (6 alarms, 17 fine). Then a fair test for TabPFN: 12 runs of 400,000 steps (4 healthy, 2 flip, 1 spin, 2 LR blow-ups, 1 physics, 1 stall, 1 crash), replayed leave-one-run-out through the thresholds, TabPFN, and a gradient-boosting baseline on the same 22 features (docs/EVAL.md, docs/EVAL_gbm.md). Neither model was tuned: TabPFN-3.5 and scikit-learn's HistGradientBoostingClassifier both ran with default settings. watcher failures caught false wakes in total on the 4 healthy runs AUROC (2,144 log windows) thresholds 6 / 8 5 2 n/a gradient boosting 6 / 8 4 1 0.888 TabPFN 6 / 8 1 0 0.93 TabPFN and gradient boosting caught the same 6 runs; the difference is how often they woke me for nothing. TabPFN's median time to wake was 7.8 minutes, against 6.3 for the thresholds. The training step alone ("how far into training are we") scores AUROC 0.793, so part of any score is knowing that failures come later in a run; without the training-progress feature, TabPFN still scores 0.932 and gradient boosting 0.89. The thresholds and TabPFN caught different runs, and the split by kind is the interesting part: Training faults (both LR blow-ups, the stall, the crash): both watchers caught 4 of 4. The LR blow-ups took seconds; the stall and crash took 12 minutes, through the plain rules. The robot itself going wrong: each caught only 2 of 4, with a median of over an hour (75 minutes for the thresholds, 70 for TabPFN). TabPFN caught one flip-bug run the thresholds missed (after 136 minutes) and the other after 3.5 minutes instead of 150. In the physics run the robot flipped onto its back by itself at step 180,000, before the physics fault fired; the thresholds woke within a minute of that, TabPFN missed it. Neither caught the spin-bug run, which also ended up on its back. Metrics are good at noticing broken training and slow at noticing a robot doing the wrong thing. That gap is what the head camera is for. What went wrong, and what it can't see The floor was a mirror. Gymnasium's floor has a reflectance of 0.5, so every snapshot also held an upside-down robot. The snapshots now use a matte floor. My first head camera caused a false alarm. I mounted it on the robot's nose; a healthy run learned a nose-down crouch, the camera went through the floor, and Gemma faithfully reported the floor in the top half. It now sits mid-torso, and the eyesight test includes crouches. My first 12-run set was flawed. Seven runs shared a seed, and three of the four "healthy" runs learned to run on their backs by themselves (a robot doesn't need a bug to find that trick). I threw it away and re-recorded with a seed per run and a correct level-keeping term for the healthy runs. 12 runs of one robot is still small. A different robot or algorithm needs its own examples first (nightwatch fit on your own runs). The head camera answers one question well: which way up. It won't notice a robot that's upright but limping. And if "is it upside down" is all you want, log the torso angle; the watcher is for the things you didn't write a check for. The first run was a compressed night: a 13-minute watch of a 600,000-step run with a check every minute, so I could kill the worker without waiting for 3 a.m. Why Does Open Innovation Matter? The pictures stay home. These are frames from unpublished research, hundreds a night, and none of them leave my laptop. It runs where the compute is. Lab and cluster machines often have no outbound internet; Ollama, TabPFN's local weights and the Temporal dev server all run offline. It costs nothing per night, so I never ration how often it looks. I could open the box when it failed. I found Gemma's blindness to the side view only because I could test it cheaply and repeatedly against simulator ground truth, and the fix came from owning the whole stack, renderer included. Reading TabPFN's source is how a check went from 96 seconds to under one. Where a closed model would be better: judging the side view; a frontier vision model would very likely see the flip my 4B model couldn't. It would also mean a bill per look and research frames leaving my machine every night. The Lazy Payoff The run it caught: flagged at step 100,000 of 600,000. Five-sixths of that compute went into teaching a robot to lie on its back; on a real night, that's a notification minutes after it rolls over instead of finding out over breakfast. The cost of watching: under a second of CPU per TabPFN check, about a minute of laptop GPU per Gemma look, $0. It survives me: I killed the watcher mid-run in both live runs, and it carried on without waking me twice. It's one week old, so the evidence is two watched runs, an eyesight test and 16 recorded runs, not a month of nights. My next long run starts before bed, with the watch on. Prize Categories Best Use of TabPFN: TabPFN-3.5 (open weights, local CPU) is the metrics watcher inside a live agent: in-context learning over past runs' log windows, scored leave-one-run-out against simulator ground truth (12 runs: AUROC 0.93 vs 0.888 for untuned gradient boosting on the same features; 6 of 8 failures with 1 false wake, vs 4 for gradient boosting and 5 for thresholds), under a second per live check with fit_with_cache. Best Use of Gemma: Gemma 4 E4B (laptop iGPU via Ollama) reads a head camera bolted to the robot in a fixed JSON schema; code turns that into a posture. Its eyesight is measured against the simulator, including the side view it couldn't judge. It also writes the number-checked morning summary. Best Use of Temporal: durable timers, retried activities for flaky model calls, continue-as-new with carried state, stop/snooze signals and a status query. It survived its worker being killed in both live runs. Best Use of GitHub Copilot (GitHub Actions): every push runs the tests, an 8-hour Temporal night in seconds, and real short training runs with injected faults, on Linux and Windows. Built solo.