Updated 23 Oct 16:07:02 BKK by kookwatch. Alert sender (leader): kookwatch. Open problems that every layer failed to fix: 7
- research-box: history rebuild moving (one-time, paid) - no progress for 9.8 h (bad 8.8 h; 18 fix(es), last: restart hist-rh (rc 0))
- research-box: M picker learning loop ran (hourtrain, hourly) - 352.2 h old (max 4.0 h) (bad 341.8 h; no automatic fix exists)
- kookwatch: machine - disk 88% used, 5555 MB RAM free, load 5.31 (bad 42.7 h; no automatic fix exists)
- kookwatch: M runtrack snapshots freshness (Test Manager, from super) - 129.1 h old (max 75 min) (bad 127.9 h; 746 fix(es), last: restart runtrack.service (rc 0))
- kookwatch: M 10 s holder and buy/sell tape freshness (Test Manager, from super) - 113.8 h old (max 30 min) (bad 113.3 h; 661 fix(es), last: restart rhfast.service (rc 0))
- kookwatch: M Picker retrain freshness (Test Manager, from super) - 191.2 h old (max 36.0 h) (bad 155.2 h; 905 fix(es), last: restart hist-retrain.service (rc 0))
- kookwatch: M Launch catch-up (reconcile) freshness (Test Manager, from super) - 153.3 h old (max 10 min) (bad 153.1 h; 893 fix(es), last: restart reconcile.timer (rc 0))
Layer 1 = each box's own guard (twice a minute). Layer 2 = the other boxes call its entry over SSH every minute (revive guard, start dead units).
Telegram only when both fail. To stop a unit on purpose: systemctl disable --now X or touch /opt/guard/hold/X.
| firstmin-trader - local guard 31 s ago | |||
| AI pick feed reaches this box | OK | 0 s old | |
| L Git snapshot cron job | OK | 1 jobs present | |
| L Git snapshot ran | OK | 80 s old | |
| L live-change log cron job | OK | 1 jobs present | |
| LIVE bot fmmc5 (real money) | OK | running, heartbeat 2 s | |
| S stop rule: 10 losses in a row (paper/live test 2 only; Ben 8 Oct: none for fmmc5) | OK | 1 jobs present | |
| dashboard pnl.html (live) | OK | 12 s old | |
| dashboard pnl2.html | OK | 17 s old | |
| dashboard pnl2v.html | OK | 17 s old | |
| dashboard tests.json | OK | 21 s old | |
| gmgnd (live sells) running | OK | running | |
| gmgnd answers | OK | answers ping {"warmOk": 1816, "warmErr": 1, "lastWarmMs": 204} | |
| guard triggers | OK | cron + timer present | |
| machine | OK | disk 38% used, 402 MB RAM free, load 5.08 | |
| outside-money card | OK | 79 s old | |
| paper test 1 fml2 | OK | running, heartbeat 0 s | |
| paper test 1b fml2v | OK | running, heartbeat 2 s | |
| safety + dashboard cron jobs | OK | 12 jobs present | |
| kookwatch - local guard -1 s ago | |||
| 60 s AI picker (feeds paper tests) | OK | running, heartbeat 2 s | |
| L Git snapshot cron job | OK | 1 jobs present | |
| L Git snapshot ran | OK | 49 min old | |
| L ROT chains.json from rot-box (Manager) | OK | 31 min old | |
| L ROT closes.json from rot-box (Manager) | OK | 116 s old | |
| L ROT map-data.json from rot-box (Manager) | OK | 15 min old | |
| L ROT rh_drill.json from rot-box (Manager) | OK | 52 min old | |
| L ROT rotation.json from rot-box (Manager) | OK | 45 min old | |
| L ROT rotmap.json from rot-box (Manager) | OK | 41 min old | |
| L ROT sync main<->rot-box (Manager) | OK | 56 s old | |
| L judge guard ok (PP Data) | OK | 6 s old | |
| L lane check ok (PP Data) | OK | 0 s old | |
| L lane price tracker (PP Data) | OK | 6 s old | |
| L lane stats page (PP Data) | OK | 6 min old | |
| L live-change log cron job | OK | 1 jobs present | |
| L nightly review cron job | OK | 1 jobs present | |
| L nightly review page | OK | 8.4 h old | |
| M 10 s holder and buy/sell tape freshness (Test Manager, from super) | BAD | 113.8 h old (max 30 min) | 661 fix(es): restart rhfast.service (rc 0) |
| M AI picker round 5 (paper) freshness (Test Manager, from super) | OK | 0 s old | |
| M Ben Telegram buttons (benbot) | OK | running | |
| M Ben Telegram line in cron | OK | 2 jobs present | |
| M Launch catch-up (reconcile) freshness (Test Manager, from super) | BAD | 153.3 h old (max 10 min) | 893 fix(es): restart reconcile.timer (rc 0) |
| M Live job checker (sentinel) freshness (Test Manager, from super) | OK | 24 s old | |
| M PP Feeds layer-2 guard (ppfeedguard2) | OK | running, heartbeat 14 s | |
| M PP data cron lines (PP Data) | OK | 3 jobs present | |
| M PP data liquidity table (liqinc) (PP Data) | OK | 43 min old | |
| M PP data picker-box copy (pbsync pull) (PP Data) | OK | 11 s old | |
| M PP data ponstape (PP Data) | OK | running | |
| M PP data ppexit honest-exit tracker (PP Data) | OK | -1 s old | |
| M PP data venuepick paper picker (PP Data) | OK | 0 s old | |
| M PP feed GMGN board (price-predict-rh) | OK | running, heartbeat 0 s | |
| M PP feed crewlive (crews) | OK | running, heartbeat 24 s | |
| M PP feed klines | OK | running, heartbeat 97 s | |
| M PP feed ponstape (Pons curve) | OK | running, heartbeat 13 s | |
| M PP feed rhledger | OK | running | |
| M PP feed runtrack (runners) | OK | heartbeat 129.1 h old but log live (not hung) | |
| M Picker guard (pickguard) freshness (Test Manager, from super) | OK | 6 s old | |
| M Picker retrain freshness (Test Manager, from super) | BAD | 191.2 h old (max 36.0 h) | 905 fix(es): restart hist-retrain.service (rc 0) |
| M SW engine w2coder (paper) | OK | running, heartbeat 23 s | |
| M bot watchdog freshness (Test Manager, from super) | OK | 4 min old | |
| M cleaner freshness (Test Manager, from super) | OK | 7 min old | |
| M daily miss review in cron | OK | 1 jobs present | |
| M gmgn feeds recorder freshness (Test Manager, from super) | OK | 70 s old | |
| M instant miss watch ran | OK | 109 s old | |
| M lane price tracker (lanetrack) freshness (Test Manager, from super) | OK | 6 s old | |
| M lane radar v3 (laneradar3) freshness (Test Manager, from super) | OK | 0 s old | |
| M lane radar v3 beat (PP Data) | OK | 0 s old | |
| M lane stats page (lanestats) freshness (Test Manager, from super) | OK | 6 min old | |
| M meta radar (metaradar) freshness (Test Manager, from super) | OK | 22 min old | |
| M paper test (pp-paper) freshness (Test Manager, from super) | OK | 29 s old | |
| M picker learning loop cron (hourly) | OK | 1 jobs present | |
| M ppinc hourly new pools + supply (PP Data) | OK | 3 min old | |
| M rate-limit review in cron | OK | 2 jobs present | |
| M rate-limit review ran (heartbeat) | OK | 41 min old | |
| M runtrack snapshots freshness (Test Manager, from super) | BAD | 129.1 h old (max 75 min) | 746 fix(es): restart runtrack.service (rc 0) |
| S AI picker (paper) freshness (Test Manager, from super) | OK | 2 s old | |
| S P3 mirror cross (a1pull@cross) | OK | running | |
| S P3 mirror early (a1pull@early) | OK | running | |
| S backup checker (livewatch2) freshness (Test Manager, from super) | OK | 6 s old | |
| S fast checker (livewatch) freshness (Test Manager, from super) | OK | 2 s old | |
| S fast checker livewatch | OK | running, heartbeat 2 s | |
| S live scanner rhfast (feeds Price Predict) | OK | running, heartbeat 78 s | |
| S live settings watchdog | OK | 1 jobs present | |
| S live settings watchdog ran | OK | 113 s old | |
| backup picker aipick5b stays off | OK | off (as decided) | |
| drpc_paid.json is free | OK | points to robinhood.drpc.org | |
| guard triggers | OK | cron + timer present | |
| machine | BAD | disk 88% used, 5555 MB RAM free, load 5.31 | |
| paid key: no stray copies | OK | paid key only in the opt-in files (scanned 37 min ago) | |
| paid key: only allowed processes | OK | only allowed processes use the paid key | |
| pick feed file l2picks.json | OK | 0 s old | |
| pick feed l2feed | OK | running, heartbeat 0 s | |
| published tests dashboard | OK | 48 s old | |
| rpc_keys.json is free | OK | points to robinhood.drpc.org | |
| rpcpool.py healthy | OK | compiles | |
| testspub cron job | OK | 1 jobs present | |
| picker-box - local guard 1 s ago | |||
| L Git snapshot cron job | OK | 1 jobs present | |
| L Git snapshot ran | OK | 49 min old | |
| M lab paper books | - | on hold (/opt/guard/hold/labbooks) | |
| M shadow picker pbshadow | OK | running | |
| free RPC relay (2nd IP) | OK | running | |
| guard triggers | OK | cron + timer present | |
| machine | OK | disk 33% used, 542 MB RAM free, load 0.28 | |
| rpc_keys.json is free | OK | points to robinhood.drpc.org | |
| research-box - local guard 2 s ago | |||
| L Git snapshot cron job | OK | 1 jobs present | |
| L Git snapshot ran | OK | 49 min old | |
| L kitb | OK | 5 min old | |
| L shadow grid report (Test Manager, nightly 02:30) | OK | 13.1 h old | |
| L trending feed | OK | 60 s old | |
| M KITFAST kit feed (PP Feeds) | OK | running, heartbeat 3 s | |
| M PP Feeds layer-1 guard (ppfeedguard) | OK | running, heartbeat 5 s | |
| M auditq ran | OK | 6 min old | |
| M crew tagger pretag | OK | running, heartbeat 16 s | |
| M datajob runner ran | OK | 59 s old | |
| M flowwatch ran | OK | 0 s old | |
| M ideaq ran | OK | 3 min old | |
| M killbot ran | OK | 55 min old | |
| M paper cards ran | OK | 118 s old | |
| M picker learning loop ran (hourtrain, hourly) | BAD | 352.2 h old (max 4.0 h) | |
| M v3 live feed v3live | OK | running, heartbeat 5 s | |
| M v3 mc5 paper book | OK | running, heartbeat 3 s | |
| S A1 early + cross feed (a1feed) | OK | running, heartbeat 0 s | |
| drpc_paid.json is free | OK | points to robinhood.drpc.org | |
| guard triggers | OK | cron + timer present | |
| history rebuild moving (one-time, paid) | BAD | no progress for 9.8 h | 18 fix(es): restart hist-rh (rc 0) |
| machine | OK | disk 88% used, 14140 MB RAM free, load 3.06 | |
| paid key: no stray copies | OK | paid key only in the opt-in files (scanned 12 min ago) | |
| paid key: only allowed processes | OK | only allowed processes use the paid key | |
| rebuild moves to free when done (cron) | OK | 1 jobs present | |
| rpc_keys.json is free | OK | points to robinhood.drpc.org | |
| rpcpool.py healthy | OK | compiles | |