S31 · Self-Monitoring, Diagnostics & Self-Healing

Declared state: complete for now.

This is the repository plan and evidence, not a live health reading. A task marked done may still depend on credentials, devices, deployment or a newer acceptance test.

Dependencies

No dependencies are declared in the catalog.

Tasks and gates

31.F1 · Four health surfaces that **agree**: structural check, functional probe, HUD, and the

Tier F · done · Verification: hermetic-test

Gate: `test_health_agreement.py`

  • unverified-import · undated · docs/SYSTEMS-DETAILS.md:3267

31.F2 · Typed error taxonomy with turn correlation. *gate:* `test_error_taxonomy.py`,

Tier F · done · Verification: hermetic-test

Gate: `test_error_taxonomy.py`,

  • unverified-import · undated · docs/SYSTEMS-DETAILS.md:3269

31.F3 · Tool-level reliability learned from the audit trail. *gate:* `test_tool_reliability.py`

Tier F · done · Verification: hermetic-test

Gate: `test_tool_reliability.py`

  • unverified-import · undated · docs/SYSTEMS-DETAILS.md:3271

31.F4 · Every background loop appears in the HUD with its last tick, its period and its budget —

Tier F · done · Verification: planned-test

Gate: new `test_loop_registry.py`

  • unverified-import · undated · docs/SYSTEMS-DETAILS.md:3272

31.R1 · Trend detection: degrading before failing (rising p95, rising retry rate) raises a

Tier R · done · Verification: hermetic-test

Gate: `test_metrics.py` [trend] - a rising p95 (newer vs older half of the window) or a rising LLM-failover / tool-error rate is named in words and becomes a routine health signal.

  • hermetic-green · 2026-10-17 · bench/test_metrics.py

31.R2 · Self-repair coverage extended to the top five recurring failure shapes from the journal.

Tier R · open · Verification: hermetic-test

Gate: `test_self_repair.py`

    31.R3 · A weekly self-report: what broke, what repaired itself, what needs the owner.

    Tier R · in-progress · Verification: hermetic-test

    Gate: `test_protocol_reports.py` extended.

    • hermetic-green · undated · bench/test_self_report.py

    31.R4 · **One correlation id per turn**, generated at the edge and stamped on every brain stage,

    Tier R · done · Verification: hermetic-test

    Gate: `test_correlation_id.py` [8][9] - one turn, one id at every stage (turn trace, audit line, error journal, the command the laptop receives); two turns in flight never share one.

    • hermetic-green · 2026-10-16 · bench/test_correlation_id.py

    31.R5 · **Governors: the budgets act, not just report.** Prefill tokens, catalogue tokens and p95

    Tier R · open · Verification: planned-test

    Gate: new `test_governors.py` - planned gate. new `test_governors.py` — each breach triggers its named degradation, the degradation

      31.R6 · **Continuous evaluation of real turns.** Every tier in this document tests what we thought

      Tier R · open · Verification: planned-test

      Gate: new `test_continuous_eval.py` - planned gate. new `test_continuous_eval.py` — a scored turn per real turn, an outcome label per

        31.E1 · A month where every incident was self-detected before the owner noticed.

        Tier E · open · Verification: manual

        Gate: incident log review.

        • recorded-note · undated · bench/edge_readiness.py (2026-10-04 diagnosis)

        31.F5 · **One correlation id, edge → brain → tool → audit, with no gaps** — instrument before observing. Jev p=0.93, conf 0.87: traces you cannot join prove nothing, and this is the precondition for every runtime-evidence claim the other forty-nine need.

        Tier F · done · Verification: hermetic-test

        Gate: `test_correlation_id.py` — one id is present and identical at all four hops (edge mint, brain adopt, turn trace, audit line); an uncorrelated turn records a blank id rather than inventing one, so a gap stays countable.

        • recorded-note · 2026-09-20 · TODO.md

        31.F6 · **Measure the real RSS and write that figure down** — replace the 250 MB ceiling with the measured number. Jev p=0.87, conf 0.83: an unrealistic gate is a gate that gets disabled.

        Tier F · done · Verification: hermetic-test

        Gate: new est_rss_ceiling.py - the ceiling in the gate EQUALS the measured figure recorded on the same day, with headroom stated

        • recorded-note · undated · scripts/preflight.sh --unpark
        • hermetic-green · 2026-10-11 · bench/test_rss_ceiling.py

        31.R7 · **Continuous behavioural evaluation** — measuring what Mushegh actually does over time in real use. Scored 3.60/4 conf 0.67, *the precondition for everything else*, and the capability with the strongest case for its own number (p=0.65) — but built as a raise here, because Jev also said none of the six should become systems (p=0.45) and numbering adds bookkeeping rather than capability.

        Tier R · open · Verification: planned-test

        Gate: new `test_continuous_eval.py` — real turns are scored against recorded expectations on a schedule, and a regression against the previous window fails.

          31.R8 · The four managed CLIs are kept current by a 24h job that installs a new release and proves it with a ping, rolling back a release that fails.

          Tier R · in-progress · Verification: hermetic-test

          Gate: `test_toolchain.py` — the rollback bound, the credential allowlist, and silence on a quiet run.

          • hermetic-green · undated · bench/test_toolchain.py

          31.R9 · A 168h improvement review: Jev ranks code-declared improvements per CLI and the CLIs are asked themselves. Advisory only — nothing is installed.

          Tier R · in-progress · Verification: hermetic-test

          Gate: `test_toolchain.py` — options come from code, and the module has no install path.

          • hermetic-green · undated · bench/test_toolchain.py

          HUD screen contracts

          All subsystems · Source: docs/systems-plan.json.