S26 · Multi-Modal Perception

Declared state: the seam is built and gated, the hardware is not — mock-only evidence, no detector and no camera.

This is the repository plan and evidence, not a live health reading. A task marked done may still depend on credentials, devices, deployment or a newer acceptance test.

Dependencies

S27

Tasks and gates

26.F1 · Vision on demand with a VLM path. *gate:* `test_vision.py`, `test_screenshot_transport.py`

Tier F · done · Verification: hermetic-test

Gate: `test_vision.py`, `test_screenshot_transport.py`

  • unverified-import · undated · docs/SYSTEMS-DETAILS.md:2856

26.F2 · A single `perceive()` snapshot that returns presence + visual + activity together, so

Tier F · done · Verification: hermetic-test

Gate: `test_perception_snapshot.py` — 36/36, including the structural check that the

  • unverified-import · undated · docs/SYSTEMS-DETAILS.md:2857

26.F3 · Every perception carries a freshness stamp; stale perception is never presented as

Tier F · done · Verification: hermetic-test

Gate: `test_perception_snapshot.py` [staleness]

  • unverified-import · undated · docs/SYSTEMS-DETAILS.md:2866

26.R1 · Audio scene classification — speech, music, TV, silence — feeding the speaker gate.

Tier R · open · Verification: planned-test

Gate: new `test_speaker_verdict.py` - planned gate. `test_speaker_verdict.py` [scene]

    26.R2 · Screen understanding: what application, what document, what state.

    Tier R · deferred · Verification: hermetic-test

    Gate: `test_vision.py` extended.

    • owner-decision · undated · docs/DECISIONS.md D-46

    26.R3 · Anomaly detection: something unusual in the room or on the screen is *noticed*.

    Tier R · deferred · Verification: hermetic-test

    Gate: `test_perception_snapshot.py` [anomaly]

    • owner-decision · undated · docs/DECISIONS.md D-46

    26.E1 · "What am I looking at / what's going on here" answered correctly ≥90% across a mixed

    Tier E · open · Verification: hermetic-test

    Gate: `test_vision.py` fixture suite.

      26.R4 · **A one-way event stream for the surfaces that need it** — SSE over the existing HTTP server, because a 15-second poll cannot show a live turn.

      Tier R · open · Verification: planned-test

      Gate: new `test_hud_stream.py` — the stream is one-way and text, adds no fourth network boundary and no second WebSocket, is gated by the same token as `/hud.json`, survives a dropped client without leaking a generator, and the page falls back to polling when it is absent.

        26.R5 · No video frame leaves the house: detection runs on the local accelerator and only event metadata — zone, label, count, timestamp — may reach the brain. The cloud vision chain serves screenshots the owner asks for and nothing else (D-61, D-66).

        Tier R · in-progress · Verification: hermetic-test

        Gate: new `test_no_frame_egress.py` — with egress blocked the detector still produces events; a frame cannot reach a remote model, a log or a crash report; and the see() path is not reachable from the detector's output channel.

        • hermetic-green · undated · bench/test_no_frame_egress.py

        HUD screen contracts

        All subsystems · Source: docs/systems-plan.json.