feat(dispatch+spec): liveness, retry, admission control (ADR-0048/0049/0050/0051)

[rita-aga]

Summary

Architectural response to the 2026-04-17 Katagami bulk-regenerate incident (8 of 11 sessions hit ask timeout, 2 stuck in Provisioning forever) and the Railway content-file HTTP 500. Four new platform primitives, each with its own ADR, delivered together so the class of bug is structurally impossible going forward.

• ADR-0048 — Dispatch retry + error taxonomy. ActorError::is_transient/is_permanent; DispatchError reshaped into Transient { source, attempts } / Permanent { source } / Deferred { retry_after_ms }. New retry::ask_with_backoff wraps every entity ask site (1 in dispatch, 6 in entity_ops). HTTP 503 + Retry-After on transient exhaustion. Idempotency-Key threaded into EntityMsg::Action; actor consults IdempotencyCache before executing and caches the successful response so retry races cannot double-execute.
• ADR-0049 — First-class state-entry timeouts. New [[state_timeout]] TOML block (state / after_seconds / on_timeout / max_occurrences / reset_on / params). Compiler auto-wires the target action's from list. Runtime StateTimeoutTracker + arm_state_timeouts_if_needed hooked into run_post_dispatch_effects; sequence-based cancellation on state exit + reset_on re-arm.
• ADR-0050 — Mandatory liveness coverage. Spec compiler rejects any non-terminal state that lacks a [[state_timeout]] or an allow_indefinite_states entry. Env-flagged via TEMPER_LIVENESS_ENFORCE (default warn-only for rollout). New verify_specs binary for CI. Violation reporter hook emits temper_spec_liveness_violations_total.
• ADR-0051 — Per-tenant admission control. AdmissionController with per-(tenant, entity, action) Tokio semaphores. Strict FIFO, test-enforced (100 interleaved acquirers, order matches arrival). [admission] spec block declares caps; enforced before ask, pulled inline from the spec registry per call — no separate registration step. /_admin/admission/{tenant}/{entity_type} PATCH endpoint for runtime overrides.

Observability

22 new OTel instruments in runtime_metrics (dispatch outcome/attempts/latency/errors; state_timeout fired/cancelled/reset; scheduler pending/overdue; spec liveness violations; admission granted/queued/deferred/wait/permits/depth/hold; actor mailbox depth/utilization/full-drops/reply latency; curation_job outcome). record_server_state_metrics samples mailbox depth + utilization per entity type.

Test plan

• 295 temper-server lib tests, 216 temper-spec, 94 temper-runtime, 48 temper-platform — 653 passing, 0 failures
• Integration test: state_timeout_fires_and_transitions_entity proves the runtime scheduler arms → sleeps → fires on_timeout → transitions the entity end-to-end
• Integration test: admin_override_applies_and_clears_caps round-trips the admin endpoint via the live router
• Load test: 120 concurrent dispatches, cap=5, queue_timeout=10s → 120 granted, 0 deferred, 0 errors, ~33k dispatches/sec, p99=2.66ms
• Load test: 300 adversarial burst, cap=2, queue_timeout=0s → 56 granted, 244 deferred (503 Retry-After), 0 errors. This is the incident-replay contract firing in a test: mailbox-full storms cannot happen.
• Load test: 1000 sustained, cap=50 → 1000 granted, 0 errors, ~20k/s
• verify_specs binary run against OpenPaw's os-apps/ — all specs pass enforce mode after migration
• Full-workspace cargo build --all-targets clean
• temper serve boots clean, PATCH /_admin/admission/... returns 200, liveness warnings emit to the log
• Enforce-mode refuses to boot a non-compliant tenant — contract validated in production-path binary, not just unit tests

Deferred (task-tracked, explicitly out of scope)

• Durable event-log-backed scheduler (ADR-0049 phase 2). Current impl is non-durable tokio::spawn + in-memory tracker — same durability profile as today's schedule effects (ADR-0012 line 138).
• Per-actor timer heap (plan specified BinaryHeap; shipped with spawn-per-timer).
• Auto-generated {state}_entered_at / {state}_timeout_seq entity state vars (shipped with external tracker — correctness-equivalent).

Companion OpenPaw PR

OpenPaw branch feat/liveness-retry-admission migrates Session + CurationJob specs to the new primitives, deletes heartbeat_scan/heartbeat_scheduler infrastructure (superseded by state_timeout), adds the Datadog dashboard/monitor widgets referencing the new metrics. That PR depends on this one.

🤖 Generated with Claude Code