test(multi-pod): cross-pod /attach + mock-ai provider scaffolding

[viktormarinho]

What is this contribution about?

Builds out the LLM-dependent half of the multi-pod test framework added in #3391. A ~140-LOC Bun mock-ai service speaks the OpenAI streaming wire protocol so mesh's existing openai-compatible adapter can drive streamText against it — no real provider budget needed. Setup helpers gain `wireMockProvider`, `createTestAgent`, and `createTestThread` for the three calls every dispatch scenario needs.

The headline scenario, `attach-cross-pod`, directly validates the deliverPolicy fix from #3387: POST a slowed-down run on pod-1, attach on pods 2 and 3, assert both see the buffered prefix. Passes locally.

The second scenario, `pod-death-dbos-replay`, is fully wired but `.skip`ped — it surfaced a real architectural finding (see below) that needs its own follow-up to fix.

Architectural finding (worth a separate issue)

The pod-death scenario times out at 120s. Two compounding gaps:

1. DBOS replay alone can't recover the run. The recovery executor re-runs `dispatchRunAndWaitStep` without `isResume: true`, which hits `claimRunStart`'s strict CAS. The dead pod still owns `run_owner_pod`, so every replay attempt fails with `RunClaimError: ... already running on another pod`.
2. Heartbeat watcher (the path that WOULD handle this via `claimOrphanedRun` + `isResume: true`) never fires. The `KV_POD_HEARTBEATS` JetStream bucket is missing from NATS, even though other JS resources initialize fine. Looks like bucket creation fails silently at `app.ts:867`.

Net: permanent pod death is currently only recovered when the dead pod itself restarts. The PR #3387 architectural claim ("DBOS replay covers what /attach orphan-resume used to do") is incomplete — full repro + diagnostic notes are in the test's docstring.

How to Test

1. Start Docker Desktop.
2. `./tests/multi-pod/run.sh` — boots cluster, runs all scenarios, tears down.
3. Expected: `6 pass / 1 skip / 0 fail` across 5 files (~5s after cluster up).

For iteration: `docker compose -f tests/multi-pod/docker-compose.yml up -d` once, then `bun test tests/multi-pod/scenarios/` repeatedly.

Migration Notes

None. Self-contained under `tests/multi-pod/` and the existing CI workflow picks up the new scenarios automatically.

Review Checklist

• PR title is clear and descriptive
• Changes are tested and working (6/6 active scenarios pass locally)
• Documentation is updated (if needed) — none; per-file docstrings carry the structure
• No breaking changes

---

Summary by cubic

Adds a mock OpenAI-compatible provider and a cross-pod /attach test to run the dispatch pipeline end-to-end across multiple pods without a real LLM, plus reliability improvements and clearer failure surfacing in the test harness.

• New Features

  • mock-ai service (OpenAI chat-completions streaming) at http://mock-ai:9000/v1; used by openai-compatible.
  • Setup helpers: wireMockProvider, createTestAgent, createTestThread; DB helper getThreadRunOwnerPod; compose POD_NAME mapping to target specific pods.
  • Scenarios: attach-cross-pod validates buffered-prefix replay across pods; pod-death-dbos-replay remains skipped with an updated note on three gaps (no cross-pod DBOS recovery scan, resume purges JetStream, heartbeat bucket fragility).

• Bug Fixes

  • mcpCall now prefers non-empty structuredContent and correctly falls back to text payloads.
  • dbQuery CLI/docstring synced to --csv; attach-cross-pod header timing corrected to slow:5x500 (~2.5s).
  • attach-cross-pod replaces a racy sleep with a JetStream-confirmation probe to deterministically test buffered replay.
  • Test hooks auto-restore stopped pods and now fail fast when docker compose ps errors, avoiding misleading waitReady timeouts.

^{Written for commit d0b9e8a. Summary will update on new commits. Review in cubic}

[cubic-dev-ai]

1 issue found across 7 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="tests/multi-pod/scenarios/attach-cross-pod.test.ts">

<violation number="1" location="tests/multi-pod/scenarios/attach-cross-pod.test.ts:121">
P2: Wait longer than the mock’s first chunk delay before attaching, otherwise this scenario can pass before any buffered content exists.</violation>
</file>

_{Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.
Re-trigger cubic}

[cubic-dev-ai]

1 issue found across 5 files (changes from recent commits).

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="tests/multi-pod/lib/hooks.ts">

<violation number="1" location="tests/multi-pod/lib/hooks.ts:33">
P2: Check the `docker compose ps -a` exit code here; otherwise compose failures are swallowed and the hook times out later with a less useful error.</violation>
</file>

_{Tip: Review your code locally with the cubic CLI to iterate faster.
Re-trigger cubic}