fix(decopilot): cross-pod recovery actually fires on pod death

[viktormarinho]

What is this contribution about?

Makes the previously-skipped `pod-death-dbos-replay` scenario pass end-to-end. Three coordinated fixes, each independently load-bearing — the recovery path was broken in three places at once.

1. Heartbeat watcher gains a polling fallback for hard kills. NATS KV's TTL-based key expiry is server-side cleanup and does NOT emit DEL/PURGE on the watcher — `kv.watch()` only catches graceful `kv.delete()`. A SIGKILL'd pod's heartbeat vanished silently and `handlePodDeath` never fired. Added a parallel `kv.keys()` poller (~10s tick) that diffs snapshots; the watcher still handles graceful shutdowns, the poller catches the rest. Detection latency: ~55s worst case.

2. Heartbeat-recovered runs now pump to JetStream. `resumeOrphanedThread` was deliberately dropping `streamBuffer` when calling `dispatchRunAndWait`, on the assumption that DBOS workflow replay would carry chunks separately. DBOS doesn't replay cross-pod, so the resumed run streamed to /dev/null and survivor `/attach` tails saw nothing. Pass `streamBuffer` through.

3. `prepareRun` skips JetStream purge on resume. `streamBuffer.purge()` ran unconditionally at every dispatch start, including the resume path — wiping the prefix chunks survivor watchers were mid-consumption of. Gate on `!input.isResume`.

Future cleanup left for follow-up: replace the NATS-KV liveness path with Postgres advisory locks (sub-second detection vs ~55s) — already discussed, deserves its own PR.

How to Test

```bash
./tests/multi-pod/run.sh
```

Expected: `7/7 pass`. The `pod-death-dbos-replay` scenario was `.skip`ped before this PR; it's now un-skipped and exercises the full recovery loop (POST → run flowing → SIGKILL owner → poller detects → recovery on survivor → /attach sees final chunk).

Migration Notes

The mesh image is now tagged `multi-pod-studio:latest` and shared across all three mesh services in test compose, instead of being built three times under three different tags. No production impact — only affects test infrastructure.

Review Checklist

• PR title is clear and descriptive
• Changes are tested and working (7/7 pass locally including the previously-skipped pod-death scenario)
• No breaking changes — heartbeat polling is additive; `streamBuffer` was always optional in deps; the `!isResume` guard is a strict subset of previous behavior

---

Summary by cubic

Fixes cross-pod recovery so pod death is detected (including after NATS reconnects) and orphaned runs resume with output visible to clients. Late /attach sees the response once (no duplicates).

• Bug Fixes

  • Heartbeat adds a KV poller (kv.keys() every ~10s) to catch hard kills missed by kv.watch(); worst-case detection ~55s.
  • Heartbeat now re-arms the watcher and poller after NATS reconnects, so death detection doesn’t silently stop.
  • Resumed runs publish to JetStream by passing streamBuffer to dispatchRunAndWait.
  • Keep unconditional streamBuffer.purge() on dispatch (including resume) to prevent duplicate replies for late /attach; in-flight watchers may still render duplicates and need a follow-up.

• Migration

  • No production changes. Test compose shares one image tag multi-pod-studio:latest across mesh services.

^{Written for commit 3b61478. Summary will update on new commits. Review in cubic}

[github-actions]

🧪 Benchmark

Should we run the Virtual MCP strategy benchmark for this PR?

React with 👍 to run the benchmark.

Reaction	Action
👍	Run quick benchmark (10 & 128 tools)

Benchmark will run on the next push after you react.

[github-actions]

Release Options

Suggested: Patch (2.330.3) — based on fix: prefix

React with an emoji to override the release type:

Reaction	Type	Next Version
👍	Prerelease	2.330.3-alpha.1
🎉	Patch	2.330.3
❤️	Minor	2.331.0
🚀	Major	3.0.0

Current version: 2.330.2

Note: If multiple reactions exist, the smallest bump wins. If no reactions, the suggested bump is used (default: patch).

[cubic-dev-ai]

1 issue found across 6 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="apps/mesh/src/nats/pod-heartbeat.ts">

<violation number="1" location="apps/mesh/src/nats/pod-heartbeat.ts:118">
P2: Starting the poller in parallel with the watcher can emit duplicate pod-death callbacks for graceful deletes, because watcher notifications don’t synchronize `knownPods` before the next poll diff.</violation>
</file>

_{Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.
Re-trigger cubic}

[cubic-dev-ai]

P2: Starting the poller in parallel with the watcher can emit duplicate pod-death callbacks for graceful deletes, because watcher notifications don’t synchronize knownPods before the next poll diff.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At apps/mesh/src/nats/pod-heartbeat.ts, line 118:

<comment>Starting the poller in parallel with the watcher can emit duplicate pod-death callbacks for graceful deletes, because watcher notifications don’t synchronize `knownPods` before the next poll diff.</comment>

<file context>
@@ -88,17 +115,65 @@ export class NatsPodHeartbeat implements PodHeartbeat {
     // Activate deferred death watcher if registered before init
     if (this.pendingDeathCallback) {
       this.startDeathWatcher(this.pendingDeathCallback);
+      this.startDeathPoller();
       this.pendingDeathCallback = null;
     }
</file context>