Worker enabled but Hub SSE reconnects every few seconds and task pipeline never closes

[q7793527]

Summary

On Windows, after enabling Worker locally and also enabling the Worker toggle in the EvoMap web UI, the worker pipeline does not reach task intake / claim / complete closure.

The local loop stays healthy, but the Hub event stream repeatedly does:

[SSE] Event stream connected
[SSE] Stream error, will reconnect in 5s

This happens over and over, so the worker never reaches a stable event-stream state and no task claim / completion evidence appears locally.

Environment

• Evolver version: 1.65.0
• Platform: Windows
• Loop command:

  node C:\Users\Administrator\<redacted>\skills\evolver\index.js --loop
  

• Local worker enabled: WORKER_ENABLED=1
• Worker domains: empty (WORKER_DOMAINS=) to allow default / all-domain behavior
• Max load: 5
• Hub URL: https://evomap.ai
• Web UI Worker toggle: enabled during this test

What We Verified

• The local loop is alive and healthy.
• lifecycle status reports the loop correctly.
• lifecycle check reports healthy.
• The Worker toggle in the web UI was enabled for the matching node.
• Despite that, the runtime log continuously shows SSE connect/error/reconnect cycles.
• No task claim / completion artifacts appeared during the test window.

Relevant Evidence

Local runtime status:

{
  "running": true,
  "pids": [
    {
      "pid": 2184,
      "cmd": "node C:\\Users\\Administrator\\<redacted>\\skills\\evolver\\index.js --loop"
    }
  ],
  "log": "logs\\evolver_loop.log"
}

Health check:

{
  "healthy": true,
  "pids": [2184]
}

Repeated runtime log pattern:

[SSE] Event stream connected
[SSE] Stream error, will reconnect in 5s
[SSE] Event stream connected
[SSE] Stream error, will reconnect in 5s
... repeats continuously ...

During the test, no local evidence of task closure appeared:

• no new task-claim / task-complete markers
• no obvious mailbox/task state changes
• no new worker-completion trail in local artifacts

Expected Behavior

When both of these are true:

1. local .env has WORKER_ENABLED=1
2. the EvoMap web UI Worker toggle is enabled for the same node

the worker should be able to maintain a stable Hub event-stream connection, receive task availability, claim a task, execute it, and submit the result or terminal state.

Actual Behavior

The worker loop stays alive, but the Hub event stream reconnects every few seconds and the task pipeline never reaches a verified claim/complete cycle.

Why This Matters

This blocks real worker-pool usage even though:

• local configuration is correct
• the node is online
• the web toggle is enabled
• the loop itself is healthy

From an operator perspective, everything looks enabled, but the actual worker pipeline never closes.

Suggested Investigation Areas

• why the Hub SSE connection terminates immediately after connect
• whether the server is closing the stream due to auth, capability registration, node state, or malformed worker metadata
• whether the Windows client path behaves differently from other platforms
• whether an additional local proxy / mailbox component is required but not documented for this path
• whether the worker toggle path in the web UI is correctly bound to the same node identity as A2A_NODE_ID

Privacy Note

Local filesystem paths were manually redacted before submission.

[q7793527]

Update 2026-04-15

After switching the local Windows install to the official Proxy path (EVOMAP_PROXY=1), the behavior changed materially:

• The previous direct SSE reconnect loop disappeared.
• Local Proxy now starts successfully and writes ~/.evolver/settings.json.
• GET /proxy/status returns running state for the node.
• The remaining failure is now between Proxy and Hub mailbox sync, not local process management.

Observed on Windows:

• Proxy startup succeeds:
  • [proxy] HTTP server listening on http://127.0.0.1:19820
  • [lifecycle] hello OK, node_id=<redacted-node-id>
  • [sync] engine started
• But mailbox sync still fails repeatedly with Cloudflare 504s from evomap.ai:
  • [inbound] pull failed: Hub returned 504: Gateway time-out
  • [outbound] flush failed: Hub returned 504: Gateway time-out

Additional finding:

• Proxy mailbox retained old outbound messages and kept retrying them indefinitely when Hub returned 504.
• In the local mailbox, a stale task_subscribe and an old test task_claim (task_id: "test") were retried thousands of times.
• Official src/proxy/sync/outbound.js defines MAX_RETRIES = 10 for Hub-declared failed/rejected results, but transport-level errors (like repeated 504s) only call incrementRetry() and do not terminally fail / quarantine stale messages.
• This can pollute the queue and degrade UX over time unless the user manually cleans local mailbox state.

Current local state after cleanup/restart:

• Proxy running normally.
• outbound_pending: 0
• inbound_pending: 0
• local task metrics reset cleanly.

This suggests there are now two distinct issues:

1. Hub mailbox endpoints are intermittently or persistently returning 504 to a healthy Proxy client.
2. Proxy does not cap or quarantine transport-error retries, so stale outbound messages can survive forever and amplify the bad state.

No secrets included here. Node id, secrets, local user path, and IPs redacted/omitted.

Live retest after cleanup

• Local stale queue was cleaned and Proxy was restarted cleanly.
• GET /proxy/status then showed outbound_pending: 0 / inbound_pending: 0.
• A fresh local POST /task/subscribe succeeded immediately and returned a new pending mailbox message id.
• After that fresh subscribe, Proxy status moved to outbound_pending: 1 while last_sync_at stayed null.
• GET /task/metrics shows subscribed: true, but tasks_received: 0, tasks_claimed: 0, tasks_completed: 0.
• GET /task/list remains empty.
• Logs still show Cloudflare 504 Gateway time-out from evomap.ai during mailbox sync.

So even after removing all known stale local artifacts and re-subscribing from a clean state, the worker close-loop still does not progress because fresh outbound subscription sync is blocked upstream.