ALIEN-59: K8s agent self-update — chart guardrails, /v1/rejoin recovery, persistence default

[yonatan-alien]

Summary

Closes ALIEN-59. Manager-driven agent self-update on Kubernetes, plus the two production-hardening fixes that came out of live e2e.

Feature

• New agent_upgrade loop in alien-agent consumes agent_target.helm on /v1/sync and spawns a Helm-runner Job that runs helm upgrade --reuse-values --set runtime.image.tag=<target> under an alien-agent-upgrader ServiceAccount.
• alien-helm chart guardrails — required management.{url,name,token,deploymentId}, /livez+/readyz probes, Recreate strategy, namespace-scoped upgrader Role/Binding, upgrader.enabled opt-out.
• Sync schema gains agent_version / agent_os / agent_arch / regime / agent_image_repository; K8s worker transport set to local; K8s ServiceAccount controller is skipped (chart owns it).
• alien-manager exposes PUT /v1/deployments/:id/target-agent-version; build_agent_target emits agent_target.helm when target ≠ reported and regime = kubernetes.

Production hardening (would silently break otherwise; both found via live e2e)

• fix(helm): enable agent runtime-data persistence by default — without a PVC, the self-update's pod roll wipes the agent's deployment_id/sync token, the new pod hits 409 from initialize, CrashLoopBackOffs, and helm --atomic rolls back. Documented opt-out.
• feat(agent,manager): /v1/rejoin endpoint — for the case where persistence is genuinely off (or the state directory is wiped for any other reason), the agent's initialize-409 falls through to a new POST /v1/rejoin that re-acquires a deployment-scoped sync token over the existing row. New RouterOptions::include_rejoin + skip_rejoin() builder lets multi-tenant embedders override, same pattern as /v1/initialize.
• fix(agent): preserve 409 from initialize so rejoin fall-through fires — the agent's .context(ConfigurationError) was clobbering the SDK's real 409 with a hardcoded 500. New DeploymentNameAlreadyExists variant (409) lets the conflict surface so the rejoin branch actually triggers.

Paired platform-side PR: https://github.com/alienplatform/platform/pull/new/yonatan/alien-59-self-update-for-the-alien-agent-across-all-pull-mode-2

Test plan

• cargo test -p alien-agent -p alien-core -p alien-deployment -p alien-helm -p alien-manager --lib
• Live e2e against local managerx + orbstack K8s: pin a target version in the dashboard → manager emits agent_target → helm-upgrade Job rolls the agent pod → PVC keeps deployment_id → new agent reports the new version
• Live e2e for rejoin: kubectl exec rm -rf /var/lib/alien-agent/* + bounce pod → agent logs Name already exists — assuming local state was wiped, rejoining… → recovers the original deployment_id and stays running
• Reviewer to confirm: client-sdks/platform/rust/openapi*.json kept ours during the main merge to preserve the rejoin/target-agent-version schemas; main's added ManagerDomainBinding* + PackageDomain schemas in those files were dropped (they were generated from a private platform branch). Next platform-side openapi regen will bring them back.

[greptile-apps]

Greptile Summary

This PR lands manager-driven K8s agent self-update (Helm-runner Job spawned from the agent on /v1/sync), the /v1/rejoin recovery flow for wiped persistent state, and the chart guardrails (Recreate strategy, PVC-on-by-default, required value guards, /livez+/readyz probes) that make the upgrade cycle safe.

• K8s self-update loop (agent_upgrade.rs): agent reads agent_target.helm from the sync response, resolves chart ref + runner image from env vars injected by the chart, and POSTs a one-shot Helm-runner Job to the in-cluster API using the alien-agent-upgrader SA.
• /v1/rejoin endpoint (manager + agent cli.rs): catches the 409 from /v1/initialize (name already exists after a state wipe), re-acquires a fresh deployment-scoped sync token over the existing row, and persists it to the local SQLite state so pod restarts don't fall back to the chart-mounted DG token.
• Chart hardening (generator.rs): PVC enabled by default, required template guards on management.*, Recreate rollout strategy, namespace-scoped upgrader Role/ClusterRoleBinding, and auto-generated encryption key via lookup.

Confidence Score: 3/5

The self-update happy path is solid, but a failed helm-runner Job silently blocks all retry attempts for up to 10 minutes with no operator warning.

The core upgrade loop, rejoin recovery, chart guardrails, and persistence defaults are well-reasoned and tested. The 409-on-failed-Job case in create_job quietly absorbs what should be a retryable error, and min_supported_version being pinned to the target version would block all upgrades once the agent enforces that field. The upgrade path is a new critical flow touching the agent self-replacement mechanism, so these gaps carry more weight than they would in a peripheral feature.

crates/alien-agent/src/loops/agent_upgrade.rs (failed-Job 409 handling) and crates/alien-manager/src/routes/sync.rs (build_agent_target min_supported_version) need the most attention before this ships to production.

Important Files Changed

Filename	Overview
crates/alien-agent/src/loops/agent_upgrade.rs	New Helm-runner Job spawner; 409 from a failed (but not yet TTL'd) Job is silently treated as idempotent success, stalling upgrades for up to 10 min with no operator warning.
crates/alien-manager/src/routes/sync.rs	New agent_sync inventory persistence and build_agent_target; min_supported_version is set equal to the target version, which would block all agents from upgrading once that field is enforced.
crates/alien-manager/src/routes/deployments.rs	New PUT /v1/deployments/:id/target-agent-version endpoint; auth, authz, and semver validation are present; is_plausible_semver rejects valid versions with both pre-release and build metadata.
crates/alien-helm/src/generator.rs	Chart guardrails (required fields, Recreate strategy, persistence default, upgrader SA/Role/ClusterRole), encryption key auto-generation, and /livez+/readyz probe defaults all look correct.
crates/alien-agent/src/cli.rs	Adds rejoin fall-through on 409 from initialize and sync-token persistence; logic and error routing look correct.
crates/alien-manager/src/stores/sqlite/deployment.rs	update_agent_metadata and set_target_agent_version are correctly scoped; neither updates the updated_at column, leaving deployment timestamps stale after inventory syncs.
crates/alien-core/src/sync.rs	SyncRequest/SyncResponse extended with new optional fields; AgentTarget, AgentHelmTarget, AgentBinaryTarget structs and back-compat tests look correct.
crates/alien-infra/src/core/executor.rs	Kubernetes ServiceAccounts are correctly skipped at controller-lookup time and auto-marked Running as platform-provided resources; logic mirrors external-binding treatment.

Sequence Diagram

sequenceDiagram
    participant Agent
    participant Manager
    participant K8sAPI
    participant HelmRunnerJob

    Note over Agent,Manager: Normal sync loop
    Agent->>Manager: "POST /v1/sync (agent_version, regime=kubernetes)"
    Manager->>Manager: update_agent_metadata()
    Manager->>Manager: build_agent_target() if target differs from reported
    Manager-->>Agent: SyncResponse with agent_target helm payload

    Note over Agent,K8sAPI: Self-update actuator
    Agent->>K8sAPI: POST /apis/batch/v1/.../jobs (helm-runner Job)
    K8sAPI-->>Agent: 201 Created (or 409 if exists)
    K8sAPI->>HelmRunnerJob: schedule pod
    HelmRunnerJob->>HelmRunnerJob: helm upgrade --reuse-values --atomic --wait

    Note over Agent,Manager: Pod rolls (Recreate strategy)
    HelmRunnerJob->>K8sAPI: Deployment updated with new image tag
    Agent->>Agent: /readyz returns 503 until first sync
    Agent->>Manager: POST /v1/sync with new agent_version
    Agent->>Agent: "first_sync_completed=true, /readyz returns 200"

    Note over Agent,Manager: Rejoin after state wipe
    Agent->>Manager: POST /v1/initialize returns 409
    Agent->>Manager: POST /v1/rejoin with name
    Manager->>Manager: get_deployment_by_name() and mint new token
    Manager-->>Agent: deployment_id and fresh token
    Agent->>Agent: db.set_sync_token() persists for next restart

Loading

Comments Outside Diff (1)

1. crates/alien-manager/src/stores/sqlite/deployment.rs, line 506-550 (link)

   updated_at is not bumped during agent metadata writes

   update_agent_metadata (and similarly set_target_agent_version) never touches the updated_at column. Any API consumer that uses updated_at to detect "something changed on this deployment" will miss both inventory syncs and operator target-version pins. Consider setting updated_at = datetime('now') inside the same UPDATE, or at least doing so for set_target_agent_version which represents an explicit admin action that callers would expect to be reflected in the record's timestamp.

   Prompt To Fix With AI

   This is a comment left during a code review.
   Path: crates/alien-manager/src/stores/sqlite/deployment.rs
   Line: 506-550
   
   Comment:
   **`updated_at` is not bumped during agent metadata writes**
   
   `update_agent_metadata` (and similarly `set_target_agent_version`) never touches the `updated_at` column. Any API consumer that uses `updated_at` to detect "something changed on this deployment" will miss both inventory syncs and operator target-version pins. Consider setting `updated_at = datetime('now')` inside the same UPDATE, or at least doing so for `set_target_agent_version` which represents an explicit admin action that callers would expect to be reflected in the record's timestamp.
   
   How can I resolve this? If you propose a fix, please make it concise.

Prompt To Fix All With AI

Fix the following 4 code review issues. Work through them one at a time, proposing concise fixes.

---

### Issue 1 of 4
crates/alien-agent/src/loops/agent_upgrade.rs:296-305
**Silently swallows a failed Job's 409 — upgrade stalls with no warning**

When the Helm-runner Job exists but is in a `Failed` state (exhausted its `backoffLimit`), the Kubernetes API still returns 409 on a new create. The code logs at `info` level and returns `Ok(())`, so `apply_agent_target` sees success. The manager keeps emitting `agent_target` on every sync tick, but the agent will keep hitting the same 409 and silently doing nothing for up to `ttlSecondsAfterFinished` seconds (10 minutes). During that window, there is no `warn` log and no error propagated — an operator watching logs will only see "Upgrader Job already exists (409); leaving in place" with no indication the actual upgrade never happened.

To distinguish "another sync already queued it" (desired idempotency) from "it exists but is already dead," the handler should query the Job status and emit at least a `warn` when the existing Job is in a `Failed` condition, so operators can investigate before the TTL window expires.

### Issue 2 of 4
crates/alien-manager/src/routes/sync.rs:1107-1112
`min_supported_version` is set equal to `version`, which will break upgrades once the field is enforced. The field is described as "the agent should refuse the upgrade if its own version is older than this." Setting it to the target version means only an agent already at version X would accept the upgrade to X — but those agents never receive the instruction (the `reported_version == Some(target_version)` guard returns `None` first). Every agent that actually needs upgrading would refuse. The MVP intent (allow all current agents to upgrade) is better expressed as an empty floor like `"1.0.0"`, or by explicitly recording the minimum agent version that is safe to self-upgrade.

```suggestion
    Some(alien_core::sync::AgentTarget {
        version: target_version.to_string(),
        // Allow all existing agents to receive the upgrade instruction.
        // Set to a meaningful floor (e.g. the first version that supports
        // self-update) once the agent enforces this field.
        min_supported_version: "1.0.0".to_string(),
        binary: None,
        helm,
    })
```

### Issue 3 of 4
crates/alien-manager/src/routes/deployments.rs:876-892
`splitn(2, |c| c == '-' || c == '+')` splits on the **first** occurrence of either separator, so versions with both a pre-release suffix and build metadata (e.g. `1.4.0-rc.1+build.123`) fail: the remainder becomes `"rc.1+build.123"` and the `+` isn't in the allowed character set `[alphanumeric, '.', '-']`. The function correctly rejects garbage input, but it silently blocks valid SemVer 2.0.0 strings that operators may reasonably try to pin.

```suggestion
fn is_plausible_semver(v: &str) -> bool {
    // Strip optional +build metadata first so the pre-release check can
    // use a simple split on '-'.
    let without_build = v.splitn(2, '+').next().unwrap_or(v);
    let mut parts = without_build.splitn(2, '-');
    let core = parts.next().unwrap_or("");
    let core_ok = {
        let segs: Vec<&str> = core.split('.').collect();
        segs.len() == 3
            && segs
                .iter()
                .all(|s| !s.is_empty() && s.chars().all(|c| c.is_ascii_digit()))
    };
    let rest_ok = parts.next().map_or(true, |r| {
        !r.is_empty()
            && r.chars()
                .all(|c| c.is_ascii_alphanumeric() || c == '.' || c == '-')
    });
    core_ok && rest_ok
}
```

### Issue 4 of 4
crates/alien-manager/src/stores/sqlite/deployment.rs:506-550
**`updated_at` is not bumped during agent metadata writes**

`update_agent_metadata` (and similarly `set_target_agent_version`) never touches the `updated_at` column. Any API consumer that uses `updated_at` to detect "something changed on this deployment" will miss both inventory syncs and operator target-version pins. Consider setting `updated_at = datetime('now')` inside the same UPDATE, or at least doing so for `set_target_agent_version` which represents an explicit admin action that callers would expect to be reflected in the record's timestamp.

_{Reviews (1): Last reviewed commit: "Merge remote-tracking branch 'origin/mai..." | Re-trigger Greptile}

[greptile-apps]

Silently swallows a failed Job's 409 — upgrade stalls with no warning

When the Helm-runner Job exists but is in a Failed state (exhausted its backoffLimit), the Kubernetes API still returns 409 on a new create. The code logs at info level and returns Ok(()), so apply_agent_target sees success. The manager keeps emitting agent_target on every sync tick, but the agent will keep hitting the same 409 and silently doing nothing for up to ttlSecondsAfterFinished seconds (10 minutes). During that window, there is no warn log and no error propagated — an operator watching logs will only see "Upgrader Job already exists (409); leaving in place" with no indication the actual upgrade never happened.

To distinguish "another sync already queued it" (desired idempotency) from "it exists but is already dead," the handler should query the Job status and emit at least a warn when the existing Job is in a Failed condition, so operators can investigate before the TTL window expires.

Prompt To Fix With AI

This is a comment left during a code review.
Path: crates/alien-agent/src/loops/agent_upgrade.rs
Line: 296-305

Comment:
**Silently swallows a failed Job's 409 — upgrade stalls with no warning**

When the Helm-runner Job exists but is in a `Failed` state (exhausted its `backoffLimit`), the Kubernetes API still returns 409 on a new create. The code logs at `info` level and returns `Ok(())`, so `apply_agent_target` sees success. The manager keeps emitting `agent_target` on every sync tick, but the agent will keep hitting the same 409 and silently doing nothing for up to `ttlSecondsAfterFinished` seconds (10 minutes). During that window, there is no `warn` log and no error propagated — an operator watching logs will only see "Upgrader Job already exists (409); leaving in place" with no indication the actual upgrade never happened.

To distinguish "another sync already queued it" (desired idempotency) from "it exists but is already dead," the handler should query the Job status and emit at least a `warn` when the existing Job is in a `Failed` condition, so operators can investigate before the TTL window expires.

How can I resolve this? If you propose a fix, please make it concise.

[greptile-apps]

min_supported_version is set equal to version, which will break upgrades once the field is enforced. The field is described as "the agent should refuse the upgrade if its own version is older than this." Setting it to the target version means only an agent already at version X would accept the upgrade to X — but those agents never receive the instruction (the reported_version == Some(target_version) guard returns None first). Every agent that actually needs upgrading would refuse. The MVP intent (allow all current agents to upgrade) is better expressed as an empty floor like "1.0.0", or by explicitly recording the minimum agent version that is safe to self-upgrade.

Suggested change

	Some(alien_core::sync::AgentTarget {
	version: target_version.to_string(),
	min_supported_version: target_version.to_string(),
	binary: None,
	helm,
	})
	Some(alien_core::sync::AgentTarget {
	version: target_version.to_string(),
	// Allow all existing agents to receive the upgrade instruction.
	// Set to a meaningful floor (e.g. the first version that supports
	// self-update) once the agent enforces this field.
	min_supported_version: "1.0.0".to_string(),
	binary: None,
	helm,
	})

Prompt To Fix With AI

This is a comment left during a code review.
Path: crates/alien-manager/src/routes/sync.rs
Line: 1107-1112

Comment:
`min_supported_version` is set equal to `version`, which will break upgrades once the field is enforced. The field is described as "the agent should refuse the upgrade if its own version is older than this." Setting it to the target version means only an agent already at version X would accept the upgrade to X — but those agents never receive the instruction (the `reported_version == Some(target_version)` guard returns `None` first). Every agent that actually needs upgrading would refuse. The MVP intent (allow all current agents to upgrade) is better expressed as an empty floor like `"1.0.0"`, or by explicitly recording the minimum agent version that is safe to self-upgrade.

```suggestion
    Some(alien_core::sync::AgentTarget {
        version: target_version.to_string(),
        // Allow all existing agents to receive the upgrade instruction.
        // Set to a meaningful floor (e.g. the first version that supports
        // self-update) once the agent enforces this field.
        min_supported_version: "1.0.0".to_string(),
        binary: None,
        helm,
    })
```

How can I resolve this? If you propose a fix, please make it concise.

[greptile-apps]

splitn(2, |c| c == '-' || c == '+') splits on the first occurrence of either separator, so versions with both a pre-release suffix and build metadata (e.g. 1.4.0-rc.1+build.123) fail: the remainder becomes "rc.1+build.123" and the + isn't in the allowed character set [alphanumeric, '.', '-']. The function correctly rejects garbage input, but it silently blocks valid SemVer 2.0.0 strings that operators may reasonably try to pin.

Suggested change

	fn is_plausible_semver(v: &str) -> bool {
	let mut parts = v.splitn(2, |c| c == '-' || c == '+');
	let core = parts.next().unwrap_or("");
	let core_ok = {
	let segs: Vec<&str> = core.split('.').collect();
	segs.len() == 3
	&& segs
	.iter()
	.all(|s| !s.is_empty() && s.chars().all(|c| c.is_ascii_digit()))
	};
	let rest_ok = parts.next().map_or(true, |r| {
	!r.is_empty()
	&& r.chars()
	.all(|c| c.is_ascii_alphanumeric() || c == '.' || c == '-')
	});
	core_ok && rest_ok
	}
	fn is_plausible_semver(v: &str) -> bool {
	// Strip optional +build metadata first so the pre-release check can
	// use a simple split on '-'.
	let without_build = v.splitn(2, '+').next().unwrap_or(v);
	let mut parts = without_build.splitn(2, '-');
	let core = parts.next().unwrap_or("");
	let core_ok = {
	let segs: Vec<&str> = core.split('.').collect();
	segs.len() == 3
	&& segs
	.iter()
	.all(|s| !s.is_empty() && s.chars().all(|c| c.is_ascii_digit()))
	};
	let rest_ok = parts.next().map_or(true, |r| {
	!r.is_empty()
	&& r.chars()
	.all(|c| c.is_ascii_alphanumeric() || c == '.' || c == '-')
	});
	core_ok && rest_ok
	}

Prompt To Fix With AI

This is a comment left during a code review.
Path: crates/alien-manager/src/routes/deployments.rs
Line: 876-892

Comment:
`splitn(2, |c| c == '-' || c == '+')` splits on the **first** occurrence of either separator, so versions with both a pre-release suffix and build metadata (e.g. `1.4.0-rc.1+build.123`) fail: the remainder becomes `"rc.1+build.123"` and the `+` isn't in the allowed character set `[alphanumeric, '.', '-']`. The function correctly rejects garbage input, but it silently blocks valid SemVer 2.0.0 strings that operators may reasonably try to pin.

```suggestion
fn is_plausible_semver(v: &str) -> bool {
    // Strip optional +build metadata first so the pre-release check can
    // use a simple split on '-'.
    let without_build = v.splitn(2, '+').next().unwrap_or(v);
    let mut parts = without_build.splitn(2, '-');
    let core = parts.next().unwrap_or("");
    let core_ok = {
        let segs: Vec<&str> = core.split('.').collect();
        segs.len() == 3
            && segs
                .iter()
                .all(|s| !s.is_empty() && s.chars().all(|c| c.is_ascii_digit()))
    };
    let rest_ok = parts.next().map_or(true, |r| {
        !r.is_empty()
            && r.chars()
                .all(|c| c.is_ascii_alphanumeric() || c == '.' || c == '-')
    });
    core_ok && rest_ok
}
```

How can I resolve this? If you propose a fix, please make it concise.

[alongubkin]

these APIs should be in alien-k8s-clients

[alongubkin]

I think we have an existing enum for this somewhere in alien-core

[alongubkin]

I think we have an existing enum for this somewhere in alien-core

[alongubkin]

regime is a really weird name haha

[alongubkin]

Can you document why did we choose --atomic?

[alongubkin]

What happens if creating this job fails? Or pulling the job's image fails? Or the command inside it fails? Do we report back those failures in any way?

How many retries do we support (I think the answer should be unlimited)? Is there exponential backoff or constant interval between retries?

[alongubkin]

We recently improved our white-labeling support, need to make sure it fits there. See other places in alien-helm

[alongubkin]

Right now this will only work for Kubernetes deployments, right? No support for local service swap etc? What'll happen in local?