feat(auth): per-sandbox authentication to gateway#1404
Open
TaylorMutch wants to merge 12 commits into
Open
Conversation
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
TaylorMutch
force-pushed
the
tmutch/gateway-config-impl
branch
2 times, most recently
from
381784e to
9bc2e11
Compare
Replaces the hard-coded sandbox-method / dual-auth / Bearer branches in
AuthGrpcRouter with a pluggable Authenticator chain that produces a
Principal::{User, Sandbox, Anonymous}. The principal is inserted into
request extensions for handler consumption.
PR-1 keeps the legacy metadata marker for sandbox principals so existing
handlers that read x-openshell-auth-source continue to work; the marker
is removed in the PR-3 wire break. The OidcAuthenticator wraps the
existing JwksCache::validate_token for User principals, and the
LegacySandboxMarkerAuthenticator preserves the pre-refactor path-based
behavior pending the gateway-minted JWT flow in PR 2/3.
Part of the per-sandbox identity series that closes #1354.
Adds the gateway-side infrastructure for per-sandbox identity tokens (the PR-2 step of the series resolving #1354): - New Ed25519 keypair generated by `certgen` alongside the existing PKI. Local mode writes `<dir>/jwt/{signing.pem,public.pem,kid}`; K8s mode creates an Opaque `<release>-jwt-keys` Secret. - `SandboxJwtIssuer` mints tokens with EdDSA-signed claims (SPIFFE-shaped `sub`, denormalised `sandbox_id`, 24h default TTL, `jti` for revocation). - `SandboxJwtAuthenticator` validates tokens through the Authenticator chain and yields `Principal::Sandbox(BootstrapJwt {..})`. Tokens with a different `kid` fall through so non-matching Bearer headers reach the OIDC authenticator unchanged. - `K8sServiceAccountAuthenticator` is path-scoped to `IssueSandboxToken`; consumes a projected SA token and produces a `K8sServiceAccount` sandbox principal that the new `IssueSandboxToken` handler exchanges for a fresh gateway JWT. - In-memory `RevocationSet` with TTL pruning, ready for the PR-3 delete-side hook and PR-5 refresh. - Helm chart mounts the JWT secret on the gateway pod and wires `[openshell.gateway.gateway_jwt]` into the rendered TOML. PR 2 is additive: no driver yet writes a sandbox token, no supervisor yet presents a Bearer JWT. PR 3 wires the consumer ends and removes the legacy path-based sandbox marker.
Switches every sandbox-to-gateway gRPC call from "path-based mTLS-only trust" to "Authorization: Bearer <gateway-minted-JWT>" presented by the sandbox supervisor. Closes the trust-boundary half of issue #1354; the per-handler sandbox_id equality check follows in PR 4. Sandbox side: - crates/openshell-sandbox/src/grpc_client.rs gains an AuthInterceptor that injects the Bearer header on every outbound RPC. The token is resolved at startup from one of three sources, in order: 1. OPENSHELL_SANDBOX_TOKEN (env, test harnesses) 2. OPENSHELL_SANDBOX_TOKEN_FILE (Docker/Podman/VM drivers) 3. OPENSHELL_K8S_SA_TOKEN_FILE (K8s driver — projected SA token exchanged for a gateway JWT via IssueSandboxToken) Gateway side: - handle_create_sandbox mints a gateway JWT and passes it through the compute layer to DriverSandboxSpec.sandbox_token. K8s sandboxes ignore the field; Docker and Podman drivers inject it as OPENSHELL_SANDBOX_TOKEN in the container env. - Removes the path-based SANDBOX_METHODS / DUAL_AUTH_METHODS branches and the x-openshell-auth-source metadata marker. The AuthGrpcRouter chain is now uniform: K8s SA -> SandboxJwt -> OIDC, all extension-based. - Removes LegacySandboxMarkerAuthenticator and the SandboxIdentitySource:: LegacyMarker variant. Handlers read Principal::Sandbox directly from request extensions. Kubernetes driver: - Sandbox pods gain a projected ServiceAccount token volume mounted at /var/run/secrets/openshell/token (audience openshell-gateway, 1h TTL, kubelet auto-rotates). - Each pod is annotated with openshell.io/sandbox-id; the gateway resolves the SA token claim's pod uid back to a sandbox id via this annotation. - Helm Role grants the gateway pods:get in the sandbox namespace. No ClusterRoleBinding to system:auth-delegator — the gateway validates SA tokens against the apiserver's anonymous JWKS endpoint instead of via TokenReview, so no cluster-scoped privilege is required. The full JWKS verifier + pod-annotation lookup lands in the follow-up that brings the K8s helm-dev demo end-to-end; PR 3 exercises the wire break with Docker/Podman as the working drivers.
ProcessHandle::spawn_impl previously inherited the supervisor's full environment when starting the sandbox entrypoint, then drop_privileges() demoted the child to the sandbox user. The combination meant a later process running as the sandbox user (e.g. an SSH-spawned shell) could read /proc/<entrypoint_pid>/environ and recover the gateway-minted JWT. Explicitly env_remove the three sandbox-token env vars before exec so the entrypoint child carries none of the supervisor's identity material. SSH session shells already use env_clear() in apply_child_env, so this plugs the only remaining inheritance path. Related to #1354 (per-sandbox identity series, PR 3 follow-up).
Adds the IDOR guard that closes the second half of the per-sandbox identity series. Every sandbox-class handler now verifies that the calling Principal::Sandbox.sandbox_id matches the canonical UUID the request body operates on. User principals bypass the check because RBAC was their gate at the router layer; anonymous callers are rejected outright. New module crates/openshell-server/src/auth/guard.rs exposes ensure_sandbox_scope / enforce_sandbox_scope. Applied at the top of: - handle_get_sandbox_config (id-keyed) - handle_get_sandbox_provider_environment (id-keyed) - handle_report_policy_status (id-keyed) - handle_push_sandbox_logs (id-keyed, first frame only — principal is stable across the stream) - handle_submit_policy_analysis (name-keyed: resolve to id, then check) - handle_get_draft_policy (name-keyed) - handle_update_config (dual-auth: enforce only when Principal::Sandbox; CLI / TUI user paths are unaffected) - handle_get_inference_bundle (no sandbox_id in body; accept any authenticated principal, reject anonymous) Existing policy.rs tests are updated to wrap their requests with a test-helper user principal so the new guard treats them as CLI calls; six new tests cover the cross-sandbox-denied / same-sandbox-allowed / user-bypasses-guard matrix.
Adds the rotation half of the per-sandbox identity series. Sandboxes holding a valid gateway-minted JWT can swap it for a fresh one without disruption; the old jti is revoked server-side before the new token is handed back, so a leaked token is unusable as soon as the rotation completes. Server side: - proto/openshell.proto gains RefreshSandboxToken plus empty request / token+expires_at_ms response messages. - handle_refresh_sandbox_token requires Principal::Sandbox with a BootstrapJwt source (K8s-SA principals are routed to IssueSandboxToken for bootstrap; user principals are rejected). The handler mints the replacement token first, then adds the old jti to the in-memory RevocationSet — so a failed mint never strands the sandbox. Sandbox side: - AuthInterceptor now reads its Bearer header from a process-wide Arc<RwLock<AsciiMetadataValue>> slot, so a single in-place token rotation is visible to every cached client (CachedOpenShellClient, the supervisor session channel, log push, etc.). - connect_channel spawns a background refresh loop once per process that sleeps for ~80% of the token's remaining lifetime (clamped to 60s-12h, plus small deterministic jitter) and calls RefreshSandboxToken, updating the token slot on success. - New parse_jwt_exp_ms helper decodes the JWT payload without signature verification — the token's origin is already trusted via the acquisition flow. Tests: - 4 server-side handler tests (round-trip, user-principal rejected, K8s-SA-principal rejected, missing-issuer returns Unavailable) - 3 sandbox-side helper tests (parse-exp, 80%-of-TTL delay, 60s floor) All existing OpenShell test impls gain a refresh_sandbox_token stub.
The projected SA token kubelet writes to each sandbox pod was previously a hardcoded 3600s literal in the driver. Operators in tighter audit regimes want to dial it lower; very large clusters may want it slightly higher to absorb token-refresh churn. Wires `sa_token_ttl_secs` through three layers: - KubernetesComputeConfig gains the field (default 3600). The driver clamps to [600, 86400] via `effective_sa_token_ttl_secs()`: 600s is kubelet's enforced minimum, 24h is the cap (the token is consumed within seconds of pod start, so longer is almost always a misconfiguration). - The openshell-driver-kubernetes binary exposes `--sa-token-ttl-secs` / `OPENSHELL_K8S_SA_TOKEN_TTL_SECS`. - `[openshell.gateway].sa_token_ttl_secs` in the gateway TOML inherits into `[openshell.drivers.kubernetes]`, mirroring the `enable_user_namespaces` plumbing. - Helm: `server.sandboxJwt.k8sSaTokenTtlSecs` (default 3600) renders into the K8s driver block of the gateway config.
Replaces the LiveK8sResolver stub with a working validator. Sandbox pods
present their projected ServiceAccount token via Authorization: Bearer
on IssueSandboxToken; the gateway:
1. Decodes the JWT header and looks up the signing key.
2. On miss, fetches the apiserver's /.well-known/openid-configuration
discovery doc + /openid/v1/jwks via kube::Client and caches the keys.
3. Validates the token's signature (RS256), issuer, audience
(openshell-gateway), and expiry.
4. Reads `kubernetes.io.pod.{name,uid}` from the claims and GETs the
pod in the gateway's sandbox namespace.
5. Verifies the live pod's UID matches the token's UID (defense against
replayed tokens from recreated pods with the same name) and reads
the openshell.io/sandbox-id annotation to derive the sandbox UUID.
The gateway needs no system:auth-delegator ClusterRoleBinding — JWKS
validation is local, so the only K8s permission it consumes is the
namespace Role's `pods: get` grant. Discovery + JWKS reads ride the
gateway's existing kube::Client auth (system:service-account-issuer-
discovery is bound to system:authenticated in every supported K8s
distro).
ServerState gains an in-cluster detection path in run_server: when
KUBERNETES_SERVICE_HOST is set AND a sandbox JWT issuer is configured,
construct the resolver and wire it as state.k8s_sa_authenticator. The
existing K8sServiceAccountAuthenticator (path-scoped to
IssueSandboxToken) becomes functional.
Tests: JWKS path parsing covers absolute URL, relative path, query
string, and garbage rejection. End-to-end validation against a real
apiserver is exercised in the helm-dev demo.
Three regressions / inefficiencies surfaced while bringing the per-sandbox identity series up end-to-end in the local helm cluster: 1. CLI returned Unauthenticated against a no-OIDC dev gateway. PR 3 removed the pre-refactor "no OIDC = pass through" behavior; with only sandbox-side authenticators in the chain, plain user CLI calls hit Unauthenticated. Add a PermissiveUserAuthenticator that installs as a final fallback when no OIDC is configured but sandbox JWT signing IS — produces a synthetic dev-anonymous user principal so the rest of the handler chain treats CLI calls as User and bypasses the IDOR guard. Production OIDC deployments are unaffected: when OIDC is configured the fallback is not installed and missing-Bearer still 401s. 2. Sandbox supervisor re-ran the K8s SA bootstrap exchange on every connect_channel() call. With multiple subsystems each building their own channels, IssueSandboxToken was firing every few seconds even though TOKEN_SLOT already had a fresh token. Change connect_channel to reuse TOKEN_SLOT when populated; only run acquire_sandbox_token on the first call per process. The refresh loop keeps the slot fresh thereafter. 3. K8s SA authenticator looked up sandbox pods in the gateway's own namespace (POD_NAMESPACE) instead of the K8s driver's configured sandbox namespace. Source from kubernetes_config_from_file() so the resolver targets the same namespace the driver creates pods in. Verified end-to-end against the helm-dev cluster: - Two sandboxes get distinct gateway JWTs with their own sandbox UUIDs. - Cross-sandbox GetSandboxConfig is rejected with PermissionDenied and the auth::guard audit log fires with both principal and requested IDs. - RefreshSandboxToken mints a new JWT and revokes the old jti; the old token is then rejected with Unauthenticated: revoked token.
…testing
Adds a small subcommand to the supervisor binary that issues one-shot
sandbox-class RPCs against the gateway using the supervisor's existing
token-acquisition pipeline. Designed to be invoked via docker exec or
kubectl exec into a running sandbox to verify the per-sandbox identity
flow end-to-end without writing a custom test binary inside the sandbox
image.
Subcommands:
- get-sandbox-config --sandbox-id <UUID> — call GetSandboxConfig
- refresh — call RefreshSandboxToken
- show-token — print raw gateway JWT bytes
- show-principal — pretty-print decoded JWT claims
Verification flow this enables (Docker path):
docker exec sandbox-a openshell-sandbox debug-rpc show-principal
docker exec sandbox-a openshell-sandbox debug-rpc \
get-sandbox-config --sandbox-id <sandbox-b-uuid>
# → exit code 7 + "PermissionDenied: cross-sandbox access denied"
K8s path: same RPCs, kubectl exec instead.
show-token and show-principal intentionally don't trigger the K8s SA
bootstrap exchange — they only read an already-cached token, so
inspection doesn't burn a fresh JWT mint per call.
TaylorMutch
force-pushed
the
tmutch/per-supervisor-authn
branch
from
834b56e to
f4daea6
Compare
TaylorMutch
changed the title
TaylorMutch
changed the title
TaylorMutch
requested review from
a team,
derekwaynecarr,
maxamillion and
mrunalp
as code owners
|
Label |
7 tasks
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds per-sandbox supervisor authentication for gateway RPCs and closes the cross-sandbox access gap tracked in #1354. Sandbox supervisors now authenticate as a specific
Principal::Sandbox; gateway handlers then enforce that the authenticated sandbox matches the sandbox named in each sandbox-scoped request.The design has two first-class bootstrap patterns:
After bootstrap, both patterns converge on the same steady-state behavior: the supervisor presents
Authorization: Bearer <gateway-jwt>, refreshes that credential in memory, and is authorized only for its own sandbox.Related Issue
Closes #1354
Changes
Authenticator/Principalrouting for gateway gRPC authentication.IssueSandboxToken.debug-rpchelpers for end-to-end authentication testing.Implementation Details
Problem Context
Before this PR, sandbox-class handlers trusted a
sandbox_idor sandbox name supplied in the request body. The shared mTLS client certificate only proved that the caller had a gateway client certificate; it did not prove that the caller was sandbox A rather than sandbox B. Any holder of that shared credential could therefore ask for another sandbox's policy, drafts, provider environment, or related sandbox-private state.This PR moves the identity decision into the gateway authentication layer. The router authenticates the caller, inserts a
Principalinto request extensions, and handlers compare that principal to the requested sandbox before serving sandbox-private data.The detailed implementation plan is captured in
architecture/plans/sandbox-service-accounts-implementation.md.Shared Gateway Auth Model
The gateway now uses a pluggable authenticator chain. Each authenticator can produce a
Principal, decline so the next authenticator can try, or reject the request fail-closed.The steady-state sandbox credential is a gateway-minted Ed25519 JWT. Validation checks issuer, audience, key ID, expiry, algorithm, and revocation state. The JWT includes sandbox identity and a
jtiso refresh and delete can invalidate previous tokens.This JWT is supervisor identity material:
CreateSandboxResponse.Docker, Podman, And VM Bootstrap
Docker, Podman, and VM deployments do not have a platform identity service equivalent to Kubernetes projected ServiceAccount tokens. For those drivers, the gateway uses a push-based bootstrap pattern.
At sandbox creation time, the gateway mints a sandbox JWT for the new sandbox and passes it to the in-process driver boundary as secret material. The driver writes that token to a supervisor-only file and starts the sandbox with
OPENSHELL_SANDBOX_TOKEN_FILEpointing at that file. The supervisor reads the file once at startup and then keeps the active token in memory.This mirrors the existing file-based secret delivery pattern used by local drivers while avoiding the unsafe parts of the old model:
Podman follows the same path as Docker. The VM path uses the same concept with the token embedded into the guest secret material at sandbox start, then refreshed in memory after the supervisor is running.
This path is the primary singleplayer/local-driver design, not a fallback from Kubernetes.
Kubernetes Bootstrap
Kubernetes uses a pull-based bootstrap pattern because kubelet can provide a short-lived, audience-bound ServiceAccount token to the sandbox pod.
The sandbox pod gets a projected ServiceAccount token mounted at a supervisor-only path. On startup, the supervisor presents that token to
IssueSandboxToken. The gateway validates the ServiceAccount token, extracts pod identity claims, fetches the pod, and reads the gateway-ownedopenshell.io/sandbox-idannotation to derive the sandbox identity. If the checks pass, the gateway returns the same kind of gateway-minted sandbox JWT used by the Docker/Podman/VM path.This avoids creating one Kubernetes Secret per sandbox. The gateway RBAC is intentionally narrow: validate token reviews and read pods in the sandbox namespace. It does not need to patch sandbox pods, and operators should avoid granting extra pod mutation permissions to the gateway identity.
Supervisor Credential Resolution
The supervisor resolves credentials in a driver-neutral order:
OPENSHELL_SANDBOX_TOKENfor tests.OPENSHELL_SANDBOX_TOKEN_FILEfor Docker, Podman, and VM.OPENSHELL_K8S_SA_TOKEN_FILEfor Kubernetes bootstrap throughIssueSandboxToken.Once resolved, every path produces a gateway-minted JWT in the same token slot. A gRPC interceptor injects it as
Authorization: Beareron all gateway calls. Refresh updates that shared slot, so existing clients do not need to be rebuilt when the token rotates.Handler Authorization
Authentication alone is not enough; handlers still need to authorize access to the requested sandbox.
Direct
sandbox_idhandlers compare the authenticatedPrincipal::Sandbox.sandbox_idto the requested ID. Name-keyed handlers resolve the sandbox name to the canonical ID and then compare. Streaming log push authorizes on the first frame, where the sandbox identity is declared.User principals continue through the normal RBAC path. Sandbox principals are limited to their own sandbox. Anonymous principals are rejected for sandbox-scoped paths.
Refresh And Revocation
RefreshSandboxTokenlets a supervisor rotate its in-memory gateway JWT before expiry. The gateway mints a replacement for the same sandbox principal and revokes the previousjti. Sandbox deletion also revokes the most recent token so replayed credentials are rejected.Kubernetes supervisors can recover from restart by repeating the ServiceAccount bootstrap exchange. Docker, Podman, and VM supervisors use their file token as bootstrap material and then rely on in-memory refresh for steady state.
Signing Key Persistence
The gateway JWT signing key is persisted through the existing local and Helm PKI paths. Helm mounts the JWT key material into the gateway even when local TLS is disabled, because per-sandbox authentication is independent from TLS enablement.
Design Decisions For Reviewers
IssueSandboxToken.CreateSandboxResponse, sandbox metadata, ordinary user environments, and logs.jti-based. Refresh and delete invalidate previous tokens without changing the stable sandbox ID.Reviewer Focus Areas
PushSandboxLogsauthorizes on the first frame.jtirejection after refresh and sandbox delete.Testing
mise run pre-commitpassessandbox list,sandbox create, andsandbox deleteagainst a local k3d deploymentChecklist