fix(gateway): make readiness health checks dependency-aware

[alangou]

Summary

This PR makes gateway readiness signals dependency-aware instead of always healthy, while keeping liveness intentionally lightweight.
It adds a configurable database readiness timeout, wires it end-to-end (CLI/env -> server config -> health router), and aligns Helm defaults so application timeout stays below Kubernetes probe timeout.
It also extends readiness coverage with Docker e2e and updates the Rust test task so openshell-server test-only coverage runs with test-support.

Related Issue

• closes OS-156
• Parent initiative: OSGH-111 (Runtime Reliability)

Changes

• Added dependency-aware health behavior in openshell-server:
  • /healthz remains liveness-only (200 when process is responsive)
  • /readyz and /health perform DB connectivity checks and return 503 when unavailable
  • readiness payload includes structured dependency details (checks.database.status, latency, error)
• Added bounded timeout handling for DB readiness checks.
• Added configurable readiness timeout:
  • CLI flag: --readiness-db-timeout-secs
  • Env var: OPENSHELL_READINESS_DB_TIMEOUT_SECS
  • Core/server config wiring and runtime validation (> 0)
• Added persistence connectivity helpers (ping / close) for both SQLite and Postgres stores.
• Added Prometheus readiness metrics:
  • openshell_server_readiness_database_healthy gauge (1 healthy, 0 unhealthy)
  • openshell_server_readiness_database_probe_duration_seconds histogram labeled by outcome (success, db_error, timeout)
  • health tests validate metric emission and /metrics exposure
• Scoped pool teardown APIs to test support only:
  • Store::close and backend close methods are behind #[cfg(any(test, feature = "test-support"))]
• Updated existing integration tests that instantiate health_router so they provide a real Store.
• Added readiness integration test coverage:
  • crates/openshell-server/tests/health_endpoint_integration.rs
• Added Docker e2e coverage for readiness:
  • e2e/rust/tests/readyz_health.rs
  • e2e test target in e2e/rust/Cargo.toml
  • Docker e2e wrapper exposes a dedicated health port for /readyz probing
• Helm chart updates for timeout customization and safe defaults:
  • server.readinessDbTimeoutSecs passed to gateway args
  • probes.readiness.timeoutSeconds default set to 2s
  • app default timeout set to 1s so app timeout < probe timeout
• Test task update for test-support coverage by default in Rust test lane:
  • tasks/test.toml runs workspace tests excluding openshell-server, then runs openshell-server with --features test-support

Why /healthz still returns 200 when DB is down

/healthz is kept as a pure liveness probe by design. If liveness depended on DB, transient DB outages could trigger unnecessary pod restarts and CrashLoop behavior without fixing the dependency outage. Readiness (/readyz) is the dependency-aware signal used to remove unhealthy pods from traffic.

Why close is test-only

close is used to simulate dependency outages in tests. Exposing it in runtime code would make it possible to tear down an active pool under live traffic. Until a dedicated graceful-shutdown flow exists, keeping it behind test support prevents accidental production use.

Testing

• mise run pre-commit passes
• Unit tests added/updated
• E2E tests added/updated (if applicable)

Validation run:

• mise run e2e
• mise run ci

Checklist

• Follows Conventional Commits
• Commits are signed off (DCO)
• Architecture docs updated (if applicable)

[TaylorMutch]

A couple of focused comments on the readiness changes.

[alangou]

@TaylorMutch I removed the hardcoded value for the database timeout some documentation and deployments resources has been updated to reflect that change (helm chart, doc)

[github-actions]

Label test:e2e applied for 05b6998. Open the existing run and click Re-run all jobs to execute with the label set. The E2E Gate check on this PR will flip green automatically once the run finishes.