CP-37371: bundle Prometheus binary in CloudZero Agent image

[evan-cz]

Official Prometheus images include a shell and utilities like curl that expand the tools available for exploitation in container escape or privilege escalation attacks in the event of a compromise. This change bundles the Prometheus binary directly into the CloudZero Agent image, which uses a minimal scratch base with no shell, to provide a bit of defense in depth.

Functional Change:

Before: The Prometheus container used the official quay.io/prometheus/prometheus image, which includes a shell and utilities that could be leveraged by an attacker.

After: The Prometheus container uses the CloudZero Agent image with the Prometheus binary bundled at /app/prometheus. The image has no shell and minimal attack surface. Users can still override to use the official image if needed.

Solution:

1. Modified docker/Dockerfile to extract and bundle the Prometheus binary:

   • Added prometheus-source build stage from official Prometheus image
   • Copy /bin/prometheus to /app/prometheus in final scratch-based image
   • Prometheus version pinned via PROMETHEUS_VERSION ARG for Renovate updates

2. Updated helm/values.yaml to use bundled binary by default:

   • Set components.prometheus.image.repository to null (falls back to agent image)
   • Added components.prometheus.command field with three-way behavior:
     • null (default): Uses /app/prometheus for bundled binary
     • []: Uses image default entrypoint (for official Prometheus image)
     • ["/custom/path"]: Uses specified command

3. Added generateContainerCommand helper to helm/templates/_helpers.tpl:

   • Accepts dict with command and default (array) parameters
   • Reusable for other containers (e.g., Alloy in future PR)

4. Updated agent-deploy.yaml and agent-daemonset.yaml to use new helper

5. Added schema validation for command field in helm/values.schema.yaml:

   • Uses oneOf to allow null or array of strings
   • References Kubernetes Container command definition

6. Updated helm/tests/README.md with correct instructions for running tests

Validation:

• Added helm/tests/prometheus_command_test.yaml with 7 unit tests covering:

  • Default command (/app/prometheus)
  • Empty array (no command, uses image entrypoint)
  • Custom command with multiple elements
  • Federated mode (daemonset) behavior
  • Args not affected by command changes

• Added 6 schema validation tests in tests/helm/schema/prometheus.command.*.yaml

• All helm test suites pass

• Deployed and verified on EKS (arm64) / GKE (amd64) clusters:

  • Pod running successfully with CloudZero agent image
  • Command correctly set to ["/app/prometheus"]
  • Prometheus scraping and forwarding cAdvisor metrics normally
  • Official image override: Verified backward compatibility with
    quay.io/prometheus/prometheus using command: [] override