Allow Fleet Server to reload TLS certificates without restarting

[ycombinator]

What is the problem this PR solves?

Fleet Server currently requires an explicit restart to reload TLS certificates used for serving HTTPS requests to Elastic Agents. In environments with frequent certificate rotation — Kubernetes (where Secrets are remounted into pods), serverless (where cert-manager manages per-project certificates), and general ops — this creates operational burden and requires external orchestration to coordinate cert rotation with server restarts.

How does this PR solve the problem?

When TLS is enabled, Fleet Server now periodically re-reads the configured cert/key files from disk and hot-swaps them into the running TLS listener without restart. This is enabled by default and can be disabled by setting ssl.certificate_reload.enabled: false.

This follows the OTel Collector's configtls approach: on each TLS handshake, if the configured reload interval has elapsed, the cert/key files are re-read from disk and validated before being swapped in. No file watchers, no extra goroutines.

The core CertReloader implementation and the CertificateReload configuration both live in elastic-agent-libs v0.37.0 (in the tlscommon package), as part of ServerConfig, so any component using tlscommon gets cert reload support. Fleet Server simply wires it up.

Key design decisions:

• Polling instead of file watching: Inspired by OTel Collector's approach. Simpler, no fsnotify dependency, no goroutines, no debounce logic. On each GetCertificate call, checks if the reload interval has elapsed; if so, re-reads files from disk
• Enabled by default: Certificate reload is on unless explicitly disabled, matching the OTel Collector pattern
• Reload interval: Defaults to 5 seconds, configurable via ssl.certificate_reload.reload_interval. Automated tooling (cert-manager, K8s secret mounts) can use the default or a shorter value, while manual rotation workflows may benefit from a longer window
• Validation: Validates the new cert/key pair with tls.LoadX509KeyPair before swapping; invalid pairs keep the old cert active and log an error
• Serving: Uses tls.Config.GetCertificate callback with sync.RWMutex and double-checked locking for concurrent reads on every TLS handshake
• Existing connections: Unaffected — only new TLS handshakes see the new cert

Modified files:

• internal/pkg/api/server.go — wires CertReloader into TLS setup when feature is enabled
• internal/pkg/api/server_test.go — adds end-to-end cert reload test
• fleet-server.reference.yml — documents the new config option

How to test this PR locally

1. Build Fleet Server

mage build:local

2. Generate a CA and server cert/key pair

mkdir -p /tmp/tls-reload-test
cd /tmp/tls-reload-test

# Generate CA
openssl req -x509 -newkey rsa:2048 -keyout ca-key.pem -out ca.pem -days 1 -nodes -subj "/CN=Test CA"

# Generate server cert signed by the CA
openssl req -newkey rsa:2048 -keyout server-key.pem -out server.csr -nodes \
  -subj "/CN=localhost" -addext "subjectAltName=DNS:localhost,IP:127.0.0.1"
openssl x509 -req -in server.csr -CA ca.pem -CAkey ca-key.pem -CAcreateserial \
  -out server-cert.pem -days 1 -copy_extensions copyall

3. Create a config file

cat > /tmp/tls-reload-test/fleet-server.yml << 'CONF'
output:
  elasticsearch:
    hosts: ['localhost:9200']
    service_token: 'fake-token'

fleet:
  agent:
    id: test-agent-id

inputs:
  - type: fleet-server
    server:
      host: localhost
      port: 8220
      ssl:
        enabled: true
        certificate: /tmp/tls-reload-test/server-cert.pem
        key: /tmp/tls-reload-test/server-key.pem
CONF

4. Start Fleet Server

bin/fleet-server -c /tmp/tls-reload-test/fleet-server.yml \
  -E logging.to_files=true \
  -E logging.to_stderr=false \
  -E logging.files.path=/tmp/tls-reload-test \
  -E logging.files.name=fleet-server.log &

Wait for the "Listening on localhost:8220" log line:

grep "Listening" /tmp/tls-reload-test/fleet-server.log-*.ndjson

5. Make a TLS connection and note the server cert fingerprint

openssl s_client -connect localhost:8220 -CAfile /tmp/tls-reload-test/ca.pem \
  -servername localhost </dev/null 2>&1 | openssl x509 -noout -serial -fingerprint

6. Replace the cert/key files with a new pair (same CA)

# Generate the new cert/key outside the watched directory
openssl req -newkey rsa:2048 -keyout /tmp/server-key2.pem -out /tmp/server2.csr -nodes \
  -subj "/CN=localhost" -addext "subjectAltName=DNS:localhost,IP:127.0.0.1"
openssl x509 -req -in /tmp/server2.csr -CA /tmp/tls-reload-test/ca.pem \
  -CAkey /tmp/tls-reload-test/ca-key.pem -CAcreateserial \
  -out /tmp/server-cert2.pem -days 1 -copy_extensions copyall

# Move the new files into the watched directory
mv /tmp/server-cert2.pem /tmp/tls-reload-test/server-cert.pem
mv /tmp/server-key2.pem /tmp/tls-reload-test/server-key.pem

7. Wait ~5 seconds for the reload interval, then make another TLS connection

openssl s_client -connect localhost:8220 -CAfile /tmp/tls-reload-test/ca.pem \
  -servername localhost </dev/null 2>&1 | openssl x509 -noout -serial -fingerprint

The fingerprint should differ from step 5.

8. Stop Fleet Server

kill %1

---

Automated tests:

• go test -run Test_server_TLSCertReload ./internal/pkg/api/... — end-to-end integration test

Design Checklist

• I have ensured my design is stateless and will work when multiple fleet-server instances are behind a load balancer.
• I have or intend to scale test my changes, ensuring it will work reliably with 100K+ agents connected.
• I have included fail safe mechanisms to limit the load on fleet-server: rate limiting, circuit breakers, caching, load shedding, etc.

Checklist

• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in ./changelog/fragments using the changelog tool

Related issues

• Closes Allow Fleet Server to reload TLS certificates without restarting #6433

[mergify]

This pull request does not have a backport label. Could you fix it @ycombinator? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-./d./d is the label to automatically backport to the 8./d branch. /d is the digit
• backport-active-all is the label that automatically backports to all active branches.
• backport-active-8 is the label that automatically backports to all active minor branches for the 8 major.
• backport-active-9 is the label that automatically backports to all active minor branches for the 9 major.

[blakerouse]

How would you feel about changing this implementation? We talked about this at a team sync meeting and I was supposed to make an issue for it, but honestly I forgot until I was reading this PR.

https://github.com/open-telemetry/opentelemetry-collector/blob/7201768d04fb628e65cb4e3d5ead69d17c43d59d/config/configtls/configtls.go#L152

That design of config reload is something we discussed in the call. We overall liked the simple design no need for fsnotify or anything. We discussed about adding it to elastic-agent-libs so it could be used in multiple places.

Maybe we could do the implementation here first in Fleet Server and then split it out later.

[ycombinator]

@blakerouse Sure, I like the simplicity of the OTel implementation. It means having to wait up to ReloadInterval for the new cert to be picked up, but I think that's an acceptable tradeoff for the simpler implementation. I'll implement it in elastic-agent-libs and then update this PR here to use that implementation.

[mergify]

This pull request is now in conflicts. Could you fix it @ycombinator? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b pr/6835 upstream/pr/6835
git merge upstream/main
git push upstream pr/6835

[ycombinator]

@blakerouse Check out elastic/elastic-agent-libs#404. Also, I've updated this PR here to use CertReloader from that PR.

[blakerouse]

Nice! Looks good.

[swiatekm]

LGTM, one non-blocking question.