feat(opamp): compute and store effective_config_hash for collector pipelines by juliaElastic · Pull Request #6872 · elastic/fleet-server

juliaElastic · 2026-04-21T14:25:29Z

Summary

Computes a SHA-256 hash of the OTel collector's pipeline topology from the OpAMP effective config and stores it as effective_config_hash in the .fleet-agents index.
Only topology fields are hashed: receivers, processors, exporters, connectors, service.pipelines, service.extensions. Non-topology fields (extensions.* config, service.telemetry, etc.) are excluded.
Keys are canonicalized via yaml.v3 Marshal (sorts alphabetically) before hashing, so key order never affects the output.
Hash is computed from the raw YAML body (before the sensitive-value redaction pass that produces effective_config).
Adds effective_config_label — a human-readable adjective-noun label (e.g. swift-hawk) derived from the first two bytes of the hash using two fixed 256-entry wordlists embedded in source. 65,536 possible combinations; stable across deployments and dependency updates since the wordlists are frozen in the codebase.

Closes: https://github.com/elastic/ingest-dev/issues/7064

To verify:

Start ES, Kibana, fleet-server locally https://github.com/elastic/fleet-server/blob/main/docs/opamp.md#setup
Go to Fleet UI, click on Add collector
Copy or download OTel config with OpAMP
Start EDOT or otel-contrib collector with the generated config
Verify that the effective_config_hash and effective_config_label is stored in .fleet-agents.

GET .fleet-agents/_search
{
    "_source": ["effective_config_hash", "effective_config_label", "effective_config"]
}

    "hits": [
      {
        "_index": ".fleet-agents-7",
        "_id": "95d87b10-299e-45ce-a5dd-72e500451e48",
        "_score": 1,
        "_source": {
          "effective_config_label": "hasty-bull",
          "effective_config": {
           ...
          },
          "effective_config_hash": "4e13e7566f2c31e563e235dd8e1a7ae8998a6677fae6124806ca842904f6aa0e"
        }
      },

Changed files

File	Change
`model/schema.json`	Added `effective_config_hash` and `effective_config_label` string properties to agent definition
`internal/pkg/model/schema.go`	Regenerated — adds `EffectiveConfigHash` and `EffectiveConfigLabel` fields to `Agent` struct
`internal/pkg/dl/constants.go`	Added `FieldEffectiveConfigHash` and `FieldEffectiveConfigLabel`
`internal/pkg/api/configHash.go`	New — `HashEffectiveConfig` + `hashConfigBody` implementation
`internal/pkg/api/configLabel.go`	New — `LabelFromHash` + 256-entry adjective/noun wordlists
`internal/pkg/api/handleOpAMP.go`	Calls `HashEffectiveConfig` and `LabelFromHash` in `updateAgent()`
`internal/pkg/checkin/bulk.go`	`effectiveConfigHash`/`effectiveConfigLabel` on `extraT`, `WithEffectiveConfigHash`/`WithEffectiveConfigLabel` options, `toUpdateBody` writes both fields
`internal/pkg/api/configHash_test.go`	New — 8 unit tests covering all hash acceptance criteria
`internal/pkg/api/configLabel_test.go`	New — 7 unit tests including wordlist uniqueness verification

Test plan

Hash (configHash_test.go)

TestHashEffectiveConfig_NilConfig — nil/empty config returns "" with no error
TestHashEffectiveConfig_EmptyBody — empty body returns "" with no error
TestHashEffectiveConfig_Determinism — same topology + different service.telemetry → same hash
TestHashEffectiveConfig_KeyOrderInvariant — shuffled key order → same hash
TestHashEffectiveConfig_TopologyChange — changed receiver → different hash
TestHashEffectiveConfig_AllowlistEnforcement — added extensions.* config → hash unchanged
TestHashEffectiveConfig_ServiceExtensionsIncluded — service.extensions list change → different hash
TestHashEffectiveConfig_HexEncoded — output is a 64-char lowercase hex string

Label (configLabel_test.go)

TestLabelFromHash_Format — produces adjective-noun format
TestLabelFromHash_Deterministic — same hash always gives same label
TestLabelFromHash_DifferentBytesGiveDifferentLabels — different first bytes → different label
TestLabelFromHash_SameLabelForSameTopology — same topology → same label
TestLabelFromHash_EmptyInput / TestLabelFromHash_InvalidHex — graceful empty return
TestWordlistsHave256UniqueEntries — both wordlists have exactly 256 unique entries

🤖 Generated with Claude Code

…pelines Computes a SHA-256 hash of the pipeline topology fields (receivers, processors, exporters, connectors, service.pipelines, service.extensions) from the OpAMP effective config and stores it as effective_config_hash in the .fleet-agents index. Non-topology fields (extensions config, service.telemetry, etc.) are excluded so the hash reflects only what the pipeline does, not how it is observed. Keys are canonicalized via yaml.v3 Marshal (which sorts alphabetically) before hashing to ensure identical topologies always produce the same hash regardless of key order. Closes elastic/ingest-dev#7064 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…h_test Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

mergify · 2026-04-21T15:02:05Z

This pull request does not have a backport label. Could you fix it @juliaElastic? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-./d./d is the label to automatically backport to the 8./d branch. /d is the digit
backport-active-all is the label that automatically backports to all active branches.
backport-active-8 is the label that automatically backports to all active minor branches for the 8 major.
backport-active-9 is the label that automatically backports to all active minor branches for the 9 major.

…onfig hash Adds an adjective-noun label (e.g. "swift-hawk") stored alongside effective_config_hash in .fleet-agents. The label is derived from the first two bytes of the SHA-256 hash using two fixed 256-entry wordlists embedded in source, giving 65,536 possible combinations. Because the wordlists are frozen in the codebase the mapping is stable across deployments and dependency updates. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

juliaElastic · 2026-04-22T15:17:10Z

CI fails due to an environmental issue, not related to changes in this PR:

The job, Package x86_64, has been canceled as it failed to get an agent after 5 tries.

The job, Package x86_64 FIPS, has been canceled as it failed to get an agent after 5 tries.

ebeahan · 2026-04-23T01:09:09Z

CI fails due to an environmental issue, not related to changes in this PR

#6881 should address the CI issues.

michel-laterman · 2026-04-23T20:13:07Z

+	if effectiveConfig.ConfigMap == nil || effectiveConfig.ConfigMap.ConfigMap[""] == nil {
+		return "", nil
+	}
+	body := effectiveConfig.ConfigMap.ConfigMap[""].Body


In theory we should hash everything in the configmap, not just the default config

Added support for multiple config files: 9ac32c8

Though I'm not sure if two separate files should hash differently from the same content in one file.

michel-laterman · 2026-04-23T20:15:40Z

+	}
+
+	topology := make(map[string]any)
+	for _, k := range []string{"receivers", "processors", "exporters", "connectors"} {


Why do we need to only use these keys when computing the sha?
How do we ensure that a change to another key is emitted to a collector?

Why do we need to only use these keys when computing the sha?

The issue scope is deliberate: the hash is meant to answer "are two collectors running the same pipeline wiring?", not "are all their configs byte-for-byte identical." The topology keys describe what data flows where. Non-topology fields like extensions.* configs or service.telemetry describe how the pipeline operates (scrape intervals, log levels, endpoints) but don't change the pipeline graph.

The use case is: two collectors that differ only in their health_check endpoint or telemetry verbosity should be considered "the same topology" and produce the same hash, so you can cluster/query them together.

How do we ensure that a change to another key is emitted to a collector?

Changes to other keys are not surfaced via the hash, but they are captured in full in effective_config (the redacted JSON blob already stored in .fleet-agents). If a non-topology field changes, effective_config changes, but effective_config_hash stays the same.

Can you add that to the doc string or in schema.json? It's important to note how this hash should differ from the RemoteConfig hash

michel-laterman · 2026-04-23T20:21:48Z

+	hash := "aabbccdd" + strings.Repeat("00", 28) // 64-char hex
+	assert.Equal(t, LabelFromHash(hash), LabelFromHash(hash))


What's the point of this test?

it tests that calling the LabelFromHash is deterministic, gives the same response when called twice

Co-authored-by: Michel Laterman <82832767+michel-laterman@users.noreply.github.com>

…entry HashEffectiveConfig now iterates all named config files in the OpAMP ConfigMap in sorted key order, feeding each file's topology fields and its key name into a single SHA-256. Previously only the default ("") file was considered. Topology extraction is refactored into extractTopologyFields for per-file reuse. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

michel-laterman · 2026-04-24T17:24:41Z

+	keys := make([]string, 0, len(effectiveConfig.ConfigMap.ConfigMap))
+	for k := range effectiveConfig.ConfigMap.ConfigMap {
+		keys = append(keys, k)
+	}
+	sort.Strings(keys)


Suggested change

keys := make([]string, 0, len(effectiveConfig.ConfigMap.ConfigMap))

for k := range effectiveConfig.ConfigMap.ConfigMap {

keys = append(keys, k)

}

sort.Strings(keys)

keys := slices.Sorted(maps.Keys(effectiveConfig.ConfigMap.ConfigMap))

michel-laterman · 2026-04-24T17:29:01Z

+	}
+
+	topology := make(map[string]any)
+	for _, k := range []string{"receivers", "processors", "exporters", "connectors"} {


Can you add that to the doc string or in schema.json? It's important to note how this hash should differ from the RemoteConfig hash

juliaElastic added enhancement New feature or request backport-skip Skip notification from the automated backport with mergify skip-changelog labels Apr 21, 2026

fix lint: extract duplicate string literals to constants in configHas…

6990501

…h_test Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

juliaElastic mentioned this pull request Apr 21, 2026

Add effective_config_hash keyword mapping to fleet-agents index elastic/elasticsearch#146975

Open

2 tasks

juliaElastic marked this pull request as ready for review April 22, 2026 07:17

juliaElastic requested a review from a team as a code owner April 22, 2026 07:17

juliaElastic requested review from blakerouse and samuelvl April 22, 2026 07:17

juliaElastic and others added 2 commits April 22, 2026 10:51

Merge branch 'main' into hash-effective-config

7e2ffd2

juliaElastic mentioned this pull request Apr 22, 2026

[Fleet] Add effective_config_hash and effective_config_label to UI elastic/kibana#265005

Open

10 tasks

juliaElastic requested a review from a team April 22, 2026 11:37

Merge branch 'main' into hash-effective-config

bbd2bf0

juliaElastic requested review from michel-laterman and ycombinator April 23, 2026 08:01

michel-laterman reviewed Apr 23, 2026

View reviewed changes

juliaElastic and others added 3 commits April 24, 2026 09:01

Update internal/pkg/api/configLabel_test.go

d1f6866

Co-authored-by: Michel Laterman <82832767+michel-laterman@users.noreply.github.com>

Update internal/pkg/api/handleOpAMP.go

95b387e

Co-authored-by: Michel Laterman <82832767+michel-laterman@users.noreply.github.com>

pierrehilbert added the Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team label Apr 24, 2026

juliaElastic requested a review from michel-laterman April 24, 2026 10:53

michel-laterman approved these changes Apr 24, 2026

View reviewed changes

		hash := "aabbccdd" + strings.Repeat("00", 28) // 64-char hex
		assert.Equal(t, LabelFromHash(hash), LabelFromHash(hash))

Conversation

juliaElastic commented Apr 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changed files

Test plan

Uh oh!

mergify Bot commented Apr 21, 2026

Uh oh!

juliaElastic commented Apr 22, 2026

Uh oh!

ebeahan commented Apr 23, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

juliaElastic commented Apr 21, 2026 •

edited

Loading