Configuration Merging Strategy via yq

[mlorenzofr]

Overview

As our configuration requirements for values-hub.yaml grow more complex, we require a robust method to overlay scenario-specific configurations (e.g., supply-chain) without duplicating the base boilerplate or losing existing comments.

I propose implementing a merging strategy using yq to perform a deep, additive merge of YAML structures.

Proposed Solution

The implementation utilizes the eval-all (or ea) yq command to process multiple files as a single stream, applying a reduction operator to combine them.

Implementation Command

yq eval-all '. as $item ireduce ({}; . *+ $item)' values-hub.yaml docs/conf/supply-chain.yaml > values-supply-chain.yaml

How it works

• eval-all: Loads all input files into memory before processing, allowing cross-file operations.
• . as $item ireduce ({}; ...): This initializes an empty object {} and iterates through each file, assigning the current file to the variable $item and merging it into the accumulator.
• *+ (Deep Additive Merge): Unlike the standard merge operator (*), the *+ operator:
  • Recursively merges objects (maps).
  • Appends arrays (sequences) instead of overwriting them.
  • Overwrites scalar values (strings, numbers) if they exist in the second file.

Comparison: yq vs. diff/patch

Feature	diff / patch	yq (Structural Merge)
Awareness	Line-based (Text)	Schema-based (Data)
Indentation	Fragile. Minor space changes break the patch.	Agnostic. Understands YAML hierarchy.
Comments	Usually preserved, but hard to target.	Preserved (yq maintains head/foot comments).
Ordering	Requires exact line matching.	Order-independent, merges by key name.
Logic	None.	Supports filtering and conditional merging.

Strengths & Benefits

• Schema Integrity: The output is guaranteed to be valid YAML.
• Maintenance: We can keep a "Clean Base" in values-hub.yaml and only define the delta in the scenario-specific configurations.
• Comment Preservation: yq is specifically designed to retain documentation comments, which is critical for our "Example-driven" configuration style.
• Idempotency: Running the merge multiple times with the same inputs consistently produces the same structured output.

Weaknesses & Considerations

• Array Duplication: If the same item exists in both the base and the override list, *+ will result in a duplicate entry. This can be mitigated by piping to unique if necessary.
• Maintenance.
  • We must create a configuration file for each scenario.
  • If we add new options or modify existing components, we must also update the scenario's configuration file.

[p-rog]

I like this approach. Having configuration file for each scenario is not a problem, it's even good idea, because project will grow over the time.

[yukinchan]

Love this approach +1.
This will streamline the automation by a lot.

[minmzzhang]

This is a very lightweight method for quick manual merge. I have a few points:

1. As you also noted, *+ appends lists unconditionally. This is correct for cases like qtodo.overrides (adding supply-chain specific entries on top of the base oidc/spire/vault entries). But if someone copies a base override into the overlay to change its value, *+ will produce a duplicate with conflicting values and no warning. Worth documenting this pitfall for anyone creating new scenario overlays.

2. Users need to know the exact yq incantation from the PR description. A Makefile target or a small wrapper script would make this more discoverable and less error-prone.

3. Shall we also consider splitting into per-feature files for composability?