Skip to content

✨ S3UTILS-230 implement CRR multi-site V2 replication format#396

Open
DarkIsDude wants to merge 4 commits into
development/1from
feature/S3UTILS-230/multi-crr
Open

✨ S3UTILS-230 implement CRR multi-site V2 replication format#396
DarkIsDude wants to merge 4 commits into
development/1from
feature/S3UTILS-230/multi-crr

Conversation

@DarkIsDude

@DarkIsDude DarkIsDude commented May 28, 2026

Copy link
Copy Markdown

Summary

  • Add V2 replication format support to `ReplicationStatusUpdater`, implementing the CRR to Multiple Sites design
  • V2 configs (rules with a `Filter` element) route through a new `_markObjectPendingV2` path that writes per-backend `destination`/`role` and removes legacy top-level `storageClass`/`storageType`/`destination`
  • V1 configs (no `Filter` on any rule) retain existing behavior unchanged
  • Arsenal static methods (`resolveBackends`, `resolveSourceRole`, `resolveDestinationRole`) are reused instead of maintaining parallel implementations; arsenal dependency pinned to `improvement/ARSN-571/crr-multi`

Changes

`CRR/ReplicationStatusUpdater.js`

  • `_isV1Format()` — V2 detection aligned with arsenal: a rule is V2 if it carries a `Filter` element
  • `_getMatchingRules()` — prefix-matches object key against enabled rules, deduplicates by site (highest priority wins); feeds the adapter for `resolveBackends`
  • `_computeTopLevelStatus()` — aggregates backend statuses: FAILED > PROCESSING > COMPLETED
  • `_removeV1Fields()` — strips legacy top-level `storageClass`/`storageType`/`destination` when migrating to V2
  • `_markObjectPendingV2()` — V2 metadata writer: adapts AWS SDK rule shape to arsenal's `ReplicationConfigurationMetadata`, calls `ReplicationConfiguration.resolveBackends` for prefix matching / priority dedup / per-backend destination+role stamping, then calls `resolveSourceRole` for the top-level role

`package.json` — arsenal pinned to `improvement/ARSN-571/crr-multi` for the new static methods

`tests/utils/crr.js` — V2 bucket replication config fixture

`tests/unit/CRR/ReplicationStatusUpdater.js` — 7 new V2 test cases (single match, dual-rule prefix overlap, no-match skip, SITE_NAME filter, all-up-to-date skip, PROCESSING aggregate status, forceUsingConfiguration)

Test plan

  • All 456 unit tests pass (`yarn test:unit`)
  • Lint clean (`yarn lint`)
  • V2 metadata written with per-backend `destination`/`role` and no legacy top-level fields
  • V1 configs produce identical output to before this change

Issue: S3UTILS-230

⚠️ The procedure at this page seems not up to date : https://scality.atlassian.net/wiki/spaces/TS/pages/2312536220/Ingestion+Replication+XDM#Step-one---Clear-the-replication-queue. The ENDPOINT should now be http://.... with the port :80. Should we ping CS or update the page directly ?

@DarkIsDude DarkIsDude self-assigned this May 28, 2026
@bert-e

This comment was marked as resolved.

@bert-e

This comment was marked as resolved.

Comment thread tests/unit/CRR/ReplicationStatusUpdater.js Outdated
@claude

This comment was marked as resolved.

@codecov

codecov Bot commented May 28, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 88.28829% with 13 lines in your changes missing coverage. Please review.
✅ Project coverage is 45.47%. Comparing base (eb9592d) to head (82575ce).

Files with missing lines Patch % Lines
CRR/ReplicationStatusUpdater.js 88.28% 10 Missing and 3 partials ⚠️
Additional details and impacted files
@@                Coverage Diff                @@
##           development/1     #396      +/-   ##
=================================================
+ Coverage          44.90%   45.47%   +0.56%     
=================================================
  Files                 88       88              
  Lines               6456     6547      +91     
  Branches            1352     1373      +21     
=================================================
+ Hits                2899     2977      +78     
- Misses              3511     3521      +10     
- Partials              46       49       +3     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Comment thread CRR/ReplicationStatusUpdater.js Outdated
Comment thread CRR/ReplicationStatusUpdater.js Outdated
Comment thread CRR/ReplicationStatusUpdater.js Outdated
Comment thread CRR/ReplicationStatusUpdater.js Outdated
Comment thread CRR/ReplicationStatusUpdater.js
@DarkIsDude DarkIsDude force-pushed the feature/S3UTILS-230/multi-crr branch from d5490b8 to f9fca0c Compare June 1, 2026 20:17
Comment thread CRR/ReplicationStatusUpdater.js Outdated
@claude

This comment was marked as resolved.

@claude

This comment was marked as resolved.

@DarkIsDude DarkIsDude requested review from a team, delthas and maeldonn June 2, 2026 14:13
@bert-e

bert-e commented Jun 2, 2026

Copy link
Copy Markdown
Contributor

Waiting for approval

The following approvals are needed before I can proceed with the merge:

  • the author

  • 2 peers

@maeldonn maeldonn left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can be simplified a lot by using arsenal logic

Comment thread CRR/ReplicationStatusUpdater.js Outdated
Comment thread CRR/ReplicationStatusUpdater.js Outdated
Comment thread CRR/ReplicationStatusUpdater.js
Comment thread CRR/ReplicationStatusUpdater.js
Comment thread CRR/ReplicationStatusUpdater.js
@bert-e

bert-e commented Jun 5, 2026

Copy link
Copy Markdown
Contributor

Waiting for approval

The following approvals are needed before I can proceed with the merge:

  • the author

  • 2 peers

The following reviewers are expecting changes from the author, or must review again:

Comment thread package.json
@claude

This comment was marked as resolved.

Comment thread package.json
"@smithy/util-retry": "^4.0.7",
"JSONStream": "^1.3.5",
"arsenal": "git+https://github.com/scality/arsenal#8.2.36",
"arsenal": "git+https://github.com/scality/Arsenal#improvement/ARSN-571/crr-multi",

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

arsenal is pinned to branch improvement/ARSN-571/crr-multi instead of a version tag. Git-based deps (arsenal, vaultclient, bucketclient, werelogs, httpagent) must pin to a tag for reproducible builds — a branch ref means different builds can resolve to different commits.

```suggestion
"arsenal": "git+https://github.com/scality/Arsenal#",

@claude

claude Bot commented Jun 8, 2026

Copy link
Copy Markdown
  • Dependency pinning: arsenal is pinned to branch improvement/ARSN-571/crr-multi instead of a version tag. Must be pinned to a tag before merge for reproducible builds.
    - Pin to the arsenal release tag that includes the ARSN-571 changes once it is published.

    Review by Claude Code

@DarkIsDude DarkIsDude requested a review from maeldonn June 8, 2026 17:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants