INTEGRATION [PR#2749 > development/9.4] BB-780: Configure CRR queue processor poll interval#2758
Merged
bert-e merged 8 commits intoJun 9, 2026
Conversation
TDD: specify the contract first. The replication queue processor accepts an optional maxPollIntervalMs in its config block; tests/config.json sets a distinctive value (350000) so the pass-through is provable, the value is optional, and values below 45000 are rejected. These tests fail until the next commit adds the schema and wiring (both pushed together so CI sees the green tip).
max.poll.interval.ms was pinned to its 5 minute default. Lowering mpu_parts_concurrency to protect Metadata makes a large-MPU replication exceed it, which evicts the queue processor's consumer and redelivers the partition - losing the partial transfer and leaving orphan parts. Let the replication queue processor set an optional maxPollIntervalMs in its config block (min 45000, the librdkafka session.timeout.ms floor); unset falls back to BackbeatConsumer's existing 300000 default. Scoped to the queue processor - the only consumer with the slow-task problem (the status processor does fast metadata writes). Adds a matching EXTENSIONS_REPLICATION_QUEUE_PROCESSOR_MAX_POLL_INTERVAL_MS mapping.
The schema only enforced a floor (min 45000). Add a 30-minute ceiling (1800000 ms) so an operator can't set an unbounded maxPollIntervalMs. This knob exists to let a single slow large-MPU transfer finish, not to mask a wedged consumer: a poll gap beyond 30 min means the task is stuck, and we want kafka to evict and redeliver rather than hide the failure behind an ever-growing timeout. Over-limit values are rejected, mirroring how the floor rejects under-limit values (no silent clamping).
…rval' into w/9.1/improvement/BB-780-crr-poll-interval
…topus/w/9.2/improvement/BB-780-crr-poll-interval
…topus/w/9.3/improvement/BB-780-crr-poll-interval
…topus/w/9.4/improvement/BB-780-crr-poll-interval
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files
... and 2 files with indirect coverage changes
@@ Coverage Diff @@
## development/9.4 #2758 +/- ##
===================================================
- Coverage 74.89% 74.88% -0.02%
===================================================
Files 200 200
Lines 13668 13670 +2
===================================================
Hits 10237 10237
- Misses 3421 3423 +2
Partials 10 10
Flags with carried forward coverage won't be shown. Click here to find out more. 🚀 New features to boost your workflow:
|
|
LGTM — clean implementation. The new optional maxPollIntervalMs config field has correct Joi bounds (45s–30min), passes through QueueProcessor to BackbeatConsumer properly, falls back to BackbeatConsumer's existing 300s default when unset, and the docker-entrypoint.sh env var override follows established patterns. Tests cover the happy path, unset case, and both boundary violations. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request has been created automatically.
It is linked to its parent pull request #2749.
Do not edit this pull request directly.
If you need to amend/cancel the changeset on branch
w/9.4/improvement/BB-780-crr-poll-interval, please follow thisprocedure:
git fetch git checkout w/9.4/improvement/BB-780-crr-poll-interval # <amend or cancel the changeset by _adding_ new commits> git push origin w/9.4/improvement/BB-780-crr-poll-intervalPlease always comment pull request #2749 instead of this one.