Add --queues CLI option for slot group configuration#1291
Merged
Add --queues CLI option for slot group configuration#1291
Conversation
One test still failing
Casing in the yaml files was causing issues
Worker: use New batch logging API
tests: deploy tests
…flag-and-an-openfn-project-clean-command feat: openfn project clean
fix package lock
stuartc
force-pushed
the
1288-queues-cli-option
branch
from
a1d3a73 to
852eb55
Compare
Introduce queue-aware slot allocation to support fast lanes. Workers can now dedicate specific slots to specific queues via a new `--queues` flag (e.g. `--queues "fast_lane:1 manual,*:4"`). - Add parseQueues() with SlotGroup type and full validation - Add --queues CLI option with WORKER_QUEUES env var support - Enforce mutual exclusivity between --queues and --capacity - Derive effectiveCapacity from slot groups in start.ts - Add queues field to ClaimPayload in lexicon
- Remove unreachable ?? 5 fallback in start.ts (capacity always has default) - Add TODO(#1289) for passing slotGroups to createWorker - Warn on duplicate queue names within a preference chain - Warn on identical queue configurations across slot groups - Add test for WORKER_QUEUES + WORKER_CAPACITY env var mutual exclusivity
capacity always receives a default value via setArg, so the type should reflect that rather than requiring a runtime fallback.
stuartc
force-pushed
the
1288-queues-cli-option
branch
from
70fee4a to
3158557
Compare
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Short Description
Add
--queuesCLI option andWORKER_QUEUESenv var for configuring worker slot groups, enabling dedicated capacity for named queues (e.g., fast lane for sync webhooks).Fixes #1288
Implementation Details
--queuesCLI option: Parses"<queues>:<count> ..."syntax into typedSlotGroup[](e.g.,--queues "fast_lane:1 manual,*:4")WORKER_QUEUESenv var: Alternative to CLI flag, same syntax--queuesand--capacitycannot be used together--queuesis not provided, derivesmanual,*:<capacity>from--capacity(default 5)args.capacityinmaxWorkflowsandmaxWorkers*wildcard only at end of preference chain, count >= 1queues?: string[]toClaimPayloadtypeSlot groups are parsed and validated but not yet wired into the workloop/claim protocol — that's #1289.
QA Notes
--capacitywithout--queuescontinues to work identically to beforeslotGroupsvariable is computed but not yet passed tocreateWorker(TODO for Implement per-group workloops and queue-aware claiming #1289)AI Usage