feat: limit concurrent HTTP outcalls with a guard#184
Conversation
Adds `HttpOutcallGuard`, a count-based guard that tracks the number of in-flight HTTP outcalls to the SOL RPC canister and rejects new ones once `MAX_CONCURRENT_HTTP_OUTCALLS` (50) are already in flight. Every SOL RPC function (`get_transaction`, `submit_transaction`, `get_recent_slot_and_blockhash`, `get_signature_statuses`, `get_signatures_for_address`) now acquires the guard before making the actual outcall and releases it when the response is received. Timer functions additionally check `too_many_http_outcalls()` after confirming there is work to do and reschedule themselves rather than starting new work when the system is at capacity. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Adds a global concurrency limit for SOL RPC–triggered HTTP outcalls by introducing an HttpOutcallGuard and wiring it into the RPC layer and timer-driven workflows, so the canister backs off when at capacity.
Changes:
- Introduces
HttpOutcallGuardbacked by a newState::active_http_outcallscounter and atoo_many_http_outcalls()helper. - Wraps all SOL RPC entrypoints in the RPC module with the guard and adds a
TooManyOutcallserror variant to each RPC error type. - Updates timer handlers to short-circuit work when outcall capacity is reached (and rely on rescheduling behavior).
Reviewed changes
Copilot reviewed 10 out of 10 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
minter/src/constants.rs |
Adds MAX_CONCURRENT_HTTP_OUTCALLS constant to define the global outcall cap. |
minter/src/state/mod.rs |
Extends State with active_http_outcalls plus accessors/mutators and initializes it in TryFrom<InitArgs>. |
minter/src/state/tests.rs |
Updates State construction assertions to include active_http_outcalls: 0. |
minter/src/guard/mod.rs |
Implements HttpOutcallGuard and too_many_http_outcalls() using state-backed accounting. |
minter/src/guard/tests.rs |
Adds unit tests for the new guard and saturation predicate. |
minter/src/rpc/mod.rs |
Acquires HttpOutcallGuard in each RPC function and adds TooManyOutcalls error variants. |
minter/src/monitor/mod.rs |
Adds capacity checks to finalize_transactions and resubmit_transactions timer handlers. |
minter/src/consolidate/mod.rs |
Adds capacity check to consolidate_deposits timer handler. |
minter/src/withdraw/mod.rs |
Adds capacity check to process_pending_withdrawals timer handler. |
minter/src/deposit/automatic/mod.rs |
Adds capacity check to poll_monitored_addresses timer handler. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| log!( | ||
| Priority::Info, | ||
| "Too many concurrent HTTP outcalls, rescheduling finalize_transactions" | ||
| ); | ||
| return; |
There was a problem hiding this comment.
Because a scopeguard above schedules an immediate (Duration::ZERO) retry, returning early here when the system is at the outcall limit can cause a tight retry loop (timer fires, sees saturation, immediately reschedules again) and waste cycles. Consider scheduling the retry with a small backoff delay (or skipping the immediate reschedule and relying on the interval timer) when too_many_http_outcalls() is true.
| log!( | ||
| Priority::Info, | ||
| "Too many concurrent HTTP outcalls, rescheduling resubmit_transactions" | ||
| ); |
There was a problem hiding this comment.
Because a scopeguard above schedules an immediate (Duration::ZERO) retry, returning early here when the system is at the outcall limit can cause a tight retry loop and waste cycles. Consider adding a small backoff delay for the retry (or avoiding the immediate reschedule and relying on the interval timer) when too_many_http_outcalls() is true.
| ); | |
| ); | |
| scopeguard::ScopeGuard::into_inner(reschedule) | |
| .set_timer(Duration::from_millis(100), resubmit_transactions); |
| log!( | ||
| Priority::Info, | ||
| "Too many concurrent HTTP outcalls, rescheduling consolidate_deposits" | ||
| ); | ||
| return; |
There was a problem hiding this comment.
Because a scopeguard above schedules an immediate (Duration::ZERO) retry, returning early here when the system is at the outcall limit can cause a tight retry loop and waste cycles. Consider scheduling the retry with a small backoff delay (or relying on the interval timer) when too_many_http_outcalls() is true.
| log!( | ||
| Priority::Info, | ||
| "Too many concurrent HTTP outcalls, rescheduling process_pending_withdrawals" | ||
| ); |
There was a problem hiding this comment.
Because a scopeguard above schedules an immediate (Duration::ZERO) retry, returning early here when the system is at the outcall limit can cause a tight retry loop and waste cycles. Consider scheduling the retry with a small backoff delay (or relying on the interval timer) when too_many_http_outcalls() is true.
| ); | |
| ); | |
| let runtime = scopeguard::ScopeGuard::into_inner(reschedule); | |
| runtime.set_timer(Duration::from_millis(100), process_pending_withdrawals); |
| log!( | ||
| Priority::Info, | ||
| "Too many concurrent HTTP outcalls, rescheduling poll_monitored_addresses" | ||
| ); |
There was a problem hiding this comment.
Because a scopeguard above schedules an immediate (Duration::ZERO) retry, returning early here when the system is at the outcall limit can cause a tight retry loop and waste cycles. Consider scheduling the retry with a small backoff delay (or relying on the interval timer) when too_many_http_outcalls() is true.
| ); | |
| ); | |
| scopeguard::ScopeGuard::into_inner(reschedule); | |
| runtime.set_timer(Duration::from_secs(1), poll_monitored_addresses); |
| let guards: Vec<_> = (0..MAX_CONCURRENT_HTTP_OUTCALLS) | ||
| .map(|_| HttpOutcallGuard::new().expect("should succeed below limit")) | ||
| .collect(); | ||
|
|
||
| // Drop one | ||
| drop(guards); | ||
|
|
||
| // Should be able to acquire a new guard | ||
| let result = HttpOutcallGuard::new(); | ||
| assert!(result.is_ok()); |
There was a problem hiding this comment.
This test intends to verify that releasing a single guard below the limit allows acquiring a new one, but drop(guards) drops the entire Vec (all guards), so it doesn’t exercise the intended behavior. Consider dropping just one element (e.g., remove one guard from the Vec) and asserting the counter decreases by 1 before acquiring a new guard.
| let guards: Vec<_> = (0..MAX_CONCURRENT_HTTP_OUTCALLS) | |
| .map(|_| HttpOutcallGuard::new().expect("should succeed below limit")) | |
| .collect(); | |
| // Drop one | |
| drop(guards); | |
| // Should be able to acquire a new guard | |
| let result = HttpOutcallGuard::new(); | |
| assert!(result.is_ok()); | |
| let mut guards: Vec<_> = (0..MAX_CONCURRENT_HTTP_OUTCALLS) | |
| .map(|_| HttpOutcallGuard::new().expect("should succeed below limit")) | |
| .collect(); | |
| // Drop one | |
| drop(guards.pop().expect("expected at least one guard to drop")); | |
| assert_eq!( | |
| read_state(|s| s.active_http_outcalls()), | |
| MAX_CONCURRENT_HTTP_OUTCALLS - 1 | |
| ); | |
| // Should be able to acquire a new guard | |
| let result = HttpOutcallGuard::new(); | |
| assert!(result.is_ok()); | |
| assert_eq!( | |
| read_state(|s| s.active_http_outcalls()), | |
| MAX_CONCURRENT_HTTP_OUTCALLS | |
| ); |
| let _guard = HttpOutcallGuard::new() | ||
| .map_err(|_: HttpOutcallGuardError| GetTransactionError::TooManyOutcalls)?; | ||
| let result = read_state(|state| state.sol_rpc_client(runtime.inter_canister_call_runtime())) |
There was a problem hiding this comment.
The new TooManyOutcalls early-return path introduced by HttpOutcallGuard::new() isn’t covered by the existing unit tests in this module (which currently cover IC/RPC/inconsistent cases). Adding a test that forces the guard limit to be reached and asserts this function returns TooManyOutcalls would prevent regressions in the guard/error mapping.
Replace the upfront too_many_http_outcalls() check in timers with inspection of the concurrent work's return values. Each timer now detects TooManyOutcalls directly from its join_all results and skips defusing the reschedule guard when any concurrent call was blocked by the limit. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Summary
HttpOutcallGuard, a count-based guard that tracks in-flight HTTP outcalls to the SOL RPC canister and refuses new ones onceMAX_CONCURRENT_HTTP_OUTCALLS(50) are already in flightget_transaction,submit_transaction,get_recent_slot_and_blockhash,get_signature_statuses,get_signatures_for_address) acquires the guard before making its outcall and releases it when the response arrives; failure to acquire returns aTooManyOutcallserror variantfinalize_transactions,resubmit_transactions,consolidate_deposits,process_pending_withdrawals,poll_monitored_addresses) checktoo_many_http_outcalls()after confirming there is work to do and reschedule themselves instead of starting new work when the system is at capacityDesign
HttpOutcallGuardfollows the same pattern asTimerGuard: it stores au32counter (active_http_outcalls) in canisterState, increments it on acquisition, and decrements it onDrop. Because ICP uses cooperative multitasking, the check-and-increment is atomic with respect to other canister messages — no races are possible.The limit of 50 aligns with the existing parallelism ceiling: 5 timer types × up to 10 concurrent RPC calls each (
MAX_CONCURRENT_RPC_CALLS).Test plan
minter/src/guard/tests.rs::http_outcall_guardcover: acquire/release, acquiring up to the limit, rejecting thelimit+1-th attempt, recovering after a guard is dropped, and independent tracking of multiple guards🤖 Generated with Claude Code