Skip to content

feat: limit concurrent HTTP outcalls with a guard#184

Draft
lpahlavi wants to merge 3 commits intomainfrom
lpahlavi/http-outcall-guard
Draft

feat: limit concurrent HTTP outcalls with a guard#184
lpahlavi wants to merge 3 commits intomainfrom
lpahlavi/http-outcall-guard

Conversation

@lpahlavi
Copy link
Copy Markdown
Contributor

Summary

  • Introduces HttpOutcallGuard, a count-based guard that tracks in-flight HTTP outcalls to the SOL RPC canister and refuses new ones once MAX_CONCURRENT_HTTP_OUTCALLS (50) are already in flight
  • Every SOL RPC function (get_transaction, submit_transaction, get_recent_slot_and_blockhash, get_signature_statuses, get_signatures_for_address) acquires the guard before making its outcall and releases it when the response arrives; failure to acquire returns a TooManyOutcalls error variant
  • All five timer functions (finalize_transactions, resubmit_transactions, consolidate_deposits, process_pending_withdrawals, poll_monitored_addresses) check too_many_http_outcalls() after confirming there is work to do and reschedule themselves instead of starting new work when the system is at capacity

Design

HttpOutcallGuard follows the same pattern as TimerGuard: it stores a u32 counter (active_http_outcalls) in canister State, increments it on acquisition, and decrements it on Drop. Because ICP uses cooperative multitasking, the check-and-increment is atomic with respect to other canister messages — no races are possible.

The limit of 50 aligns with the existing parallelism ceiling: 5 timer types × up to 10 concurrent RPC calls each (MAX_CONCURRENT_RPC_CALLS).

Test plan

  • Unit tests in minter/src/guard/tests.rs::http_outcall_guard cover: acquire/release, acquiring up to the limit, rejecting the limit+1-th attempt, recovering after a guard is dropped, and independent tracking of multiple guards
  • All 169 existing unit tests pass
  • All 25 existing PocketIC integration tests pass (verifies the guard does not block normal operation)

🤖 Generated with Claude Code

Adds `HttpOutcallGuard`, a count-based guard that tracks the number of
in-flight HTTP outcalls to the SOL RPC canister and rejects new ones once
`MAX_CONCURRENT_HTTP_OUTCALLS` (50) are already in flight.

Every SOL RPC function (`get_transaction`, `submit_transaction`,
`get_recent_slot_and_blockhash`, `get_signature_statuses`,
`get_signatures_for_address`) now acquires the guard before making the
actual outcall and releases it when the response is received. Timer
functions additionally check `too_many_http_outcalls()` after confirming
there is work to do and reschedule themselves rather than starting new
work when the system is at capacity.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Copilot AI review requested due to automatic review settings April 27, 2026 12:07
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a global concurrency limit for SOL RPC–triggered HTTP outcalls by introducing an HttpOutcallGuard and wiring it into the RPC layer and timer-driven workflows, so the canister backs off when at capacity.

Changes:

  • Introduces HttpOutcallGuard backed by a new State::active_http_outcalls counter and a too_many_http_outcalls() helper.
  • Wraps all SOL RPC entrypoints in the RPC module with the guard and adds a TooManyOutcalls error variant to each RPC error type.
  • Updates timer handlers to short-circuit work when outcall capacity is reached (and rely on rescheduling behavior).

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 7 comments.

Show a summary per file
File Description
minter/src/constants.rs Adds MAX_CONCURRENT_HTTP_OUTCALLS constant to define the global outcall cap.
minter/src/state/mod.rs Extends State with active_http_outcalls plus accessors/mutators and initializes it in TryFrom<InitArgs>.
minter/src/state/tests.rs Updates State construction assertions to include active_http_outcalls: 0.
minter/src/guard/mod.rs Implements HttpOutcallGuard and too_many_http_outcalls() using state-backed accounting.
minter/src/guard/tests.rs Adds unit tests for the new guard and saturation predicate.
minter/src/rpc/mod.rs Acquires HttpOutcallGuard in each RPC function and adds TooManyOutcalls error variants.
minter/src/monitor/mod.rs Adds capacity checks to finalize_transactions and resubmit_transactions timer handlers.
minter/src/consolidate/mod.rs Adds capacity check to consolidate_deposits timer handler.
minter/src/withdraw/mod.rs Adds capacity check to process_pending_withdrawals timer handler.
minter/src/deposit/automatic/mod.rs Adds capacity check to poll_monitored_addresses timer handler.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread minter/src/monitor/mod.rs Outdated
Comment on lines +67 to +71
log!(
Priority::Info,
"Too many concurrent HTTP outcalls, rescheduling finalize_transactions"
);
return;
Copy link

Copilot AI Apr 27, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because a scopeguard above schedules an immediate (Duration::ZERO) retry, returning early here when the system is at the outcall limit can cause a tight retry loop (timer fires, sees saturation, immediately reschedules again) and waste cycles. Consider scheduling the retry with a small backoff delay (or skipping the immediate reschedule and relying on the interval timer) when too_many_http_outcalls() is true.

Copilot uses AI. Check for mistakes.
Comment thread minter/src/monitor/mod.rs Outdated
log!(
Priority::Info,
"Too many concurrent HTTP outcalls, rescheduling resubmit_transactions"
);
Copy link

Copilot AI Apr 27, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because a scopeguard above schedules an immediate (Duration::ZERO) retry, returning early here when the system is at the outcall limit can cause a tight retry loop and waste cycles. Consider adding a small backoff delay for the retry (or avoiding the immediate reschedule and relying on the interval timer) when too_many_http_outcalls() is true.

Suggested change
);
);
scopeguard::ScopeGuard::into_inner(reschedule)
.set_timer(Duration::from_millis(100), resubmit_transactions);

Copilot uses AI. Check for mistakes.
Comment thread minter/src/consolidate/mod.rs Outdated
Comment on lines +61 to +65
log!(
Priority::Info,
"Too many concurrent HTTP outcalls, rescheduling consolidate_deposits"
);
return;
Copy link

Copilot AI Apr 27, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because a scopeguard above schedules an immediate (Duration::ZERO) retry, returning early here when the system is at the outcall limit can cause a tight retry loop and waste cycles. Consider scheduling the retry with a small backoff delay (or relying on the interval timer) when too_many_http_outcalls() is true.

Copilot uses AI. Check for mistakes.
Comment thread minter/src/withdraw/mod.rs Outdated
log!(
Priority::Info,
"Too many concurrent HTTP outcalls, rescheduling process_pending_withdrawals"
);
Copy link

Copilot AI Apr 27, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because a scopeguard above schedules an immediate (Duration::ZERO) retry, returning early here when the system is at the outcall limit can cause a tight retry loop and waste cycles. Consider scheduling the retry with a small backoff delay (or relying on the interval timer) when too_many_http_outcalls() is true.

Suggested change
);
);
let runtime = scopeguard::ScopeGuard::into_inner(reschedule);
runtime.set_timer(Duration::from_millis(100), process_pending_withdrawals);

Copilot uses AI. Check for mistakes.
Comment thread minter/src/deposit/automatic/mod.rs Outdated
log!(
Priority::Info,
"Too many concurrent HTTP outcalls, rescheduling poll_monitored_addresses"
);
Copy link

Copilot AI Apr 27, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because a scopeguard above schedules an immediate (Duration::ZERO) retry, returning early here when the system is at the outcall limit can cause a tight retry loop and waste cycles. Consider scheduling the retry with a small backoff delay (or relying on the interval timer) when too_many_http_outcalls() is true.

Suggested change
);
);
scopeguard::ScopeGuard::into_inner(reschedule);
runtime.set_timer(Duration::from_secs(1), poll_monitored_addresses);

Copilot uses AI. Check for mistakes.
Comment thread minter/src/guard/tests.rs
Comment on lines +167 to +176
let guards: Vec<_> = (0..MAX_CONCURRENT_HTTP_OUTCALLS)
.map(|_| HttpOutcallGuard::new().expect("should succeed below limit"))
.collect();

// Drop one
drop(guards);

// Should be able to acquire a new guard
let result = HttpOutcallGuard::new();
assert!(result.is_ok());
Copy link

Copilot AI Apr 27, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test intends to verify that releasing a single guard below the limit allows acquiring a new one, but drop(guards) drops the entire Vec (all guards), so it doesn’t exercise the intended behavior. Consider dropping just one element (e.g., remove one guard from the Vec) and asserting the counter decreases by 1 before acquiring a new guard.

Suggested change
let guards: Vec<_> = (0..MAX_CONCURRENT_HTTP_OUTCALLS)
.map(|_| HttpOutcallGuard::new().expect("should succeed below limit"))
.collect();
// Drop one
drop(guards);
// Should be able to acquire a new guard
let result = HttpOutcallGuard::new();
assert!(result.is_ok());
let mut guards: Vec<_> = (0..MAX_CONCURRENT_HTTP_OUTCALLS)
.map(|_| HttpOutcallGuard::new().expect("should succeed below limit"))
.collect();
// Drop one
drop(guards.pop().expect("expected at least one guard to drop"));
assert_eq!(
read_state(|s| s.active_http_outcalls()),
MAX_CONCURRENT_HTTP_OUTCALLS - 1
);
// Should be able to acquire a new guard
let result = HttpOutcallGuard::new();
assert!(result.is_ok());
assert_eq!(
read_state(|s| s.active_http_outcalls()),
MAX_CONCURRENT_HTTP_OUTCALLS
);

Copilot uses AI. Check for mistakes.
Comment thread minter/src/rpc/mod.rs
Comment on lines +28 to 30
let _guard = HttpOutcallGuard::new()
.map_err(|_: HttpOutcallGuardError| GetTransactionError::TooManyOutcalls)?;
let result = read_state(|state| state.sol_rpc_client(runtime.inter_canister_call_runtime()))
Copy link

Copilot AI Apr 27, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new TooManyOutcalls early-return path introduced by HttpOutcallGuard::new() isn’t covered by the existing unit tests in this module (which currently cover IC/RPC/inconsistent cases). Adding a test that forces the guard limit to be reached and asserts this function returns TooManyOutcalls would prevent regressions in the guard/error mapping.

Copilot uses AI. Check for mistakes.
lpahlavi and others added 2 commits April 27, 2026 14:23
Replace the upfront too_many_http_outcalls() check in timers with
inspection of the concurrent work's return values. Each timer now detects
TooManyOutcalls directly from its join_all results and skips defusing
the reschedule guard when any concurrent call was blocked by the limit.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Copilot AI review requested due to automatic review settings April 27, 2026 12:55
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants