Skip to content

fix(lambda): rate-limit retry warn log to avoid log storms#6318

Open
fmassot wants to merge 4 commits intomainfrom
fmassot/rate-limit-lambda-retry-warn
Open

fix(lambda): rate-limit retry warn log to avoid log storms#6318
fmassot wants to merge 4 commits intomainfrom
fmassot/rate-limit-lambda-retry-warn

Conversation

@fmassot
Copy link
Copy Markdown
Collaborator

@fmassot fmassot commented Apr 19, 2026

Summary

  • Lambda invocation retries emit a WARN log on every attempt. Under heavy rate-limiting this floods customer logs, 700k+ logs was observed.
  • Switch to quickwit_common::rate_limited_warn!(limit_per_min = 10, ...) so at most 10 warnings are emitted per minute regardless of retry volume.

Test plan

  • cargo check -p quickwit-lambda-client passes
  • Manual: trigger Lambda rate-limiting and confirm log volume is bounded

🤖 Generated with Claude Code

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@fmassot fmassot requested a review from trinity-1686a April 19, 2026 17:22
fmassot and others added 3 commits April 21, 2026 17:22
When the rate limiter opens a new window after suppressing messages,
it now emits a preceding log at the same level:
"suppressed N similar log messages in the last minute"

Closes #6322.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
… enum

The three possible outcomes (log normally, log after suppression, suppress)
are now distinct enum variants, removing the implicit invariant that the
skipped count is only meaningful when the bool is true.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@fmassot fmassot mentioned this pull request Apr 26, 2026
11 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants