feat(storage): emit per-RPC latency log for put/get handlers#95
Merged
jacderida merged 1 commit intoMay 12, 2026
Merged
Conversation
Adds a structured `info!` log event per chunk-store PUT and GET RPC with duration_ms, outcome, and address (and chunk_size on PUT). The target is `ant_node::storage::rpc_latency` so Elasticsearch / Kibana can build p50/p95/p99 store-RPC latency histograms from the existing telegraf log forwarding without any node-side metric pipeline. Motivates DEV-01 post-mortem: aggregate upload throughput halved over 12h before any node failures, file-size sensitivity points at CLOSE_GROUP quorum / slowest-peer cost on the client, but the node-side per-RPC service time is currently unobservable. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Adds structured per-RPC latency tracing for chunk-store PUT/GET handlers so latency histograms can be built from forwarded logs (without adding a metrics pipeline), by wrapping the existing handler logic and emitting a single info! event with duration_ms, outcome, and chunk address (plus chunk_size on PUT).
Changes:
- Wrapped
handle_put/handle_getwith timing and a structuredinfo!event targetingant_node::storage::rpc_latency. - Extracted the previous PUT/GET logic into
handle_put_inner/handle_get_innerso the wrapper can log on all return paths.
Comments suppressed due to low confidence (1)
src/storage/handler.rs:296
- Same as PUT:
Instant::now(),hex::encode(...), andduration_ms/outcomeare computed regardless of whether therpc_latencyinfo event will actually be emitted. Please gate this work withcrate::logging::enabled!(preferably for this target) and avoid the extra hex-encode in the wrapper (the inner handler already encodes for its debug/warn logs).
let start = std::time::Instant::now();
let addr_hex = hex::encode(request.address);
let response = self.handle_get_inner(request).await;
let duration_ms = u64::try_from(start.elapsed().as_millis()).unwrap_or(u64::MAX);
let outcome: &'static str = match &response {
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+174
to
+194
| let start = std::time::Instant::now(); | ||
| let addr_hex = hex::encode(request.address); | ||
| let chunk_size = request.content.len(); | ||
| let response = self.handle_put_inner(request).await; | ||
| let duration_ms = u64::try_from(start.elapsed().as_millis()).unwrap_or(u64::MAX); | ||
| let outcome: &'static str = match &response { | ||
| ChunkPutResponse::Success { .. } => "success", | ||
| ChunkPutResponse::AlreadyExists { .. } => "already_exists", | ||
| ChunkPutResponse::PaymentRequired { .. } => "payment_required", | ||
| ChunkPutResponse::Error(_) => "error", | ||
| _ => "unknown", | ||
| }; | ||
| info!( | ||
| target: "ant_node::storage::rpc_latency", | ||
| duration_ms, | ||
| chunk_size, | ||
| outcome, | ||
| addr = %addr_hex, | ||
| "put_rpc" | ||
| ); | ||
| response |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a structured
info!log event per chunk-store PUT and GET RPC withduration_ms,outcome, and address (andchunk_sizeon PUT). The tracing target isant_node::storage::rpc_latencyso Elasticsearch / Kibana can build p50/p95/p99 store-RPC latency histograms from the existing telegraf log forwarding without any node-side metric pipeline.Motivation: a recent testnet (DEV-01) saw aggregate upload throughput halve over 12h before any node failures, with file-size-sensitive degradation that strongly suggests CLOSE_GROUP quorum / slowest-peer-dominates cost on the client. Confirming requires both client-side per-chunk timing (in flight as WithAutonomi/ant-client#87) and node-side per-RPC service time — this PR adds the latter.
Changes
src/storage/handler.rs: extract the existinghandle_put/handle_getbodies into_innerhelpers; wrap withInstant::now()and emit a tracing event on every exit path, including the early-return validation paths. Existing per-RPCinfo!/warn!log lines are kept for human-readable diagnostics.Test plan
cargo check --workspacecleancargo clippy --workspace --testsclean🤖 Generated with Claude Code