fix(puller): prevent massive log lines from joined errors#5331
fix(puller): prevent massive log lines from joined errors#5331gacevicljubisa wants to merge 2 commits intomasterfrom
Conversation
janos
left a comment
There was a problem hiding this comment.
The approach is maybe too high level. It looks to me that errors.Join is used in a way that is not ideal, maybe even misused. This is solving the problem with long log lines, but does not solve the root cause of using errors as lists of errors.
I would say that this is ok solution but ideally it would be better in the future to address the usage of errors.Join.
pkg/puller/error_summary_test.go
Outdated
| @@ -0,0 +1,72 @@ | |||
| // Copyright 2025 The Swarm Authors. All rights reserved. | |||
I agree. |
pkg/puller/puller.go
Outdated
| errCount := countErrors(err) | ||
| if errCount > 1 { | ||
| p.logger.Debug("syncWorker interval failed", "error_count", errCount, "example_error", errors.Unwrap(err), "peer_address", address, "bin", bin, "cursor", cursor, "start", start, "topmost", top) | ||
| } else { |
There was a problem hiding this comment.
errors.Unwrap() looks for Unwrap() error, but errors.Join() implements Unwrap() []error.
Would this return nil here?
There was a problem hiding this comment.
I'm not sure I understand the question?
By the way @gacevicljubisa, maybe it is just better to remove the if statement and log both ways with the error_count field? Not sure if such granularity is needed
There was a problem hiding this comment.
So I didn't find a way to test and see errors locally in puller logs, so I created this unit test in error_summary_test.go to validate it but i got the issue that i meantioned.
t.Run("errors.Unwrap returns nil for joined errors", func(t *testing.T) {
err1 := errors.New("storage: not found")
err2 := errors.New("storage: not found")
joined := errors.Join(err1, err2)
unwrapped := errors.Unwrap(joined)
if unwrapped == nil {
t.Errorf("expected not nil, got %v", unwrapped)
}
})
There was a problem hiding this comment.
errors.Join() creates an error with Unwrap() []error but errors.Unwrap() looks for Unwrap() error.
Since the signatures don't match, errors.Unwrap() returns nil.
Checklist
Description
Previously, when many errors were joined together during sync
operations, all errors were logged in a single line, creating
log entries up to 30KB (e.g., 245 identical "batch not found"
errors).
This commit adds countErrors() to count total errors and logs
only the count plus one example error, keeping entries concise
while maintaining diagnostic value.
Changes:
Example output:
Open API Spec Version Changes (if applicable)
Motivation and Context (Optional)
Related Issue (Optional)
Screenshots (if appropriate):