Add rezolus_memory example comparing Histogram vs sparse footprint#8
Draft
thinkingfish wants to merge 3 commits intomainfrom
Draft
Add rezolus_memory example comparing Histogram vs sparse footprint#8thinkingfish wants to merge 3 commits intomainfrom
thinkingfish wants to merge 3 commits intomainfrom
Conversation
…print
Reads histogram columns from Rezolus parquet recordings, reconstructs each
sample as `Histogram`, `SparseHistogram`, and `CumulativeROHistogram`, and
prints per-column and overall distribution statistics for the in-memory size
of each representation.
Usage:
cargo run --release --example rezolus_memory -- \
path/to/recording1.parquet path/to/recording2.parquet ...
Also bumps the crate to 1.1.1-alpha.0 per CLAUDE.md.
Member
Author
|
We didn't need this code to be actually in histogram |
Member
Author
|
Not sure if we actually want this code to be checked in. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds an example that measures the in-memory footprint of the
Histogram,SparseHistogram, andCumulativeROHistogramrepresentations for each row of everyList<UInt64>histogram column in one or more Rezolus parquet recordings. Reports per-column and overall distribution statistics (mean/min/max bytes, non-zero bucket counts, relative size).Also bumps the crate version to
1.1.1-alpha.0perCLAUDE.md.Usage
Results for the Rezolus sample recordings
Run against the three parquet files shipped under
rezolus/site/viewer/data/:HistogramtotalSparseHistogramtotaldemo.parquetcachecannon.parquetvllm.parquetCumulativeROHistogramuses the same columnar layout asSparseHistogram, so its memory numbers match exactly; the example prints both for completeness.Test plan
cargo build --release --example rezolus_memorydemo.parquet,cachecannon.parquet, andvllm.parquet— per-column and aggregate output rendered correctlycargo test(no new tests; example exercised manually)