feat: CAR-stream session for trustless-gateway block broker#1052
Draft
SgtPooki wants to merge 2 commits into
Draft
feat: CAR-stream session for trustless-gateway block broker#1052SgtPooki wants to merge 2 commits into
SgtPooki wants to merge 2 commits into
Conversation
Opt-in via trustlessGateway({ carStreamSessions: true }): fetch a DAG over
one ?format=car request and serve each block from the stream, with verified
?format=raw gap-fill for anything the CAR omits. Collapses the default
session's one-request-per-block cost for many-small-block DAGs.
Refs ipfs#1051 for the root-scoping design question.
Browser HTTP cache satisfied getCar's force-cache across tests, hiding CAR requests from the test counter; the test gateway now sends no-store. Add backpressures/dups to the cspell dictionary.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements the block-broker side of #1051.
What changed
An opt-in CAR-stream session for the trustless-gateway broker:
trustlessGateway({ carStreamSessions: true }). Instead of one?format=rawrequest per block, a session fetches the whole DAG over one?format=car&dag-scope=allrequest to one gateway and serves each block the graph walk asks for from the stream. Blocks the CAR omits, or that fail the per-retrievevalidateFn, fall back to a single?format=rawfetch.For a many-small-block DAG (a sharded HAMT directory is hundreds of tiny blocks) that is one request instead of hundreds. A local bench (249-block DAG, 25ms/request) measured ~24x.
Default behaviour is unchanged (
carStreamSessions: false). The session does not extendAbstractSession, which models per-block, per-provider racing; it implementsSessionBlockBrokerdirectly.Two design questions (why this is a draft)
createSessionis not passed the root, so the session infers it from the firstretrieve(the same wayAbstractSessionscopes provider discovery to the first CID). Passingroottobroker.createSession(root, options)would make this first-class.SessionBlockBrokerhas no close hook, so an abandoned stream cannot be cancelled eagerly. Today it is bounded by the buffer cap plus a 60s idle-timeout teardown; a close hook would let it cancel immediately.I would rather settle those before polishing, since both touch the session contract.
Notes / risks
validateFnbefore it is returned, so a truncated or wrong CAR cannot reach the blockstore as a verified block.getCarstreams the whole DAG, so it is notlimitedResponse-capped likegetRawBlock. Per-block bytes above the 2 MiB ceiling are dropped, but@ipld/carreads each declared block before that check, so an untrusted gateway is an opt-in DoS surface.SessionStorage/NetworkedStorageintegration test (the new abort-simulation test is a proxy for theraceBlockRetrieverscleanup), full progress-event parity on the stream path, and an oversize-block test.Verify
cd packages/block-brokers && npm run test:node(Node 24). 30 tests; build, lint, and dep-check green.