Skip to content

feat: CAR-stream session for trustless-gateway block broker#1052

Draft
SgtPooki wants to merge 2 commits into
ipfs:mainfrom
SgtPooki:feat/trustless-gateway-car-session
Draft

feat: CAR-stream session for trustless-gateway block broker#1052
SgtPooki wants to merge 2 commits into
ipfs:mainfrom
SgtPooki:feat/trustless-gateway-car-session

Conversation

@SgtPooki

@SgtPooki SgtPooki commented Jun 9, 2026

Copy link
Copy Markdown
Member

Implements the block-broker side of #1051.

What changed

An opt-in CAR-stream session for the trustless-gateway broker: trustlessGateway({ carStreamSessions: true }). Instead of one ?format=raw request per block, a session fetches the whole DAG over one ?format=car&dag-scope=all request to one gateway and serves each block the graph walk asks for from the stream. Blocks the CAR omits, or that fail the per-retrieve validateFn, fall back to a single ?format=raw fetch.

For a many-small-block DAG (a sharded HAMT directory is hundreds of tiny blocks) that is one request instead of hundreds. A local bench (249-block DAG, 25ms/request) measured ~24x.

Default behaviour is unchanged (carStreamSessions: false). The session does not extend AbstractSession, which models per-block, per-provider racing; it implements SessionBlockBroker directly.

Two design questions (why this is a draft)

  1. createSession is not passed the root, so the session infers it from the first retrieve (the same way AbstractSession scopes provider discovery to the first CID). Passing root to broker.createSession(root, options) would make this first-class.
  2. SessionBlockBroker has no close hook, so an abandoned stream cannot be cancelled eagerly. Today it is bounded by the buffer cap plus a 60s idle-timeout teardown; a close hook would let it cancel immediately.

I would rather settle those before polishing, since both touch the session contract.

Notes / risks

  • Verification is unchanged: every block passes through the caller's validateFn before it is returned, so a truncated or wrong CAR cannot reach the blockstore as a verified block.
  • getCar streams the whole DAG, so it is not limitedResponse-capped like getRawBlock. Per-block bytes above the 2 MiB ceiling are dropped, but @ipld/car reads each declared block before that check, so an untrusted gateway is an opt-in DoS surface.
  • Follow-ups: a SessionStorage/NetworkedStorage integration test (the new abort-simulation test is a proxy for the raceBlockRetrievers cleanup), full progress-event parity on the stream path, and an oversize-block test.

Verify

cd packages/block-brokers && npm run test:node (Node 24). 30 tests; build, lint, and dep-check green.

SgtPooki added 2 commits June 9, 2026 13:28
Opt-in via trustlessGateway({ carStreamSessions: true }): fetch a DAG over
one ?format=car request and serve each block from the stream, with verified
?format=raw gap-fill for anything the CAR omits. Collapses the default
session's one-request-per-block cost for many-small-block DAGs.

Refs ipfs#1051 for the root-scoping design question.
Browser HTTP cache satisfied getCar's force-cache across tests, hiding CAR
requests from the test counter; the test gateway now sends no-store. Add
backpressures/dups to the cspell dictionary.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant