Skip to content

Stream SQL, CSV, and JSON exports with paged reads#225

Open
BobbyZ2026 wants to merge 8 commits into
outerbase:mainfrom
BobbyZ2026:main
Open

Stream SQL, CSV, and JSON exports with paged reads#225
BobbyZ2026 wants to merge 8 commits into
outerbase:mainfrom
BobbyZ2026:main

Conversation

@BobbyZ2026
Copy link
Copy Markdown

Purpose

Fixes #59
/claim #59

This refactors the export endpoints so large SQLite exports do not have to materialize the full response in Durable Object memory before returning.

Changes

  • Adds shared streaming export helpers that return ReadableStream responses instead of Blob-backed full strings.
  • Pages table reads with a bounded default page size and cooperative runtime yields between full pages.
  • Uses keyset pagination on primary keys or rowid cursors, with OFFSET only as a fallback when cursor values are unavailable.
  • Quotes SQLite identifiers, serializes SQL values safely, preserves CSV escaping, and keeps rowid cursor aliases out of exported rows.
  • Updates SQL dump, CSV, and JSON routes plus regression tests for streaming helpers and route behavior.

Verification

  • npx pnpm@10.12.1 exec vitest run src/export/dump.test.ts src/export/csv.test.ts src/export/json.test.ts src/export/streaming.test.ts
  • npx pnpm@10.12.1 run build

Full-suite note: npx pnpm@10.12.1 exec vitest run still has four pre-existing RLS failures in src/rls/index.test.ts that are unrelated to this export change.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Database dumps do not work on large databases

1 participant