Skip to content

Stream large database exports#224

Open
sureshchouksey8 wants to merge 1 commit into
outerbase:mainfrom
sureshchouksey8:codex/large-database-export-jobs
Open

Stream large database exports#224
sureshchouksey8 wants to merge 1 commit into
outerbase:mainfrom
sureshchouksey8:codex/large-database-export-jobs

Conversation

@sureshchouksey8
Copy link
Copy Markdown

/claim #59

Summary

Fixes #59 with a shared streaming export path for the SQL dump, CSV, and JSON export endpoints so large tables are exported in bounded pages instead of being materialized into one in-memory payload.

What changed

  • Added shared export streaming helpers for:
    • safe SQL identifier quoting
    • SQL value serialization, including NULL, booleans, finite numbers, bigint, and binary values
    • CSV value escaping
    • sanitized download filenames
    • bounded table-row iteration with runtime yielding
  • Updated /export/dump to stream schema and INSERT statements progressively.
  • Updated /export/csv/:tableName and /export/json/:tableName to stream rows progressively instead of loading the whole table before creating the response.
  • Uses deterministic pagination order: primary-key columns when available, falling back to rowid for tables without a primary key.
  • Keeps table existence checks parameterized and excludes SQLite internal tables from full database dumps.
  • Added focused coverage for the helpers plus SQL/CSV/JSON export routes.

Why this should help large databases

The previous implementation loaded all exported rows into memory before sending a response. This branch sends a ReadableStream response and reads rows in bounded pages, which keeps memory bounded and starts returning bytes immediately for large exports.

Verification

  • npx -y pnpm@9.15.4 install --frozen-lockfile --ignore-scripts
  • npx -y pnpm@9.15.4 exec vitest --run src/export ✅ 5 files / 34 tests
  • git diff --check

I also ran npx -y pnpm@9.15.4 exec tsc --noEmit --pretty false; it still reports existing unrelated project type errors in CDC/cache/DO/operation tests, with no errors from the export files changed here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Database dumps do not work on large databases

1 participant