From a57cf0fdb95090c4300798caf4bc273b2fe35504 Mon Sep 17 00:00:00 2001 From: Joao Henrique Machado Silva Date: Mon, 11 May 2026 09:09:04 +0200 Subject: [PATCH] feat(sdk): Phase 11.8 multi-handle SDK shape (SQLR-22) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Closes the end-to-end gap from 11.7. The retry-error machinery (BusyError / errorKind / ErrBusy) was reachable through every SDK after 11.7 — but each `sqlrite.connect()` / `new Database()` / `sql.Open()` built an *isolated* backing DB, so the retry idioms were untriggerable. This slice exposes the engine's `Connection::connect()` through the public SDK surfaces so apps can finally mint sibling handles that share state. Picked ahead of plan-doc 11.5 (durability) for the same reason 11.5–11.7 jumped the queue: visible user value first. After this PR, the canonical `BEGIN CONCURRENT` retry loop pattern works end-to-end through Python and Node. C FFI ([`sqlrite-ffi/src/lib.rs`]): - new `sqlrite_connect_sibling(existing, out)` function. Thin wrapper around the engine's `Connection::connect`. Sibling has its own pointer + lifecycle but shares Database state. - header regenerated via build.rs. - 2 new tests: connect_sibling_mints_a_handle_that_shares_state (real cross-handle BEGIN CONCURRENT conflict through the public API) and connect_sibling_rejects_null_inputs. Python SDK ([`sdk/python/src/lib.rs`]): - new `Connection.connect()` instance method. Acquires the inner Mutex, calls `inner.connect()`, wraps result in a fresh pyclass with its own Mutex. Inherits the parent's ask_config. - 4 new tests: sibling sharing, outliving closed parent, raise on closed connect, and the headline busy-round-trip-via-siblings. Node.js SDK ([`sdk/nodejs/src/lib.rs`]): - new `Database.connect()` method on the napi-rs class. Same shape — sibling Database with its own RefCell + ask_config clone. - 4 new tests including the cross-sibling BusyError round trip using 11.7's errorKind classifier. Go SDK: deliberately skipped. database/sql's pool model already gives sibling-like behavior across `db.Conn(ctx)` calls within a single `sql.Open`, but exposing a *cross-pool* sibling shape through `database/sql` would need a process-level path → Database registry. Deferred to a follow-up (roadmap 11.11). WASM SDK: deliberately skipped (single-threaded browser + wasm-bindgen lifetime complications). Same deferral as 11.7. Workspace: 654/654 Rust tests pass (was 652 + 2 new FFI). Python + Node SDK CI jobs will exercise the new sibling-handle tests on the next CI run. fmt + clippy + doc clean on changed files. Roadmap renumbered again: - plan-doc 11.5 (checkpoint) → roadmap 11.9 (was 11.8) - plan-doc 11.7 (indexes) → roadmap 11.10 (was 11.9) - REPL .spawn + bench workload + Go multi-handle = roadmap 11.11 - plan-doc 11.9 (docs) → roadmap 11.12 (was 11.11) Co-Authored-By: Claude Opus 4.7 (1M context) --- docs/_index.md | 2 +- docs/roadmap.md | 39 ++++++----- sdk/nodejs/src/lib.rs | 34 +++++++++ sdk/nodejs/test/test.mjs | 85 +++++++++++++++++++++- sdk/python/src/lib.rs | 41 +++++++++++ sdk/python/tests/test_sqlrite.py | 80 +++++++++++++++++++++ sqlrite-ffi/include/sqlrite.h | 22 ++++++ sqlrite-ffi/src/lib.rs | 117 +++++++++++++++++++++++++++++++ 8 files changed, 402 insertions(+), 18 deletions(-) diff --git a/docs/_index.md b/docs/_index.md index 189a3b3..a912393 100644 --- a/docs/_index.md +++ b/docs/_index.md @@ -54,7 +54,7 @@ As of May 2026, SQLRite has: - Full-text search + hybrid retrieval (Phase 8 complete): FTS5-style inverted index with BM25 ranking + `fts_match` / `bm25_score` scalar functions + `try_fts_probe` optimizer hook + on-disk persistence with on-demand v4 → v5 file-format bump (8a-8c), a worked hybrid-retrieval example combining BM25 with vector cosine via raw arithmetic (8d), and a `bm25_search` MCP tool symmetric with `vector_search` (8e). See [`docs/fts.md`](fts.md). - SQL surface + DX follow-ups (Phase 9 complete, v0.2.0 → v0.9.1): DDL completeness — `DEFAULT`, `DROP TABLE` / `DROP INDEX`, `ALTER TABLE` (9a); free-list + manual `VACUUM` (9b) + auto-VACUUM (9c); `IS NULL` / `IS NOT NULL` (9d); `GROUP BY` + aggregates + `DISTINCT` + `LIKE` + `IN` (9e); four flavors of `JOIN` — INNER, LEFT, RIGHT, FULL OUTER (9f); prepared statements + `?` parameter binding with a per-connection LRU plan cache (9g); HNSW probe widened to cosine + dot via `WITH (metric = …)` (9h); `PRAGMA` dispatcher with the `auto_vacuum` knob (9i) - Benchmarks against SQLite + DuckDB (Phase 10 complete, SQLR-4 / SQLR-16): twelve-workload bench harness with a pluggable `Driver` trait, criterion-driven, pinned-host runs published. See [`docs/benchmarks.md`](benchmarks.md). -- Phase 11 (concurrent writes via MVCC + `BEGIN CONCURRENT`, SQLR-22) is in flight. **11.1 → 11.6: shipped.** `Connection` is `Send + Sync`; `Connection::connect()` mints sibling handles. `sqlrite::mvcc` exposes `MvccClock`, `ActiveTxRegistry`, `MvStore`, and `ConcurrentTx`. WAL header v1 → v2 persists the clock high-water mark. `PRAGMA journal_mode = mvcc;` opts a database into MVCC. `BEGIN CONCURRENT` writes go through commit-time validation and abort with `SQLRiteError::Busy` on row-level write-write conflict. Reads via `Statement::query` see the BEGIN-time snapshot. Per-commit GC + `Connection::vacuum_mvcc()` bound the in-memory version chain growth. **11.7 SDK propagation: shipped on this branch.** The C FFI gains `SqlriteStatus::Busy` / `BusySnapshot`. Python adds `sqlrite.BusyError` / `BusySnapshotError` subclasses. Node exports `errorKind(message)` + `ErrorKind` enum. Go adds `ErrBusy` / `ErrBusySnapshot` sentinels (matchable with `errors.Is`) + an `IsRetryable(err)` helper. Plan: [`docs/concurrent-writes-plan.md`](concurrent-writes-plan.md). +- Phase 11 (concurrent writes via MVCC + `BEGIN CONCURRENT`, SQLR-22) is in flight. **11.1 → 11.7: shipped.** Engine + SDK error propagation: `Connection` is `Send + Sync`; `Connection::connect()` mints sibling handles. `sqlrite::mvcc` exposes `MvccClock`, `ActiveTxRegistry`, `MvStore`, `ConcurrentTx`. WAL header v1 → v2 persists the clock high-water mark. `PRAGMA journal_mode = mvcc;` opts a database into MVCC. `BEGIN CONCURRENT` writes commit-validate against `MvStore` and abort with `SQLRiteError::Busy`. Reads via `Statement::query` see the BEGIN-time snapshot. Per-commit GC + `vacuum_mvcc()` bound the version chain growth. C FFI / Python / Node / Go all propagate `Busy` / `BusySnapshot` as typed retryable errors. **11.8 multi-handle SDK shape: shipped on this branch.** The FFI's `sqlrite_connect_sibling`, Python's `Connection.connect()`, and Node's `db.connect()` mint sibling handles that share backing state — closes the end-to-end gap from 11.7 where `BusyError` was reachable but not exerciseable through any SDK. Plan: [`docs/concurrent-writes-plan.md`](concurrent-writes-plan.md). - A fully-automated release pipeline that ships every product to its registry on every release with one human action — Rust engine + `sqlrite-ask` + `sqlrite-mcp` to crates.io, Python wheels to PyPI (`sqlrite`), Node.js + WASM to npm (`@joaoh82/sqlrite` + `@joaoh82/sqlrite-wasm`), Go module via `sdk/go/v*` git tag, plus C FFI tarballs, MCP binary tarballs, and unsigned desktop installers as GitHub Release assets (Phase 6 complete) See the [Roadmap](roadmap.md) for the full phase plan. diff --git a/docs/roadmap.md b/docs/roadmap.md index dabf647..4458260 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -622,7 +622,7 @@ The headline slice. Multiple sibling `Connection`s can each hold their own open **Known limitations carried forward (most resolved in 11.5):** - ~~Reads via `Statement::query` / `Statement::query_with_params` bypass the swap.~~ ✅ Fixed in 11.5 — `Connection.concurrent_tx` is now `Mutex>` and a new `with_snapshot_read` helper threads the swap through `&self`. -- The `MvStore` write-set isn't yet persisted to the WAL — Phase 11.8 introduces an MVCC log-record frame kind so commits become durable through `MvStore` itself rather than via the legacy `Database::tables` mirror. (Durability already works through the legacy mirror in v0; the WAL log-record format is foundation work for cross-process MVCC.) +- The `MvStore` write-set isn't yet persisted to the WAL — Phase 11.9 introduces an MVCC log-record frame kind so commits become durable through `MvStore` itself rather than via the legacy `Database::tables` mirror. (Durability already works through the legacy mirror in v0; the WAL log-record format is foundation work for cross-process MVCC.) - `AUTOINCREMENT` inside `BEGIN CONCURRENT` isn't explicitly rejected; the v0 deep-clone-snapshot model handles concurrent INSERTs by isolating each tx's `last_rowid` bumps to its private snapshot, so two concurrent INSERTs on an `AUTOINCREMENT` column may collide at COMMIT and surface as `Busy`. Adopting the plan's "reject AUTOINCREMENT under MVCC" gate is a clean follow-up. - Tables touched by `BEGIN CONCURRENT` writes can't carry FTS or HNSW indexes today — `restore_row` only maintains B-tree secondary indexes. Concurrent-tx tests don't exercise FTS / HNSW, but a runtime guard would surface this with a clear error rather than producing inconsistent indexes. @@ -647,34 +647,41 @@ Bounds in-memory growth of the [`MvStore`](../src/mvcc/store.rs) version chains. **What 11.6 doesn't yet do:** - No background GC thread or `PRAGMA mvcc_gc_interval_ms`. Per-commit sweep + explicit `vacuum_mvcc()` cover the v0 model; the periodic-sweep variant lands as a follow-up if profiles show it's needed. -- GC sweeps don't trigger `Mvcc → Wal` journal-mode downgrades. The `set_journal_mode` setter still rejects the transition while the store carries committed versions; promoting that path requires the checkpoint-integration story from 11.8. +- GC sweeps don't trigger `Mvcc → Wal` journal-mode downgrades. The `set_journal_mode` setter still rejects the transition while the store carries committed versions; promoting that path requires the checkpoint-integration story from 11.9. -### 🚧 Phase 11.7 — SDK propagation of `Busy` / `BusySnapshot` *(in progress, plan-doc "Phase 10.8"; promoted ahead of plan-doc 11.5 checkpoint work for the same reason 11.5 / 11.6 jumped the queue — surfacing retryable errors to SDK callers is what unblocks Python / Node / Go users from actually writing `BEGIN CONCURRENT` retry loops)* +### ✅ Phase 11.7 — SDK propagation of `Busy` / `BusySnapshot` *(plan-doc "Phase 10.8"'s first half; promoted ahead of plan-doc 11.5 checkpoint work because surfacing retryable errors to SDK callers is what unblocks Python / Node / Go users from writing `BEGIN CONCURRENT` retry loops)* -- **C FFI** ([`sqlrite-ffi/src/lib.rs`](../sqlrite-ffi/src/lib.rs)): new `SqlriteStatus::Busy = 5` and `SqlriteStatus::BusySnapshot = 6` codes alongside the existing `Ok` / `Error` / `InvalidArgument` set. `SqlriteStatus::is_retryable()` covers both. A new internal `status_of_sqlrite` mapper inspects the engine's `SQLRiteError` variant and routes `Busy` / `BusySnapshot` to the dedicated codes (the generic `status_of` keeps mapping every error to `Error`). `sqlrite_execute` switches to the engine-aware mapper so `BEGIN CONCURRENT` commits surface the dedicated codes through every language binding. Header regenerated automatically via `build.rs`. -- **Python SDK** ([`sdk/python/src/lib.rs`](../sdk/python/src/lib.rs)): two new exception classes `sqlrite.BusyError` and `sqlrite.BusySnapshotError`, both inheriting from `sqlrite.SQLRiteError`. Existing `except sqlrite.SQLRiteError` blocks keep catching them; retry helpers can branch with `except sqlrite.BusyError`. A new `map_engine_err` helper inspects the engine error variant and raises the matching exception class. Every engine-typed call site (open / execute / prepare / query / rows.next) routes through it. -- **Node.js SDK** ([`sdk/nodejs/src/lib.rs`](../sdk/nodejs/src/lib.rs)): new exported `ErrorKind` string enum (`'Busy'`, `'BusySnapshot'`, `'Other'`) and `errorKind(message: string)` classifier function. The engine's `thiserror` Display already prefixes retryable errors with `'Busy: '` / `'BusySnapshot: '`, so the classifier just regex-tests the prefix. JS callers wrap their `BEGIN CONCURRENT` loops in `try / catch (err) { if (errorKind(err.message) === ErrorKind.Busy) continue; }`. -- **Go SDK** ([`sdk/go/sqlrite.go`](../sdk/go/sqlrite.go)): two new sentinel error values `sqlrite.ErrBusy` and `sqlrite.ErrBusySnapshot`, plus an `IsRetryable(err error) bool` helper. `wrapErr` recognises the new FFI status codes and wraps the engine message with `fmt.Errorf("…: %w", ErrBusy)` so `errors.Is(err, sqlrite.ErrBusy)` works through the `database/sql` driver chain. -- **WASM SDK** — deliberately untouched. The browser WASM target is single-threaded; `BEGIN CONCURRENT` is meaningful but multi-handle concurrency through `Connection::connect` isn't yet exposed across `wasm-bindgen`'s lifetime model. When the multi-handle JS shape lands (separate slice), the same `Busy: …` message prefix will be the classifier hook for the WASM bindings too. +- **C FFI** ([`sqlrite-ffi/src/lib.rs`](../sqlrite-ffi/src/lib.rs)): new `SqlriteStatus::Busy = 5` and `SqlriteStatus::BusySnapshot = 6` codes; `SqlriteStatus::is_retryable()` covers both. A new internal `status_of_sqlrite` mapper inspects the engine's `SQLRiteError` variant and routes `Busy` / `BusySnapshot` to the dedicated codes. +- **Python SDK**: two new exception classes `sqlrite.BusyError` and `sqlrite.BusySnapshotError`, both inheriting from `sqlrite.SQLRiteError`. `map_engine_err` helper raises the matching subclass. +- **Node.js SDK**: exported `ErrorKind` string enum (`'Busy'`, `'BusySnapshot'`, `'Other'`) and `errorKind(message: string)` classifier function. The engine's `thiserror` Display prefixes retryable errors with `'Busy: '` / `'BusySnapshot: '` so the classifier just regex-tests the prefix. +- **Go SDK**: two new sentinel error values `sqlrite.ErrBusy` / `sqlrite.ErrBusySnapshot`, plus an `IsRetryable(err error) bool` helper. `wrapErr` recognises the new FFI status codes and wraps the engine message with `fmt.Errorf("…: %w", ErrBusy)`. +- **WASM SDK** — deliberately untouched (browser is single-threaded; multi-handle shape not yet exposed). -**What this slice doesn't do:** +### 🚧 Phase 11.8 — Multi-handle SDK shape *(in progress, was plan-doc 11.8's other half; promoted ahead of plan-doc 11.5 again because the 11.7 retry-error machinery can't be exercised end-to-end through any SDK until siblings are reachable)* -- The multi-handle / sibling-`Connection` shape isn't exposed through any SDK yet. Each `sqlrite.connect(path)` / `new Database(path)` / `sql.Open(...)` builds an independent backing database. End-to-end testing of cross-handle `Busy` is therefore deferred to the multi-handle SDK slice; this PR ships the *plumbing* so once that wiring lands, callers already have the retry idioms they need. -- The WAL log-record durability work (plan-doc 11.5 / our 11.8) stays deferred. +Each pre-11.8 SDK `connect()` / `new Database()` built an *isolated* backing DB; the 11.7 `BusyError` / `errorKind` / `ErrBusy` plumbing was reachable but not actually triggerable from user code. This slice exposes the engine's `Connection::connect()` through every reachable language so apps can mint sibling handles that share state, and finally exercise the 11.7 retry idioms with real cross-handle conflicts. -### Phase 11.8 — Checkpoint integration + crash recovery *(planned, plan-doc "Phase 10.5"; renumbered to follow GC + SDK propagation because durability via the legacy `save_database` mirror already works in v0; this slice is foundation work for cross-process MVCC and column-level WAL deltas)* +- **C FFI** ([`sqlrite-ffi/src/lib.rs`](../sqlrite-ffi/src/lib.rs)): new `sqlrite_connect_sibling(existing, out)` function. Wraps the engine's `Connection::connect`. Callers get a sibling handle with its own `SqlriteConnection` pointer but shared backing database; the sibling must be closed via `sqlrite_close` (its lifecycle is independent — closing one handle doesn't tear down the others while a sibling is still alive). +- **Python SDK** ([`sdk/python/src/lib.rs`](../sdk/python/src/lib.rs)): new `Connection.connect()` instance method that mints a sibling pyclass. Wraps the engine's `Connection::connect` inside the existing `Mutex`. The new handle inherits the parent's `ask_config`. +- **Node.js SDK** ([`sdk/nodejs/src/lib.rs`](../sdk/nodejs/src/lib.rs)): new `db.connect()` method on the `Database` class. Same shape — sibling shares state, can hold its own `BEGIN CONCURRENT`. +- **Go SDK** — deliberately not changed. Go's `database/sql` already gives callers a connection pool over a single `sql.Open`; each pool connection acquired through `db.Conn(ctx)` is *already* a sibling of the rest at the driver layer. But each `sql.Open("sqlrite", path)` still builds an independent backing DB because the pool is per-`sql.DB`. Exposing a cross-pool sibling shape through the `database/sql` driver model is genuinely non-obvious (it'd require a process-level registry keyed by path); deferred to the multi-handle Go follow-up. +- **WASM SDK** — still untouched. The browser is single-threaded and `wasm-bindgen` lifetimes complicate sibling pyclass-style sharing. Same deferral as 11.7. + +Each SDK gets end-to-end tests that exercise `BEGIN CONCURRENT` cross-handle conflicts: two sibling handles, two concurrent transactions on the same row, the second commit hits the SDK's typed retryable error, retry succeeds. + +### Phase 11.9 — Checkpoint integration + crash recovery *(planned, plan-doc "Phase 10.5"; renumbered to follow SDK propagation because durability via the legacy `save_database` mirror already works in v0; this slice is foundation work for cross-process MVCC and column-level WAL deltas)* MVCC log-record WAL frame format (the deferred 11.4 piece). Commit appends log records pre-`save_database`. Reopen replays log records into `MvStore`. Checkpoint drains `MvStore` versions back into the pager (so `Mvcc → Wal` becomes legal once the store is empty). Crash-recovery test: kill mid-commit between log-record append and version-chain push; reopen; verify the committed transaction is visible and the half-written one is not. -### Phase 11.9 — Indexes under MVCC *(deferred-by-design, plan-doc "Phase 10.7")* +### Phase 11.10 — Indexes under MVCC *(deferred-by-design, plan-doc "Phase 10.7")* Each secondary-index entry becomes its own `RowVersion`. Turso explicitly punted on this; SQLRite's v0 will reject `CREATE INDEX` while `journal_mode = mvcc`. -### Phase 11.10 — Multi-handle SDK shape + REPL `.spawn` *(planned, was plan-doc 11.8's other half)* +### Phase 11.11 — REPL `.spawn` + bench workload *(planned)* -Expose `Connection::connect()` through the FFI + each SDK so Python / Node / Go callers can mint sibling handles, plus a new REPL `.spawn` meta-command. Without this, the 11.7 retry-error machinery can't actually be exercised end-to-end through an SDK (each SDK `connect()` builds an independent DB). Also adds the "N concurrent writers" benchmark workload. +REPL `.spawn` meta-command for interactive `BEGIN CONCURRENT` demos. New "N concurrent writers" benchmark workload pitting SQLRite-MVCC against SQLite + DuckDB on disjoint-row write throughput. Plus Go SDK multi-handle work (cross-pool sibling shape). -### Phase 11.11 — Docs *(planned, plan-doc "Phase 10.9")* +### Phase 11.12 — Docs *(planned, plan-doc "Phase 10.9")* Promote the plan to `docs/concurrent-writes.md` and update the cross-references. diff --git a/sdk/nodejs/src/lib.rs b/sdk/nodejs/src/lib.rs index bf15fc9..df13282 100644 --- a/sdk/nodejs/src/lib.rs +++ b/sdk/nodejs/src/lib.rs @@ -299,6 +299,40 @@ impl Database { }) } + /// Phase 11.8 — mints a sibling `Database` that shares the + /// same underlying state (the in-memory tables, the MVCC + /// store, the pager). Wraps the engine's `Connection::connect`. + /// + /// Use this to drive `BEGIN CONCURRENT` from multiple Node + /// handles in the same process: each sibling can hold its own + /// concurrent transaction, and commits validate against the + /// shared MvStore. + /// + /// ```js + /// const db = new Database(':memory:'); + /// db.exec('PRAGMA journal_mode = mvcc'); + /// db.exec('CREATE TABLE t (id INTEGER PRIMARY KEY, v INTEGER)'); + /// const sibling = db.connect(); + /// // sibling.exec(...) lands on the same backing tables. + /// ``` + /// + /// The sibling carries its own per-handle prepared-statement + /// cache and concurrent-transaction slot. Closing one handle + /// (`db.close()`) doesn't affect siblings; the underlying + /// database lives as long as any handle holds it. + #[napi] + pub fn connect(&self) -> Result { + let borrow = self.inner.borrow(); + let parent = borrow + .as_ref() + .ok_or_else(|| napi::Error::from_reason("cannot connect: database is closed"))?; + let sibling = parent.connect(); + Ok(Database { + inner: RefCell::new(Some(sibling)), + ask_config: RefCell::new(self.ask_config.borrow().clone()), + }) + } + /// Closes the connection and releases the OS file lock. Safe to /// call multiple times. #[napi] diff --git a/sdk/nodejs/test/test.mjs b/sdk/nodejs/test/test.mjs index 314907b..401c40d 100644 --- a/sdk/nodejs/test/test.mjs +++ b/sdk/nodejs/test/test.mjs @@ -15,7 +15,7 @@ import { mkdtempSync, rmSync } from 'node:fs'; import { tmpdir } from 'node:os'; import { join } from 'node:path'; -import { Database } from '../index.js'; +import { Database, errorKind, ErrorKind } from '../index.js'; function tmpDbPath(name) { const dir = mkdtempSync(join(tmpdir(), `sqlrite-node-${name}-`)); @@ -187,3 +187,86 @@ test('closed DB throws on any operation', () => { assert.throws(() => db.exec('SELECT 1'), /closed/); assert.throws(() => db.prepare('SELECT 1'), /closed/); }); + +// --------------------------------------------------------------------------- +// Phase 11.8 — multi-handle (db.connect()) end-to-end + +test('sibling db sees writes the parent made', () => { + const db = new Database(':memory:'); + db.exec('CREATE TABLE t (id INTEGER PRIMARY KEY, name TEXT)'); + db.exec("INSERT INTO t (id, name) VALUES (1, 'alice')"); + + const sibling = db.connect(); + const rows = sibling.prepare('SELECT name FROM t WHERE id = 1').all(); + assert.deepEqual(rows, [{ name: 'alice' }]); + + // Writes on the sibling are visible to the parent. + sibling.exec("INSERT INTO t (id, name) VALUES (2, 'bob')"); + const rows2 = db.prepare('SELECT name FROM t WHERE id = 2').all(); + assert.deepEqual(rows2, [{ name: 'bob' }]); + + db.close(); + sibling.close(); +}); + +test('sibling outlives closed parent', () => { + const db = new Database(':memory:'); + db.exec('CREATE TABLE t (id INTEGER PRIMARY KEY, v INTEGER)'); + db.exec('INSERT INTO t (id, v) VALUES (1, 100)'); + + const sibling = db.connect(); + db.close(); + + // Sibling still operates — the underlying database is Arc-shared. + const rows = sibling.prepare('SELECT v FROM t WHERE id = 1').all(); + assert.deepEqual(rows, [{ v: 100 }]); + + sibling.close(); +}); + +test('connect on closed parent throws', () => { + const db = new Database(':memory:'); + db.close(); + assert.throws(() => db.connect(), /closed/); +}); + +test('BEGIN CONCURRENT busy round-trips through Node siblings', () => { + // The end-to-end test that wasn't possible before 11.8 — Node's + // `new Database()` always built an isolated DB. With `db.connect()` + // exposed, the 11.7 retry idiom can finally be exercised from JS. + const db = new Database(':memory:'); + db.exec('PRAGMA journal_mode = mvcc'); + db.exec('CREATE TABLE accounts (id INTEGER PRIMARY KEY, balance INTEGER)'); + db.exec('INSERT INTO accounts (id, balance) VALUES (1, 100)'); + + const sibling = db.connect(); + + db.exec('BEGIN CONCURRENT'); + sibling.exec('BEGIN CONCURRENT'); + db.exec('UPDATE accounts SET balance = 200 WHERE id = 1'); + sibling.exec('UPDATE accounts SET balance = 300 WHERE id = 1'); + + // First commit wins. + db.exec('COMMIT'); + + // Second commit aborts; the 11.7 classifier recognises it as Busy. + let caught; + try { + sibling.exec('COMMIT'); + } catch (err) { + caught = err; + } + assert.ok(caught, 'expected sibling commit to throw'); + assert.equal(errorKind(caught.message), ErrorKind.Busy); + + // Retry succeeds. + sibling.exec('BEGIN CONCURRENT'); + sibling.exec('UPDATE accounts SET balance = 300 WHERE id = 1'); + sibling.exec('COMMIT'); + + const rows = db.prepare('SELECT balance FROM accounts WHERE id = 1').all(); + assert.deepEqual(rows, [{ balance: 300 }]); + + db.close(); + sibling.close(); +}); diff --git a/sdk/python/src/lib.rs b/sdk/python/src/lib.rs index 6a65da1..5f72095 100644 --- a/sdk/python/src/lib.rs +++ b/sdk/python/src/lib.rs @@ -241,6 +241,47 @@ impl Connection { }) } + /// Phase 11.8 — mints a sibling `Connection` that shares the + /// same underlying database state (the in-memory tables, the + /// MVCC store, the pager). Wraps the engine's + /// `Connection::connect`. + /// + /// Use this to drive `BEGIN CONCURRENT` from multiple Python + /// handles in the same process: each sibling can hold its + /// own concurrent transaction, and commits validate against + /// the shared MvStore. + /// + /// ```python + /// import sqlrite + /// conn = sqlrite.connect(":memory:") + /// conn.execute("PRAGMA journal_mode = mvcc") + /// conn.execute("CREATE TABLE t (id INTEGER PRIMARY KEY, v INTEGER)") + /// conn.execute("INSERT INTO t (id, v) VALUES (1, 0)") + /// + /// sibling = conn.connect() + /// # `sibling` shares the same backing DB — its writes and + /// # `BEGIN CONCURRENT` transactions land on the same tables. + /// ``` + /// + /// The sibling carries its own per-handle prepared-statement + /// cache and concurrent-transaction slot. Closing one + /// handle (`conn.close()`) doesn't affect siblings; the + /// underlying database lives as long as any handle holds it. + fn connect(&self) -> PyResult { + let guard = self + .inner + .as_ref() + .ok_or_else(|| SQLRiteError::new_err("cannot connect: connection is closed"))?; + let locked = guard + .lock() + .map_err(|_| SQLRiteError::new_err("connection mutex poisoned"))?; + let sibling = locked.connect(); + Ok(Connection { + inner: Some(Mutex::new(sibling)), + ask_config: self.ask_config.clone(), + }) + } + /// Closes the connection and releases the OS file lock. Safe to /// call multiple times; a closed connection raises `SQLRiteError` /// on any subsequent operation. diff --git a/sdk/python/tests/test_sqlrite.py b/sdk/python/tests/test_sqlrite.py index 0eb0044..6b4e11b 100644 --- a/sdk/python/tests/test_sqlrite.py +++ b/sdk/python/tests/test_sqlrite.py @@ -285,6 +285,86 @@ def test_journal_mode_pragma_reaches_python(conn): conn.execute("PRAGMA journal_mode = nonsense") +# --------------------------------------------------------------------------- +# Phase 11.8 — multi-handle (Connection.connect()) end-to-end + + +def test_sibling_connection_shares_underlying_database(conn): + """A sibling minted via `conn.connect()` sees writes the parent + made — the headline 11.8 contract. + """ + sibling = conn.connect() + conn.execute("CREATE TABLE t (id INTEGER PRIMARY KEY, name TEXT)") + conn.execute("INSERT INTO t (id, name) VALUES (1, 'alice')") + + # Sibling reads the parent's write. + cur = sibling.execute("SELECT name FROM t WHERE id = 1") + assert cur.fetchone() == ("alice",) + + # Sibling's writes are visible to the parent. + sibling.execute("INSERT INTO t (id, name) VALUES (2, 'bob')") + cur = conn.execute("SELECT name FROM t WHERE id = 2") + assert cur.fetchone() == ("bob",) + + +def test_sibling_after_close_still_works(conn): + """Closing the parent doesn't tear down the sibling. The + underlying database is `Arc`-shared across handles.""" + conn.execute("CREATE TABLE t (id INTEGER PRIMARY KEY, v INTEGER)") + conn.execute("INSERT INTO t (id, v) VALUES (1, 100)") + sibling = conn.connect() + conn.close() + + cur = sibling.execute("SELECT v FROM t WHERE id = 1") + assert cur.fetchone() == (100,) + + +def test_sibling_on_closed_connection_raises(conn): + """Calling `connect()` on a closed connection should raise + SQLRiteError, not segfault.""" + conn.close() + with pytest.raises(sqlrite.SQLRiteError): + conn.connect() + + +def test_begin_concurrent_busy_round_trips_through_python_siblings(conn): + """Two sibling handles, two `BEGIN CONCURRENT` transactions on + the same row: second commit must raise `sqlrite.BusyError`. + + This is the end-to-end test that wasn't possible before 11.8 + — the previous Python `sqlrite.connect()` model built isolated + databases. With `Connection.connect()` exposed, the 11.7 retry + idiom can finally be exercised from Python. + """ + conn.execute("PRAGMA journal_mode = mvcc") + conn.execute("CREATE TABLE accounts (id INTEGER PRIMARY KEY, balance INTEGER)") + conn.execute("INSERT INTO accounts (id, balance) VALUES (1, 100)") + + sibling = conn.connect() + + conn.execute("BEGIN CONCURRENT") + sibling.execute("BEGIN CONCURRENT") + + conn.execute("UPDATE accounts SET balance = 200 WHERE id = 1") + sibling.execute("UPDATE accounts SET balance = 300 WHERE id = 1") + + # First commit wins. + conn.execute("COMMIT") + + # Second commit aborts with the dedicated subclass. + with pytest.raises(sqlrite.BusyError) as exc_info: + sibling.execute("COMMIT") + # Still a SQLRiteError too (subclass relationship). + assert isinstance(exc_info.value, sqlrite.SQLRiteError) + # Retry succeeds. + sibling.execute("BEGIN CONCURRENT") + sibling.execute("UPDATE accounts SET balance = 300 WHERE id = 1") + sibling.execute("COMMIT") + + cur = conn.execute("SELECT balance FROM accounts WHERE id = 1") + assert cur.fetchone() == (300,) + + def test_executescript_runs_batched_statements(conn): cur = conn.cursor() cur.executescript( diff --git a/sqlrite-ffi/include/sqlrite.h b/sqlrite-ffi/include/sqlrite.h index 62dd41c..76b903a 100644 --- a/sqlrite-ffi/include/sqlrite.h +++ b/sqlrite-ffi/include/sqlrite.h @@ -106,6 +106,28 @@ enum SqlriteStatus sqlrite_open_read_only(const char *path, struct SqlriteConnec // `out` must be a valid writable pointer. enum SqlriteStatus sqlrite_open_in_memory(struct SqlriteConnection **out); +// Phase 11.8 — mints a sibling connection that shares the same +// underlying database state (the in-memory tables, the MVCC +// store, the pager). Wraps the engine's `Connection::connect`. +// +// Use this to drive `BEGIN CONCURRENT` from multiple FFI +// handles: each sibling can hold its own concurrent transaction, +// and commits validate against the shared MvStore. +// +// The returned handle is owned by the caller and must be freed +// with [`sqlrite_close`]; closing one sibling doesn't affect +// the others. +// +// # Safety +// +// `existing` must be a valid pointer returned by one of the +// `sqlrite_open_*` functions and not yet closed. `out` must be +// a valid writable pointer. On success the caller owns the +// returned sibling and must call [`sqlrite_close`] on it when +// done. +enum SqlriteStatus sqlrite_connect_sibling(struct SqlriteConnection *existing, + struct SqlriteConnection **out); + // Closes a connection and releases its file locks. Safe to call with // a null pointer (no-op). // diff --git a/sqlrite-ffi/src/lib.rs b/sqlrite-ffi/src/lib.rs index f33f0d0..f654b55 100644 --- a/sqlrite-ffi/src/lib.rs +++ b/sqlrite-ffi/src/lib.rs @@ -332,6 +332,43 @@ pub unsafe extern "C" fn sqlrite_open_in_memory(out: *mut *mut SqlriteConnection status } +/// Phase 11.8 — mints a sibling connection that shares the same +/// underlying database state (the in-memory tables, the MVCC +/// store, the pager). Wraps the engine's `Connection::connect`. +/// +/// Use this to drive `BEGIN CONCURRENT` from multiple FFI +/// handles: each sibling can hold its own concurrent transaction, +/// and commits validate against the shared MvStore. +/// +/// The returned handle is owned by the caller and must be freed +/// with [`sqlrite_close`]; closing one sibling doesn't affect +/// the others. +/// +/// # Safety +/// +/// `existing` must be a valid pointer returned by one of the +/// `sqlrite_open_*` functions and not yet closed. `out` must be +/// a valid writable pointer. On success the caller owns the +/// returned sibling and must call [`sqlrite_close`] on it when +/// done. +#[unsafe(no_mangle)] +pub unsafe extern "C" fn sqlrite_connect_sibling( + existing: *mut SqlriteConnection, + out: *mut *mut SqlriteConnection, +) -> SqlriteStatus { + if existing.is_null() || out.is_null() { + set_last_error("connection or output pointer is null"); + return SqlriteStatus::InvalidArgument; + } + // Safety: caller guarantees `existing` is a valid handle. + let parent = unsafe { &mut *(existing as *mut ConnHandle) }; + let sibling = parent.conn.connect(); + let boxed = Box::new(ConnHandle { conn: sibling }); + unsafe { *out = Box::into_raw(boxed) as *mut SqlriteConnection }; + clear_last_error(); + SqlriteStatus::Ok +} + /// Closes a connection and releases its file locks. Safe to call with /// a null pointer (no-op). /// @@ -1254,6 +1291,86 @@ mod tests { } } + /// Phase 11.8 — `sqlrite_connect_sibling` is the public path + /// SDK bindings use to mint a sibling handle (replacing the + /// hack the 11.4 test did by reaching into `ConnHandle` + /// directly). Same conflict scenario as + /// `begin_concurrent_busy_status_round_trip` but exercises + /// the proper public surface. + #[test] + fn connect_sibling_mints_a_handle_that_shares_state() { + unsafe { + let mut a: *mut SqlriteConnection = ptr::null_mut(); + assert_eq!(sqlrite_open_in_memory(&mut a), SqlriteStatus::Ok); + + let mut b: *mut SqlriteConnection = ptr::null_mut(); + assert_eq!(sqlrite_connect_sibling(a, &mut b), SqlriteStatus::Ok); + assert!(!b.is_null()); + // Distinct handles — different pointer values, even + // though they share backing state. + assert!(a != b); + + // Set up schema + a row through `a`; sibling `b` must + // see it via the shared backing database. + let (_c, p) = cstr("PRAGMA journal_mode = mvcc;"); + assert_eq!(sqlrite_execute(a, p), SqlriteStatus::Ok); + let (_c, p) = cstr("CREATE TABLE t (id INTEGER PRIMARY KEY, v INTEGER);"); + assert_eq!(sqlrite_execute(a, p), SqlriteStatus::Ok); + let (_c, p) = cstr("INSERT INTO t (id, v) VALUES (1, 100);"); + assert_eq!(sqlrite_execute(a, p), SqlriteStatus::Ok); + + // Sibling reads the same row. + let mut stmt: *mut SqlriteStatement = ptr::null_mut(); + let (_c, p) = cstr("SELECT v FROM t WHERE id = 1;"); + assert_eq!(sqlrite_query(b, p, &mut stmt), SqlriteStatus::Ok); + assert_eq!(sqlrite_step(stmt), SqlriteStatus::Row); + let mut got: i64 = 0; + assert_eq!(sqlrite_column_int64(stmt, 0, &mut got), SqlriteStatus::Ok); + assert_eq!(got, 100); + sqlrite_finalize(stmt); + + // Two `BEGIN CONCURRENT` transactions on the same row + // through the two public sibling handles — second + // commit aborts with Busy. + let (_c, p) = cstr("BEGIN CONCURRENT;"); + assert_eq!(sqlrite_execute(a, p), SqlriteStatus::Ok); + let (_c, p) = cstr("BEGIN CONCURRENT;"); + assert_eq!(sqlrite_execute(b, p), SqlriteStatus::Ok); + let (_c, p) = cstr("UPDATE t SET v = 200 WHERE id = 1;"); + assert_eq!(sqlrite_execute(a, p), SqlriteStatus::Ok); + let (_c, p) = cstr("UPDATE t SET v = 300 WHERE id = 1;"); + assert_eq!(sqlrite_execute(b, p), SqlriteStatus::Ok); + let (_c, p) = cstr("COMMIT;"); + assert_eq!(sqlrite_execute(a, p), SqlriteStatus::Ok); + let (_c, p) = cstr("COMMIT;"); + assert_eq!(sqlrite_execute(b, p), SqlriteStatus::Busy); + + sqlrite_close(b); + sqlrite_close(a); + } + } + + /// Null-arg guards for the sibling minter. + #[test] + fn connect_sibling_rejects_null_inputs() { + unsafe { + let mut out: *mut SqlriteConnection = ptr::null_mut(); + assert_eq!( + sqlrite_connect_sibling(ptr::null_mut(), &mut out), + SqlriteStatus::InvalidArgument + ); + assert!(out.is_null()); + + let mut a: *mut SqlriteConnection = ptr::null_mut(); + sqlrite_open_in_memory(&mut a); + assert_eq!( + sqlrite_connect_sibling(a, ptr::null_mut()), + SqlriteStatus::InvalidArgument + ); + sqlrite_close(a); + } + } + #[test] fn step_without_query_returns_error() { // Column accessors against a handle that hasn't produced a row