Skip to content

Improve save protocol and SQLite pragmas#1036

Merged
GlassOfWhiskey merged 1 commit intomasterfrom
improve-persistence
Apr 19, 2026
Merged

Improve save protocol and SQLite pragmas#1036
GlassOfWhiskey merged 1 commit intomasterfrom
improve-persistence

Conversation

@GlassOfWhiskey
Copy link
Copy Markdown
Member

Replace persistence_lock: Lock with _saving: Event in PersistableEntity so concurrent callers wait on a shared event rather than serialize through a mutex. Each save method also gains an early-return guard and moves follow-on writes (ports, steps, dependencies) inside the guarded block to avoid redundant work on re-entry. Remove LocalTarget._get_deployment_config() class-level singleton as it is no longer needed.

Replace PRAGMA wal_autocheckpoint with PRAGMA temp_store = MEMORY, PRAGMA foreign_keys = ON, and PRAGMA mmap_size in the SQLite connection setup. Expose mmap_size as a configurable option (default 256 MiB) in the schema and SqliteDatabase constructor.

Replace `persistence_lock: Lock` with `_saving: Event` in
`PersistableEntity` so concurrent callers wait on a shared event
rather than serialize through a mutex. Each `save` method also
gains an early-return guard and moves follow-on writes (ports,
steps, dependencies) inside the guarded block to avoid redundant
work on re-entry. Remove `LocalTarget._get_deployment_config()`
class-level singleton as it is no longer needed.

Replace `PRAGMA wal_autocheckpoint` with `PRAGMA temp_store = MEMORY`,
`PRAGMA foreign_keys = ON`, and `PRAGMA mmap_size` in the SQLite
connection setup. Expose `mmap_size` as a configurable option
(default 256 MiB) in the schema and `SqliteDatabase` constructor.
@codecov
Copy link
Copy Markdown

codecov bot commented Apr 19, 2026

Codecov Report

❌ Patch coverage is 80.64516% with 12 lines in your changes missing coverage. Please review.
✅ Project coverage is 74.20%. Comparing base (7a61258) to head (70c8251).
⚠️ Report is 1 commits behind head on master.
✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
streamflow/core/workflow.py 64.28% 6 Missing and 4 partials ⚠️
streamflow/core/deployment.py 90.90% 1 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #1036      +/-   ##
==========================================
- Coverage   74.25%   74.20%   -0.06%     
==========================================
  Files          90       90              
  Lines       12461    12478      +17     
  Branches     2182     2186       +4     
==========================================
+ Hits         9253     9259       +6     
- Misses       2689     2696       +7     
- Partials      519      523       +4     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@GlassOfWhiskey GlassOfWhiskey merged commit 252d21f into master Apr 19, 2026
41 checks passed
@GlassOfWhiskey GlassOfWhiskey deleted the improve-persistence branch April 19, 2026 16:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants