feat(minivm): register VM bytecode pipeline + debugger (Phase 0/1)#35
feat(minivm): register VM bytecode pipeline + debugger (Phase 0/1)#35cuzzo wants to merge 19 commits into
Conversation
|
| Branch | register-machine |
| Testbed | ubuntu-latest |
⚠️ WARNING: No Threshold found!Without a Threshold, no Alerts will ever be generated.
Click here to create a new Threshold
For more information, see the Threshold documentation.
To only post results if a Threshold exists, set the--ci-only-thresholdsflag.
Click to view all benchmark results
| Benchmark | leak-build-ms | Measure (units) x 1e3 | leak-count | Measure (units) | leak-run-ms | Measure (units) |
|---|---|---|---|---|---|---|
| benchmarks/concurrent/04_fanout_fanin/bench | 📈 view plot | 5.49 units x 1e3 | 📈 view plot | 0.00 units | 📈 view plot | 3,315.21 units |
| benchmarks/concurrent/09_kvstore/bench | 📈 view plot | 5.41 units x 1e3 | 📈 view plot | 0.00 units | 📈 view plot | 60,003.74 units |
| benchmarks/concurrent/14_nested_lock/bench | 📈 view plot | 5.29 units x 1e3 | 📈 view plot | 0.00 units | 📈 view plot | 391.29 units |
| benchmarks/inter-clear/02_concurrent_fsm_vs_stackful/bench_fsm | 📈 view plot | 5.21 units x 1e3 | 📈 view plot | 0.00 units | 📈 view plot | 151.21 units |
| benchmarks/inter-clear/02_concurrent_fsm_vs_stackful/bench_stackful | 📈 view plot | 5.23 units x 1e3 | 📈 view plot | 0.00 units | 📈 view plot | 195.09 units |
| benchmarks/sequential/11_pipeline_overhead/bench | 📈 view plot | 5.22 units x 1e3 | 📈 view plot | 0.00 units | 📈 view plot | 12,495.82 units |
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #35 +/- ##
==========================================
- Coverage 92.83% 92.78% -0.05%
==========================================
Files 206 206
Lines 52596 52636 +40
Branches 12325 12343 +18
==========================================
+ Hits 48829 48840 +11
- Misses 3767 3796 +29
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
340f8a5 to
4180c4b
Compare
Brings over the register-VM work onto current master (post-hotfix). The compiler/MIR pre-reqs now landed via the hotfix branch are already in place (PipelineHost, IfBind defensive lookups, cross-file FunctionSignature reconstruction, items_access pointer-deref, CheatLib polymorphic helpers, @enumFromInt CAST, K[]@list keys(), fixed-array literal storage, raw fixed-array indexed-set, setAt / cleanupAt / join pointer-deref). This commit adds: VM emitter and runtime - examples/minivm/register_bc_emitter.rb — register-allocated bytecode emitter with shared-event recording groundwork for future loom - examples/minivm/_bc_runner.cht — Zig-compiled register VM with byebug-style debugger (breakpoints, step, frame, time-travel) - examples/minivm/vm.cht — VM template using bare T[N] register files - 90+ golden harness tests under examples/minivm/vm-tests/ - 20 cross-language VM benchmarks under benchmarks/vm/ Compiler hooks needed to drive the :bc target - src/mir/mir.rb: MIR::Stmt#source_line / source_column for debugger attribution; MIR::Let#alias_safe field for bc emitter aliasing hints - src/mir/mir_lowering.rb: stamp_source_line! propagates AST source positions through lowered MIR - src/backends/importer.rb: CompiledModule.mir_items field for cross-file MIR access from the bc emitter; attr_reader :module_cache - src/ast/std_lib.rb: byteAt builtin (O(1) byte-level numeric access) - zig/runtime/runtime-header.zig: CheatLib.byteAt impl Bare T[N] MUTABLE syntax (used by VM register-file declarations) - src/ast/parser.rb: parse_mutable_var_decl handles `MUTABLE x: T[N];` with synthesized zero-defaults for primitive element types - src/ast/diagnostic_registry.rb: 3 new MUTABLE_BARE_* codes 5 RSpec files under spec/minivm_*_spec.rb cover golden harness, debugger, source-line attribution, file limits, and pipeline lowering. Baseline: 4712 specs / 539 transpile-tests / 0 leaks; register-VM allowlist 175/243 (see register-transpile-allowlist.txt for the roadmap of remaining work).
The register VM has already proven its value as a transpiler bug- finder: stress-testing it against transpile-tests surfaced 13 master bugs that landed as the hotfix-master-bugs branch. Lock that signal in by gating every PR on a baseline ratchet — no regression below the current pass count. Adds: - `--min-pass=N` flag to examples/minivm/run_tests.rb. The runner exits 0 if `passed >= N` regardless of pending/failed counts; exits 1 (with "baseline REGRESSION" message) if it slips below. This is the ratchet: bump N as we land coverage work that crosses tests fail->pass. - New CI job `register-vm-allowlist` after transpile-tests, sharing the clear-cache key for warm builds. Initial baseline: 175/243. Side effect: the runner's old all-pass-or-failure semantics (`failed.zero? && pending.zero?`) is preserved structurally but strict-mode callers should now use `run_vm_target_suite_with_count` + explicit predicate. The default `--vm=register` invocation still returns the pass count and exits 0 if any tests pass.
The bytecode VM cannot execute raw Zig, so `try_lower_equality_assert` (which produces a `MIR::InlineZig` calling `std.testing.expectEqual*`) left every `ASSERT a == b` test PENDING with reason "register emitter does not support MIR::InlineZig stmt assert_eq_*". Skip the equality-assert lowering when `@target == :bc` so the condition flows through the existing `lower(node.condition)` + `CheatLib.assert` path. Failure messages are slightly less rich than `expectEqualStrings`'s structured diff, but the tests run. Also bumps the CI register-VM baseline 175 → 186 (15 PENDING tests resolved: 11 PASS + 4 fail/missing on real correctness issues that the bc emitter now reaches and exposes). Found by VM stress-test against the transpile-tests corpus.
`collect_type_defs` iterated `@frontend_result.union_schemas` with
the assumption that each value is a raw `{ variant_name => Type }`
hash. Master's typed-cleanup work wrapped these in
`Schemas::UnionSchema` objects, which expose the hash via a
`.variants` accessor rather than being one. Result:
`undefined method 'map' for #<Schemas::UnionSchema>` on any test
that defines a generic union (e.g. `UNION Optional<T> { Some: T,
None }`) and matches it across file boundaries.
Normalize to `schema.variants` if available, fall back to the raw
hash otherwise (older importer paths still produce the latter).
Found by VM stress-test: 57_match_union_capture; baseline 186 -> 189.
The FSM transform consumes a BG body into Zig text and leaves
`run_body=[]` in the resulting `MIR::BgBlock`. The bc emitter cannot
walk Zig text; it needs structured MIR. Without this guard, every
BG block test failed with `register emitter requires structured
run_body for BgBlock`.
Skip the transform when `@target == :bc` so the legacy stackful-
fiber lowering populates run_body with the actual MIR of the body.
The bc VM is single-threaded, so synchronous inline execution of
the body is the correct semantics for `BG { ... }` -- the future
loom-mode scheduler (the `record_shared_event` scaffolding already
present) will exhaust interleavings later, but for now we just run
each BG body inline.
This was in the original 72-commit register-machine branch but got
lost when I rebuilt the branch cleanly on the current master.
Bumps CI baseline 189 -> 215. The 26 newly-passing tests cover BG
basics (58_bg, 59_bg_concurrent), bounded streams, BG STREAM, and
the FSM-with-suspend / FSM-multi-cap test cluster -- exactly the
PENDING block from the previous report. ALL 29 PENDING tests in the
allowlist resolved (0 PENDING remain); 26 PASS + 3 surfaced new
failures that the bc emitter now reaches.
After hotfix-stack-check-require lands (importer fix + huge-frame prologue verifier fix), the bc-VM register allowlist passes 238 of 243 tests. Lock the new floor. Remaining 5 are not memory corruption -- 1 lower_where Sorbet sig issue, 1 UTF-8 encoding mismatch, 3 missing test files (allowlist drift). Tracked separately. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
| Branch | register-machine |
| Testbed | ubuntu-latest |
⚠️ WARNING: No Threshold found!Without a Threshold, no Alerts will ever be generated.
Click here to create a new Threshold
For more information, see the Threshold documentation.
To only post results if a Threshold exists, set the--ci-only-thresholdsflag.
Click to view all benchmark results
| Benchmark | leak-build-ms | Measure (units) x 1e3 | leak-count | Measure (units) | leak-run-ms | Measure (units) |
|---|---|---|---|---|---|---|
| benchmarks/concurrent/02_concurrent_search/bench | 📈 view plot | 1.30 units x 1e3 | 📈 view plot | 0.00 units | 📈 view plot | 5.35 units |
| benchmarks/concurrent/07_stream_merge/bench | 📈 view plot | 1.15 units x 1e3 | 📈 view plot | 0.00 units | 📈 view plot | 28.96 units |
| benchmarks/concurrent/12_false_sharing/bench | 📈 view plot | 1.19 units x 1e3 | 📈 view plot | 0.00 units | 📈 view plot | 1,084.23 units |
| benchmarks/concurrent/19_atomic_ptr/bench | 📈 view plot | 1.16 units x 1e3 | 📈 view plot | 0.00 units | 📈 view plot | 117.10 units |
| benchmarks/inter-clear/05_concurrent_mvcc_pure_read/bench | 📈 view plot | 1.16 units x 1e3 | 📈 view plot | 0.00 units | 📈 view plot | 683.14 units |
| benchmarks/sequential/04_hashmap/bench | 📈 view plot | 1.30 units x 1e3 | 📈 view plot | 0.00 units | 📈 view plot | 1,748.58 units |
| benchmarks/sequential/09_frame_vs_heap/bench | 📈 view plot | 1.10 units x 1e3 | 📈 view plot | 0.00 units | 📈 view plot | 1,358.84 units |
| benchmarks/sequential/14_iterator/bench | 📈 view plot | 1.11 units x 1e3 | 📈 view plot | 0.00 units | 📈 view plot | 279.19 units |
|
| Branch | register-machine |
| Testbed | ubuntu-latest |
⚠️ WARNING: No Threshold found!Without a Threshold, no Alerts will ever be generated.
Click here to create a new Threshold
For more information, see the Threshold documentation.
To only post results if a Threshold exists, set the--ci-only-thresholdsflag.
Click to view all benchmark results
| Benchmark | leak-build-ms | Measure (units) x 1e3 | leak-count | Measure (units) | leak-run-ms | Measure (units) |
|---|---|---|---|---|---|---|
| benchmarks/concurrent/03_atomic_contention/bench | 📈 view plot | 4.86 units x 1e3 | 📈 view plot | 0.00 units | 📈 view plot | 53.22 units |
| benchmarks/concurrent/08_pubsub/bench | 📈 view plot | 4.80 units x 1e3 | 📈 view plot | 0.00 units | 📈 view plot | 2,881.69 units |
| benchmarks/concurrent/13_rwlock_starvation/bench | 📈 view plot | 4.86 units x 1e3 | 📈 view plot | 0.00 units | 📈 view plot | 1,029.97 units |
| benchmarks/inter-clear/06_concurrent_mvcc_writer_pressure/bench | 📈 view plot | 4.85 units x 1e3 | 📈 view plot | 0.00 units | 📈 view plot | 1,204.17 units |
| benchmarks/sequential/05_string_builder/bench | 📈 view plot | 4.62 units x 1e3 | 📈 view plot | 0.00 units | 📈 view plot | 23,123.10 units |
| benchmarks/sequential/10_pool_vs_multiowned/bench | 📈 view plot | 4.62 units x 1e3 | 📈 view plot | 0.00 units | 📈 view plot | 632.62 units |
| benchmarks/server/01_tcp_kvstore/server | 📈 view plot | 4.85 units x 1e3 | 📈 view plot | 0.00 units | 📈 view plot | 1,002.38 units |
|
| Branch | register-machine |
| Testbed | ubuntu-latest |
⚠️ WARNING: No Threshold found!Without a Threshold, no Alerts will ever be generated.
Click here to create a new Threshold
For more information, see the Threshold documentation.
To only post results if a Threshold exists, set the--ci-only-thresholdsflag.
Click to view all benchmark results
| Benchmark | leak-build-ms | Measure (units) x 1e3 | leak-count | Measure (units) | leak-run-ms | Measure (units) |
|---|---|---|---|---|---|---|
| benchmarks/concurrent/01_socket_throughput/bench | 📈 view plot | 5.38 units x 1e3 | 📈 view plot | 0.00 units | 📈 view plot | 1,442.08 units |
| benchmarks/concurrent/06_dynamic_spawn/bench | 📈 view plot | 5.22 units x 1e3 | 📈 view plot | 0.00 units | 📈 view plot | 3,684.60 units |
| benchmarks/concurrent/11_parallel_aggregation/bench | 📈 view plot | 5.30 units x 1e3 | 📈 view plot | 0.00 units | 📈 view plot | 7,344.67 units |
| benchmarks/concurrent/18_atomic_counter/bench | 📈 view plot | 5.21 units x 1e3 | 📈 view plot | 0.00 units | 📈 view plot | 45.70 units |
| benchmarks/inter-clear/04_concurrent_mvcc_fat_struct/bench | 📈 view plot | 5.29 units x 1e3 | 📈 view plot | 0.00 units | 📈 view plot | 342.66 units |
| benchmarks/sequential/03_alloc_throughput/bench | 📈 view plot | 5.20 units x 1e3 | 📈 view plot | 0.00 units | 📈 view plot | 11,508.36 units |
| benchmarks/sequential/13_soa_layout/bench | 📈 view plot | 5.17 units x 1e3 | 📈 view plot | 0.00 units | 📈 view plot | 755.09 units |
|
| Branch | register-machine |
| Testbed | ubuntu-latest |
⚠️ WARNING: No Threshold found!Without a Threshold, no Alerts will ever be generated.
Click here to create a new Threshold
For more information, see the Threshold documentation.
To only post results if a Threshold exists, set the--ci-only-thresholdsflag.
Click to view all benchmark results
| Benchmark | leak-build-ms | Measure (units) x 1e3 | leak-count | Measure (units) | leak-run-ms | Measure (units) |
|---|---|---|---|---|---|---|
| benchmarks/concurrent/05_backpressure/bench | 📈 view plot | 5.27 units x 1e3 | 📈 view plot | 0.00 units | 📈 view plot | 1,894.18 units |
| benchmarks/concurrent/10_shard_vs_locked/bench | 📈 view plot | 5.24 units x 1e3 | 📈 view plot | 0.00 units | 📈 view plot | 60,004.63 units |
| benchmarks/concurrent/16_observables/bench | 📈 view plot | 5.20 units x 1e3 | 📈 view plot | 0.00 units | 📈 view plot | 103.10 units |
| benchmarks/inter-clear/03_concurrent_mvcc_vs_rwlock/bench | 📈 view plot | 5.95 units x 1e3 | 📈 view plot | 0.00 units | 📈 view plot | 298.39 units |
| benchmarks/sequential/07_pointer_chase/bench | 📈 view plot | 5.25 units x 1e3 | 📈 view plot | 0.00 units | 📈 view plot | 521.40 units |
| benchmarks/sequential/12_weak_ref_graph/bench | 📈 view plot | 5.18 units x 1e3 | 📈 view plot | 0.00 units | 📈 view plot | 269.25 units |
| benchmarks/server/03_pathological/server | 📈 view plot | 5.29 units x 1e3 | 📈 view plot | 0.00 units | 📈 view plot | 1,002.88 units |
|
| Branch | register-machine |
| Testbed | ubuntu-latest |
⚠️ WARNING: No Threshold found!Without a Threshold, no Alerts will ever be generated.
Click here to create a new Threshold
For more information, see the Threshold documentation.
To only post results if a Threshold exists, set the--ci-only-thresholdsflag.
Click to view all benchmark results
| Benchmark | leak-build-ms | Measure (units) x 1e3 | leak-count | Measure (units) | leak-run-ms | Measure (units) |
|---|---|---|---|---|---|---|
| benchmarks/concurrent/04_fanout_fanin/bench | 📈 view plot | 5.63 units x 1e3 | 📈 view plot | 0.00 units | 📈 view plot | 3,253.07 units |
| benchmarks/concurrent/09_kvstore/bench | 📈 view plot | 5.72 units x 1e3 | 📈 view plot | 0.00 units | 📈 view plot | 60,004.15 units |
| benchmarks/concurrent/14_nested_lock/bench | 📈 view plot | 5.61 units x 1e3 | 📈 view plot | 0.00 units | 📈 view plot | 383.34 units |
| benchmarks/inter-clear/02_concurrent_fsm_vs_stackful/bench_fsm | 📈 view plot | 5.53 units x 1e3 | 📈 view plot | 0.00 units | 📈 view plot | 152.22 units |
| benchmarks/inter-clear/02_concurrent_fsm_vs_stackful/bench_stackful | 📈 view plot | 5.52 units x 1e3 | 📈 view plot | 0.00 units | 📈 view plot | 202.06 units |
| benchmarks/sequential/11_pipeline_overhead/bench | 📈 view plot | 5.58 units x 1e3 | 📈 view plot | 0.00 units | 📈 view plot | 12,381.34 units |
| benchmarks/server/02_json_api/server | 📈 view plot | 5.77 units x 1e3 | 📈 view plot | 0.00 units | 📈 view plot | 1,002.83 units |
Summary
Brings the register VM onto current master (post-hotfix-master-bugs merge) and lands the first two coverage unlocks of the pre-v0.1 bug-finding sprint. The VM is the primary stress-tester for the transpiler — every test category it can run shakes out fresh transpiler bugs.
Status: baseline raised 175 → 189 / 243 register-VM allowlist tests, gated in CI.
Commits
cf1beac2feat(minivm): register VM bytecode pipeline + debugger + golden tests — the VM itself: bc_emitter, runtime, vm.cht template, byebug-style debugger, time-travel scaffolding, 90+ golden tests, 20 cross-language benchmarks. Plus the small compiler hooks the:bctarget needs (MIR source-line metadata, MIR::Let alias_safe, CompiledModule mir_items field, byteAt builtin, bare T[N] MUTABLE syntax).763933d4ci(minivm): gate PRs on register-VM allowlist baseline — adds--min-pass=Nratchet toexamples/minivm/run_tests.rband a new CI job that fails on any regression below the current pass count. Bump--min-pass=Nas we land coverage work.0afef11cfeat(minivm): skip InlineZig equality assert lowering for :bc target — the bytecode VM cannot run raw Zig, sotry_lower_equality_assert(which producesMIR::InlineZigcallingstd.testing.expectEqual*) left everyASSERT a == btest PENDING. Skip when@target == :bcso the condition flows throughCheatLib.assert. +11 tests passing.e8a74c71feat(minivm): unwrap UnionSchema when collecting cross-file variants —collect_type_defsiteratedunion_schemasassuming raw{ variant => Type }hashes, but master's typed-cleanup work wrapped them inSchemas::UnionSchema. Normalize toschema.variantsif available. +3 tests passing.Test plan
bundle exec prspec spec/— 4712 specs, 0 failures./clear test transpile-tests/— 540 passed, 0 leaksbundle exec ruby examples/minivm/run_tests.rb --vm=register --min-pass=189— baseline gate greenWhy this matters
The VM is on track to expose more transpiler bugs as coverage grows. The hotfix-master-bugs branch (already merged) was 13 fixes that came from running the VM against transpile-tests; the next phases of this sprint will surface more.
Each coverage unlock here gates merge on the new baseline so future PRs can't silently regress what's already working.
🤖 Generated with Claude Code