Memory-safety: generalize borrow-escape rule + loop-escape promotion + EXTERN STRUCT CLOSE#47
Open
cuzzo wants to merge 21 commits into
Open
Memory-safety: generalize borrow-escape rule + loop-escape promotion + EXTERN STRUCT CLOSE#47cuzzo wants to merge 21 commits into
cuzzo wants to merge 21 commits into
Conversation
|
Bencher Report| Branch | hotfix-bg-promise-captures |
| Testbed | ubuntu-latest |
⚠️ WARNING: No Threshold found!Without a Threshold, no Alerts will ever be generated.
Click here to create a new Threshold
For more information, see the Threshold documentation.
To only post results if a Threshold exists, set the--ci-only-thresholdsflag.
Click to view all benchmark results
| Benchmark | leak-build-ms | Measure (units) x 1e3 | leak-count | Measure (units) | leak-run-ms | Measure (units) |
|---|---|---|---|---|---|---|
| benchmarks/concurrent/04_fanout_fanin/bench | 📈 view plot | 5.48 units x 1e3 | 📈 view plot | 0.00 units | 📈 view plot | 3,589.46 units |
| benchmarks/concurrent/09_kvstore/bench | 📈 view plot | 5.35 units x 1e3 | 📈 view plot | 0.00 units | 📈 view plot | 60,004.19 units |
| benchmarks/concurrent/14_nested_lock/bench | 📈 view plot | 5.29 units x 1e3 | 📈 view plot | 0.00 units | 📈 view plot | 412.00 units |
| benchmarks/inter-clear/02_concurrent_fsm_vs_stackful/bench_fsm | 📈 view plot | 5.31 units x 1e3 | 📈 view plot | 0.00 units | 📈 view plot | 167.73 units |
| benchmarks/inter-clear/02_concurrent_fsm_vs_stackful/bench_stackful | 📈 view plot | 5.29 units x 1e3 | 📈 view plot | 0.00 units | 📈 view plot | 194.28 units |
| benchmarks/sequential/11_pipeline_overhead/bench | 📈 view plot | 5.24 units x 1e3 | 📈 view plot | 0.00 units | 📈 view plot | 14,016.45 units |
| benchmarks/server/02_json_api/server | 📈 view plot | 5.38 units x 1e3 | 📈 view plot | 0.00 units | 📈 view plot | 1,002.80 units |
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #47 +/- ##
==========================================
+ Coverage 90.01% 90.12% +0.10%
==========================================
Files 185 185
Lines 50226 50363 +137
Branches 12044 12139 +95
==========================================
+ Hits 45212 45388 +176
+ Misses 5014 4975 -39
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
Modernizes the MAL example for current CLEAR syntax, fallible-return discipline, ownership copies, and pool env access patterns. This is compile-only: the interpreter still needs follow-up work before its runtime assertions pass correctly.
… forms
The compiler had two specialized rules — RETURN-of-WITH-alias and IF-AS
non_escaping propagation — that left a memory-safety hole: borrows
extracted via MATCH-AS, WHILE-AS, FOR-EACH, struct destructure, or
pipeline AS could be returned, container-stored, or BG-captured with no
diagnostic, producing UAF.
Discovered while debugging examples/mal/interpreter.cht's "tagged pair
car" test: prStr's `Value.Symbol AS s -> RETURN s;` returned a slice
into a caller-owned Value whose cleanup-defer fired before the function
actually returned. Caller dereferenced freed memory.
Universal rule replacing the specialized ones:
Any binding produced by a borrow-extracting pattern (WITH-AS,
IF-AS, MATCH-AS, WHILE-AS, FOR-EACH, struct destructure in
IF/MATCH, pipeline `AS $name`) is marked non_escaping AND
borrowed_alias when the payload is non-Copy. visit_ReturnNode
rejects RETURN of any chain rooted in such a binding (identifier,
GetField, GetIndex) regardless of WITH context. The existing
has_non_escaping_capture check rejects fiber capture via the same
flag.
This is mostly subtractive: the @with_block_depth>0 gate is lifted,
the IF-AS conditional propagation is replaced with unconditional, and
a single helper (mark_borrow_binding_non_escaping!) becomes the
single source of truth. New diagnostic codes RETURN_OF_BORROW,
RETURN_FIELD_OF_BORROW, RETURN_INDEX_OF_BORROW for non-WITH cases;
existing WITH codes preserved for in-WITH context.
Tests / specs updated to use `RETURN COPY s` (the safe form) where
they used the unsafe pattern: 7 transpile-tests (119, 160, 162, 174,
175, 176, 179) plus spec/affine_ownership_spec.rb and
spec/cleanup_plan_spec.rb. The test intent (cleanup-plan classification)
is preserved.
Regression artifacts:
- transpile-tests/_xfail/383_uaf_match_as_borrow_returned.cht:
minimal 25-line runtime repro; lives outside the auto-glob.
- spec/transpiler_spec.rb: 7 new specs covering identifier, field,
index, BG-capture, FOR-EACH, MATCH-destructure shapes, plus a
`RETURN COPY` positive case.
- spec/schemas_spec.rb: 7 unit tests for as_struct_schema and
as_resource_schema fallbacks (closes 3-line coverage gap from
the existing PR scope).
- docs/postmortems/383_uaf_match_as_borrow_returned.md: full
root-cause writeup, table of every borrow-introducing site,
LEND-roadmap alignment.
examples/mal/interpreter.cht updated with COPY at every flagged site
so it transpiles cleanly. It still does not run end-to-end: prStr
mixes frame-allocated string-concat returns with heap-allocated COPY
returns, hitting an INV-1 (single-allocator-per-binding) violation
that escape analysis currently does not unify across return branches.
That is a separate compiler bug filed for follow-up.
Verified: 4722 examples / 0 failures / 3 pending; Sorbet clean;
all transpile-tests/*.cht produce valid Zig; 0 uncovered changed
lines in src/.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
cuzzo
changed the title
Template-based program generator that drives .cht source through `./clear test` end-to-end, exercising the MIR static checker (9 invariants) and runtime leak detection (`std.testing.allocator`). Templates (90 active cells, 30 reserved :in_dev for unlanded features): - escape_via_return (18) — E2 :always_returned, :heap_ptr_return - loop_carry_collection (8) — E2 :loop_carry_string + frame-rewind - mutable_collection_param (8) — INV-CROSS-FRAME-PARAM-ALLOC - nested_loop_escape (8) — loop-local list/map escape (commit 9fa2192) - stream_into_boundary (48) — NEXT × {BG, DO, BG STREAM} × ownership/sync/move/value - lifetimed_return (6) — bg_lifetime_sources stamping → enforcement check Cells carry an `expected:` annotation (:pass / :compile_error / :in_dev) so the matrix accommodates documented capability boundaries (e.g., DO + @Local) and unlanded features (LEND, @atomic BG-body capture) without losing cell count. Findings on the current tree: - 1 leak: (bg_stream, local, copy, String) — heap addresses leak after BG STREAM consumer with WHILE TRUE and outer-scope String COPY. - 2 UNEXPECTED-PASS: BG handle capturing @Local can RETURN or be stored in a heap struct field. Compiler accepts; runtime crashes with SIGABRT. Real UAF surface — bg_lifetime_sources stamps but doesn't enforce these cases. - 5 MIR-FAIL: phase B outstanding work (BG-body atomic auto-load, edge case in BG+versioned+copy). docs/agents/formal-verification-testing.md inventories the 12 testing layers, what each covers, what's intentionally not covered (with reasons), and a ranked TODO of 13 combinatoric sets to add. Folds in takeaways from an older draft at ~/manual/clear/docs/agents/. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Verifies CLAUDE.md's non-escaping rule on WITH aliases (EXCLUSIVE / BORROWED
/ RESTRICT / SNAPSHOT) against 10 patterns per alias-perm tuple: 2 baselines
(use, RETURN COPY) + 8 escape attempts (return alias, return field, BG / DO
/ BG STREAM capture, TAKES consume, store in heap field, list append).
5 alias-perm tuples × 10 patterns = 50 cells.
Findings on current tree:
- 30/30 escape-attempt cells correctly reject with the right diagnostic
("Cannot RETURN 'ref' from inside a WITH block. WITH aliases are
borrows of locked data and cannot escape their scope."). Solid.
- 16/20 baseline cells pass.
- 4 baseline_copy_return cells fail with Zig codegen error "expected
error_union, found *T": RETURN COPY ref lowers to a *T pointer
instead of a Counter value for EXCLUSIVE / RESTRICT / SNAPSHOT
aliases. Only BORROWED's COPY-return path is correctly lowered.
The unit test spec/with_alias_escape_spec.rb "allows RETURN COPY of an
EXCLUSIVE alias" passes because it stops at annotation. The matrix
running end-to-end through codegen surfaces the type mismatch — exactly
the unit/e2e gap motivating the harness.
Per branch policy: matrix surfaces the bugs; fixes land elsewhere.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
A function whose return type is non-Copy String/list/map hands its
result to the caller, who expects to own it on a single allocator
(heap, by convention — what the caller's cleanup defer assumes).
Before this commit, escape analysis only marked the FUNCTION's
return_provenance, never propagating that decision down into the
individual return-position expressions. A function with mixed
branches like
RETURNS !String ->
PARTIAL MATCH v START
Value.Symbol AS s -> RETURN COPY s;, -- heap (dupe to heap)
Value.List AS items -> RETURN out + ")"; -- frame (string concat)
END
would emit `concat(rt.frameAlloc(), ...)` for the List branch while
the caller emits `defer rt.heapAlloc().free(returned_slice)`. INV-1
violation that surfaces as "Invalid free" under the debug allocator.
Surfaced by examples/mal/interpreter.cht's `prStr` after the
borrow-escape rejection landed: prStr was previously dodging this by
returning borrows (no caller-side cleanup at all); once the borrow
return was rejected and rewritten to `RETURN COPY s`, the mixed-
allocator bug became live.
Three pieces:
1. src/mir/escape_analysis.rb — new "Condition 9" in per_fn_scan!:
if the function's return type (after `!T` unwrap) is a non-Copy
String/list/map, walk every ReturnNode and call both
`e2_promote_frame_concats!` (BinaryOp / StringConcat / list &
struct literals) and `LoopFrameAnalysis.promote_value_to_heap!`
(FuncCall / MethodCall / Identifier root resolution).
2. src/mir/control_flow.rb — `promote_value_to_heap!` for
FuncCall/MethodCall now ALSO sets `node.storage = :heap` so the
intrinsic dispatch's `:node_storage` allocator resolution
produces `:heap` and the InlineZig substitutes
`rt.heapAlloc()` for `{alloc}`. Previously it only set
`heap_dupe_result`, which lower_intrinsic never reaches because
intrinsics return early.
3. src/mir/mir_lowering.rb — `hoist_cleanup_entry` gains a case for
allocating `MIR::InlineZig`, deriving the cleanup kind from the
originating AST node's return type (string / list / non-Copy
union). Without this, a heap-promoted intrinsic that triggers
`mir_allocates?` would crash hoist_alloc with
"unhandled allocating MIR node MIR::InlineZig".
Effect on examples/mal/interpreter.cht: tests 1-57 now run
correctly (arithmetic, def!/let*/if/do, comparison, list/count/
empty?/not, recursion, define/lambda/let/begin, set!, TCO,
vector-ref, cons/car/cdr, quote). String-append (test ~58) still
fails — separate INV-1 bug class around loop-carry strings whose
declared allocator is heap but reassignments produce frame.
Filed as a follow-up.
No spec regressions: 4722 examples / 0 failures / 3 pending.
Sorbet clean. All transpile-tests/*.cht produce valid Zig.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three bugs fixed in this PR (universal borrow-escape; mixed-allocator
returns / INV-1; allocating InlineZig in hoist_cleanup_entry); two
filed for follow-up:
4. Loop-carry frame->heap binding: a heap-declared binding
(e.g. `MUTABLE out = getStr(...)`) reassigned from a frame
string-concat (`out = out + getStr(...)`) violates INV-1 at
scope-end cleanup. The existing loop_carry_string rule only
fires for loops with mark_per_iter; this loop body has no
frame allocs, so Cond 5 is silent. MAL stops at test ~58
(string-append) on this.
5. Universal borrow-escape misses plain assignment: the same
check that catches `RETURN borrow.field` and
`outer.append(borrow.field)` should catch `x = borrow.field`
when `x`'s storage is heap. Today the user must add `COPY`
manually (see eval's `ast = COPY tco.tcoAst`).
The audit rule applied: `COPY` added at borrow-return sites is
correct language semantics, not a workaround. A workaround is
anything that hides a compiler bug behind MAL-level code. None of
the MAL changes in PR #47 fall in that category.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Loop-carry-string allocator mismatch: a binding declared with
storage=:heap (initialized from a heap-returning fn) reassigned from
a frame string-concat would store frame memory in a heap-tagged
binding. The cleanup defer at scope-end then frees frame slice via
heapAlloc.free — INV-1 violation, "Invalid free" under the debug
allocator.
Surfaced by examples/mal/interpreter.cht's `applyNative` string-append
branch:
IF id == 26 THEN
MUTABLE out = getStr(evaled[1]); # heap (getStr returns heap)
FOR si IN (...) DO
out = out + getStr(evaled[si]); # frame concat -> heap binding
END
RETURN Value{ Str: COPY out };
END
Existing Cond 5 (loop_carry_string) handled a subset: same shape but
gated on loops with `mark_per_iter`. The string-append loop body has
no frame-allocating decls of its own, so Cond 5 was silent.
New Cond 10 in escape_analysis.rb#per_fn_scan!:
1. Walk decls, collect names declared with storage=:heap (or whose
type already has heap_provenance) and a non-Copy String/list/map
type.
2. Walk assignments. For each `name = value` where `name` is in the
heap_string_decls set, run `e2_promote_frame_concats!` and
`LoopFrameAnalysis.promote_value_to_heap!` on the value.
This is the broader sibling of Cond 9 (RETURN-site unification): same
underlying rule (binding allocator must be uniform across all stores
into it), applied at every assignment instead of only at returns.
Effect on examples/mal/interpreter.cht: ALL 75 INTERPRETER TESTS
PASSED. (The debug allocator still reports leaks — separate bug class
about COPY-at-return cleanup matching, filed in mal-bugs.md.)
No spec regressions: 4722 examples / 0 failures / 3 pending. Sorbet
clean. All transpile-tests/*.cht produce valid Zig.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Verifies which (callee signature × caller binding) tuples the annotator admits today. 6 callee forms × 6 caller bindings = 36 cells. Callees: concrete, SHARED Counter (parametric), REQUIRES c: LOCKED, REQUIRES c: VERSIONED, REQUIRES c: LOCAL, REQUIRES c: LOCKED | LOCAL. Callers: @locked, @writeLocked, @versioned, @Local, @multiowned, plain. Findings on current tree (1 UNEXPECTED-PASS + 12 MIR-FAIL): - (concrete, @Local) UNEXPECTED-PASS: concrete callee admits @Local despite docs/sharing-capabilities.md stating concrete admits plain T only. The viralization-risk surface — @Local is structurally a *T pointer so admission rules treat it like a plain local. - SHARED T param rejects @locked / @writeLocked / @versioned short forms (3 cells) — short forms don't coerce to @shared:* at call sites. Test 349 uses the explicit @shared:locked form throughout. - WITH MATCH syntax not parsed — "Unknown WITH capability 'MATCH'" (5 cells, all REQUIRES LOCKED | LOCAL). CLAUDE.md describes the syntax; parser doesn't accept it yet. - Codegen failures for several legitimately-admitted cells (req_locked + @locked, req_versioned + @versioned, req_local + @Local) — pointer-deref / error-union-ignored Zig errors. Same patterns test 349 uses successfully with full @shared:locked form; short-form callers trip a different lowering path. Per branch policy: matrix surfaces; fixes land elsewhere. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Cleanup after the assignment-site allocator unification landed: - src/mir/escape_analysis.rb: drop the STDERR debug instrumentation used to verify Cond 10 fires on applyNative::out and friends. - docs/agents/mal-bugs.md: bug #4 (loop-carry frame->heap binding) marked fixed; new bug #7 filed for the 72 leaks the debug allocator reports after all 75 MAL tests pass — sample trace points to the COPY-at-borrow-return path in getStr. MAL state at end of this PR: - All 75 interpreter tests PASS - 4722 spec examples / 0 failures / 3 pending - Sorbet clean - 72 DebugAllocator leaks (filed, not addressed) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Verifies the modifier × ownership rules from src/ast/diagnostic_registry.rb: what can / can't cross BG / DO / BG STREAM with @parallel or @Pinned. Cross-product: - boundary ∈ {bg, do, bg_stream} - modifier ∈ {none, @parallel, @Pinned} - ownership ∈ {@Local, @shared:locked, @multiowned} = 27 cells. Findings on the current tree: Solid enforcement: - (any boundary, @parallel, @Local) → correctly rejected (3/3 cells) - (any boundary, @parallel, @multiowned) → correctly rejected (3/3 cells) - All BG cells pass (none / @parallel / @Pinned × all 3 ownerships) Gaps surfaced (8 cells fail end-to-end): - DO + @Local / DO + @multiowned (modifier none or @Pinned, 4 cells): DO branches lower to inner Zig fns that don't close over outer @Local / @multiowned bindings. The existing test corpus only uses DO with @shared:locked state. Either DO should learn to capture these, or docs should clarify that DO requires @shared. - BG STREAM + @parallel / BG STREAM + @Pinned (4 cells): the BG STREAM parser has no equivalent of parse_bg_prefix — modifier sigils inside the stream body don't parse. Inconsistent with BG. The @parallel-with-@local and @parallel-with-@multiowned diagnostics fire correctly across all 3 boundaries — the canonical boundary- crossing rules from sharing-capabilities.md are enforced. Combined matrix: 203 active cells across 9 templates, 25 distinct findings (1 leak + 21 mir-error + 3 unexpected-pass). Per branch policy: matrix surfaces; fixes land elsewhere. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closes the remaining gaps in the borrow-escape rule so every borrow producer feeds into one helper and every escape category fires the same way at every shape (identifier, field-chain, index-chain). Producer gaps closed (one helper): - WITH-AS aliases (capabilities.rb 6 sites: BORROWED, RESTRICT, VIEW, SNAPSHOT, plain field, plain synced) now route through mark_borrow_binding_non_escaping! with `force: true`. Lock-bounded lifetime semantics share the helper with heap-payload borrow semantics. The `force:` keyword bypasses the Copy short-circuit — WITH-AS aliases need non_escaping even for Copy payloads since the lock can be released independently of payload Copy-ness. - IF-AS and WHILE-AS now also set @og[name].kind = :borrowed for symmetry with MATCH-AS / FOR-EACH / pipeline-AS. Enforcement gaps closed (chain-aware everywhere): - ensure_owned_value!: container-store check extended via ifbind_source_root. New diag STORE_FIELD_OF_BORROW_INTO_CONTAINER for chain shapes; existing STORE_WITH_SCOPED_INTO_CONTAINER kept for single-level identifiers (better diagnostics). - handle_assign_move: NEW check for plain assignment of a borrow chain. Walks ifbind_source_root; gates on (a) is_reassign — only Assignment nodes and BindExpr `:assign` mode, fresh decls are accepted, and (b) extracted-value type is non-Copy — so e.g. `v = y.value` where `.value` is Int64 stays accepted. New diag ASSIGN_BORROW_TO_OUTER_BINDING. This was MAL bug #5 (mal-bugs.md): the eval-loop's `ast = tco.tcoAst` previously slipped through. Categories now uniformly enforced at the annotator level: ESCAPE / RETURN visit_ReturnNode (chain) ESCAPE / container-store ensure_owned_value! (chain) ESCAPE / outer-binding-assign handle_assign_move (chain) ESCAPE / plain-assign chain handle_assign_move (chain) [NEW] BOUNDARY / fiber-capture has_non_escaping_capture (chain) GATED / WITH RESTRICT/BORROWED BorrowChecker (existing) Defense layer: spec/borrow_escape_matrix_spec.rb is a combinatorial (producer × category) regression suite. 16 cells covered today; expandable as new patterns appear. Plain-assign repro: transpile-tests/_xfail/borrow_escape_via_plain_assign.cht. A separate MIR-level checker pass is intentionally NOT included — the annotator IS the rejection gate, and the regression matrix provides the safety net. A defensive MIR pass remains a future follow-up if/when MIR-level rewrites generate synthetic escape patterns the annotator can't see. Verified: 4738 examples / 0 failures / 3 pending. Sorbet clean. All transpile-tests/*.cht produce valid Zig. MAL still passes 75/75 (no false positives). Files updated: - docs/agents/mal-bugs.md: bug #5 marked fixed - docs/postmortems/383_uaf_match_as_borrow_returned.md: universal coverage table Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three focused templates sharing _cleanup_dimensions.rb so adding a new alloc kind propagates to all three; control-flow dimension is template- specific (loop disruptors, error patterns, branch shapes). Templates: - loop_cleanup (40 cells) — INV-2/INV-6: alloc inside loop bodies with break/continue/early-return/raise disruptors - error_cleanup (24 cells) — INV-9: alloc-cleanup on error paths (OR PASS / RAISE / DEFAULT × success/failure trigger) - branch_cleanup (48 cells) — INV-2: alloc-cleanup across IF/ELSE with then-only / else-only / both-same / both-diff shapes and optional early-return disruptors Shared: - _cleanup_dimensions.rb — ALLOC_KINDS, VALUE_DESTS, alloc/use helpers. Loaded but doesn't register a template. docs/agents/formal-verification-bugs.md — catalogue of 10 active bugs the matrix has surfaced so far across all templates, plus 10 emitter workarounds tracked for removal when bugs are fixed. Each entry has repro, symptom, and notes on which spec layer missed it. Combined matrix (12 templates, 315 active cells, 30 :in_dev): - 203 ok - 6 leak - 103 mir-error - 3 unexpected-pass = 112 distinct findings. Many cleanup-correctness MIR-FAILs are likely related to bug #10 (`x = expr OR PASS` leaks for heap-returning fn), since the cleanup templates use OR PASS as the caller wrapper. Once #10 is fixed expect a cleaner baseline; remaining failures are real findings. Per branch policy: matrix surfaces; fixes land elsewhere. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…actic position Bug #10 in formal-verification-bugs.md (`result = expr OR PASS` leaks) was only the tip. This matrix exercises every syntactic position OR can appear in × action × inner outcome. Cell schema: position × action × outcome × inner_t. Positions tested: - :assign_rhs — `result = inner() OR …` - :fn_arg — `consume(inner() OR …)` - :method_arg — `outer.append(inner() OR …)` - :return_expr — `RETURN inner() OR …` - :with_source — `WITH EXCLUSIVE (inner() OR …) AS x { … }` - :collection_lit — `[inner() OR …]` Actions: PASS, RAISE, <default>. Outcomes: inner succeeds vs raises. Inner types: heap_list, heap_string. = 60 cells total. Findings: 15 ok / 45 fail. The bug surface is much wider than originally suspected — `OR <action>` cleanup pairing is broken across most positions: - All assign_rhs cells with inner=success leak (5 cells) - All method_arg + heap_string + success leak (3 cells) - All return_expr cells fail (LEAK or MIR-FAIL, 7 cells) - All with_source cells MIR-FAIL universally (10 cells) - Various others depending on action / outcome / type Updated bug #10 in formal-verification-bugs.md with the broader surface. Combined matrix now: 13 templates, 375 active cells, 30 :in_dev. Per branch policy: matrix surfaces; fixes land elsewhere. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The lifetime check at function_analysis.rb#verify_return already rejects borrow returns without COPY or RETURNS x:T — but its gate used Type#copyable? / implicitly_copyable?, which treat strings, slices, and any struct/union recursively containing them as Copy. That answers "can I memcpy this value?" — true for slice headers (ptr+len) — but conflates value-Copy with self-containment. The bytes the slice points to live in someone else's heap; returning the slice without a lifetime contract is a shallow-copy escape that surfaces as UAF the moment the caller drops the source. Repro: RETURN e.errMsg where e is a parameter compiled cleanly, yielded a slice into caller-owned memory, and was relied on by existing transpile-tests (77, 78) and a spec (caller_cleanup_spec). Same hole exists for plain T[] slices and for structs/unions with String fields. Fix: new Type#self_contained?(resolver) in src/ast/type.rb. Stricter than copyable?: returns false for string?, array?, heap?, sync/wrapper types, containers, @indirect; recurses into structs and union variants requiring all to be self-contained. function_analysis.rb#verify_return gates on this instead of is_copyable. The existing RETURNS x:T lifetime annotation IS the mechanism for shallow-copy returns. No new MIR infrastructure needed (user asked whether MIR-level dangling-pointer tracking was warranted — it is not, the existing lifetime check handles it once the gate is correct). Blast radius: 2 transpile-tests + 1 spec that did `RETURN x.name` where x was a local. Real shallow-copy escapes; fixed by binding `name = COPY x.name; RETURN name;`. The naive `RETURN COPY x.name` form hits a separate lowering bug (UNHOISTED_ALLOC for DeepCopy in non-Let-init position) — filed as task #27. Matrix coverage: 4 new cells in spec/borrow_escape_matrix_spec.rb under "function parameter (shallow-copy escape via param.field)". Verified: 4738 examples / 0 failures / 3 pending. Sorbet clean. All transpile-tests/*.cht produce valid Zig. MAL still passes 75/75 (already used COPY at every borrow-return site). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`lower_return` for catch_string_dupe_ret functions wraps every return value in `MIR::DupeSlice.new(value, :heap)` to ensure both success and error paths return heap-backed strings (consistent caller cleanup contract). That's correct for frame-allocated values (literals, slice-of-param, frame concats) — the wrap promotes them to heap. But when `value` is itself already heap-allocating (`MIR::DeepCopy` from `RETURN COPY x.field`, `MIR::ConcatStr` from a heap-promoted concat, etc.), the wrap NESTS the inner alloc inside DupeSlice. The MIR checker walks the Let.init with allow_top: true and recurses with allow_top: false; the nested DeepCopy lands in non-Let-init position and trips UNHOISTED_ALLOC. Surfaced by `RETURN COPY validUser.name` patterns in caller_cleanup spec, transpile-tests/77, and transpile-tests/78 — those were originally fixed with `name = COPY x.name; RETURN name;` workarounds to sidestep this lowering bug. With this fix the cleaner form works directly. Fix: gate the DupeSlice wrap on `mir_allocates?(value)`. If the value is already heap-allocating, hoist it as-is. Otherwise wrap in DupeSlice as before. Verified: 4742 examples / 0 failures / 3 pending. Sorbet clean. All transpile-tests/*.cht produce valid Zig. MAL still passes 75/75. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
cuzzo
added a commit
that referenced
this pull request
May 16, 2026
Locals sourced from `x.type_info rescue nil` are provably nil|Type post-#45, so the `Type.new(ti) if ti && !ti.is_a?(Type)` coercions are dead and `if ti.is_a?(Type)` is a redundant nil-check. Collapsed: escape_analysis per_fn_scan!(238), e2_loop_carry_names!(decl_ti, outer_ti), e3_mark_carry_expr!(904,910); control_flow _collect_share_moves; promotion_plan stamp_field_pre_cleanups!. #52's 327/374 are .symbol.type (heterogeneous, = #47), left alone. Gates green. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
cuzzo
added a commit
that referenced
this pull request
May 16, 2026
User's model validated (target nil|Type|FunctionSignature; String/ Symbol slop; sigs -> T.any(Type,FunctionSignature)). Carrier disambiguated: Literal#type is a separate token-kind field, no blast radius. Real obstacle is semantic, not mechanical: the 11 is_a?(Type) sites double as a resolved-vs-unresolved gate, so normalizing the seam changes which decls get processed. Documented as the spec for a focused reviewed PR; not auto-run (gates-green != provably-correct for semantic change). #48-#51,#53,#57,#59 same. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
cuzzo
added a commit
that referenced
this pull request
May 17, 2026
Locals sourced from `x.type_info rescue nil` are provably nil|Type post-#45, so the `Type.new(ti) if ti && !ti.is_a?(Type)` coercions are dead and `if ti.is_a?(Type)` is a redundant nil-check. Collapsed: escape_analysis per_fn_scan!(238), e2_loop_carry_names!(decl_ti, outer_ti), e3_mark_carry_expr!(904,910); control_flow _collect_share_moves; promotion_plan stamp_field_pre_cleanups!. #52's 327/374 are .symbol.type (heterogeneous, = #47), left alone. Gates green. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
cuzzo
added a commit
that referenced
this pull request
May 17, 2026
User's model validated (target nil|Type|FunctionSignature; String/ Symbol slop; sigs -> T.any(Type,FunctionSignature)). Carrier disambiguated: Literal#type is a separate token-kind field, no blast radius. Real obstacle is semantic, not mechanical: the 11 is_a?(Type) sites double as a resolved-vs-unresolved gate, so normalizing the seam changes which decls get processed. Documented as the spec for a focused reviewed PR; not auto-run (gates-green != provably-correct for semantic change). #48-#51,#53,#57,#59 same. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Started as a multi-bug hotfix; grew into a substantive memory-safety release after we discovered a use-after-free in MAL whose root cause was a missing universal borrow-escape rule.
Conceptual changes
1.
src/annotator.rb— Universal borrow-escape rejection (new)Every borrow-introducing AST construct now feeds into a single helper,
mark_borrow_binding_non_escaping!, which sets bothnon_escaping(rejects RETURN / container-store) andborrowed_alias(rejects BG / lambda / stream capture) when the payload is non-Copy.Sites covered:
WITH ... AS alias(pre-existing)IF expr AS x(was conditional, now unconditional for non-Copy)MATCH v AS s(new)WHILE expr AS s(new)FOR x IN coll(new)MATCH v { Variant{a, b} -> ... }andIF expr AS { a, b }struct destructure (new)coll AS $name(pipeline AS-binding) (new)visit_ReturnNode's previously WITH-only return-of-borrow rejection is now universal. Walks GetField/GetIndex chains viaifbind_source_rootto catchpp.p.left-style chains.3 new diagnostic codes:
RETURN_OF_BORROW,RETURN_FIELD_OF_BORROW,RETURN_INDEX_OF_BORROW. WITH-specific codes preserved for in-WITH context.Postmortem:
docs/postmortems/383_uaf_match_as_borrow_returned.md. Minimal repro:transpile-tests/_xfail/383_uaf_match_as_borrow_returned.cht.2.
src/mir/control_flow.rb— Loop-escape promotionLoop-local frame values (lists, maps, arrays, strings) that escape into outer containers are now heap-promoted at declaration time. Without this, the per-iteration frame rewind invalidates the pointer the outer container holds, producing UAF.
3.
src/mir/mir_lowering.rb+src/annotator-helpers/function_analysis.rb— Mutable value params by referenceSemantic change.
MUTABLE x: Int64parameters now lower to*i64at the Zig level with a writebackdeferon every exit path, so callee mutations propagate to the caller. Call sites pass&binding. Newmutable_ref_targetflag onSymbolEntryforces Zigvarstorage (so&bindingyields*T, not*const T).Two existing specs updated to match the new behavior (
mir_lowering_spec.rb,generics_spec.rb).4.
src/ast/schemas.rb— EXTERN STRUCT CLOSE fields fallbackas_struct_schemaandas_resource_schemanow accept the bare-key hash shape produced byvisit_ExternStructDecl, soEXTERN STRUCT { ... } CLOSE "deinit"resolves field accesses correctly.Test surface
spec/transpiler_spec.rb: 7 new specs underMATCH-AS borrow escape (memory safety)covering identifier / field / index / FOR-EACH / MATCH-destructure / BG-capture /RETURN COPYpositive casespec/schemas_spec.rb(new): 7 unit tests for the schema fallbacksspec/loop_frame_analysis_spec.rb: 4 new specs for escape-into-outer-container scenariosspec/annotator_spec.rb: 1 new spec for EXTERN STRUCT CLOSE field accesstranspile-tests/382_mutable_param_updates_caller.cht(new): mutable-param-by-reference contractRETURN COPY s(the safe form) where they used the unsafe pattern: 119, 160, 162, 174, 175, 176, 179. Test intent (cleanup-plan classification) preserved.affine_ownership_spec.rb,cleanup_plan_spec.rbVerification
transpile-tests/*.chtproduce valid Zigsrc/per simplecovTest plan
.chtfiles intranspile-tests/transpile cleanlyall-tests.zig— fails on a pre-existing@hasField/AtomicPtr issue unrelated to this PR; same failure on master baselineexamples/mal/interpreter.chtruntime — still does not pass; surfaces a separate INV-1 bug in escape analysis (allocator unification across return branches) filed for follow-upKnown follow-up (NOT in this PR)
examples/mal/interpreter.chtnow compile-errors at every unsafe site (correct). WithCOPYadded it transpiles. At runtime,prStrmixes frame-allocated string-concat returns with heap-allocatedCOPYreturns; the caller's cleanup defer assumes one allocator. This is a genuine INV-1 violation that escape analysis should prevent — files for separate compiler work.Generated with Claude Code