feat: rivet mechanical oracle (validate + impact → findings)#26
Merged
Conversation
## Why The hardened AI review (PR #24) closes the model's escape hatches but the model itself is still the only source of findings. For repos that ship rivet traceability (rivet, spar, ...), we have a *mechanical* oracle that already emits the exact shape of finding we want — name-anchored, deterministic, falsifiable. Wire it in. This is the pulseengine.eu "model proposes, oracle decides" pattern. The discovery here is that `rivet validate --format json` and `rivet impact --since=<ref> --format json` already produce the per-artifact diagnostic output we need; no model is involved in the *judgement* — only in the proposal of which lines to look at, which we get for free from the diff. ## What New module `src/rivet-oracle.js` with three exports: | Function | Job | |---|---| | `isRivetProject(repoPath)` | Detect `rivet.yaml` at the tree root. | | `runRivetValidate(binary, repoPath, opts)` | Run the validate CLI, parse JSON, convert error+warning diagnostics + lifecycle gaps + broken cross-refs to Finding records. Severity 'info' is dropped (66 of them on rivet/main alone — too noisy). | | `runRivetImpact(binary, repoPath, baseRef, opts)` | Run the impact CLI against a baseline ref. Surfaces removed artifacts (risky) and directly_affected (the trace contract this PR is changing). Transitive list is summarised, not enumerated. `added`/`changed` are *not* surfaced — those are diff metadata, not concerns. | | `runRivetOracle(binary, repoPath, opts)` | One-shot helper: runs both, merges findings, returns degraded-but-useful results when one of the two fails. | The module is *side-effect-free* on the caller's repo: it does not clone, checkout, or install. The caller provides a working tree path. The `runner` option is injected for tests so we can stub `execFile` deterministically. Each finding carries `{source: 'oracle:rivet-validate' | 'oracle:rivet-impact', severity, artifact_id, claim}`. Claims are constructed from the rivet output verbatim — no hedging language is possible by construction. The PR-A slop filter is therefore a no-op on these findings (which is fine — they don't go through the model path anyway). ## What's NOT in this PR - Integration with `ai-review.js` — that's PR-C. This PR is the building block; wiring it requires repo-tarball fetching, base-ref handling, and finding-source styling in the renderer. - Auto-installer + scheduled update task — that's PR-D. - The binary itself on netcup — manual install (one shot) until the installer ships. ## Test plan - [x] All 752 tests pass (was 737 — added 15 covering parse, severity-drop, non-zero-exit handling, spawn-failure, summary aggregation, degraded-mode behaviour) - [x] eslint clean - [x] Manual: ran `rivet validate --format json` against `pulseengine/rivet@v0.4.3` locally — it emits 6 errors / 10 warnings / 66 infos / 1 lifecycle gap. Module correctly converts to 17 findings (errors + warnings + gaps + broken cross-refs; infos dropped). - [x] Manual: ran `rivet impact --since=v0.4.2 --format json` on the same — JSON shape matches what the parser expects. ## Risk & rollout - Risk: **low**. No behaviour change yet — module is exported but not yet called from `ai-review.js`. Pure addition. - Rollout: self-update on merge is a no-op for end users. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The hardened AI review (PR #24) closes the model's escape hatches but the model itself is still the only source of findings. For repos that ship rivet traceability (rivet, spar, ...), we have a mechanical oracle that already emits the exact shape of finding we want — name-anchored, deterministic, falsifiable. Wire it in.
This is the pulseengine.eu "model proposes, oracle decides" pattern. The discovery here is that `rivet validate --format json` and `rivet impact --since= --format json` already produce the per-artifact diagnostic output we need; no model is involved in the judgement, only in proposing which lines to look at — which we get for free from the diff.
What
New module `src/rivet-oracle.js` with three exports:
The module is side-effect-free: no cloning, no checkout, no install. The caller provides a working tree path. The `runner` option is injected for tests so we can stub `execFile` deterministically.
Each finding carries `{source, severity, artifact_id, claim}`. Claims are constructed from rivet output verbatim — no hedging is possible by construction. PR-A's slop filter is a no-op on oracle findings (correct: oracle findings don't pass through the model path).
What's NOT here
Test plan
Risk & rollout
🤖 Generated with Claude Code