v1.3: fail-on-findings + in-band suppressions + first calibrated Terraform shape#65
Merged
Conversation
Two coupled changes that respond to real engineering critique on v1.0-1.2: warnings get ignored in 2026 unless they have teeth, and intentional exceptions need a first-class mechanism that doesn't break the audit chain. Policy-driven blocking: - New Action input fail-on-findings (default false). When true, the workflow exits 1 if there are non-suppressed findings. - The receipt's calibrated severity is unchanged (warning). The receipt represents the methodology; the CI failure represents the user's policy. Severity belongs to the discipline; enforcement belongs to the team. - Suppressed findings do not trigger the failure. They are still recorded in the receipt and covered by the digest. In-band suppressions: - New comment syntax: # verify:ignore <SHAPE-ID> reason:"<text>" - Trailing-only scope: same line as trigger, or line immediately above. File-scoped and resource-scoped variants were rejected as abuse vectors. - Strict shape-ID matching. Comments with typo'd or unknown shape IDs are recorded as suppression warnings, not silently treated as no-ops. - Non-empty reason required. Empty-reason comments do not suppress. - Suppressed findings move out of the primary findings block into a new MANIFEST INTENT / SUPPRESSIONS block (position 5 in the receipt). - The packet digest covers the suppressions and warnings arrays. A suppression cannot be silently removed without changing the digest. Receipt schema (verify-receipt/v1) extended with two arrays: - suppressions: SuppressedFinding[] - suppression_warnings: SuppressionWarning[] Both are part of the digest. Empty arrays in the no-suppression case. Bundle rebuilt with --legal-comments=none --minify-whitespace. Size: 234.6 KB (was 227 KB at v1.2). Behaviour: same 7 detectors, same detection logic. Canonical demo digest with the v1.3 schema: sha256:2040419351bc062a508ecac06f24d5c458db3e365d668168ae1967a6984f284d (replaces the v1.2 digest sha256:bceed6d3...). METHODOLOGY.md adds two sections: 'Methodology vs. policy' explaining the severity-vs-enforcement separation, and 'Suppressions and intent' documenting the comment syntax and the digest-covered audit chain. The semantics:intent entry in the Not Checked block now reads: 'Verify does not infer author intent. It only records intent that operators declare explicitly via in-band suppression comments.'
Checks with no findings
Not checked Terraform, CloudFormation, Helm-templated YAML, Kustomize overlays, runtime cloud state, business logic, intent, recall, uncalibrated shapes. Receipt artifacts: |
Eighth calibrated check shipping in the v1.3 receipt path: TF-SG-WORLD-OPEN-INGRESS-01 (AWS security-group ingress opening sensitive ports to the world). Calibrated/warning under the v1.1 Terraform substrate (same-module var/local resolver), promoted via strong-single-corpus on the devops-in-the-cloud corpus. Discipline reminder: this is one calibrated Terraform shape, narrow but real, warning-only, single-corpus. The receipt's Not-Checked block names the rest of the Terraform surface as still uncovered. Not "Terraform solved", not blocking-tier, not two-corpus. Bundle: - dist/action/index.cjs rebuilt from the engine repo's src/action-v2/index-receipt.ts entry. Bundle bytes-identical to the engine-side bundle. Contains TF-SG-WORLD-OPEN-INGRESS-01 and the narrowed Not-Checked Terraform line; the stale "Terraform .tf files are not parsed" string is removed. - dist/action/calibration/shapes.json + attempts.jsonl regenerated from the engine ledger; eight shipped shapes / eleven ratified attempts (was seven / ten). Surface metadata: - README.md table adds the TF-SG row; description and "What is not checked" block updated to reflect the narrow Terraform coverage. - action.yml description updated to mention the Terraform shape. Sample documentation: - docs/VERIFY-RECEIPT-SAMPLE.md preserved verbatim as the v1 frozen sample (its sha256:bceed6d3... digest is an audit anchor); header relabeled "(historical, v1 schema; frozen)". - docs/VERIFY-RECEIPT-SAMPLE-v1.3.md (new) is the current-state sample for the v1.3 schema with eight calibrated checks and the TF-SG row. Canonical demo digest: sha256:0d7d9d941a672e37f2a31e0d8db8b20ce7ca416d57cf989c89deac7ca958b0e4. - docs/GITHUB-ACTION-MVP.md surface table, calibrated table, and Not-Checked block updated; the embedded v1 sample PR comment block is wrapped in a historical/frozen header. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
v1.3 adds three coupled features in response to engineering critique on v1.0-1.2 plus the first calibrated Terraform shape:
fail-on-findingsAction input (default false). Opt-in CI failure on non-suppressed findings. The receipt's calibrated severity stays at warning; the failure is the team's policy, not the methodology's claim.# verify:ignore <SHAPE-ID> reason:"<text>"). Trailing-only scope. Strict shape-ID matching (typos surface as warnings, not silent no-ops). Suppressions go in their own receipt block and are part of the packet digest.TF-SG-WORLD-OPEN-INGRESS-01): AWS security-group ingress opening sensitive ports to the world. Calibrated/warning under the v1.1 same-module Terraform resolver, promoted via strong-single-corpus on the devops-in-the-cloud corpus (32 TP / 0 FP / 16 AMB / 100% precision / 33.33% ambiguity).Discipline reminder on the Terraform shape
Narrow but real: one shape, warning-only, single-corpus. The receipt's Not-Checked block names the rest of the Terraform surface as still uncovered. Not "Terraform solved", not blocking-tier, not two-corpus. Future Terraform shapes ship when each one earns its own ledger row.
Receipt schema (verify-receipt/v1) extended
Two new arrays, both part of the digest:
suppressions: SuppressedFinding[]suppression_warnings: SuppressionWarning[]Empty arrays when no suppressions are declared.
Surfaceextended to includeterraform.CalibratedIaCShapeIdextended to includeTF-SG-WORLD-OPEN-INGRESS-01.New canonical demo digest
sha256:0d7d9d941a672e37f2a31e0d8db8b20ce7ca416d57cf989c89deac7ca958b0e4(Was
sha256:bceed6d3...at v1.2; superseded the pre-Terraform v1.3 digestsha256:2040419...once TF-SG landed.)Behaviour preserved on prior shapes
Born14/verify@v1install line unchanged.What changed at the runtime
src/action-v2/index-receipt.ts(the receipt entry, not the legacy migration entry).*.tffiles now classify as the newterraformsurface.extractTerraformSurface->resolveSurface->runTfSgWorldOpenIngress01->classifyFindings. Only TP findings appear as receipt findings; AMB findings are dropped (the substrate's resolution boundaries are named in the Not-Checked block).dist/action/calibration/{shapes.json,attempts.jsonl}) now contain 8 shapes / 11 ratified attempts.Test plan
dist/action/index.cjson GitHub containsTF-SG-WORLD-OPEN-INGRESS-01.dist/action/index.cjson GitHub does not contain the staleTerraform .tf files are not parsed.# verify:ignorecomment routes the finding to suppressions and reduces non-suppressed count.Decisions locked in this release
fail-on-findings: trueand suppressions: counts non-suppressed findings only.resolved_fromprovenance in evidence when same-module resolution backed the literal.Not in this release (banked for later)
Tag move
The
v1tag is not moved by this PR. After merge, tag move requires explicit operator confirmation.🤖 Generated with Claude Code