Skip to content

Harden review lineage stale reporting#688

Merged
sungjunlee merged 2 commits into
mainfrom
issue-682-review-lineage
Jun 8, 2026
Merged

Harden review lineage stale reporting#688
sungjunlee merged 2 commits into
mainfrom
issue-682-review-lineage

Conversation

@sungjunlee

Copy link
Copy Markdown
Owner

Summary

  • Add stale to the relay-review verdict lineage contract, validator, schema, reviewer prompt guidance, and lineage docs.
  • Surface lineage counts in redispatch prompts, prior-round summaries, manifest review state, and review_apply/escalation event payloads.
  • Add deterministic same-HEAD stale candidate guidance without script-only semantic stale inference, and aggregate review_lineage counts in reliability-report JSON by run and round.

Verification

  • Commit reviewed: 9d0d31b
  • node --test tests/relay-review/scripts/review-runner.test.js tests/relay-review/scripts/review-runner-redispatch.test.js tests/relay-review/scripts/review-runner-verdict.test.js tests/relay-review/scripts/review-schema.test.js tests/relay-dispatch/scripts/reliability-report.test.js
    • Passed: 225 tests, 0 failures
  • git diff --check
    • Passed

Fixes #682
Related #678

@coderabbitai

coderabbitai Bot commented Jun 8, 2026

Copy link
Copy Markdown

Warning

Review limit reached

@sungjunlee, we couldn't start this review because you've reached your PR review rate limit.

More reviews will be available in 2 minutes and 34 seconds. Learn how PR review limits work.

Your organization has run out of usage credits. Purchase more in the billing tab.

⌛ How to resolve this issue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans include higher PR review limits than trial, open-source, and free plans. In all cases, reviews become available again over time. During sustained high-volume PR review activity, CodeRabbit may temporarily slow when the next review becomes available.

Please see our Fair Usage Limits Policy for further information.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 5c524d25-6e29-49d3-bbfd-510a3b95e08c

📥 Commits

Reviewing files that changed from the base of the PR and between c50f743 and 66d4683.

📒 Files selected for processing (15)
  • skills/relay-dispatch/scripts/reliability-report.js
  • skills/relay-review/references/review-lineage.md
  • skills/relay-review/references/reviewer-prompt.md
  • skills/relay-review/scripts/review-runner.js
  • skills/relay-review/scripts/review-runner/manifest-apply.js
  • skills/relay-review/scripts/review-runner/prompt.js
  • skills/relay-review/scripts/review-runner/redispatch.js
  • skills/relay-review/scripts/review-runner/verdict.js
  • skills/relay-review/scripts/review-schema.js
  • tests/relay-dispatch/scripts/reliability-report.test.js
  • tests/relay-review/scripts/review-runner-prompt.test.js
  • tests/relay-review/scripts/review-runner-redispatch.test.js
  • tests/relay-review/scripts/review-runner-verdict.test.js
  • tests/relay-review/scripts/review-runner.test.js
  • tests/relay-review/scripts/review-schema.test.js
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch issue-682-review-lineage

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9d0d31be9c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Use `lineage: "new"` when this is a first-time finding with no prior-round ancestor; omit `relates_to` unless a specific prior note helps.
Use `lineage: "deepening"` when the prior issue was valid but the current finding exposes a narrower or deeper edge case; set `relates_to` to the prior issue title or stable id when known.
Use `lineage: "repeat"` when the same issue still blocks the PR; set `relates_to` to the prior issue title or stable id when known.
Use `lineage: "stale"` when deterministic prior-round metadata shows the same reviewed HEAD or stale artifact is being evaluated again; set `relates_to` to the stale round, artifact, prior issue title, or stable id when known.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Provide stale metadata before requiring stale labels

In automated review rounds, this instruction asks the reviewer to use lineage: "stale" only when deterministic prior-round metadata proves the same reviewed HEAD or a stale artifact, but the primary review prompt built by buildPrompt/formatPriorRoundContext only includes prior verdict text and not the current/prior head_sha or artifact identity; the new same-HEAD section is added later to the re-dispatch prompt after the reviewer has already emitted the verdict. As a result, normal reviewers have no allowed evidence to select stale, so the new stale lineage accounting and fail-closed behavior will be unreachable for the same-HEAD scenario unless a manual review file supplies it.

Useful? React with 👍 / 👎.

@sungjunlee

Copy link
Copy Markdown
Owner Author

Relay Review Round 1

Verdict: CHANGES_REQUESTED
Summary: Diff overview: 변경은 lineage enum/docs/prompt/validator, redispatch stale same-HEAD metadata, manifest/event lineage summary, reliability-report aggregation, 관련 테스트에 집중되어 있어 범위 자체는 맞습니다. 다만 변경된 formatPriorVerdictSummary 출력과 기존 direct prompt unit test 기대값이 불일치하여 기존 review-runner 계열 테스트가 깨지는 상태입니다.
Contract: FAIL
Quality Review: NOT_RUN
Quality Execution: PASS
Issues:

  • tests/relay-review/scripts/review-runner-prompt.test.js:226 — 기존 prompt unit test가 새 lineage 출력과 불일치합니다: skills/relay-review/scripts/review-runner/prompt.js:183은 prior verdict summary에 lineage counts를 항상 포함하도록 바뀌었지만, tests/relay-review/scripts/review-runner-prompt.test.js:226의 기존 assert는 여전히 lineage 없는 문자열을 기대합니다. 이 테스트는 formatPriorVerdictSummary의 직접 caller/import 검증이며 현재 변경과 함께 실패합니다. 새 의도가 lineage summary를 포함하는 것이라면 이 테스트 기대값과 테스트명을 함께 갱신해야 합니다.

@sungjunlee

Copy link
Copy Markdown
Owner Author

Relay Review

Verdict: LGTM
Summary: Diff overview: 변경은 lineage enum/docs/prompt/validator, redispatch same-HEAD stale candidate metadata, manifest/event lineage summaries, reliability-report additive aggregation, 관련 회귀 테스트에 집중되어 있습니다. 변경 함수의 caller/import를 확인했으며 Round 1의 prompt test 기대값 불일치는 갱신되어 더 이상 정적 통합 문제가 보이지 않습니다. Done Criteria는 diff 기준으로 모두 VERIFIED입니다.
Contract: PASS
Quality Review: PASS
Quality Execution: PASS
Rounds: 2

@sungjunlee sungjunlee merged commit 3eb3dd6 into main Jun 8, 2026
2 checks passed
@sungjunlee sungjunlee deleted the issue-682-review-lineage branch June 8, 2026 12:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

relay-review: distinguish repeated/stale findings from meaningful deepening

1 participant