Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -2,3 +2,10 @@
node_modules/
dist/
benchmarks/.cache/
autoresearch.md
autoresearch.sh
autoresearch.checks.sh
autoresearch.ideas.md
benchmarks/results/autoresearch-candidate-rule.json
reports/autoresearch-candidate-rule.md
scripts/benchmark-experimental-rule.ts
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -134,6 +134,7 @@ Current checks focus on patterns that often show up in unreviewed generated code
- [log-and-continue catch blocks](src/rules/error-swallowing/README.md)
- [error-obscuring catch blocks](src/rules/error-obscuring/README.md) (default-return or generic replacement error)
- [empty catch blocks](src/rules/empty-catch/README.md)
- [promise `.catch()` default fallbacks](src/rules/promise-default-fallbacks/README.md)
- [async wrapper / `return await` noise](src/rules/async-noise/README.md)
- [pass-through wrappers](src/rules/pass-through-wrappers/README.md)
- [barrel density](src/rules/barrel-density/README.md)
Expand Down
23 changes: 23 additions & 0 deletions benchmarks/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,22 @@ If a repo did not exist yet for an older backfill date, that weekly point is ski

The rolling history is intentionally separate from the pinned benchmark snapshot so reproducible benchmark claims still point at exact SHAs.

## Per-rule signal benchmark

A separate pinned mini cohort runs each built-in rule **in isolation** so we can compare which rules separate the explicit-AI and mature-OSS cohorts most cleanly.

Refresh it locally with:

```bash
bun run benchmark:rules
```

That writes:

- an aggregate JSON summary at `benchmarks/results/rule-signal-mini.json`
- a markdown leaderboard/report at `reports/rule-signal-mini.md`
- benchmark summary sections into each `src/rules/*/README.md`

## Artifacts

For the current pinned set:
Expand All @@ -93,6 +109,13 @@ For rolling history:
- latest summary: `benchmarks/history/known-ai-vs-solid-oss/latest.json`
- generated history report: `reports/known-ai-vs-solid-oss-history.md`

For per-rule signal benchmarking:

- manifest: `benchmarks/sets/rule-signal-mini.json`
- summary: `benchmarks/results/rule-signal-mini.json`
- generated report: `reports/rule-signal-mini.md`
- per-rule docs: `src/rules/*/README.md`

## Notes

- Checkouts are stored under `benchmarks/.cache/` and are gitignored.
Expand Down
Loading
Loading