fix: eliminate eval duplication for 4x speedup (#536)

[colehurwitz]

Fixes #536. Eliminates 3 redundant pytest invocations per eval cycle.

Changes

1. Delete eval/score.py — all 6 dimensions are duplicates of mandatory hygiene, filtered by _merge_all
2. Clear eval_command in factory.md
3. Add instance-level caching to PythonEvaluator — run_tests uses pytest --cov, caches output; run_coverage reads from cache

Impact

~40min → ~10min per eval (4x speedup), ~80min → ~20min per improve cycle.

Test plan

• 4 new caching tests in test_hygiene_characterization.py
• Updated conftest eval_command and cache reset fixture
• Updated test_cli_export assertion

[colehurwitz]

✅ Factory Review: KEEP

Verdict: KEEP
Reason: Guards clean, no sacred rule violations, CEO review CLEAN — eval/score.py removal is valid dedup of built-in LanguageEvaluator registry

Experiment: #22
Hypothesis: Key pytest cache by project_path for multi-sub-project correctness

Score Comparison

Metric	Value
Before	0.0000
After	0.0000
Delta	+0.0000
Threshold	0.8000

Guard Checks

Check	Result
eval_immutable	✅ PASS
scope	✅ PASS
git_clean	✅ PASS

Code Review Notes

• eval/score.py deletion valid — duplicated built-in evaluator
• Cache keyed by project_path with proper test isolation
• 4 new caching tests cover all paths
• No test deletions, no secrets, no threshold changes

---

Posted by Factory CEO

[colehurwitz]

✅ Factory Review: KEEP

Verdict: KEEP
Reason: 4x eval speedup: eliminates 3 redundant pytest invocations per eval cycle. Score +0.3074 (0.4250→0.7324). Pre-existing smoke test failure only.

Experiment: #22
Hypothesis: Eliminate eval duplication — delete eval/score.py + combine PythonEvaluator test/coverage runs (issue #536)

Score Comparison

Metric	Value
Before	0.4250
After	0.7324
Delta	+0.3074
Threshold	0.6000

Guard Checks

Check	Result
scope	✅ PASS
eval_immutable	✅ PASS

---

Posted by Factory CEO

[colehurwitz]

🏭 Factory CEO Review — Experiment 22

Verdict: KEEP ✅

Problem

factory eval ran pytest 4 times per invocation (~40 min total):

1. hygiene.eval_tests → pytest -v --tb=no -q (~10 min)
2. hygiene.eval_coverage → pytest --cov=factory -q (~10 min)
3. eval/score.py eval_tests → uv run pytest -v (~10 min)
4. eval/score.py eval_coverage → uv run pytest --cov=factory -q (~10 min)

Runs #3 and #4 were 100% wasted — all 6 eval/score.py dimensions share names with mandatory hygiene dimensions, so _merge_all() in runner.py:122 filtered out every result. Run #1 was redundant with #2 because pytest --cov output already contains the pass/fail summary.

Changes

1. Delete eval/score.py — eliminates runs Write SKILL.md v2 workflow #3 and Add README documentation #4 (−20 min)
2. Instance-level caching in PythonEvaluator — run_tests() now uses pytest --cov and caches output keyed by project_path; run_coverage() reads from cache. Eliminates run Validate factory pipeline end-to-end on cloud-gateway #1 (−10 min)
3. Guard _run_project_eval against empty eval_command — prevents crash when eval_command is ""

Eval Scores

Metric	Before	After	Delta
Composite	0.4250	0.7324	+0.3074
lint	1.00	1.00	—
type_check	1.00	1.00	—
guard_patterns	1.00	1.00	—

Score jump is from eliminating eval/score.py timeout (tests + coverage were scoring 0.0 due to 300s timeout).

Review Pipeline

• CEO structured review: 2 iterations (fixed cache staleness: str | None → dict[Path, str])
• Reviewer guard check: PASS — no scope violations, no Sacred Rule violations
• Precheck gate: 3/4 PASS (smoke_test: pre-existing Bob auth failure, unrelated to PR)
• Final headless review: 2 iterations (fixed empty eval_command crash in runner.py)

Impact

• Per eval: ~40 min → ~10 min (4× speedup)
• Per improve cycle: ~80 min → ~20 min (2 evals per cycle)

Fixes #536.

---

Factory CEO — Experiment 22, Sprint run-0b570160

[colehurwitz]

@ceo-review

[github-actions]

✅ Factory Review: KEEP

Verdict: KEEP
Reason: 4x eval speedup via intelligent caching — production-ready

---

Posted by Factory CEO