feat: add meta mode Phase 3 — classify and contribute upstream#376
feat: add meta mode Phase 3 — classify and contribute upstream#376Maxusmusti wants to merge 3 commits into
Conversation
Add the ability for meta mode to distinguish general improvements from project-specific ones, and contribute the general items back upstream as PRs. This closes the self-improvement loop: the more the factory is used across diverse projects, the better its default playbooks become. New CLI command `factory contribute` with three modes: - `--classify`: scores evolved playbook items on a general-vs-specific spectrum using four weighted signals (cross-project prevalence 40%, domain independence 25%, evidence strength 20%, category signal 15%) - `--submit`: creates a PR against the factory repo with approved items - `--status`: shows pending contribution candidates The classification engine uses cross-project experiment data to identify items that appear across 3+ unrelated projects as "general" (upstream candidates), single-project items as "specific" (local only), and everything in between as "uncertain" (needs more data). The CEO prompt is updated with Phase 3 (M4/M5/M6) which runs after ACE evolution, presents the user with a terminal summary showing the distinction, and lets them opt in to contributing. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #376 +/- ##
==========================================
+ Coverage 87.54% 87.56% +0.02%
==========================================
Files 60 62 +2
Lines 9170 9734 +564
==========================================
+ Hits 8028 8524 +496
- Misses 1142 1210 +68 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add 26 tests covering uncovered paths: classify_evolved_playbooks pipeline, package_evidence, prepare_contribution, execute_contribution (mocked subprocess), explain_specificity/uncertainty branches, load_candidates edge cases, and cmd_contribute CLI handler. Fix lint: remove unused imports, rename ambiguous variable, drop unused locals. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
akashgit
added
the
enhancement
|
I tried running, but unfortunately the meta mode didn't have enough to update the placebook. So the classify and contribute upstream didn't find anything meaningful from that small pool to contribute upstream @akashgit could you give these two command a try? |
|
@ceo-review |
✅ Factory CEO Review: KEEPVerdict: KEEP Code Quality Review
Key Strengths
Test Results
RecommendationThis PR is production-ready. The classification methodology is empirically grounded, the implementation is robust, and the user experience (opt-in contribution, clear terminal summary) is excellent. Code follows factory conventions and is highly maintainable. Review completed by Factory CEO at 2026-06-03T20:56:09Z |
|
❌ CEO review failed. Check the workflow run for details. |
|
Triage: parking this one deliberately. Before the factory grows an upstream contribution pipeline (edit: earlier wording said "auto-submit", which was wrong; submission is user-approved, see discussion below), we want the spec-driven workflow in place first — every change implemented against an explicit spec with a problem statement — otherwise this multiplies the current intent-opacity problem upstream. @akashgit deferring to you if you want to prioritize it differently. |
|
@osilkin98 I'm not sure I follow, this PR does not "give factory the ability to auto-submit PRs". It specifically is adding a step to meta mode runs exclusively, which classifies potential general learnings across projects vs project specific learnings. It gives users the option to review and contribute those learnings back to factory upstream. It just reviews and provides information over long-term factory use, it doesn't submit or act automatically. |
osilkin98
added
kind:capability
|
You're right, I overstated it. I went back and read the diff properly: M6 only runs The narrower thing I'd still raise: the no-auto-submit rule lives in the prompt, and |
5 participants
Summary
factory contributeCLI command with--classify,--submit, and--statussubcommandsMotivation
Currently, meta mode evolves playbooks locally via ACE — all learnings stay in
~/.factory/playbooks/and never flow back upstream. This means every user independently re-discovers the same improvements. With this change, the factory can identify patterns that are universally useful across diverse projects and contribute them back to the default playbooks, closing the self-improvement loop: the more the factory is used, the better it becomes for everyone.How it works
Classification engine (
factory/ace/contributor.py)Each evolved playbook item is scored on a general-vs-specific spectrum using four weighted signals:
Items scoring ≥ 0.65 are classified as general, ≤ 0.35 as specific, and between as uncertain.
User experience
At the end of a meta mode run, users see a terminal summary:
Users can then run
factory contribute --submitto create a PR, or skip — contribution is always opt-in.CLI commands
CEO prompt changes
Phase 3 (steps M4/M5/M6) is added after Phase 2 (ACE). The CEO:
factory contribute --classifyto score evolved itemsFiles changed
factory/ace/contributor.pyfactory/cli.pyfactory contributecommand with--classify/--submit/--statussubcommandsfactory/agents/prompts/ceo.mdtests/test_contributor.pyDesign decisions
ClassifiedItemwrappingPlaybookItem(sincePlaybookItemhasextra="forbid")Playbook.from_markdown(),classify_hypothesis(),discover_projects(),load_all_histories(),DEFAULTS_DIR,user_playbooks_dir()prepare_contribution()returns specs without executing git — keeps the module testable;execute_contribution()handles the actual git/gh commands separatelyTest plan
pytest tests/test_contributor.py)factory contribute --helpshows correct usagefactory contribute --classifyon a project with evolved playbooksfactory contribute --submit --dry-runto verify PR spec generation🤖 Generated with Claude Code