From c27dbfaa52e97edb3411a5c8ce2e5be805eb6014 Mon Sep 17 00:00:00 2001 From: Rafael Richards Date: Sun, 10 May 2026 22:50:26 -0400 Subject: [PATCH 1/5] docs(plans): add Phase 3 implementation plan (3b + 3c) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Convert §6.2 of language-cli-survey.md into an actionable punch list for the remaining Phase 3 work — m fix, m profile, m bench, and m debug — with explicit dependencies, priority ordering, exit criteria, and per-subcommand risks. Phase 3a is shipped, so the plan picks up at 3b. Add a pointer from language-cli-survey.md so the plan is discoverable from the source survey. Co-Authored-By: Claude Opus 4.7 (1M context) --- docs/plans/language-cli-survey.md | 4 + docs/plans/phase-3-plan.md | 267 ++++++++++++++++++++++++++++++ 2 files changed, 271 insertions(+) create mode 100644 docs/plans/phase-3-plan.md diff --git a/docs/plans/language-cli-survey.md b/docs/plans/language-cli-survey.md index e77a4f4..afadc27 100644 --- a/docs/plans/language-cli-survey.md +++ b/docs/plans/language-cli-survey.md @@ -6,6 +6,10 @@ > software development life cycle (SDLC). Closes with a rank-ordered gap analysis > for `m-cli` — the canonical CLI for the M (MUMPS) language. > +> **Implementation:** the actionable punch list for Phase 3b/3c (with +> dependencies, priorities, and exit criteria) lives in +> [`phase-3-plan.md`](phase-3-plan.md). Phase 3a is shipped. +> > **Scope.** Five languages: **Rust, Go, Python, JavaScript/TypeScript, Java.** > Chosen for top consistent rankings (TIOBE / Stack Overflow / GitHub Octoverse > 2024–2026) and for spanning the design space — single-binary unified CLIs diff --git a/docs/plans/phase-3-plan.md b/docs/plans/phase-3-plan.md new file mode 100644 index 0000000..7d4b0ca --- /dev/null +++ b/docs/plans/phase-3-plan.md @@ -0,0 +1,267 @@ +# Phase 3 Implementation Plan — `m fix`, `m profile`, `m bench`, `m debug` + +**Status:** working plan. Phase 3a is shipped; this document scopes Phase 3b and Phase 3c with explicit dependencies and priorities. + +**Derived from:** [`language-cli-survey.md`](language-cli-survey.md) §6.2 (phased roadmap) and §4.1 (capability ranks). The phase labels and shape come from there — this doc converts the prose into an actionable punch list. + +**Audience:** m-cli contributors picking up the next subcommand; reviewers tracking what's queued. + +**Scope:** + +- Phase 3b: `m fix`, `m profile`, `m bench`. +- Phase 3c: `m debug` (DAP server + VS Code wiring). +- Cross-cutting: VistA gate, library API surface, performance budgets. + +**Not in scope:** Phase 4 (`m pkg` / `m audit` / `m publish` — ecosystem work, year-2+) and Phase 5 polish (`m typecheck`, `m toolchain`, `$PATH`-discovered subcommands). The IRIS portability track in [`iris-ydb-portability.md`](iris-ydb-portability.md) runs in parallel; this plan only notes touchpoints, not deliverables. + +--- + +## 0. Current state — Phase 3a shipped + +| Subcommand | Source | Status | +|---|---|---| +| `m doctor` | [`src/m_cli/doctor/cli.py`](../../src/m_cli/doctor/cli.py) | Shipped | +| `m new` | [`src/m_cli/new/cli.py`](../../src/m_cli/new/cli.py) | Shipped | +| `m ci init` | [`src/m_cli/ci/cli.py`](../../src/m_cli/ci/cli.py) | Shipped | +| `m run` | [`src/m_cli/run/cli.py`](../../src/m_cli/run/cli.py) | Shipped | +| `m build` | [`src/m_cli/build/cli.py`](../../src/m_cli/build/cli.py) | Shipped | +| `m doc` | [`src/m_cli/doc/cli.py`](../../src/m_cli/doc/cli.py) | Shipped | + +Phase 3a exit criterion is met: `m new` produces a project that passes `m fmt --check && m lint && m test && m coverage` on a clean clone. Everything below builds on the Phase 3a / Tier 1+2 foundation; nothing in this plan revisits it. + +--- + +## 1. Dependency graph + +``` + Tier 1+2 foundation (parser · fmt rule engine · lint runner · + WorkspaceIndex · YDB trace · test discovery) + │ + ┌─────────────────────────┼─────────────────────────┐ + │ │ │ + ▼ ▼ ▼ + ┌──────────┐ ┌──────────┐ ┌──────────┐ + │ m fix │ │ m profile│ │ m bench │ + │ (3b-1) │ │ (3b-2) │ │ (3b-3) │ + └────┬─────┘ └────┬─────┘ └──────────┘ + │ │ + │ shares fixer_id │ shares TRACE view + │ contract with LSP │ with `m coverage` + ▼ ▼ + (Quick Fix code-actions (flamegraph output · + expand to ≥10 rules) folded-stack format) + + ┌────────────────────────┐ + │ engine ZBREAK / ZSTEP │ + │ ZSHOW (existing) │ + └──────────┬─────────────┘ + ▼ + ┌──────────┐ ┌──────────────────┐ + │ m debug │ ────▶ │ VS Code DAP wiring│ + │ (3c-1) │ │ (3c-2) │ + └──────────┘ └──────────────────┘ +``` + +**Critical-path observations:** + +1. `m fix` is the highest-leverage 3b item — every `fixer_id`-tagged lint rule becomes an LSP Quick Fix the day it ships, and future lint rules can register a fixer alongside the rule. +2. `m profile` and `m bench` are independent of `m fix` and of each other. They can ship in parallel. +3. `m debug` depends on no other Phase 3 work. It's gated by DAP-server scope (large, ~4–6 weeks) and can run in parallel with all of 3b. +4. Nothing in 3b/3c blocks Phase 4. The ecosystem work is independent and deliberately a year-2+ commitment. + +--- + +## 2. Priorities — what to ship first + +Ranked by leverage (developer time saved per engineering week invested) and by how much existing infrastructure each item reuses. + +| Rank | Item | Why first | Effort | +|:---:|---|---|:---:| +| 1 | `m fix` | Promotes every existing `fixer_id`-tagged lint rule into an auto-fix; LSP Quick Fix coverage jumps from 2 → 10+ on day one. Reuses `m fmt` rule engine wholesale. | M (1–2 wk) | +| 2 | `m profile` | Closes the biggest "I can't tell where my routine is slow" gap. Reuses `view "TRACE"` machinery from `m coverage`; folded-stack output is one new formatter. | M (1–2 wk) | +| 3 | `m bench` | Smallest of the three; immediately useful for m-cli's own perf-budget discipline. Reuses test discovery 1:1 (just a different label prefix and a `.m` file-name suffix). | S (3–5 d) | +| 4 | `m debug` (DAP server) | Largest single developer-felt missing feature, but largest scope. Parallelizable, not blocking. | L (4–6 wk) | +| 5 | VS Code DAP wiring | Trivial once `m debug` exists; lands in `tree-sitter-m-vscode`. | S (3–5 d) | + +**Recommended sequencing.** Start `m fix` first — it has the steepest payoff curve and the smallest blast radius. Begin `m debug` in parallel if a second contributor is available; otherwise queue 3c after 3b. `m profile` and `m bench` can interleave with `m fix` reviews without contention. + +--- + +## 3. Phase 3b — Generalize existing infrastructure + +### 3b-1 · `m fix` — auto-fix recipes + structural search/replace + +**Builds on:** `m fmt` rule engine ([`src/m_cli/fmt/`](../../src/m_cli/fmt/)), lint `fixer_id` linkage ([`src/m_cli/lint/__init__.py`](../../src/m_cli/lint/__init__.py) `fixer_for`). + +**Deliverables:** + +- New subcommand package `src/m_cli/fix/` with `cli.py` + `recipes.py` + `engine.py`. +- Named recipes that wrap a single fmt rule (`m fix trim-trailing-whitespace`). +- Structural search/replace driven by tree-sitter-m queries (`m fix --query 'SET $x=$x+1' --replace 'SET $x=$x+1 ; iterate'`). +- Every lint rule with a `fixer_id` becomes auto-fixable via `m fix --rule M-XINDX-013`. +- LSP `code_actions_for_uri` updated to expose all `fixer_id`-tagged rules (currently exposes 2, target ≥10). +- Idempotency tests: running `m fix` twice on the same source produces no second diff. + +**Exit criterion:** + +- All currently-registered `fixer_id` mappings are reachable via `m fix --rule `. +- `make check-fixer-linkage` (new gate) asserts every lint rule with a `fixer_id` resolves to a real fmt rule. +- LSP Quick Fix code-action count ≥10 on a fixture file that triggers all auto-fixable rules. +- `make lint-vista` over the configured corpus shows zero regressions in finding counts (auto-fixes change source, not finding semantics). + +**Risks:** + +- Tree-sitter structural-replace UX is unproven in M; start with named recipes only, defer the `--query` mode if it complicates the first ship. +- Recipe-on-recipe ordering matters once we ship more than one rule that touches the same node type. Document the conflict resolution rule upfront — likely "left-to-right in CLI invocation, deterministic-order if discovered from `fixer_id` set." + +--- + +### 3b-2 · `m profile` — flat profile + folded-stack output + +**Builds on:** YDB `view "TRACE"` driver from [`src/m_cli/coverage/runner.py`](../../src/m_cli/coverage/runner.py). + +**Deliverables:** + +- New subcommand package `src/m_cli/profile/` with `cli.py` + `runner.py` + `output.py`. +- Flat profile: line/label execution counts × time per call, sorted by self-time. +- Folded-stack format compatible with [`flamegraph.pl`](https://github.com/brendangregg/FlameGraph). +- Optional `--svg` flag that shells to `flamegraph.pl` if available; otherwise prints folded stacks for the user to pipe. +- `--baseline FILE` for comparison runs (delta self-time per label). +- Test fixture: profile a real `*TST.m` suite from m-stdlib; assert the slowest label surfaces in the top-N. + +**Exit criterion:** + +- `m profile FILE.m::tLabel` produces a flat profile with non-zero self-times. +- `m profile --format=folded` output renders correctly in `flamegraph.pl` (manually verified once; CI checks format only, not rendering). +- `make check` includes a smoke test that profiles a known-slow fixture and asserts the hot label is at the top of the flat profile. + +**Risks:** + +- `view "TRACE"` resolution granularity is per-line, not per-call. Document the limitation; don't promise call-graph reconstruction we can't deliver. +- Time measurement uses `$ZH` (or `$ZHOROLOG`); confirm sub-millisecond resolution on the target YDB version. Fall back to "count-only profiling" if the timer is too coarse for short tests. + +--- + +### 3b-3 · `m bench` — micro-benchmark runner + +**Builds on:** `m test` discovery ([`src/m_cli/test/discovery.py`](../../src/m_cli/test/discovery.py)) and runner ([`src/m_cli/test/runner.py`](../../src/m_cli/test/runner.py)). + +**Deliverables:** + +- New subcommand package `src/m_cli/bench/` with `cli.py` + `discovery.py` + `runner.py` + `output.py`. +- Discovery: `b` labels in `*BCH.m` files (parser-aware, same pattern as test discovery). +- Timing via `$ZH`; report ops/sec and ns/op. +- `--baseline FILE` for comparison mode (delta ops/sec per label, with significance threshold). +- Output formats: text (default), JSON. +- Document the benchmark-writing convention in [`docs/guide.md`](../guide.md) (analogous to the test section). + +**Exit criterion:** + +- `m bench` discovers and runs at least one bench file in a real M project (m-stdlib `STDFMT` or `STDREGEX` are obvious candidates). +- Comparison mode flags a deliberate 2× regression in a fixture and surfaces it in the output. +- Zero new dependencies beyond what `m test` already pulls in. + +**Risks:** + +- Warmup / JIT-like effects are absent in M, but YDB block-cache state isn't. Document that benches should be self-contained or include explicit setup; don't try to auto-warm. + +--- + +## 4. Phase 3c — `m debug` (DAP server + editor wiring) + +### 3c-1 · `m debug` — DAP server + +**Builds on:** YDB engine primitives (`ZBREAK`, `ZSTEP`, `ZSHOW`) — already used elsewhere in the codebase via [`src/m_cli/engine.py`](../../src/m_cli/engine.py); no new engine work required. + +**Deliverables:** + +- New subcommand package `src/m_cli/debug/` exposing a [Debug Adapter Protocol](https://microsoft.github.io/debug-adapter-protocol/) server over stdio. +- Capabilities: breakpoints (line, conditional), step in/over/out, locals/watch (via `ZSHOW`), call stack, exception breakpoints. +- Driver wraps a `ydb -direct` subprocess; multiplexes DAP messages on stdin/stdout while controlling the YDB process via `ZBREAK` / `ZSTEP`. +- Engine targeting: YDB-first. The `--target-engine` flag accepts `iris` but exit-1's with "not supported yet" — the hook stays in for the IRIS portability track. +- Tests use a fake engine (same pattern as `m test`'s injectable `RunnerFn`) so unit tests don't need a live ydb. + +**Exit criterion:** + +- A DAP client (manually invoked, no editor) can set a breakpoint, run, hit it, inspect locals, and step through a routine. +- `m debug` survives a 10-minute soak with breakpoints set on a real m-stdlib test suite (no leaks, no orphaned ydb processes). + +**Risks:** + +- `ZBREAK` semantics across YDB versions: confirm the documented behavior on the pinned minimum YDB version, pin the test suite to that version in CI. +- Variable scoping: M's local variables don't have explicit scopes the way most DAP clients expect. The DAP `Scope` for "locals" maps to "all currently-defined locals via `ZSHOW \"V\"`"; document the mapping upfront. +- Multi-process: YDB can fork worker processes (`JOB`); we will not attempt multi-process debugging in v1. Document and exit-1 if encountered. + +--- + +### 3c-2 · VS Code DAP wiring + +**Builds on:** `m debug` server + existing [`tree-sitter-m-vscode`](https://github.com/m-dev-tools/tree-sitter-m-vscode) extension. + +**Deliverables:** + +- Register a debug adapter contribution in `tree-sitter-m-vscode/package.json`. +- Launch configuration template: `m debug ` or `m debug --test FILE.m::tLabel`. +- F5 launches a debug session; breakpoints set in the gutter round-trip to the DAP server. +- Documentation update in `tree-sitter-m-vscode/README.md`. + +**Exit criterion:** + +- Setting a gutter breakpoint, pressing F5 on a `*TST.m` file, hitting the breakpoint, inspecting locals, and stepping through a routine all work without manual `ydb` terminal interaction. + +--- + +## 5. Cross-cutting investments + +These thread through every phase and should be budgeted explicitly, not as line items. + +| Investment | When | Why | +|---|---|---| +| **VistA gate maintenance.** Re-run `make vista`, `make lint-vista`, smoke tests on every new subcommand that touches the parser or rule engines. | Every PR | The 39,330-routine corpus is the only honest stress test we have. Regressions caught here cost 1 hour; in prod they cost weeks. | +| **Library API stability.** Anything new that out-of-tree tools should consume goes into `m_cli.__all__` and is pinned by [`tests/test_library_api.py`](../../tests/test_library_api.py). | Phase 3b onward | LSP, pre-commit, future IDE integrations all consume the library API. Breaking it is a tax on every consumer. | +| **Performance budget.** Every new subcommand that walks the corpus declares a budget and verifies it. | Phase 3b onward | Lint already costs ~22 s on 16 cores. A `fix` / `profile` / `bench` run that scales linearly to corpus size needs the same `--jobs` discipline up front. | +| **`make check-manifest` drift gate.** Every new subcommand must register a `cli.py` so `make manifest` picks it up into `dist/commands.json`. | Phase 3b onward | The manifest is the tier-1 contract for downstream tooling discovery. | + +--- + +## 6. Decision points to revisit per phase + +| Decision | When | Default | Watch for | +|---|---|---|---| +| Continue extending Python codebase vs port hot paths to Rust | End of 3b | Stay in Python | Lint perf regressing past 60 s on VistA; test runner startup overhead becoming visible. | +| `m debug` on YDB only or design for engine-portable DAP | Start of 3c | YDB-first; abstraction comes later if Caché / IRIS users appear | Demand signal from non-YDB users; sponsor for Caché support. | +| Tree-sitter structural-replace UX | During 3b-1 | Ship named recipes only in v1; defer `--query` mode | Recipe coverage stays flat; users hand-rolling `sed` for cross-rule fixes. | +| Profile call-graph reconstruction | During 3b-2 | Flat profile + folded stacks only; no synthetic call graph | Repeated user requests for true call-graph view. | + +--- + +## 7. Suggested Gantt + +``` + W1 W2 W3 W4 W5 W6 W7 W8 W9 W10 W11 W12 … +m fix (3b-1) █████████████ +m profile (3b-2) █████████████ +m bench (3b-3) ████████ +m debug (3c-1) ░░░░░░░░░░░░░░░░░░░░░░░░░████████████████████ +VS Code DAP (3c-2) ████████ +VistA gate ◄────────── continuous ──────────────────────────► +``` + +- 3b lands inside ~8 weeks if sequential, ~5 if `m profile` and `m bench` overlap. +- 3c can start at W1 with a second contributor; otherwise picks up at W8. Either way, lands inside one quarter. +- The "two productivity-defining phases (3b + 3c) ship inside one quarter" target from the survey is achievable; the constraint is contributor-count, not technical scope. + +--- + +## 8. Exit criterion for the phase as a whole + +Phase 3 is complete when: + +1. `m fix`, `m profile`, `m bench`, `m debug` are all documented in [`guide.md`](../guide.md), shipped with tests, and reachable via `m --help`. +2. `make check-manifest` passes — every subcommand is in `dist/commands.json`. +3. LSP Quick Fix coverage is ≥10 rules. +4. A flamegraph generated by `m profile` is checked in as a worked example under [`examples/`](../../examples/). +5. A debug session in VS Code (gutter breakpoint → hit → step → inspect) is documented with screenshots in `tree-sitter-m-vscode/README.md`. +6. The `m-cli` capability surface in `dist/commands.json` matches §6 of [`language-cli-survey.md`](language-cli-survey.md) rows 1–12 (the Tier-1 essentials + Phase 3 deliverables). + +When all six are true, the next planning doc is Phase 4 (`m pkg` / `m audit` / `m publish`) — explicitly a year-2+ commitment per the survey, contingent on ecosystem demand signal. From 51775f521b9bfc1e458b8e50a5582f9be6bdf0b3 Mon Sep 17 00:00:00 2001 From: Rafael Richards Date: Sun, 10 May 2026 22:53:40 -0400 Subject: [PATCH 2/5] Revert "docs(plans): add Phase 3 implementation plan (3b + 3c)" MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit This reverts c27dbfa. The doc was scoped to m-cli's internal Phase 3 from language-cli-survey.md §6.2 (m fix / m profile / m bench / m debug), but "Phase 3" in this org refers to the org-level AI-discoverability Phase 3 (recipes + handshake test) tracked in .github/docs/. The plan now lands there instead. Co-Authored-By: Claude Opus 4.7 (1M context) --- docs/plans/phase-3-plan.md | 267 ------------------------------------- 1 file changed, 267 deletions(-) delete mode 100644 docs/plans/phase-3-plan.md diff --git a/docs/plans/phase-3-plan.md b/docs/plans/phase-3-plan.md deleted file mode 100644 index 7d4b0ca..0000000 --- a/docs/plans/phase-3-plan.md +++ /dev/null @@ -1,267 +0,0 @@ -# Phase 3 Implementation Plan — `m fix`, `m profile`, `m bench`, `m debug` - -**Status:** working plan. Phase 3a is shipped; this document scopes Phase 3b and Phase 3c with explicit dependencies and priorities. - -**Derived from:** [`language-cli-survey.md`](language-cli-survey.md) §6.2 (phased roadmap) and §4.1 (capability ranks). The phase labels and shape come from there — this doc converts the prose into an actionable punch list. - -**Audience:** m-cli contributors picking up the next subcommand; reviewers tracking what's queued. - -**Scope:** - -- Phase 3b: `m fix`, `m profile`, `m bench`. -- Phase 3c: `m debug` (DAP server + VS Code wiring). -- Cross-cutting: VistA gate, library API surface, performance budgets. - -**Not in scope:** Phase 4 (`m pkg` / `m audit` / `m publish` — ecosystem work, year-2+) and Phase 5 polish (`m typecheck`, `m toolchain`, `$PATH`-discovered subcommands). The IRIS portability track in [`iris-ydb-portability.md`](iris-ydb-portability.md) runs in parallel; this plan only notes touchpoints, not deliverables. - ---- - -## 0. Current state — Phase 3a shipped - -| Subcommand | Source | Status | -|---|---|---| -| `m doctor` | [`src/m_cli/doctor/cli.py`](../../src/m_cli/doctor/cli.py) | Shipped | -| `m new` | [`src/m_cli/new/cli.py`](../../src/m_cli/new/cli.py) | Shipped | -| `m ci init` | [`src/m_cli/ci/cli.py`](../../src/m_cli/ci/cli.py) | Shipped | -| `m run` | [`src/m_cli/run/cli.py`](../../src/m_cli/run/cli.py) | Shipped | -| `m build` | [`src/m_cli/build/cli.py`](../../src/m_cli/build/cli.py) | Shipped | -| `m doc` | [`src/m_cli/doc/cli.py`](../../src/m_cli/doc/cli.py) | Shipped | - -Phase 3a exit criterion is met: `m new` produces a project that passes `m fmt --check && m lint && m test && m coverage` on a clean clone. Everything below builds on the Phase 3a / Tier 1+2 foundation; nothing in this plan revisits it. - ---- - -## 1. Dependency graph - -``` - Tier 1+2 foundation (parser · fmt rule engine · lint runner · - WorkspaceIndex · YDB trace · test discovery) - │ - ┌─────────────────────────┼─────────────────────────┐ - │ │ │ - ▼ ▼ ▼ - ┌──────────┐ ┌──────────┐ ┌──────────┐ - │ m fix │ │ m profile│ │ m bench │ - │ (3b-1) │ │ (3b-2) │ │ (3b-3) │ - └────┬─────┘ └────┬─────┘ └──────────┘ - │ │ - │ shares fixer_id │ shares TRACE view - │ contract with LSP │ with `m coverage` - ▼ ▼ - (Quick Fix code-actions (flamegraph output · - expand to ≥10 rules) folded-stack format) - - ┌────────────────────────┐ - │ engine ZBREAK / ZSTEP │ - │ ZSHOW (existing) │ - └──────────┬─────────────┘ - ▼ - ┌──────────┐ ┌──────────────────┐ - │ m debug │ ────▶ │ VS Code DAP wiring│ - │ (3c-1) │ │ (3c-2) │ - └──────────┘ └──────────────────┘ -``` - -**Critical-path observations:** - -1. `m fix` is the highest-leverage 3b item — every `fixer_id`-tagged lint rule becomes an LSP Quick Fix the day it ships, and future lint rules can register a fixer alongside the rule. -2. `m profile` and `m bench` are independent of `m fix` and of each other. They can ship in parallel. -3. `m debug` depends on no other Phase 3 work. It's gated by DAP-server scope (large, ~4–6 weeks) and can run in parallel with all of 3b. -4. Nothing in 3b/3c blocks Phase 4. The ecosystem work is independent and deliberately a year-2+ commitment. - ---- - -## 2. Priorities — what to ship first - -Ranked by leverage (developer time saved per engineering week invested) and by how much existing infrastructure each item reuses. - -| Rank | Item | Why first | Effort | -|:---:|---|---|:---:| -| 1 | `m fix` | Promotes every existing `fixer_id`-tagged lint rule into an auto-fix; LSP Quick Fix coverage jumps from 2 → 10+ on day one. Reuses `m fmt` rule engine wholesale. | M (1–2 wk) | -| 2 | `m profile` | Closes the biggest "I can't tell where my routine is slow" gap. Reuses `view "TRACE"` machinery from `m coverage`; folded-stack output is one new formatter. | M (1–2 wk) | -| 3 | `m bench` | Smallest of the three; immediately useful for m-cli's own perf-budget discipline. Reuses test discovery 1:1 (just a different label prefix and a `.m` file-name suffix). | S (3–5 d) | -| 4 | `m debug` (DAP server) | Largest single developer-felt missing feature, but largest scope. Parallelizable, not blocking. | L (4–6 wk) | -| 5 | VS Code DAP wiring | Trivial once `m debug` exists; lands in `tree-sitter-m-vscode`. | S (3–5 d) | - -**Recommended sequencing.** Start `m fix` first — it has the steepest payoff curve and the smallest blast radius. Begin `m debug` in parallel if a second contributor is available; otherwise queue 3c after 3b. `m profile` and `m bench` can interleave with `m fix` reviews without contention. - ---- - -## 3. Phase 3b — Generalize existing infrastructure - -### 3b-1 · `m fix` — auto-fix recipes + structural search/replace - -**Builds on:** `m fmt` rule engine ([`src/m_cli/fmt/`](../../src/m_cli/fmt/)), lint `fixer_id` linkage ([`src/m_cli/lint/__init__.py`](../../src/m_cli/lint/__init__.py) `fixer_for`). - -**Deliverables:** - -- New subcommand package `src/m_cli/fix/` with `cli.py` + `recipes.py` + `engine.py`. -- Named recipes that wrap a single fmt rule (`m fix trim-trailing-whitespace`). -- Structural search/replace driven by tree-sitter-m queries (`m fix --query 'SET $x=$x+1' --replace 'SET $x=$x+1 ; iterate'`). -- Every lint rule with a `fixer_id` becomes auto-fixable via `m fix --rule M-XINDX-013`. -- LSP `code_actions_for_uri` updated to expose all `fixer_id`-tagged rules (currently exposes 2, target ≥10). -- Idempotency tests: running `m fix` twice on the same source produces no second diff. - -**Exit criterion:** - -- All currently-registered `fixer_id` mappings are reachable via `m fix --rule `. -- `make check-fixer-linkage` (new gate) asserts every lint rule with a `fixer_id` resolves to a real fmt rule. -- LSP Quick Fix code-action count ≥10 on a fixture file that triggers all auto-fixable rules. -- `make lint-vista` over the configured corpus shows zero regressions in finding counts (auto-fixes change source, not finding semantics). - -**Risks:** - -- Tree-sitter structural-replace UX is unproven in M; start with named recipes only, defer the `--query` mode if it complicates the first ship. -- Recipe-on-recipe ordering matters once we ship more than one rule that touches the same node type. Document the conflict resolution rule upfront — likely "left-to-right in CLI invocation, deterministic-order if discovered from `fixer_id` set." - ---- - -### 3b-2 · `m profile` — flat profile + folded-stack output - -**Builds on:** YDB `view "TRACE"` driver from [`src/m_cli/coverage/runner.py`](../../src/m_cli/coverage/runner.py). - -**Deliverables:** - -- New subcommand package `src/m_cli/profile/` with `cli.py` + `runner.py` + `output.py`. -- Flat profile: line/label execution counts × time per call, sorted by self-time. -- Folded-stack format compatible with [`flamegraph.pl`](https://github.com/brendangregg/FlameGraph). -- Optional `--svg` flag that shells to `flamegraph.pl` if available; otherwise prints folded stacks for the user to pipe. -- `--baseline FILE` for comparison runs (delta self-time per label). -- Test fixture: profile a real `*TST.m` suite from m-stdlib; assert the slowest label surfaces in the top-N. - -**Exit criterion:** - -- `m profile FILE.m::tLabel` produces a flat profile with non-zero self-times. -- `m profile --format=folded` output renders correctly in `flamegraph.pl` (manually verified once; CI checks format only, not rendering). -- `make check` includes a smoke test that profiles a known-slow fixture and asserts the hot label is at the top of the flat profile. - -**Risks:** - -- `view "TRACE"` resolution granularity is per-line, not per-call. Document the limitation; don't promise call-graph reconstruction we can't deliver. -- Time measurement uses `$ZH` (or `$ZHOROLOG`); confirm sub-millisecond resolution on the target YDB version. Fall back to "count-only profiling" if the timer is too coarse for short tests. - ---- - -### 3b-3 · `m bench` — micro-benchmark runner - -**Builds on:** `m test` discovery ([`src/m_cli/test/discovery.py`](../../src/m_cli/test/discovery.py)) and runner ([`src/m_cli/test/runner.py`](../../src/m_cli/test/runner.py)). - -**Deliverables:** - -- New subcommand package `src/m_cli/bench/` with `cli.py` + `discovery.py` + `runner.py` + `output.py`. -- Discovery: `b` labels in `*BCH.m` files (parser-aware, same pattern as test discovery). -- Timing via `$ZH`; report ops/sec and ns/op. -- `--baseline FILE` for comparison mode (delta ops/sec per label, with significance threshold). -- Output formats: text (default), JSON. -- Document the benchmark-writing convention in [`docs/guide.md`](../guide.md) (analogous to the test section). - -**Exit criterion:** - -- `m bench` discovers and runs at least one bench file in a real M project (m-stdlib `STDFMT` or `STDREGEX` are obvious candidates). -- Comparison mode flags a deliberate 2× regression in a fixture and surfaces it in the output. -- Zero new dependencies beyond what `m test` already pulls in. - -**Risks:** - -- Warmup / JIT-like effects are absent in M, but YDB block-cache state isn't. Document that benches should be self-contained or include explicit setup; don't try to auto-warm. - ---- - -## 4. Phase 3c — `m debug` (DAP server + editor wiring) - -### 3c-1 · `m debug` — DAP server - -**Builds on:** YDB engine primitives (`ZBREAK`, `ZSTEP`, `ZSHOW`) — already used elsewhere in the codebase via [`src/m_cli/engine.py`](../../src/m_cli/engine.py); no new engine work required. - -**Deliverables:** - -- New subcommand package `src/m_cli/debug/` exposing a [Debug Adapter Protocol](https://microsoft.github.io/debug-adapter-protocol/) server over stdio. -- Capabilities: breakpoints (line, conditional), step in/over/out, locals/watch (via `ZSHOW`), call stack, exception breakpoints. -- Driver wraps a `ydb -direct` subprocess; multiplexes DAP messages on stdin/stdout while controlling the YDB process via `ZBREAK` / `ZSTEP`. -- Engine targeting: YDB-first. The `--target-engine` flag accepts `iris` but exit-1's with "not supported yet" — the hook stays in for the IRIS portability track. -- Tests use a fake engine (same pattern as `m test`'s injectable `RunnerFn`) so unit tests don't need a live ydb. - -**Exit criterion:** - -- A DAP client (manually invoked, no editor) can set a breakpoint, run, hit it, inspect locals, and step through a routine. -- `m debug` survives a 10-minute soak with breakpoints set on a real m-stdlib test suite (no leaks, no orphaned ydb processes). - -**Risks:** - -- `ZBREAK` semantics across YDB versions: confirm the documented behavior on the pinned minimum YDB version, pin the test suite to that version in CI. -- Variable scoping: M's local variables don't have explicit scopes the way most DAP clients expect. The DAP `Scope` for "locals" maps to "all currently-defined locals via `ZSHOW \"V\"`"; document the mapping upfront. -- Multi-process: YDB can fork worker processes (`JOB`); we will not attempt multi-process debugging in v1. Document and exit-1 if encountered. - ---- - -### 3c-2 · VS Code DAP wiring - -**Builds on:** `m debug` server + existing [`tree-sitter-m-vscode`](https://github.com/m-dev-tools/tree-sitter-m-vscode) extension. - -**Deliverables:** - -- Register a debug adapter contribution in `tree-sitter-m-vscode/package.json`. -- Launch configuration template: `m debug ` or `m debug --test FILE.m::tLabel`. -- F5 launches a debug session; breakpoints set in the gutter round-trip to the DAP server. -- Documentation update in `tree-sitter-m-vscode/README.md`. - -**Exit criterion:** - -- Setting a gutter breakpoint, pressing F5 on a `*TST.m` file, hitting the breakpoint, inspecting locals, and stepping through a routine all work without manual `ydb` terminal interaction. - ---- - -## 5. Cross-cutting investments - -These thread through every phase and should be budgeted explicitly, not as line items. - -| Investment | When | Why | -|---|---|---| -| **VistA gate maintenance.** Re-run `make vista`, `make lint-vista`, smoke tests on every new subcommand that touches the parser or rule engines. | Every PR | The 39,330-routine corpus is the only honest stress test we have. Regressions caught here cost 1 hour; in prod they cost weeks. | -| **Library API stability.** Anything new that out-of-tree tools should consume goes into `m_cli.__all__` and is pinned by [`tests/test_library_api.py`](../../tests/test_library_api.py). | Phase 3b onward | LSP, pre-commit, future IDE integrations all consume the library API. Breaking it is a tax on every consumer. | -| **Performance budget.** Every new subcommand that walks the corpus declares a budget and verifies it. | Phase 3b onward | Lint already costs ~22 s on 16 cores. A `fix` / `profile` / `bench` run that scales linearly to corpus size needs the same `--jobs` discipline up front. | -| **`make check-manifest` drift gate.** Every new subcommand must register a `cli.py` so `make manifest` picks it up into `dist/commands.json`. | Phase 3b onward | The manifest is the tier-1 contract for downstream tooling discovery. | - ---- - -## 6. Decision points to revisit per phase - -| Decision | When | Default | Watch for | -|---|---|---|---| -| Continue extending Python codebase vs port hot paths to Rust | End of 3b | Stay in Python | Lint perf regressing past 60 s on VistA; test runner startup overhead becoming visible. | -| `m debug` on YDB only or design for engine-portable DAP | Start of 3c | YDB-first; abstraction comes later if Caché / IRIS users appear | Demand signal from non-YDB users; sponsor for Caché support. | -| Tree-sitter structural-replace UX | During 3b-1 | Ship named recipes only in v1; defer `--query` mode | Recipe coverage stays flat; users hand-rolling `sed` for cross-rule fixes. | -| Profile call-graph reconstruction | During 3b-2 | Flat profile + folded stacks only; no synthetic call graph | Repeated user requests for true call-graph view. | - ---- - -## 7. Suggested Gantt - -``` - W1 W2 W3 W4 W5 W6 W7 W8 W9 W10 W11 W12 … -m fix (3b-1) █████████████ -m profile (3b-2) █████████████ -m bench (3b-3) ████████ -m debug (3c-1) ░░░░░░░░░░░░░░░░░░░░░░░░░████████████████████ -VS Code DAP (3c-2) ████████ -VistA gate ◄────────── continuous ──────────────────────────► -``` - -- 3b lands inside ~8 weeks if sequential, ~5 if `m profile` and `m bench` overlap. -- 3c can start at W1 with a second contributor; otherwise picks up at W8. Either way, lands inside one quarter. -- The "two productivity-defining phases (3b + 3c) ship inside one quarter" target from the survey is achievable; the constraint is contributor-count, not technical scope. - ---- - -## 8. Exit criterion for the phase as a whole - -Phase 3 is complete when: - -1. `m fix`, `m profile`, `m bench`, `m debug` are all documented in [`guide.md`](../guide.md), shipped with tests, and reachable via `m --help`. -2. `make check-manifest` passes — every subcommand is in `dist/commands.json`. -3. LSP Quick Fix coverage is ≥10 rules. -4. A flamegraph generated by `m profile` is checked in as a worked example under [`examples/`](../../examples/). -5. A debug session in VS Code (gutter breakpoint → hit → step → inspect) is documented with screenshots in `tree-sitter-m-vscode/README.md`. -6. The `m-cli` capability surface in `dist/commands.json` matches §6 of [`language-cli-survey.md`](language-cli-survey.md) rows 1–12 (the Tier-1 essentials + Phase 3 deliverables). - -When all six are true, the next planning doc is Phase 4 (`m pkg` / `m audit` / `m publish`) — explicitly a year-2+ commitment per the survey, contingent on ecosystem demand signal. From f21d77e074f109f6d3c3756b693dbb82dbe8be90 Mon Sep 17 00:00:00 2001 From: Rafael Richards Date: Mon, 11 May 2026 10:14:57 -0400 Subject: [PATCH 3/5] docs: add index README and YAML frontmatter to every doc Add docs/README.md indexing every document in docs/ with a shared TYPE/CONNECTION vocabulary used consistently across all m-dev-tools repos. Add YAML frontmatter (created, last_modified, revisions, doc_type) to every existing doc. Existing frontmatter is merged, not replaced. Co-Authored-By: Claude Opus 4.7 (1M context) --- docs/README.md | 41 +++++++++++++++++++++ docs/evolution.md | 7 ++++ docs/guide.md | 7 ++++ docs/m-linting-user-guide.md | 7 ++++ docs/plans/iris-ydb-portability.md | 7 ++++ docs/plans/language-cli-survey.md | 7 ++++ docs/plans/linter-profiles-guide.md | 7 ++++ docs/plans/m-cli-history-and-evolution.md | 4 ++ docs/plans/m-corpus-catalog.md | 4 ++ docs/plans/m-env-implementation-plan.md | 7 ++++ docs/plans/m-environment-tool.md | 7 ++++ docs/plans/m-linter-status-2026-04-30.md | 7 ++++ docs/plans/m-linting-implementation-plan.md | 7 ++++ docs/plans/m-linting-survey.md | 7 ++++ docs/plugin-development.md | 7 ++++ docs/pre-commit.md | 7 ++++ docs/vista-meta-bootstrap.md | 7 ++++ docs/worked-example-accsum.md | 7 ++++ 18 files changed, 154 insertions(+) create mode 100644 docs/README.md diff --git a/docs/README.md b/docs/README.md new file mode 100644 index 0000000..e601dd3 --- /dev/null +++ b/docs/README.md @@ -0,0 +1,41 @@ +--- +created: 2026-05-11 +last_modified: 2026-05-11 +revisions: 0 +doc_type: [REFERENCE] +--- + +# m-cli — Documentation Index + +> First-pass index generated 2026-05-11. Labels follow the shared vocabulary below; the same vocabulary is used across all m-dev-tools repos. + +## Vocabulary + +Each doc is labeled `[TYPE · type? · connection · connection?]`. + +**Types** — `HISTORY` · `ARCHITECTURE` · `DESIGN` · `ADR` · `SPEC` · `REFERENCE` · `GUIDE` · `TUTORIAL` · `ROADMAP` · `PLAN` · `RESEARCH` · `SURVEY` · `GAP-ANALYSIS` · `STATUS` · `EXPLAINER` · `NOTES` · `WORKED-EXAMPLE` · `SETUP` · `INTEGRATION` · `PROPOSAL` · `BUILD-LOG` · `CHANGELOG` · `POSTMORTEM` + +**Repo connections** — `history` · `function` · `design` · `architecture` · `planning` · `implementation` + +## Top-level + +- **`evolution.md`** — `[HISTORY · BUILD-LOG · history · implementation]` Chronological narrative of how m-cli was built tier by tier, milestone by milestone, with performance journey and deferred items. +- **`guide.md`** — `[GUIDE · REFERENCE · function · architecture]` Comprehensive user-facing reference covering every subcommand, configuration, profiles, and the four-tier framework m-cli implements. +- **`m-linting-user-guide.md`** — `[GUIDE · function]` How-to guide for `m lint`: profiles, severity, thresholds, output formats, inline disables, and CLI flags. +- **`plugin-development.md`** — `[GUIDE · SPEC · function · architecture]` Contract and walkthrough for registering out-of-tree subcommands via the `m_cli.plugins` entry-point group. +- **`pre-commit.md`** — `[GUIDE · INTEGRATION · function]` Downstream pre-commit integration recipe for `m-fmt-check`, `m-fmt`, and `m-lint` hooks via `language: system`. +- **`vista-meta-bootstrap.md`** — `[HISTORY · EXPLAINER · history · design]` Records how the vista-meta YottaDB container bootstrapped m-cli development and why m-cli is now engine-independent. +- **`worked-example-accsum.md`** — `[WORKED-EXAMPLE · TUTORIAL · function]` End-to-end TDD walkthrough building an `accsum` access-log summariser to demonstrate every `m` subcommand in context. + +## `plans/` — design proposals, surveys, status reports, and implementation plans tracking m-cli's roadmap + +- **`plans/iris-ydb-portability.md`** — `[PLAN · RESEARCH · planning · architecture]` Function-by-function IRIS vs YottaDB CLI comparison with an engine-adapter refactor plan for cross-engine dispatch. +- **`plans/language-cli-survey.md`** — `[SURVEY · GAP-ANALYSIS · planning · design]` Survey of Rust/Go/Python/JS/Java CLI toolchains scored on productivity and quality, closing with a rank-ordered m-cli gap analysis. +- **`plans/linter-profiles-guide.md`** — `[DESIGN · PROPOSAL · design · planning]` Proposes splitting the `sac` profile into four mechanism-grounded profiles (KIDS-build / vista / safety / sac-style) by gatekeeper. +- **`plans/m-cli-history-and-evolution.md`** — `[HISTORY · EXPLAINER · history · architecture]` Chronicles the six-week sprint birthing m-tools, vista-meta, m-standard, tree-sitter-m, m-cli, and m-stdlib as cooperating siblings. +- **`plans/m-corpus-catalog.md`** — `[REFERENCE · RESEARCH · planning]` Catalog of non-VistA open-source M corpora vetted as candidates for the M-MOD-NN regression gate. +- **`plans/m-env-implementation-plan.md`** — `[PLAN · planning · implementation]` Self-contained implementation guide for an `m-env` POC proving containerized YottaDB/IRIS environments before folding into m-cli. +- **`plans/m-environment-tool.md`** — `[PROPOSAL · DESIGN · design · planning]` Design proposal for `m init` / `m env` / `m doctor` commands managing Dev Container-based M execution environments. +- **`plans/m-linter-status-2026-04-30.md`** — `[STATUS · POSTMORTEM · implementation]` Comprehensive audit of every shipped lint rule against a 4,215-routine non-VA corpus with prioritized fixes and landed deltas. +- **`plans/m-linting-implementation-plan.md`** — `[PLAN · BUILD-LOG · planning · implementation]` Phase-by-phase tracker for the M-MOD-NN modernization track, vista split, thresholds, engine-aware rules, and data-flow research. +- **`plans/m-linting-survey.md`** — `[SURVEY · GAP-ANALYSIS · design · planning]` Audits the 42 XINDEX/SAC rules for modern relevance and proposes a rank-ordered greenfield rule set drawn from first principles. diff --git a/docs/evolution.md b/docs/evolution.md index c9fbe86..d40480d 100644 --- a/docs/evolution.md +++ b/docs/evolution.md @@ -1,3 +1,10 @@ +--- +created: 2026-05-10 +last_modified: 2026-05-10 +revisions: 1 +doc_type: [HISTORY, BUILD-LOG] +--- + # m-cli — evolution How m-cli was built, in chronological order. This is **archaeology** — read the diff --git a/docs/guide.md b/docs/guide.md index 5bdc707..2cba499 100644 --- a/docs/guide.md +++ b/docs/guide.md @@ -1,3 +1,10 @@ +--- +created: 2026-04-27 +last_modified: 2026-05-10 +revisions: 12 +doc_type: [GUIDE, REFERENCE] +--- + # m-cli — Comprehensive Guide **Document type:** Reference + roadmap diff --git a/docs/m-linting-user-guide.md b/docs/m-linting-user-guide.md index c18fe3c..2ba990f 100644 --- a/docs/m-linting-user-guide.md +++ b/docs/m-linting-user-guide.md @@ -1,3 +1,10 @@ +--- +created: 2026-04-30 +last_modified: 2026-05-10 +revisions: 2 +doc_type: [GUIDE] +--- + # m lint — User Guide The `m lint` command checks `.m` source for engine-neutral logic, style, diff --git a/docs/plans/iris-ydb-portability.md b/docs/plans/iris-ydb-portability.md index e852814..28b96fa 100644 --- a/docs/plans/iris-ydb-portability.md +++ b/docs/plans/iris-ydb-portability.md @@ -1,3 +1,10 @@ +--- +created: 2026-04-30 +last_modified: 2026-05-10 +revisions: 2 +doc_type: [PLAN, RESEARCH] +--- + # IRIS ↔ YottaDB CLI Comparison — Portability Plan for `m-cli` > **Purpose.** A function-by-function comparison of the command-line surface diff --git a/docs/plans/language-cli-survey.md b/docs/plans/language-cli-survey.md index afadc27..061ff00 100644 --- a/docs/plans/language-cli-survey.md +++ b/docs/plans/language-cli-survey.md @@ -1,3 +1,10 @@ +--- +created: 2026-04-30 +last_modified: 2026-05-10 +revisions: 4 +doc_type: [SURVEY, GAP-ANALYSIS] +--- + # Language CLI Survey — Tooling Landscape and `m-cli` Gap Analysis > **Purpose.** A comprehensive survey of the CLI toolchains shipped by five of the diff --git a/docs/plans/linter-profiles-guide.md b/docs/plans/linter-profiles-guide.md index 7fe7bd5..5e719e9 100644 --- a/docs/plans/linter-profiles-guide.md +++ b/docs/plans/linter-profiles-guide.md @@ -1,3 +1,10 @@ +--- +created: 2026-05-06 +last_modified: 2026-05-10 +revisions: 2 +doc_type: [DESIGN, PROPOSAL] +--- + # Linter Profiles Guide **Status:** design / review artifact. **Drafted 2026-05-06**, rewritten same day after the realization that the right organizing axis for SAC rules is *which gatekeeper enforces them*, not *which era they came from*. This doc proposes splitting today's `sac` profile into four mechanism-grounded profiles and reorganizing tags accordingly. diff --git a/docs/plans/m-cli-history-and-evolution.md b/docs/plans/m-cli-history-and-evolution.md index b5093e9..a9ae5e8 100644 --- a/docs/plans/m-cli-history-and-evolution.md +++ b/docs/plans/m-cli-history-and-evolution.md @@ -3,6 +3,10 @@ title: m-cli — history and evolution status: live (2026-05-06) audience: anyone trying to understand how m-cli came to exist, what its precursors were, and where it sits among the M-language tooling sibling projects companion: ../README.md (current capability surface) +created: 2026-05-06 +last_modified: 2026-05-10 +revisions: 3 +doc_type: [HISTORY, EXPLAINER] --- # m-cli — history and evolution diff --git a/docs/plans/m-corpus-catalog.md b/docs/plans/m-corpus-catalog.md index 34a65c0..b536507 100644 --- a/docs/plans/m-corpus-catalog.md +++ b/docs/plans/m-corpus-catalog.md @@ -4,6 +4,10 @@ purpose: Candidate corpora for m-cli's modern-rule-set regression gate inventory_date: 2026-04-29 inventoried_by: research pass for m-cli M-MOD-NN rule design scope: Public, post-2010-active, NOT VistA-derived, substantial enough to surface real lint findings +created: 2026-04-30 +last_modified: 2026-05-10 +revisions: 2 +doc_type: [REFERENCE, RESEARCH] --- # Catalog: open-source M / MUMPS / ObjectScript corpora (non-VistA) diff --git a/docs/plans/m-env-implementation-plan.md b/docs/plans/m-env-implementation-plan.md index d4b9a59..f2ccd10 100644 --- a/docs/plans/m-env-implementation-plan.md +++ b/docs/plans/m-env-implementation-plan.md @@ -1,3 +1,10 @@ +--- +created: 2026-05-07 +last_modified: 2026-05-10 +revisions: 4 +doc_type: [PLAN] +--- + # `m-env` Implementation Plan **Document type:** self-contained implementation guide diff --git a/docs/plans/m-environment-tool.md b/docs/plans/m-environment-tool.md index b1bce61..5a9032f 100644 --- a/docs/plans/m-environment-tool.md +++ b/docs/plans/m-environment-tool.md @@ -1,3 +1,10 @@ +--- +created: 2026-05-07 +last_modified: 2026-05-10 +revisions: 4 +doc_type: [PROPOSAL, DESIGN] +--- + # `m environment` / `m doctor` / `m init` — Engine Environment Strategy **Document type:** design proposal diff --git a/docs/plans/m-linter-status-2026-04-30.md b/docs/plans/m-linter-status-2026-04-30.md index 4ab78b8..0d3dc27 100644 --- a/docs/plans/m-linter-status-2026-04-30.md +++ b/docs/plans/m-linter-status-2026-04-30.md @@ -1,3 +1,10 @@ +--- +created: 2026-04-30 +last_modified: 2026-05-10 +revisions: 4 +doc_type: [STATUS, POSTMORTEM] +--- + # `m lint` — Status & Audit Report **Date:** 2026-04-30 *(filename was requested as `m-linter-status-2024-04-40.md`; date corrected — 2024 was a typo and `04-40` is not a valid date)* diff --git a/docs/plans/m-linting-implementation-plan.md b/docs/plans/m-linting-implementation-plan.md index 15e7849..1b5aed8 100644 --- a/docs/plans/m-linting-implementation-plan.md +++ b/docs/plans/m-linting-implementation-plan.md @@ -1,3 +1,10 @@ +--- +created: 2026-04-30 +last_modified: 2026-05-10 +revisions: 4 +doc_type: [PLAN, BUILD-LOG] +--- + # M Linting Implementation Plan **Status:** working plan, derived from [`docs/m-linting-survey.md`](m-linting-survey.md) §7 (greenfield rule list) and §8 (roadmap), informed by [`docs/m-corpus-catalog.md`](m-corpus-catalog.md) for the regression gate. diff --git a/docs/plans/m-linting-survey.md b/docs/plans/m-linting-survey.md index 55fe3ac..c1b8be0 100644 --- a/docs/plans/m-linting-survey.md +++ b/docs/plans/m-linting-survey.md @@ -1,3 +1,10 @@ +--- +created: 2026-04-30 +last_modified: 2026-05-10 +revisions: 2 +doc_type: [SURVEY, GAP-ANALYSIS] +--- + # M Linting Survey & Greenfield Recommendations **Status:** the four open questions in the original draft (§9) are now diff --git a/docs/plugin-development.md b/docs/plugin-development.md index f53666b..c64b5a8 100644 --- a/docs/plugin-development.md +++ b/docs/plugin-development.md @@ -1,3 +1,10 @@ +--- +created: 2026-05-09 +last_modified: 2026-05-09 +revisions: 1 +doc_type: [GUIDE, SPEC] +--- + # Plugin development m-cli's built-in subcommands (`m fmt`, `m lint`, `m test`, `m doc`, …) diff --git a/docs/pre-commit.md b/docs/pre-commit.md index 28e6e3d..e0e2651 100644 --- a/docs/pre-commit.md +++ b/docs/pre-commit.md @@ -1,3 +1,10 @@ +--- +created: 2026-04-27 +last_modified: 2026-05-07 +revisions: 2 +doc_type: [GUIDE, INTEGRATION] +--- + # Pre-commit integration `m-cli` ships a [pre-commit](https://pre-commit.com) hook scaffold so diff --git a/docs/vista-meta-bootstrap.md b/docs/vista-meta-bootstrap.md index fff3256..5bd8169 100644 --- a/docs/vista-meta-bootstrap.md +++ b/docs/vista-meta-bootstrap.md @@ -1,3 +1,10 @@ +--- +created: 2026-05-10 +last_modified: 2026-05-10 +revisions: 1 +doc_type: [HISTORY, EXPLAINER] +--- + # vista-meta bootstrap — and why m-cli is independent of it now This document records how the `vista-meta` YottaDB container was used during diff --git a/docs/worked-example-accsum.md b/docs/worked-example-accsum.md index d29b2ff..e564673 100644 --- a/docs/worked-example-accsum.md +++ b/docs/worked-example-accsum.md @@ -1,3 +1,10 @@ +--- +created: 2026-05-09 +last_modified: 2026-05-09 +revisions: 2 +doc_type: [WORKED-EXAMPLE, TUTORIAL] +--- + # Worked example — building `accsum` with `m`, TDD, and m-stdlib **Document type:** end-to-end walkthrough From 599970a1f5dd705a238bd2c3d5e5bdc6bbd2752c Mon Sep 17 00:00:00 2001 From: Rafael Richards Date: Mon, 11 May 2026 11:09:00 -0400 Subject: [PATCH 4/5] doctor: actionable Docker-engine hints from vendored manifest (Phase 1b) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Implements Phase 1b of the `m engine` plan (see m-test-engine/docs/m-engine-implementation-plan.md): * dist/m-test-engine.json — vendored from m-test-engine's tier-1 contract * .gitignore — allowlist the vendored manifest * Makefile — `manifest` target re-vendors from $(M_TEST_ENGINE) when upstream is newer; existing `git diff --exit-code dist/` is the drift gate * m_cli.engine_manifest — typed loader (EngineManifest / BindMount / Volume / RunArgs) with protocol-version handshake * m_cli.doctor._runtime — five injectable docker/filesystem probes; tests monkeypatch the runtime surface instead of faking subprocess Doctor extensions: * Status.SKIPPED for prerequisite-failed downstream checks * Fix dataclass (command tuple + destructive bool) * Check.prerequisites declares the dependency chain * run_all_checks() skips downstream when any declared prereq is non-OK, emitting a SKIPPED Check that names the failing prereq * 5 new Docker checks (docker_installed, docker_daemon, engine_image, engine_container, engine_bind_mount) — hint+fix derived from manifest * Text output renders `hint:` and `fix:` lines under failing checks * JSON output exposes fix.command (list) + fix.destructive (bool) and prerequisites (list) per check Local YDB checks (ydb_dist, ydb_routines, parser, keywords, ydb_binary) remain unchanged — they handle their own missing-input cases. Tests: 18 new in test_doctor.py + 11 in test_engine_manifest.py. Full suite 1393 passing, 1 skipped (pre-existing). Lint + mypy clean. Co-Authored-By: Claude Opus 4.7 (1M context) --- .gitignore | 1 + Makefile | 18 ++- dist/m-test-engine.json | 33 +++++ src/m_cli/doctor/_runtime.py | 73 ++++++++++ src/m_cli/doctor/checks.py | 255 +++++++++++++++++++++++++++++++-- src/m_cli/doctor/cli.py | 68 ++++++--- src/m_cli/engine_manifest.py | 132 +++++++++++++++++ tests/test_doctor.py | 258 +++++++++++++++++++++++++++++++++- tests/test_engine_manifest.py | 152 ++++++++++++++++++++ 9 files changed, 961 insertions(+), 29 deletions(-) create mode 100644 dist/m-test-engine.json create mode 100644 src/m_cli/doctor/_runtime.py create mode 100644 src/m_cli/engine_manifest.py create mode 100644 tests/test_engine_manifest.py diff --git a/.gitignore b/.gitignore index ea8a758..ea75dce 100644 --- a/.gitignore +++ b/.gitignore @@ -14,6 +14,7 @@ __pycache__/ !/dist/commands.json !/dist/lint-rules.json !/dist/fmt-rules.json +!/dist/m-test-engine.json /build/ *.egg-info/ .mypy_cache/ diff --git a/Makefile b/Makefile index 46298f5..f0967a3 100644 --- a/Makefile +++ b/Makefile @@ -111,7 +111,23 @@ dist/fmt-rules.json: $(FMT_SOURCES) @mkdir -p dist $(M) fmt --list-rules --json > $@ -manifest: dist/commands.json dist/lint-rules.json dist/fmt-rules.json +# Vendored from m-dev-tools/m-test-engine. The source of truth is +# $(M_TEST_ENGINE)/dist/m-test-engine.json — copied here at release +# time so a fresh `pip install m-cli` carries the engine contract +# without needing network access. The drift gate (`git diff --exit-code +# dist/` below) catches missed re-vendoring after an upstream bump. +# +# If M_TEST_ENGINE is not a local checkout, the rule is a no-op — the +# vendored copy already in git is treated as authoritative. +M_TE_MANIFEST := $(wildcard $(M_TEST_ENGINE)/dist/m-test-engine.json) +ifneq ($(M_TE_MANIFEST),) +dist/m-test-engine.json: $(M_TE_MANIFEST) + @mkdir -p dist + cp $< $@ + @echo "vendored dist/m-test-engine.json from $(M_TEST_ENGINE)" +endif + +manifest: dist/commands.json dist/lint-rules.json dist/fmt-rules.json dist/m-test-engine.json check-manifest: manifest git diff --exit-code dist/ diff --git a/dist/m-test-engine.json b/dist/m-test-engine.json new file mode 100644 index 0000000..4016c5d --- /dev/null +++ b/dist/m-test-engine.json @@ -0,0 +1,33 @@ +{ + "$schema": "https://raw.githubusercontent.com/m-dev-tools/m-test-engine/main/dist/m-test-engine.schema.json", + "protocol": 1, + "image": "ghcr.io/m-dev-tools/m-test-engine", + "default_tag": "r2.02", + "container": "m-test-engine", + "bind_mount": { + "host": "/m-work", + "container": "/m-work", + "mode": "rw" + }, + "compose_file": "docker/compose.yml", + "repo_url": "https://github.com/m-dev-tools/m-test-engine", + "min_docker": "20.10", + "ydb_version": "r2.02", + "run_args": { + "hostname": "m-test-engine", + "working_dir": "/m-work", + "restart": "unless-stopped", + "volumes": [ + { + "name": "m-test-engine-globals", + "target": "/data" + } + ], + "command": [ + "tail", + "-f", + "/dev/null" + ] + }, + "verified_on": "2026-05-11" +} \ No newline at end of file diff --git a/src/m_cli/doctor/_runtime.py b/src/m_cli/doctor/_runtime.py new file mode 100644 index 0000000..65f3a1b --- /dev/null +++ b/src/m_cli/doctor/_runtime.py @@ -0,0 +1,73 @@ +"""Runtime probes used by Docker-engine checks in ``m doctor``. + +These functions wrap the shell-out / filesystem calls that determine +whether the Docker engine path is healthy. They live in their own +module so tests can monkeypatch them without faking ``subprocess`` or +``shutil`` at large. + +Failure modes are intentionally non-throwing: every probe returns +``bool``. The diagnostic richness (which exact failure occurred) is +handled by the check function that wraps the probe. +""" + +from __future__ import annotations + +import shutil +import subprocess +from pathlib import Path + +_TIMEOUT_SECONDS = 5 + + +def docker_available() -> bool: + """``docker`` CLI is on ``PATH``.""" + return shutil.which("docker") is not None + + +def docker_daemon_reachable() -> bool: + """``docker info`` returns 0 — the daemon is up and accessible.""" + if not docker_available(): + return False + try: + result = subprocess.run( + ["docker", "info"], + capture_output=True, + timeout=_TIMEOUT_SECONDS, + ) + return result.returncode == 0 + except (subprocess.TimeoutExpired, OSError): + return False + + +def docker_image_present(ref: str) -> bool: + """``docker image inspect `` returns 0 — image is pulled locally.""" + try: + result = subprocess.run( + ["docker", "image", "inspect", ref], + capture_output=True, + timeout=_TIMEOUT_SECONDS, + ) + return result.returncode == 0 + except (subprocess.TimeoutExpired, OSError): + return False + + +def docker_container_running(name: str) -> bool: + """A container with this exact name is in ``docker ps`` output.""" + try: + result = subprocess.run( + ["docker", "ps", "--filter", f"name=^{name}$", "--format", "{{.Names}}"], + capture_output=True, + timeout=_TIMEOUT_SECONDS, + text=True, + ) + if result.returncode != 0: + return False + return name in result.stdout.split() + except (subprocess.TimeoutExpired, OSError): + return False + + +def path_exists(path: str) -> bool: + """Host filesystem path resolves.""" + return Path(path).exists() diff --git a/src/m_cli/doctor/checks.py b/src/m_cli/doctor/checks.py index 301d246..a053ff9 100644 --- a/src/m_cli/doctor/checks.py +++ b/src/m_cli/doctor/checks.py @@ -1,7 +1,22 @@ """Environment-health checks for ``m doctor``. -Each ``check_*`` function is independent and returns a :class:`Check`. -``run_all_checks()`` runs them in a stable order and returns the list. +Two paths share this surface: + +1. **Docker engine path** (canonical for m-cli users without local + YottaDB) — driven by the vendored manifest in + ``dist/m-test-engine.json`` (see ``m_cli.engine_manifest``). Probes + live in :mod:`m_cli.doctor._runtime`; checks declare prerequisites + so downstream checks emit ``SKIPPED`` instead of redundant ``WARN`` + when an upstream cause has already been reported. + +2. **Local YottaDB path** (alternative, for hosts with a system-level + YDB install) — the original five checks. These do **not** declare + prerequisites because each handles its own missing-input case + gracefully (e.g. ``check_ydb_binary`` falls back to ``$PATH`` when + ``$ydb_dist`` is unset). + +Each ``check_*`` function is independent. Prerequisite handling lives +in :func:`run_all_checks`. """ from __future__ import annotations @@ -9,15 +24,27 @@ import enum import os import shutil -from dataclasses import dataclass +from dataclasses import dataclass, field from pathlib import Path from typing import Callable +from m_cli.doctor import _runtime +from m_cli.engine_manifest import EngineManifest, load_engine_manifest + class Status(enum.Enum): OK = "OK" WARN = "WARN" FAIL = "FAIL" + SKIPPED = "SKIPPED" + + +@dataclass(frozen=True) +class Fix: + """A copy-pasteable fix command derived from manifest data.""" + + command: tuple[str, ...] + destructive: bool = False @dataclass(frozen=True) @@ -26,10 +53,15 @@ class Check: status: Status message: str hint: str | None = None + prerequisites: tuple[str, ...] = field(default_factory=tuple) + fix: Fix | None = None + + +# ── Local YDB path (legacy, unchanged behaviour) ───────────────────── def check_ydb_dist() -> Check: - """Check `$ydb_dist` env var: set, exists, contains a `ydb` binary.""" + """Check ``$ydb_dist`` env var: set, exists, contains a ``ydb`` binary.""" val = os.environ.get("ydb_dist") if not val: return Check( @@ -72,7 +104,7 @@ def check_ydb_dist() -> Check: def check_ydb_routines() -> Check: - """Check `$ydb_routines` is set (the routine search path).""" + """Check ``$ydb_routines`` is set (the routine search path).""" val = os.environ.get("ydb_routines") if not val: return Check( @@ -144,7 +176,7 @@ def check_keywords() -> Check: def check_ydb_binary() -> Check: - """Locate the `ydb` binary via `$YDB`, `$ydb_dist/ydb`, or PATH.""" + """Locate the ``ydb`` binary via ``$YDB``, ``$ydb_dist/ydb``, or PATH.""" explicit = os.environ.get("YDB") if explicit: path = Path(explicit) @@ -187,7 +219,177 @@ def check_ydb_binary() -> Check: ) +# ── Docker engine path (canonical for m-cli users) ─────────────────── + + +def _manifest() -> EngineManifest | None: + """Best-effort manifest load — None if vendoring hasn't happened.""" + try: + return load_engine_manifest() + except (FileNotFoundError, ValueError, KeyError): + return None + + +def check_docker_installed() -> Check: + """Top of the Docker chain: is the ``docker`` CLI on ``$PATH``?""" + if _runtime.docker_available(): + return Check( + name="docker_installed", + status=Status.OK, + message="docker CLI on PATH", + ) + return Check( + name="docker_installed", + status=Status.WARN, + message="docker CLI not found on PATH", + hint=( + "Install Docker Engine (linux) or Docker Desktop (mac/win). " + "See https://docs.docker.com/engine/install/ for distro-" + "specific instructions." + ), + ) + + +def check_docker_daemon() -> Check: + """``docker info`` succeeds — the daemon is up and the user can reach it.""" + if _runtime.docker_daemon_reachable(): + return Check( + name="docker_daemon", + status=Status.OK, + message="docker daemon reachable", + prerequisites=("docker_installed",), + ) + return Check( + name="docker_daemon", + status=Status.WARN, + message="docker daemon not reachable", + hint=( + "Start the daemon: `sudo systemctl start docker` (linux) or " + "launch Docker Desktop (mac/win). Also verify your user is in " + "the `docker` group: `groups | grep docker`." + ), + prerequisites=("docker_installed",), + fix=Fix( + command=("sudo", "systemctl", "start", "docker"), + destructive=False, + ), + ) + + +def check_engine_image() -> Check: + """The canonical m-test-engine image is pulled locally.""" + m = _manifest() + if m is None: + return Check( + name="engine_image", + status=Status.WARN, + message="manifest dist/m-test-engine.json not loadable", + hint=( + "Re-vendor the engine contract: " + "`make manifest M_TEST_ENGINE=/path/to/m-test-engine`." + ), + prerequisites=("docker_installed", "docker_daemon"), + ) + ref = m.image_ref() + if _runtime.docker_image_present(ref): + return Check( + name="engine_image", + status=Status.OK, + message=f"image {ref} present", + prerequisites=("docker_installed", "docker_daemon"), + ) + return Check( + name="engine_image", + status=Status.WARN, + message=f"image {ref} not pulled", + hint=f"Pull the canonical engine image: `docker pull {ref}`.", + prerequisites=("docker_installed", "docker_daemon"), + fix=Fix(command=("docker", "pull", ref), destructive=False), + ) + + +def check_engine_container() -> Check: + """The canonical m-test-engine container is running.""" + m = _manifest() + if m is None: + return Check( + name="engine_container", + status=Status.WARN, + message="manifest dist/m-test-engine.json not loadable", + hint="See engine_image hint.", + prerequisites=("docker_installed", "docker_daemon"), + ) + if _runtime.docker_container_running(m.container): + return Check( + name="engine_container", + status=Status.OK, + message=f"container `{m.container}` running", + prerequisites=("docker_installed", "docker_daemon"), + ) + return Check( + name="engine_container", + status=Status.WARN, + message=f"container `{m.container}` not running", + hint=( + f"Start it from the m-test-engine checkout: `docker compose -f {m.compose_file} up -d`." + ), + prerequisites=("docker_installed", "docker_daemon"), + fix=Fix( + command=("docker", "compose", "-f", m.compose_file, "up", "-d"), + destructive=False, + ), + ) + + +def check_engine_bind_mount() -> Check: + """Host bind-mount directory exists. + + Independent of the rest of the Docker chain — the user can create + the bind-mount directory before docker is installed (and should). + """ + m = _manifest() + if m is None: + return Check( + name="engine_bind_mount", + status=Status.WARN, + message="manifest dist/m-test-engine.json not loadable", + hint="See engine_image hint.", + ) + host = m.bind_mount.host + if _runtime.path_exists(host): + return Check( + name="engine_bind_mount", + status=Status.OK, + message=f"host {host} exists", + ) + return Check( + name="engine_bind_mount", + status=Status.WARN, + message=f"host {host} does not exist", + hint=( + f"Create the shared m-* working dir: " + f"`sudo install -d -o $USER -g $USER {host}` then " + f"`cd {host} && git clone ` for each m-* repo you " + "want available inside the engine." + ), + fix=Fix( + command=("sudo", "install", "-d", "-o", "$USER", "-g", "$USER", host), + destructive=False, + ), + ) + + +# ── Registry + dependency-aware runner ─────────────────────────────── + + _CHECKS: tuple[Callable[[], Check], ...] = ( + # Docker engine path first — canonical runtime + check_docker_installed, + check_docker_daemon, + check_engine_image, + check_engine_container, + check_engine_bind_mount, + # Local YDB path second — alternative runtime check_ydb_dist, check_ydb_routines, check_parser, @@ -196,6 +398,43 @@ def check_ydb_binary() -> Check: ) +def _skip_for(prereq_name: str) -> Check: + """SKIPPED placeholder check pointing at the failed prerequisite.""" + return Check( + name="", # caller fills this in + status=Status.SKIPPED, + message=f"skipped — waiting on {prereq_name}", + ) + + def run_all_checks() -> list[Check]: - """Run every registered check in order and return the results.""" - return [fn() for fn in _CHECKS] + """Run every registered check, honouring prerequisite chains. + + A check is SKIPPED (not run) when any of its declared prerequisites + landed in a non-OK status. This implements the root-cause grouping + pattern: one warning per cause instead of N warnings per N + downstream effects. + + Local YDB checks declare no prerequisites and behave exactly as + before. Only the Docker chain participates in the SKIPPED grouping. + """ + results: dict[str, Check] = {} + for fn in _CHECKS: + # Peek at the function's first-pass output to discover its + # declared prerequisites and its own name. (The check function + # is the source of truth — registry has no separate metadata.) + provisional = fn() + prereqs = provisional.prerequisites + blockers = [p for p in prereqs if p in results and results[p].status is not Status.OK] + if blockers: + # One pointer is enough — the user follows the chain up. + skipped = Check( + name=provisional.name, + status=Status.SKIPPED, + message=f"skipped — waiting on {blockers[0]}", + prerequisites=prereqs, + ) + results[provisional.name] = skipped + else: + results[provisional.name] = provisional + return list(results.values()) diff --git a/src/m_cli/doctor/cli.py b/src/m_cli/doctor/cli.py index f19ee8a..8436502 100644 --- a/src/m_cli/doctor/cli.py +++ b/src/m_cli/doctor/cli.py @@ -1,46 +1,80 @@ """`m doctor` command — environment diagnostics. Runs every registered check, prints a per-check line, and exits ``1`` -if any check is FAIL (WARN does not fail the run). +if any check is FAIL (WARN/SKIPPED do not fail the run). + +Output formats: + +* ``text`` (default) — human-readable, one line per check; failing + checks render ``hint:`` and ``fix:`` lines below. +* ``json`` — structured payload. Each entry includes ``name``, + ``status``, ``message``, ``hint`` and (when the check provides one) + ``fix.command`` + ``fix.destructive``. Agents walk failure → fix + edges programmatically without re-parsing prose. """ from __future__ import annotations import argparse import json +import shlex from m_cli.doctor.checks import Check, Status, run_all_checks +_MARKERS = { + Status.OK: "✓", + Status.WARN: "!", + Status.FAIL: "x", + Status.SKIPPED: "-", +} + def doctor_command(args: argparse.Namespace) -> int: checks = run_all_checks() fmt = getattr(args, "format", "text") if fmt == "json": - payload = [ - { - "name": c.name, - "status": c.status.value, - "message": c.message, - "hint": c.hint, - } - for c in checks - ] - print(json.dumps(payload, indent=2)) + print(json.dumps(_to_json(checks), indent=2)) else: _write_text(checks) return 1 if any(c.status is Status.FAIL for c in checks) else 0 +def _to_json(checks: list[Check]) -> list[dict]: + out: list[dict] = [] + for c in checks: + entry: dict = { + "name": c.name, + "status": c.status.value, + "message": c.message, + "hint": c.hint, + } + if c.fix is not None: + entry["fix"] = { + "command": list(c.fix.command), + "destructive": c.fix.destructive, + } + else: + entry["fix"] = None + if c.prerequisites: + entry["prerequisites"] = list(c.prerequisites) + out.append(entry) + return out + + def _write_text(checks: list[Check]) -> None: width = max((len(c.name) for c in checks), default=0) for c in checks: - label = c.status.value.ljust(4) - marker = {"OK": "✓", "WARN": "!", "FAIL": "x"}[c.status.value] - print(f" {marker} {label} {c.name.ljust(width)} {c.message}") - if c.hint and c.status is not Status.OK: + label = c.status.value.ljust(7) + marker = _MARKERS[c.status] + print(f" {marker} {label} {c.name.ljust(width)} {c.message}") + if c.status is Status.OK or c.status is Status.SKIPPED: + continue + if c.hint: print(f" hint: {c.hint}") + if c.fix is not None: + print(f" fix: {shlex.join(c.fix.command)}") fails = sum(1 for c in checks if c.status is Status.FAIL) warns = sum(1 for c in checks if c.status is Status.WARN) oks = sum(1 for c in checks if c.status is Status.OK) - summary = f"\n{oks} OK, {warns} warning, {fails} fail" - print(summary) + skipped = sum(1 for c in checks if c.status is Status.SKIPPED) + print(f"\n{oks} OK, {warns} warning, {fails} fail, {skipped} skipped") diff --git a/src/m_cli/engine_manifest.py b/src/m_cli/engine_manifest.py new file mode 100644 index 0000000..1d0ca97 --- /dev/null +++ b/src/m_cli/engine_manifest.py @@ -0,0 +1,132 @@ +"""In-process loader for the vendored m-test-engine manifest. + +m-cli vendors ``dist/m-test-engine.json`` from the m-test-engine repo at +release time (see Makefile's ``manifest`` target). This module parses +that file into typed dataclasses and is the single in-process source of +truth for the engine contract: image registry, container name, +bind-mount layout, compose file path, YDB release. + +Consumed by: + +* ``m doctor`` — emits actionable Docker-engine hints derived from + these fields (Phase 1b). +* The ``m engine`` subcommand family — builds ``docker`` / + ``docker compose`` invocations from these fields (Phase 2). + +The legacy hardcoded constants in :class:`m_cli.engine.DockerEngine` +(``container = "m-test-engine"`` / ``bind_root = /work``) are kept in +sync with this manifest during the Phase 1 → Phase 2 transition. + +Protocol handshake — see ``docs/m-engine-implementation-plan.md`` §1.5 +in the m-test-engine repo for the bump policy. m-cli understands a +fixed set of protocol versions and rejects manifests claiming a higher +one, with a clear "upgrade m-cli" hint. +""" + +from __future__ import annotations + +import json +from dataclasses import dataclass +from pathlib import Path + +# Protocol versions this build of m-cli can consume. Add to this set +# when a new protocol revision is adopted; never remove an entry until +# the corresponding manifest version is provably gone from the wild. +SUPPORTED_PROTOCOLS: frozenset[int] = frozenset({1}) + + +@dataclass(frozen=True) +class BindMount: + host: str + container: str + mode: str + + +@dataclass(frozen=True) +class Volume: + name: str + target: str + + +@dataclass(frozen=True) +class RunArgs: + hostname: str + working_dir: str + restart: str + volumes: tuple[Volume, ...] + command: tuple[str, ...] + + +@dataclass(frozen=True) +class EngineManifest: + protocol: int + image: str + default_tag: str + container: str + bind_mount: BindMount + compose_file: str + repo_url: str + min_docker: str + ydb_version: str + run_args: RunArgs + verified_on: str + + def image_ref(self) -> str: + """Full image reference including tag (e.g. ``ghcr.io/...:r2.02``).""" + return f"{self.image}:{self.default_tag}" + + +def vendored_manifest_path() -> Path: + """Absolute path to the vendored manifest inside this m-cli checkout.""" + # src/m_cli/engine_manifest.py → repo_root/dist/m-test-engine.json + return Path(__file__).resolve().parent.parent.parent / "dist" / "m-test-engine.json" + + +def load_engine_manifest(path: Path | None = None) -> EngineManifest: + """Parse the manifest into an :class:`EngineManifest`. + + Pass ``path`` to load from a non-default location (tests, the future + ``m engine --manifest=...`` flag). Raises ``FileNotFoundError`` if + the file does not exist, ``ValueError`` for protocol mismatches or + malformed structures, and ``KeyError`` for missing required fields. + """ + src = path if path is not None else vendored_manifest_path() + data = json.loads(src.read_text(encoding="utf-8")) + + protocol = data["protocol"] + if not isinstance(protocol, int): + raise ValueError(f"protocol must be an integer, got {type(protocol).__name__}") + if protocol not in SUPPORTED_PROTOCOLS: + supported = sorted(SUPPORTED_PROTOCOLS) + raise ValueError( + f"manifest protocol {protocol} is not supported by this " + f"build of m-cli (supports: {supported}). " + "Upgrade m-cli or vendor a compatible manifest." + ) + + bm = data["bind_mount"] + bind_mount = BindMount(host=bm["host"], container=bm["container"], mode=bm["mode"]) + + ra = data["run_args"] + volumes = tuple(Volume(name=v["name"], target=v["target"]) for v in ra["volumes"]) + run_args = RunArgs( + hostname=ra["hostname"], + working_dir=ra["working_dir"], + restart=ra["restart"], + volumes=volumes, + command=tuple(ra["command"]), + ) + + return EngineManifest( + protocol=protocol, + image=data["image"], + default_tag=data["default_tag"], + container=data["container"], + bind_mount=bind_mount, + compose_file=data["compose_file"], + repo_url=data["repo_url"], + min_docker=data["min_docker"], + ydb_version=data["ydb_version"], + run_args=run_args, + verified_on=data["verified_on"], + ) diff --git a/tests/test_doctor.py b/tests/test_doctor.py index f5df4f8..03eab3f 100644 --- a/tests/test_doctor.py +++ b/tests/test_doctor.py @@ -15,11 +15,20 @@ from __future__ import annotations import argparse +import json + +import pytest from m_cli.doctor import doctor_command from m_cli.doctor.checks import ( Check, + Fix, Status, + check_docker_daemon, + check_docker_installed, + check_engine_bind_mount, + check_engine_container, + check_engine_image, check_keywords, check_parser, check_ydb_binary, @@ -39,8 +48,36 @@ def test_check_dataclass_carries_status_and_message(): assert c.hint is None -def test_status_enum_has_three_levels(): - assert {Status.OK, Status.WARN, Status.FAIL} == set(Status) +def test_status_enum_has_four_levels(): + # SKIPPED is added in Phase 1b for prerequisite-failed downstream + # checks (root-cause grouping in m doctor). + assert {Status.OK, Status.WARN, Status.FAIL, Status.SKIPPED} == set(Status) + + +def test_fix_dataclass_carries_command_and_destructive_flag(): + f = Fix(command=("docker", "pull", "x"), destructive=False) + assert f.command == ("docker", "pull", "x") + assert f.destructive is False + # Destructive defaults to False + f2 = Fix(command=("docker", "stop", "y")) + assert f2.destructive is False + + +def test_check_accepts_optional_prerequisites_and_fix(): + c = Check( + name="x", + status=Status.WARN, + message="m", + prerequisites=("y",), + fix=Fix(command=("a", "b")), + ) + assert c.prerequisites == ("y",) + assert c.fix is not None + assert c.fix.command == ("a", "b") + # Existing call sites without these fields still work + c2 = Check(name="x", status=Status.OK, message="m") + assert c2.prerequisites == () + assert c2.fix is None # ------------------------------------------------------------- Per-check tests @@ -190,9 +227,224 @@ def test_doctor_cli_json_format(monkeypatch, tmp_path, capsys): monkeypatch.setenv("ydb_routines", ".") rc = doctor_command(_ns(format="json")) out = capsys.readouterr().out - import json payload = json.loads(out) assert isinstance(payload, list) assert all("name" in c and "status" in c for c in payload) assert rc in (0, 1) + + +# ───────────────────────────────────────────────────────────────── +# Phase 1b — Docker engine checks driven by the vendored manifest +# ───────────────────────────────────────────────────────────────── + + +@pytest.fixture +def docker_world(monkeypatch): + """Inject all five Docker-runtime probes with controllable defaults. + + Tests adjust individual probes via ``world.set(...)`` to model the + cell of the truth table they're exercising. No actual docker / + filesystem calls fire. + """ + from m_cli.doctor import _runtime + + state = { + "docker_available": True, + "docker_daemon_reachable": True, + "docker_image_present": True, + "docker_container_running": True, + "path_exists": True, + } + + def _set(**kw): + state.update(kw) + + monkeypatch.setattr(_runtime, "docker_available", lambda: state["docker_available"]) + monkeypatch.setattr( + _runtime, + "docker_daemon_reachable", + lambda: state["docker_daemon_reachable"], + ) + monkeypatch.setattr( + _runtime, + "docker_image_present", + lambda ref: state["docker_image_present"], + ) + monkeypatch.setattr( + _runtime, + "docker_container_running", + lambda name: state["docker_container_running"], + ) + monkeypatch.setattr(_runtime, "path_exists", lambda p: state["path_exists"]) + + class _World: + set = staticmethod(_set) + + return _World() + + +def test_check_docker_installed_ok_when_present(docker_world): + docker_world.set(docker_available=True) + c = check_docker_installed() + assert c.status is Status.OK + + +def test_check_docker_installed_warn_with_hint_when_missing(docker_world): + docker_world.set(docker_available=False) + c = check_docker_installed() + assert c.status is Status.WARN + assert c.hint is not None # must be actionable + + +def test_check_docker_daemon_ok_when_reachable(docker_world): + c = check_docker_daemon() + assert c.status is Status.OK + + +def test_check_docker_daemon_warn_with_fix_when_unreachable(docker_world): + docker_world.set(docker_daemon_reachable=False) + c = check_docker_daemon() + assert c.status is Status.WARN + assert c.fix is not None + cmd = " ".join(c.fix.command) + # Linux daemon start: systemctl. Mac: starts Docker Desktop. Either + # way the fix command is non-destructive. + assert "docker" in cmd or "systemctl" in cmd + assert c.fix.destructive is False + + +def test_check_engine_image_ok_when_present(docker_world): + c = check_engine_image() + assert c.status is Status.OK + + +def test_check_engine_image_warn_with_pull_fix_when_missing(docker_world): + from m_cli.engine_manifest import load_engine_manifest + + docker_world.set(docker_image_present=False) + c = check_engine_image() + assert c.status is Status.WARN + assert c.fix is not None + # Pull command derived from the manifest + assert c.fix.command[:2] == ("docker", "pull") + m = load_engine_manifest() + assert m.image_ref() in c.fix.command + assert c.fix.destructive is False + + +def test_check_engine_container_ok_when_running(docker_world): + c = check_engine_container() + assert c.status is Status.OK + + +def test_check_engine_container_warn_with_start_fix_when_stopped(docker_world): + docker_world.set(docker_container_running=False) + c = check_engine_container() + assert c.status is Status.WARN + assert c.fix is not None + # Compose-first per the implementation plan §1.3 + cmd = " ".join(c.fix.command) + assert "compose" in cmd or "docker" in cmd + assert c.fix.destructive is False + + +def test_check_engine_bind_mount_ok_when_path_exists(docker_world): + c = check_engine_bind_mount() + assert c.status is Status.OK + + +def test_check_engine_bind_mount_warn_when_missing(docker_world): + from m_cli.engine_manifest import load_engine_manifest + + docker_world.set(path_exists=False) + c = check_engine_bind_mount() + assert c.status is Status.WARN + m = load_engine_manifest() + # Message references the host bind-mount path declared in the manifest + assert m.bind_mount.host in (c.message or "") or m.bind_mount.host in (c.hint or "") + + +# ───────────────────────────────────────────────────────────────── +# Phase 1b.4 — Root-cause grouping: SKIPPED downstream +# ───────────────────────────────────────────────────────────────── + + +def test_run_all_checks_marks_docker_downstream_skipped_when_docker_missing( + docker_world, +): + docker_world.set(docker_available=False) + checks = run_all_checks() + by_name = {c.name: c for c in checks} + + # Top of the chain — WARN with hint + assert by_name["docker_installed"].status is Status.WARN + + # Everything that prerequisites docker_installed is SKIPPED, not WARN + for downstream in ("docker_daemon", "engine_image", "engine_container"): + c = by_name[downstream] + assert c.status is Status.SKIPPED, ( + f"{downstream} should be SKIPPED when docker_installed fails, got {c.status}" + ) + # Skipped reason references the failing prereq + assert "docker_installed" in (c.message or "") + + +def test_run_all_checks_marks_engine_chain_skipped_when_daemon_down(docker_world): + docker_world.set(docker_daemon_reachable=False) + checks = run_all_checks() + by_name = {c.name: c for c in checks} + + assert by_name["docker_installed"].status is Status.OK + assert by_name["docker_daemon"].status is Status.WARN + # engine_image and engine_container both depend on daemon being up + for downstream in ("engine_image", "engine_container"): + assert by_name[downstream].status is Status.SKIPPED + + +# ───────────────────────────────────────────────────────────────── +# Phase 1b.3 — JSON schema extension: fix.command + fix.destructive +# ───────────────────────────────────────────────────────────────── + + +def test_doctor_json_emits_fix_block_when_check_provides_one(docker_world, capsys): + docker_world.set(docker_image_present=False) + doctor_command(_ns(format="json")) + out = capsys.readouterr().out + + payload = json.loads(out) + by_name = {c["name"]: c for c in payload} + assert "engine_image" in by_name + fx = by_name["engine_image"].get("fix") + assert fx is not None + assert isinstance(fx["command"], list) + assert fx["command"][:2] == ["docker", "pull"] + assert fx["destructive"] is False + + +def test_doctor_json_omits_fix_when_check_does_not_provide_one(docker_world, capsys): + doctor_command(_ns(format="json")) + out = capsys.readouterr().out + payload = json.loads(out) + ok_check = next(c for c in payload if c["status"] == "OK") + # Healthy checks need no fix + assert ok_check.get("fix") is None + + +def test_doctor_text_shows_fix_command_after_hint(docker_world, capsys): + docker_world.set(docker_image_present=False) + doctor_command(_ns(format="text")) + out = capsys.readouterr().out + # Fix line uses a "fix:" marker, copy-pasteable command + assert "fix:" in out + assert "docker pull" in out + + +def test_doctor_text_renders_skipped_with_prereq_reference(docker_world, capsys): + docker_world.set(docker_available=False) + doctor_command(_ns(format="text")) + out = capsys.readouterr().out + # SKIPPED checks visible in output + assert "SKIP" in out + # The prereq that caused the skip is named in the output + assert "docker_installed" in out diff --git a/tests/test_engine_manifest.py b/tests/test_engine_manifest.py new file mode 100644 index 0000000..9ca356a --- /dev/null +++ b/tests/test_engine_manifest.py @@ -0,0 +1,152 @@ +"""Tests for the vendored m-test-engine manifest loader. + +m-cli vendors `dist/m-test-engine.json` from the m-test-engine repo at +release time. The loader in `m_cli.engine_manifest` is the in-process +source of truth for the engine contract — image registry, container +name, bind-mount layout, compose file path, YDB release. + +`m doctor` and (later) `m engine ` consume the loaded manifest; +hardcoded strings in `m_cli.engine.DockerEngine` collapse onto it once +Phase 2 lands. + +See docs/m-engine-implementation-plan.md (in the m-test-engine repo) +for the full design rationale and the protocol bump policy. +""" + +from __future__ import annotations + +import json +from pathlib import Path + +import pytest + +from m_cli.engine_manifest import ( + BindMount, + EngineManifest, + load_engine_manifest, + vendored_manifest_path, +) + + +def test_vendored_manifest_path_resolves_to_dist_file(): + p = vendored_manifest_path() + assert isinstance(p, Path) + assert p.name == "m-test-engine.json" + assert p.exists(), "Phase 1b.1 vendoring step must have run" + + +def test_load_engine_manifest_returns_dataclass(): + m = load_engine_manifest() + assert isinstance(m, EngineManifest) + + +def test_manifest_protocol_is_one_at_phase_one(): + m = load_engine_manifest() + assert m.protocol == 1 + + +def test_manifest_image_ref_combines_image_and_default_tag(): + m = load_engine_manifest() + assert m.image_ref() == f"{m.image}:{m.default_tag}" + + +def test_manifest_container_name_matches_existing_DockerEngine_default(): + # Cross-check the manifest matches the legacy hardcoded default in + # DockerEngine. When Phase 2 collapses the hardcoded value onto the + # manifest, this assertion still passes — but it documents the + # consistency requirement during the transition. + from m_cli.engine import DockerEngine + + m = load_engine_manifest() + assert m.container == DockerEngine.container + + +def test_manifest_bind_mount_is_typed(): + m = load_engine_manifest() + assert isinstance(m.bind_mount, BindMount) + assert m.bind_mount.host.startswith("/") + assert m.bind_mount.container.startswith("/") + assert m.bind_mount.mode in ("ro", "rw") + + +def test_manifest_compose_file_relative_to_engine_repo(): + m = load_engine_manifest() + # Compose file path is repo-relative inside m-test-engine, NOT + # resolvable from m-cli. The contract just declares it; m-cli uses + # it to construct hints. + assert m.compose_file + assert not Path(m.compose_file).is_absolute() + + +def test_manifest_run_args_carry_fallback_docker_run_payload(): + m = load_engine_manifest() + args = m.run_args + assert args.hostname + assert args.working_dir.startswith("/") + assert args.restart in ("no", "always", "on-failure", "unless-stopped") + assert isinstance(args.command, tuple) + assert all(isinstance(s, str) for s in args.command) + + +def test_manifest_load_from_explicit_path(tmp_path): + # Loader accepts an override path — useful for tests and for the + # future `m engine --manifest=/custom/path` flag. + payload = { + "protocol": 1, + "image": "example/img", + "default_tag": "v1", + "container": "x", + "bind_mount": {"host": "/h", "container": "/c", "mode": "rw"}, + "compose_file": "compose.yml", + "repo_url": "https://example.com/repo", + "min_docker": "20.10", + "ydb_version": "r2.02", + "run_args": { + "hostname": "x", + "working_dir": "/c", + "restart": "unless-stopped", + "volumes": [{"name": "v", "target": "/v"}], + "command": ["sleep", "infinity"], + }, + "verified_on": "2026-05-11", + } + p = tmp_path / "manifest.json" + p.write_text(json.dumps(payload)) + m = load_engine_manifest(p) + assert m.image == "example/img" + assert m.image_ref() == "example/img:v1" + assert m.bind_mount.container == "/c" + + +def test_manifest_load_missing_required_field_raises(tmp_path): + payload = {"protocol": 1} # everything else missing + p = tmp_path / "bad.json" + p.write_text(json.dumps(payload)) + with pytest.raises((KeyError, TypeError, ValueError)): + load_engine_manifest(p) + + +def test_manifest_load_unknown_protocol_version_raises(tmp_path): + payload_base = { + "protocol": 999, # higher than anything m-cli understands + "image": "x", + "default_tag": "y", + "container": "z", + "bind_mount": {"host": "/h", "container": "/c", "mode": "rw"}, + "compose_file": "compose.yml", + "repo_url": "https://example.com", + "min_docker": "20.10", + "ydb_version": "r2.02", + "run_args": { + "hostname": "x", + "working_dir": "/c", + "restart": "unless-stopped", + "volumes": [], + "command": ["sleep", "infinity"], + }, + "verified_on": "2026-05-11", + } + p = tmp_path / "future.json" + p.write_text(json.dumps(payload_base)) + with pytest.raises(ValueError, match="protocol"): + load_engine_manifest(p) From 5016b2da7b512ee19c4ae821dac8cac50eb65842 Mon Sep 17 00:00:00 2001 From: Rafael Richards Date: Mon, 11 May 2026 11:27:31 -0400 Subject: [PATCH 5/5] engine: m engine subcommand family + DockerDriver (Phase 2) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Implements Phase 2 of the m-engine plan from m-test-engine repo (docs/m-engine-implementation-plan.md). Coordinated with m-test-engine commit 694094e which flipped the runtime bind mount from /work to /m-work. * src/m_cli/engine_driver.py — EngineDriver Protocol + DockerDriver. All command construction reads from the vendored manifest (no hardcoded strings). Injectable `runner` for tests so no real docker calls fire. Compose-first start path with docker-run fallback reconstructed from manifest.run_args. * src/m_cli/engine_cli.py — argparse + dispatcher for 12 verbs: status (text + --json), install, start, stop, restart, logs [--follow], shell, exec , version, upgrade, reset [--confirm], capabilities --json. `set_driver_factory()` is the test injection point. * src/m_cli/engine.py — DockerEngine.bind_root default flipped to /m-work (matches manifest); stage_routines() learns to map host /m-work/ → container /m-work/ (legacy single-mount fallback retained for explicit bind_root overrides). * src/m_cli/cli.py — `m engine` registered via add_engine_arguments; appears in `m capabilities --json` (dist/commands.json regenerated). * docs/plugin-development.md — new "Engine drivers (out-of-tree)" section documenting the `m_cli_engines` entry-point group seam. Built-in `docker` driver is NOT registered through the group; out-of-tree drivers (future m-cli-iris-engine, etc.) register a class implementing EngineDriver. Locked under PLUGIN_API_VERSION=1. 35 new tests (17 driver + 16 CLI + 2 entry-point) across test_engine_driver.py and test_engine_cli.py, plus 3 in test_engine_transports.py and 1 in test_engine_manifest.py for the /m-work cutover. Full suite 1431 passing / 1 skipped (pre-existing). ruff + mypy clean on the new modules. Phase 2.4 (m doctor --fix delegation) deferred — design ambiguity around sudo'd fixes documented in the tracker narrative log. Co-Authored-By: Claude Opus 4.7 (1M context) --- dist/commands.json | 5 + docs/plugin-development.md | 22 ++ src/m_cli/cli.py | 4 + src/m_cli/engine.py | 37 ++- src/m_cli/engine_cli.py | 283 ++++++++++++++++++++ src/m_cli/engine_driver.py | 456 ++++++++++++++++++++++++++++++++ tests/test_engine_cli.py | 276 +++++++++++++++++++ tests/test_engine_driver.py | 305 +++++++++++++++++++++ tests/test_engine_manifest.py | 12 + tests/test_engine_transports.py | 28 ++ 10 files changed, 1418 insertions(+), 10 deletions(-) create mode 100644 src/m_cli/engine_cli.py create mode 100644 src/m_cli/engine_driver.py create mode 100644 tests/test_engine_cli.py create mode 100644 tests/test_engine_driver.py diff --git a/dist/commands.json b/dist/commands.json index 6d90e03..92c462d 100644 --- a/dist/commands.json +++ b/dist/commands.json @@ -159,6 +159,11 @@ ], "examples": [] }, + "engine": { + "purpose": "Manage the m-test-engine container (install/start/stop/...)", + "options": [], + "examples": [] + }, "errors": { "purpose": "List every U-STD* error code and the labels that raise it", "options": [ diff --git a/docs/plugin-development.md b/docs/plugin-development.md index c64b5a8..5c00034 100644 --- a/docs/plugin-development.md +++ b/docs/plugin-development.md @@ -131,6 +131,28 @@ def my_command(args): if no transport is available — let the exception propagate; the user gets a useful message. +## Engine drivers (out-of-tree) + +A second, narrower entry-point group exists for engine lifecycle +drivers: **`m_cli_engines`**. Out-of-tree drivers implement the +[`m_cli.engine_driver.EngineDriver`][] Protocol and register the class +(not a function) under their name: + +```toml +[project.entry-points."m_cli_engines"] +iris = "m_cli_iris_engine.driver:IrisDriver" +podman = "m_cli_podman_engine.driver:PodmanDriver" +``` + +The built-in `docker` driver is **not** registered through this group — +m-cli always falls back to `DockerDriver` when no override is +requested. The group is the seam for adding non-Docker engines without +forking core. `m_cli.engine_driver.discover_drivers()` walks the group +for `m engine`'s capability output and future driver-selection UX. + +Same `PLUGIN_API_VERSION = 1` envelope: the group name + the +`EngineDriver` Protocol shape are part of the v1 contract. + ## Versioning `m_cli.plugins.PLUGIN_API_VERSION` is currently `1`. We bump it on a diff --git a/src/m_cli/cli.py b/src/m_cli/cli.py index 75576b0..101eeee 100644 --- a/src/m_cli/cli.py +++ b/src/m_cli/cli.py @@ -21,6 +21,7 @@ from m_cli.doc.manifest import manifest_command from m_cli.doc.search import search_command from m_cli.doctor import doctor_command +from m_cli.engine_cli import add_engine_arguments from m_cli.fmt import fmt_command from m_cli.lint import lint_command from m_cli.lsp import lsp_command @@ -508,6 +509,9 @@ def build_parser() -> argparse.ArgumentParser: ) lsp_parser.set_defaults(func=lsp_command) + # `m engine` — lifecycle for m-test-engine container + add_engine_arguments(subparsers) + # `m doctor` doctor_parser = subparsers.add_parser( "doctor", diff --git a/src/m_cli/engine.py b/src/m_cli/engine.py index dc797b3..410a3de 100644 --- a/src/m_cli/engine.py +++ b/src/m_cli/engine.py @@ -169,18 +169,29 @@ def stage_routines(self, start: Path) -> str: # ── DockerEngine — YottaDB in a container (m-test-engine) ───────────── +_HOST_SHARED_ROOT = Path("/m-work") + + @dataclass(frozen=True) class DockerEngine: """Runs ``mumps`` via ``docker exec`` against a long-running container. The container is typically ``m-test-engine`` started via ``m-dev-tools/m-test-engine``'s compose file. The compose file - bind-mounts the consumer project's root as ``/work``, so the - in-container path for any project file is ``/work/``. + bind-mounts the host's ``/m-work`` directory (containing every + participating m-* repo checkout) as ``/m-work`` inside the + container. A consumer project at ``/m-work/m-cli/`` on the host is + therefore visible at ``/m-work/m-cli/`` inside the container — the + mapping is identity once you're under ``/m-work``. + + Consumers whose project root is **not** under host ``/m-work`` fall + through to the legacy single-mount path: ``bind_root`` is assumed + to be the project root in the container, so the user is responsible + for arranging that bind separately (e.g. via M_TEST_ENGINE_BIND). """ container: str = "m-test-engine" - bind_root: Path = field(default_factory=lambda: Path("/work")) + bind_root: Path = field(default_factory=lambda: Path("/m-work")) def _exec_prefix(self) -> list[str]: return ["docker", "exec", self.container] @@ -205,19 +216,25 @@ def build_direct_cmd(self, stage: str) -> list[str]: return [*self._exec_prefix(), "bash", "-lc", script] def stage_routines(self, start: Path) -> str: - """Return the in-container path the host project root is bound to. + """Return the in-container routine path(s) for the host project. - Assumes the project root mounts to ``self.bind_root`` (the - m-test-engine compose default). Routine dirs under it are - space-separated, mirroring LocalEngine semantics. + Shared-mount model: when the host project root lives under + ``/m-work``, the in-container path is ``self.bind_root / + ``. Otherwise falls back to + ``self.bind_root`` directly (legacy single-mount). """ - root = project_root(start) + root = project_root(start).resolve() + try: + rel = root.relative_to(_HOST_SHARED_ROOT.resolve()) + in_container_root = self.bind_root / rel + except ValueError: + in_container_root = self.bind_root dirs = [ - str(self.bind_root / sub) + str(in_container_root / sub) for sub in _ROUTINE_DIRS if (root / sub).is_dir() ] - return " ".join(dirs) if dirs else str(self.bind_root) + return " ".join(dirs) if dirs else str(in_container_root) # ── SSHEngine — remote YottaDB over SSH (legacy / vista-meta) ───────── diff --git a/src/m_cli/engine_cli.py b/src/m_cli/engine_cli.py new file mode 100644 index 0000000..a2de76e --- /dev/null +++ b/src/m_cli/engine_cli.py @@ -0,0 +1,283 @@ +"""`m engine` subcommand surface — nested verbs over an EngineDriver. + +Wires argparse subparsers for each lifecycle verb (status / install / +start / stop / restart / logs / shell / exec / version / upgrade / +reset / capabilities), dispatches to the active +:class:`m_cli.engine_driver.EngineDriver`, and emits status output in +text or JSON. + +The driver itself is resolved by :func:`select_driver` — the built-in +``DockerDriver`` is the only option in core; out-of-tree drivers will +register via the ``m_cli_engines`` entry-point group (Phase 2.3) and +hook in here without touching this file. + +See ``m-test-engine/docs/m-engine-implementation-plan.md`` §2.1. +""" + +from __future__ import annotations + +import argparse +import json +from typing import Callable + +from m_cli.engine_driver import DockerDriver, EngineDriver +from m_cli.engine_manifest import load_engine_manifest + +# Cached driver factory so tests can inject a fake without touching +# Docker. Production code uses the default (DockerDriver from +# manifest); tests monkeypatch :data:`_DRIVER_FACTORY`. +_DriverFactory = Callable[[], EngineDriver] + + +def _default_driver_factory() -> EngineDriver: + manifest = load_engine_manifest() + return DockerDriver(manifest=manifest) + + +_DRIVER_FACTORY: _DriverFactory = _default_driver_factory + + +def set_driver_factory(factory: _DriverFactory) -> None: + """Override the driver resolution (tests, future entry-point integration).""" + global _DRIVER_FACTORY + _DRIVER_FACTORY = factory + + +def select_driver() -> EngineDriver: + return _DRIVER_FACTORY() + + +# ── argparse wiring ────────────────────────────────────────────────── + + +def add_engine_arguments(subparsers: argparse._SubParsersAction) -> None: + """Register the ``engine`` subcommand tree on a top-level parser.""" + engine_parser = subparsers.add_parser( + "engine", + help="Manage the m-test-engine container (install/start/stop/...)", + description=( + "Lifecycle management for the canonical m-test-engine " + "Docker container. Every verb shells out to docker / docker " + "compose; configuration is driven by the vendored " + "dist/m-test-engine.json contract. See `m doctor` for " + "diagnostics with fix-pointers; `m engine status` is the " + "single-line health summary." + ), + ) + actions = engine_parser.add_subparsers( + dest="engine_action", + required=True, + ) + + # status + status_p = actions.add_parser( + "status", + help="Print container/image/daemon state", + ) + status_p.add_argument("--json", action="store_true", help="Emit JSON") + status_p.set_defaults(func=_cmd_status) + + # install + install_p = actions.add_parser( + "install", + help="Pull the canonical engine image (`docker pull`)", + ) + install_p.set_defaults(func=_cmd_install) + + # start + start_p = actions.add_parser( + "start", + help="Start the engine container (compose-first; docker-run fallback)", + ) + start_p.set_defaults(func=_cmd_start) + + # stop + stop_p = actions.add_parser( + "stop", + help="Stop the engine container (globals volume preserved)", + ) + stop_p.set_defaults(func=_cmd_stop) + + # restart + restart_p = actions.add_parser( + "restart", + help="Stop + start", + ) + restart_p.set_defaults(func=_cmd_restart) + + # logs + logs_p = actions.add_parser( + "logs", + help="Print container logs (use --follow to stream)", + ) + logs_p.add_argument("--follow", "-f", action="store_true", help="Stream logs continuously") + logs_p.set_defaults(func=_cmd_logs) + + # shell + shell_p = actions.add_parser( + "shell", + help="Interactive bash shell inside the container", + ) + shell_p.set_defaults(func=_cmd_shell) + + # exec + exec_p = actions.add_parser( + "exec", + help="Run a one-shot M command via `mumps -run %%XCMD`", + ) + exec_p.add_argument( + "m_cmd", + help="M command to execute (e.g. 'write $ZVERSION,!')", + ) + exec_p.set_defaults(func=_cmd_exec) + + # version + version_p = actions.add_parser( + "version", + help="Print manifest-declared vs container-reported versions", + ) + version_p.set_defaults(func=_cmd_version) + + # upgrade + upgrade_p = actions.add_parser( + "upgrade", + help="Pull latest image and recreate container (globals preserved)", + ) + upgrade_p.set_defaults(func=_cmd_upgrade) + + # reset (destructive) + reset_p = actions.add_parser( + "reset", + help="DESTRUCTIVE: stop + remove + drop globals volume", + description=( + "Wipes the running container AND the persistent globals " + "volume. Useful when a stuck global/lock state poisons " + "tests. Refuses to run without --confirm." + ), + ) + reset_p.add_argument( + "--confirm", + action="store_true", + help="Required acknowledgement that this is destructive", + ) + reset_p.set_defaults(func=_cmd_reset) + + # capabilities — mirrors top-level `m capabilities` + caps_p = actions.add_parser( + "capabilities", + help="Emit the engine namespace's machine-readable capabilities", + ) + caps_p.add_argument("--json", action="store_true", help="JSON (default: pretty-printed JSON)") + caps_p.set_defaults(func=_cmd_capabilities) + + +def engine_command(args: argparse.Namespace) -> int: + """Top-level dispatcher (matches the pattern of every other subcommand).""" + return args.func(args) + + +# ── verb handlers ──────────────────────────────────────────────────── + + +def _cmd_status(args: argparse.Namespace) -> int: + driver = select_driver() + status = driver.status() + if getattr(args, "json", False): + print(json.dumps(status.to_dict(), indent=2)) + else: + marks = {True: "✓", False: "✗", None: "-"} + print(f"driver: {status.driver}") + print(f"image: {status.image_ref}") + print(f"container: {status.container}") + print(f" cli installed: {marks[status.installed]}") + print(f" daemon up: {marks[status.daemon_reachable]}") + print(f" image present: {marks[status.image_present]}") + print(f" container up: {marks[status.container_running]}") + print(f" healthy: {marks[status.container_healthy]}") + return 0 if status.container_running else 1 + + +def _cmd_install(args: argparse.Namespace) -> int: + return select_driver().install() + + +def _cmd_start(args: argparse.Namespace) -> int: + return select_driver().start() + + +def _cmd_stop(args: argparse.Namespace) -> int: + return select_driver().stop() + + +def _cmd_restart(args: argparse.Namespace) -> int: + return select_driver().restart() + + +def _cmd_logs(args: argparse.Namespace) -> int: + return select_driver().logs(follow=getattr(args, "follow", False)) + + +def _cmd_shell(args: argparse.Namespace) -> int: + return select_driver().shell() + + +def _cmd_exec(args: argparse.Namespace) -> int: + return select_driver().exec(args.m_cmd) + + +def _cmd_version(args: argparse.Namespace) -> int: + return select_driver().version() + + +def _cmd_upgrade(args: argparse.Namespace) -> int: + return select_driver().upgrade() + + +def _cmd_reset(args: argparse.Namespace) -> int: + return select_driver().reset(confirm=args.confirm) + + +def _cmd_capabilities(args: argparse.Namespace) -> int: + """Emit the engine namespace as JSON. Mirrors `m capabilities --json`.""" + manifest = load_engine_manifest() + payload = { + "namespace": "engine", + "driver": "docker", + "manifest": { + "protocol": manifest.protocol, + "image": manifest.image, + "default_tag": manifest.default_tag, + "image_ref": manifest.image_ref(), + "container": manifest.container, + "ydb_version": manifest.ydb_version, + "bind_mount": { + "host": manifest.bind_mount.host, + "container": manifest.bind_mount.container, + "mode": manifest.bind_mount.mode, + }, + }, + "verbs": [ + {"name": "status", "destructive": False, "read_only": True}, + {"name": "install", "destructive": False, "read_only": False}, + {"name": "start", "destructive": False, "read_only": False}, + {"name": "stop", "destructive": False, "read_only": False}, + {"name": "restart", "destructive": False, "read_only": False}, + {"name": "logs", "destructive": False, "read_only": True}, + {"name": "shell", "destructive": False, "read_only": False}, + {"name": "exec", "destructive": False, "read_only": False}, + {"name": "version", "destructive": False, "read_only": True}, + {"name": "upgrade", "destructive": False, "read_only": False}, + {"name": "reset", "destructive": True, "read_only": False, "requires_confirm": True}, + {"name": "capabilities", "destructive": False, "read_only": True}, + ], + } + print(json.dumps(payload, indent=2)) + return 0 + + +__all__ = [ + "add_engine_arguments", + "engine_command", + "select_driver", + "set_driver_factory", +] diff --git a/src/m_cli/engine_driver.py b/src/m_cli/engine_driver.py new file mode 100644 index 0000000..e213149 --- /dev/null +++ b/src/m_cli/engine_driver.py @@ -0,0 +1,456 @@ +"""Engine lifecycle drivers — the ``m engine`` subcommand backend. + +Distinct from :mod:`m_cli.engine` (which handles **routine execution** — +"given a running engine, build the command to exec inside it"). This +module handles **engine lifecycle** — "install / start / stop / inspect +the engine itself". + +The :class:`EngineDriver` Protocol is the public seam for out-of-tree +drivers (IRIS, podman, …). The built-in :class:`DockerDriver` shells +out to ``docker`` / ``docker compose`` and is driven entirely by the +vendored :mod:`m_cli.engine_manifest`. Tests inject a custom +``runner`` to avoid actually running ``docker`` calls. + +Implements the Phase 2 surface decided in +``m-test-engine/docs/m-engine-implementation-plan.md``. +""" + +from __future__ import annotations + +import importlib.metadata as _md +import shutil +import subprocess +from dataclasses import dataclass, field +from typing import Callable, Protocol + +from m_cli.engine_manifest import EngineManifest + +# Entry-point group that out-of-tree drivers register against. +# Locked under PLUGIN_API_VERSION = 1 (see m_cli.plugins). +# +# Example downstream pyproject.toml: +# +# [project.entry-points."m_cli_engines"] +# iris = "m_cli_iris_engine.driver:IrisDriver" +# +# The named attribute must be a class implementing :class:`EngineDriver`. +ENGINE_DRIVER_ENTRY_POINT_GROUP = "m_cli_engines" + + +# Result type from a single shell-out: returncode + captured stdout/stderr. +@dataclass(frozen=True) +class CommandResult: + returncode: int + stdout: str = "" + stderr: str = "" + + @property + def ok(self) -> bool: + return self.returncode == 0 + + +# A `runner` is any callable that takes argv + optional kwargs and returns +# a CommandResult. The default runner shells out via subprocess; tests +# inject a fake that returns canned results without touching the real +# Docker daemon. +Runner = Callable[..., CommandResult] + + +def default_runner( + argv: list[str], + *, + capture: bool = True, + timeout: float | None = None, +) -> CommandResult: + """Shell-out via :mod:`subprocess`. Used by every built-in driver.""" + try: + result = subprocess.run( + argv, + capture_output=capture, + text=True, + timeout=timeout, + check=False, + ) + except FileNotFoundError as exc: + return CommandResult(returncode=127, stderr=str(exc)) + except subprocess.TimeoutExpired as exc: + # text=True is set on subprocess.run above so stdout/stderr are + # str when the call completes; on timeout, they remain bytes|None + # per CPython stdlib. Decode defensively. + out_raw: bytes | str | None = exc.stdout + err_raw: bytes | str | None = exc.stderr + out_str: str = ( + out_raw.decode(errors="replace") if isinstance(out_raw, bytes) else (out_raw or "") + ) + err_str: str = ( + err_raw.decode(errors="replace") if isinstance(err_raw, bytes) else (err_raw or "") + ) + return CommandResult( + returncode=124, + stdout=out_str, + stderr=err_str + f"\ntimeout after {timeout}s", + ) + return CommandResult( + returncode=result.returncode, + stdout=result.stdout or "", + stderr=result.stderr or "", + ) + + +# ── Status payload ─────────────────────────────────────────────────── + + +@dataclass(frozen=True) +class EngineStatus: + """Snapshot of the engine state. Used by ``m engine status`` text/JSON output.""" + + driver: str # e.g. "docker" + installed: bool # CLI present + daemon_reachable: bool # docker info reachable (or n/a) + image_present: bool # image is pulled locally + container_running: bool # canonical container is up + container_healthy: bool | None # None until Phase 3 healthcheck lands + image_ref: str + container: str + + def to_dict(self) -> dict: + return { + "driver": self.driver, + "installed": self.installed, + "daemon_reachable": self.daemon_reachable, + "image_present": self.image_present, + "container_running": self.container_running, + "container_healthy": self.container_healthy, + "image_ref": self.image_ref, + "container": self.container, + } + + +# ── Protocol ───────────────────────────────────────────────────────── + + +class EngineDriver(Protocol): + """Public lifecycle protocol for engine drivers. + + Locked under ``PLUGIN_API_VERSION = 1``. Out-of-tree drivers + (e.g. ``m-cli-iris-engine``) register via the ``m_cli_engines`` + Python entry-point group; the built-in :class:`DockerDriver` is the + only driver registered in core. + + Every method returns an exit code (``0`` for success) so callers + can compose lifecycle ops with shell-style semantics. Output that + should reach the user is printed by the driver; the caller decides + whether to also act on it. + """ + + name: str # e.g. "docker" + + def status(self) -> EngineStatus: ... + def install(self) -> int: ... + def start(self) -> int: ... + def stop(self) -> int: ... + def restart(self) -> int: ... + def logs(self, follow: bool = False) -> int: ... + def shell(self) -> int: ... + def exec(self, m_cmd: str) -> int: ... + def version(self) -> int: ... + def upgrade(self) -> int: ... + def reset(self, *, confirm: bool = False) -> int: ... + + +# ── DockerDriver — the only built-in driver ───────────────────────── + + +@dataclass +class DockerDriver: + """Default driver: shells out to ``docker`` / ``docker compose``. + + All command construction reads from ``manifest`` — image refs, + container name, compose file path, run_args fallback. No hardcoded + strings. + """ + + manifest: EngineManifest + runner: Runner = field(default=default_runner) + name: str = "docker" + + # ── primitives ─────────────────────────────────────────────────── + + def _docker_available(self) -> bool: + return shutil.which("docker") is not None + + def _daemon_reachable(self) -> bool: + if not self._docker_available(): + return False + return self.runner(["docker", "info"], capture=True, timeout=5).ok + + def _image_present(self) -> bool: + return self.runner( + ["docker", "image", "inspect", self.manifest.image_ref()], + capture=True, + timeout=5, + ).ok + + def _container_running(self) -> bool: + result = self.runner( + [ + "docker", + "ps", + "--filter", + f"name=^{self.manifest.container}$", + "--format", + "{{.Names}}", + ], + capture=True, + timeout=5, + ) + return result.ok and self.manifest.container in result.stdout.split() + + def _has_compose_plugin(self) -> bool: + return self.runner(["docker", "compose", "version"], capture=True, timeout=5).ok + + # ── lifecycle verbs ────────────────────────────────────────────── + + def status(self) -> EngineStatus: + installed = self._docker_available() + daemon = installed and self._daemon_reachable() + image = daemon and self._image_present() + running = daemon and self._container_running() + return EngineStatus( + driver=self.name, + installed=installed, + daemon_reachable=daemon, + image_present=image, + container_running=running, + container_healthy=None, # Phase 3 will source this from HEALTHCHECK + image_ref=self.manifest.image_ref(), + container=self.manifest.container, + ) + + def install(self) -> int: + """Pull the canonical engine image.""" + result = self.runner( + ["docker", "pull", self.manifest.image_ref()], + capture=False, + ) + return result.returncode + + def _start_via_compose(self) -> int: + result = self.runner( + [ + "docker", + "compose", + "-f", + self.manifest.compose_file, + "up", + "-d", + ], + capture=False, + ) + return result.returncode + + def _start_via_run(self) -> int: + """Fallback when the compose plugin is unavailable. + + Constructs an equivalent ``docker run`` from the manifest's + ``run_args`` block + ``bind_mount``. Functionally equivalent to + compose for the single-service m-test-engine case. + """ + bm = self.manifest.bind_mount + ra = self.manifest.run_args + argv = [ + "docker", + "run", + "-d", + "--name", + self.manifest.container, + "--hostname", + ra.hostname, + "--restart", + ra.restart, + "-w", + ra.working_dir, + "-v", + f"{bm.host}:{bm.container}:{bm.mode}", + ] + for vol in ra.volumes: + argv.extend(["-v", f"{vol.name}:{vol.target}"]) + argv.append(self.manifest.image_ref()) + argv.extend(ra.command) + result = self.runner(argv, capture=False) + return result.returncode + + def start(self) -> int: + """Start the engine container; compose-first, ``docker run`` fallback.""" + if self._has_compose_plugin(): + return self._start_via_compose() + return self._start_via_run() + + def stop(self) -> int: + """Stop the engine container. Globals volume preserved.""" + if self._has_compose_plugin(): + result = self.runner( + [ + "docker", + "compose", + "-f", + self.manifest.compose_file, + "down", + ], + capture=False, + ) + return result.returncode + result = self.runner( + ["docker", "stop", self.manifest.container], + capture=False, + ) + return result.returncode + + def restart(self) -> int: + rc = self.stop() + if rc != 0: + return rc + return self.start() + + def logs(self, follow: bool = False) -> int: + argv = ["docker", "logs"] + if follow: + argv.append("--follow") + argv.append(self.manifest.container) + # follow streams; never capture + result = self.runner(argv, capture=not follow) + if not follow and result.stdout: + print(result.stdout, end="") + return result.returncode + + def shell(self) -> int: + """Drop into an interactive bash shell inside the container.""" + # Interactive: tty must be inherited. runner caller is responsible + # for not capturing; default_runner with capture=False suffices. + result = self.runner( + ["docker", "exec", "-it", self.manifest.container, "bash"], + capture=False, + ) + return result.returncode + + def exec(self, m_cmd: str) -> int: + """One-shot M command via ``mumps -run %XCMD``. + + Useful for ad-hoc inspection. Exit code matters; output goes to + stdout/stderr unchanged. + """ + import shlex + + # Inside the container, bash -lc sources /etc/profile.d/ydb-env.sh + # so $ydb_dist is set. + inner = f"$ydb_dist/mumps -run %XCMD {shlex.quote(m_cmd)}" + result = self.runner( + ["docker", "exec", self.manifest.container, "bash", "-lc", inner], + capture=False, + ) + return result.returncode + + def version(self) -> int: + """Print manifest-declared vs container-reported versions.""" + m = self.manifest + print(f"manifest: image={m.image_ref()} ydb={m.ydb_version} protocol={m.protocol}") + result = self.runner( + ["docker", "inspect", "--format", "{{.Image}}", m.container], + capture=True, + timeout=5, + ) + if result.ok and result.stdout.strip(): + print(f"container: image-id={result.stdout.strip()}") + else: + print("container: not running (manifest is the only available version)") + return 0 + + def upgrade(self) -> int: + """Pull latest image and recreate container. + + Equivalent to ``install`` + ``stop`` + ``start``. Globals + volume is preserved. + """ + rc = self.install() + if rc != 0: + return rc + rc = self.stop() + if rc != 0: + return rc + return self.start() + + def reset(self, *, confirm: bool = False) -> int: + """Destructive: stop + remove + drop globals volume. + + Refuses to run without ``confirm=True``. After reset, the next + ``start`` produces a fresh container with empty globals — useful + when a stuck global/lock state poisons tests. + """ + if not confirm: + print( + "refusing: `m engine reset` is destructive (drops the " + "globals volume). Re-run with --confirm.", + flush=True, + ) + return 2 + if self._has_compose_plugin(): + result = self.runner( + [ + "docker", + "compose", + "-f", + self.manifest.compose_file, + "down", + "-v", + ], + capture=False, + ) + return result.returncode + # docker run fallback: stop + rm + volume rm + self.runner(["docker", "stop", self.manifest.container], capture=False) + self.runner(["docker", "rm", self.manifest.container], capture=False) + for vol in self.manifest.run_args.volumes: + self.runner(["docker", "volume", "rm", vol.name], capture=False) + return 0 + + +# ── Entry-point driver discovery ───────────────────────────────────── + + +@dataclass(frozen=True) +class DriverInfo: + """Metadata for a discovered engine driver.""" + + name: str + package: str + entry_point: str + + +def discover_drivers() -> list[DriverInfo]: + """Walk the ``m_cli_engines`` entry-point group. + + The built-in ``docker`` driver is **not** included here — only + out-of-tree drivers. m-cli's CLI always falls back to + :class:`DockerDriver` when no override is requested. Today this + function returns an empty list on most installs; the seam exists so + a future ``m-cli-iris-engine`` package can register. + """ + out: list[DriverInfo] = [] + for ep in _md.entry_points(group=ENGINE_DRIVER_ENTRY_POINT_GROUP): + dist = getattr(ep, "dist", None) + pkg = getattr(dist, "name", None) or ep.value.split(":", 1)[0].split(".", 1)[0] + out.append(DriverInfo(name=ep.name, package=pkg, entry_point=ep.value)) + return out + + +__all__ = [ + "CommandResult", + "DockerDriver", + "DriverInfo", + "ENGINE_DRIVER_ENTRY_POINT_GROUP", + "EngineDriver", + "EngineStatus", + "Runner", + "default_runner", + "discover_drivers", +] diff --git a/tests/test_engine_cli.py b/tests/test_engine_cli.py new file mode 100644 index 0000000..b1d534f --- /dev/null +++ b/tests/test_engine_cli.py @@ -0,0 +1,276 @@ +"""Tests for the ``m engine`` subcommand surface. + +The CLI module is a thin argparse + dispatch layer over +:class:`~m_cli.engine_driver.EngineDriver`. Tests inject a fake driver +via ``set_driver_factory`` so no docker calls fire. + +Phase 2 of the m-engine plan +(see m-test-engine/docs/m-engine-implementation-plan.md §2.1). +""" + +from __future__ import annotations + +import argparse +import json + +import pytest + +from m_cli.engine_cli import set_driver_factory +from m_cli.engine_driver import EngineStatus + + +class FakeDriver: + """Records every verb invocation. Verbs return a configurable rc.""" + + name = "docker" + + def __init__(self, *, status_payload: EngineStatus | None = None, rc: int = 0): + self.calls: list[tuple[str, tuple, dict]] = [] + self.rc = rc + self._status = status_payload or EngineStatus( + driver="docker", + installed=True, + daemon_reachable=True, + image_present=True, + container_running=True, + container_healthy=None, + image_ref="ghcr.io/example:v1", + container="m-test-engine", + ) + + def _record(self, name, *args, **kw): + self.calls.append((name, args, kw)) + return self.rc + + def status(self): + self.calls.append(("status", (), {})) + return self._status + + def install(self): + return self._record("install") + + def start(self): + return self._record("start") + + def stop(self): + return self._record("stop") + + def restart(self): + return self._record("restart") + + def logs(self, follow=False): + return self._record("logs", follow=follow) + + def shell(self): + return self._record("shell") + + def exec(self, m_cmd): + return self._record("exec", m_cmd) + + def version(self): + return self._record("version") + + def upgrade(self): + return self._record("upgrade") + + def reset(self, *, confirm=False): + return self._record("reset", confirm=confirm) + + +@pytest.fixture +def fake_driver(): + drv = FakeDriver() + set_driver_factory(lambda: drv) + try: + yield drv + finally: + # restore default factory + from m_cli.engine_cli import _default_driver_factory + + set_driver_factory(_default_driver_factory) + + +def _ns(**kw) -> argparse.Namespace: + return argparse.Namespace(**kw) + + +# ── status ─────────────────────────────────────────────────────────── + + +def test_engine_status_text_output_shows_container_up(fake_driver, capsys): + from m_cli.engine_cli import _cmd_status + + rc = _cmd_status(_ns(json=False)) + out = capsys.readouterr().out + assert rc == 0 # container_running=True in default fake status + assert "m-test-engine" in out + assert "ghcr.io/example:v1" in out + + +def test_engine_status_returns_one_when_container_down(capsys): + from m_cli.engine_cli import _cmd_status + + drv = FakeDriver( + status_payload=EngineStatus( + driver="docker", + installed=True, + daemon_reachable=True, + image_present=True, + container_running=False, + container_healthy=None, + image_ref="x:y", + container="m-test-engine", + ) + ) + set_driver_factory(lambda: drv) + try: + rc = _cmd_status(_ns(json=False)) + assert rc == 1 + finally: + from m_cli.engine_cli import _default_driver_factory + + set_driver_factory(_default_driver_factory) + + +def test_engine_status_json_output(fake_driver, capsys): + from m_cli.engine_cli import _cmd_status + + rc = _cmd_status(_ns(json=True)) + out = capsys.readouterr().out + payload = json.loads(out) + assert payload["driver"] == "docker" + assert payload["container"] == "m-test-engine" + assert payload["container_running"] is True + assert rc == 0 + + +# ── install / start / stop / restart ───────────────────────────────── + + +def test_engine_install_invokes_driver_install(fake_driver): + from m_cli.engine_cli import _cmd_install + + rc = _cmd_install(_ns()) + assert rc == 0 + assert any(c[0] == "install" for c in fake_driver.calls) + + +def test_engine_start_invokes_driver_start(fake_driver): + from m_cli.engine_cli import _cmd_start + + _cmd_start(_ns()) + assert any(c[0] == "start" for c in fake_driver.calls) + + +def test_engine_stop_invokes_driver_stop(fake_driver): + from m_cli.engine_cli import _cmd_stop + + _cmd_stop(_ns()) + assert any(c[0] == "stop" for c in fake_driver.calls) + + +def test_engine_restart_invokes_driver_restart(fake_driver): + from m_cli.engine_cli import _cmd_restart + + _cmd_restart(_ns()) + assert any(c[0] == "restart" for c in fake_driver.calls) + + +# ── logs / shell / exec / version / upgrade ────────────────────────── + + +def test_engine_logs_forwards_follow_flag(fake_driver): + from m_cli.engine_cli import _cmd_logs + + _cmd_logs(_ns(follow=True)) + call = next(c for c in fake_driver.calls if c[0] == "logs") + assert call[2]["follow"] is True + + +def test_engine_shell_invokes_driver_shell(fake_driver): + from m_cli.engine_cli import _cmd_shell + + _cmd_shell(_ns()) + assert any(c[0] == "shell" for c in fake_driver.calls) + + +def test_engine_exec_passes_m_command(fake_driver): + from m_cli.engine_cli import _cmd_exec + + _cmd_exec(_ns(m_cmd="write 1+2,!")) + call = next(c for c in fake_driver.calls if c[0] == "exec") + assert call[1] == ("write 1+2,!",) + + +def test_engine_version_invokes_driver_version(fake_driver): + from m_cli.engine_cli import _cmd_version + + _cmd_version(_ns()) + assert any(c[0] == "version" for c in fake_driver.calls) + + +def test_engine_upgrade_invokes_driver_upgrade(fake_driver): + from m_cli.engine_cli import _cmd_upgrade + + _cmd_upgrade(_ns()) + assert any(c[0] == "upgrade" for c in fake_driver.calls) + + +# ── reset (destructive) ────────────────────────────────────────────── + + +def test_engine_reset_forwards_confirm_flag(fake_driver): + from m_cli.engine_cli import _cmd_reset + + _cmd_reset(_ns(confirm=True)) + call = next(c for c in fake_driver.calls if c[0] == "reset") + assert call[2]["confirm"] is True + + +def test_engine_reset_without_confirm_still_forwards_false(fake_driver): + from m_cli.engine_cli import _cmd_reset + + _cmd_reset(_ns(confirm=False)) + call = next(c for c in fake_driver.calls if c[0] == "reset") + assert call[2]["confirm"] is False + + +# ── capabilities ───────────────────────────────────────────────────── + + +def test_engine_capabilities_emits_manifest_summary(capsys): + from m_cli.engine_cli import _cmd_capabilities + + rc = _cmd_capabilities(_ns(json=True)) + out = capsys.readouterr().out + payload = json.loads(out) + assert rc == 0 + assert payload["namespace"] == "engine" + assert payload["driver"] == "docker" + assert "manifest" in payload + assert payload["manifest"]["protocol"] == 1 + # verbs section advertises the safe vs destructive verbs + verbs = {v["name"]: v for v in payload["verbs"]} + assert verbs["status"]["read_only"] is True + assert verbs["reset"]["destructive"] is True + assert verbs["reset"].get("requires_confirm") is True + + +# ── argparse wiring (top-level dispatch) ───────────────────────────── + + +def test_main_dispatches_engine_status_via_argparse(monkeypatch, capsys): + from m_cli.cli import main + + drv = FakeDriver() + set_driver_factory(lambda: drv) + try: + rc = main(["engine", "status", "--json"]) + out = capsys.readouterr().out + payload = json.loads(out) + assert rc == 0 + assert payload["container_running"] is True + finally: + from m_cli.engine_cli import _default_driver_factory + + set_driver_factory(_default_driver_factory) diff --git a/tests/test_engine_driver.py b/tests/test_engine_driver.py new file mode 100644 index 0000000..31ba20d --- /dev/null +++ b/tests/test_engine_driver.py @@ -0,0 +1,305 @@ +"""Tests for the EngineDriver Protocol + built-in DockerDriver. + +The driver layer manages engine **lifecycle** (install / start / stop / +inspect) — distinct from :mod:`m_cli.engine` which handles routine +execution against a running engine. The two collaborate but neither +imports the other. + +Tests inject a fake ``runner`` so no actual ``docker`` calls fire. The +fake records every argv it sees so we can assert command shape + +manifest-derived values. + +Phase 2 of the m-engine plan +(see m-test-engine/docs/m-engine-implementation-plan.md §2.2). +""" + +from __future__ import annotations + +import pytest + +from m_cli.engine_driver import ( + CommandResult, + DockerDriver, + EngineDriver, + EngineStatus, +) +from m_cli.engine_manifest import load_engine_manifest + + +@pytest.fixture +def manifest(): + return load_engine_manifest() + + +@pytest.fixture +def fake_runner(): + """Returns a runner callable that records calls and dispatches via a rules dict. + + Tests configure ``world.rule(prefix=tuple_of_args, result=CommandResult)`` + or ``world.default = CommandResult(...)`` to drive specific argv + matches without faking subprocess module-wide. + """ + calls: list[list[str]] = [] + rules: list[tuple[tuple[str, ...], CommandResult]] = [] + default = CommandResult(returncode=0, stdout="", stderr="") + + def _runner(argv, *, capture=True, timeout=None): + calls.append(list(argv)) + for prefix, result in rules: + if tuple(argv[: len(prefix)]) == prefix: + return result + return default + + class _World: + def __init__(self): + self.calls = calls + self.rules = rules + self.runner = _runner + + def rule(self, prefix, result): + self.rules.append((tuple(prefix), result)) + + def set_default(self, result): + nonlocal default + default = result + + return _World() + + +# ── Protocol shape ─────────────────────────────────────────────────── + + +def test_docker_driver_satisfies_engine_driver_protocol(manifest, fake_runner): + drv: EngineDriver = DockerDriver(manifest=manifest, runner=fake_runner.runner) + assert drv.name == "docker" + + +def test_engine_status_dataclass_roundtrips_to_dict(): + s = EngineStatus( + driver="docker", + installed=True, + daemon_reachable=True, + image_present=False, + container_running=False, + container_healthy=None, + image_ref="ghcr.io/example:tag", + container="m-test-engine", + ) + d = s.to_dict() + assert d["driver"] == "docker" + assert d["installed"] is True + assert d["container_healthy"] is None # serialised as null in JSON + + +# ── DockerDriver.status() ──────────────────────────────────────────── + + +def test_docker_driver_status_uses_manifest_image_and_container(manifest, fake_runner): + drv = DockerDriver(manifest=manifest, runner=fake_runner.runner) + s = drv.status() + assert s.image_ref == manifest.image_ref() + assert s.container == manifest.container + + +def test_docker_driver_status_reports_container_not_running_when_ps_empty(manifest, fake_runner): + # docker info → ok; image inspect → ok; docker ps → empty stdout + fake_runner.rule(prefix=["docker", "info"], result=CommandResult(returncode=0)) + fake_runner.rule( + prefix=["docker", "image", "inspect"], + result=CommandResult(returncode=0), + ) + fake_runner.rule( + prefix=["docker", "ps"], + result=CommandResult(returncode=0, stdout=""), + ) + drv = DockerDriver(manifest=manifest, runner=fake_runner.runner) + s = drv.status() + assert s.container_running is False + + +def test_docker_driver_status_reports_running_when_ps_returns_name(manifest, fake_runner): + fake_runner.rule(prefix=["docker", "info"], result=CommandResult(returncode=0)) + fake_runner.rule( + prefix=["docker", "image", "inspect"], + result=CommandResult(returncode=0), + ) + fake_runner.rule( + prefix=["docker", "ps"], + result=CommandResult(returncode=0, stdout=f"{manifest.container}\n"), + ) + drv = DockerDriver(manifest=manifest, runner=fake_runner.runner) + s = drv.status() + assert s.container_running is True + + +# ── install / start (compose-first, run fallback) ──────────────────── + + +def test_docker_driver_install_pulls_image_ref(manifest, fake_runner): + drv = DockerDriver(manifest=manifest, runner=fake_runner.runner) + rc = drv.install() + assert rc == 0 + pulls = [c for c in fake_runner.calls if c[:2] == ["docker", "pull"]] + assert len(pulls) == 1 + assert pulls[0] == ["docker", "pull", manifest.image_ref()] + + +def test_docker_driver_start_uses_compose_when_plugin_available(manifest, fake_runner): + fake_runner.rule( + prefix=["docker", "compose", "version"], + result=CommandResult(returncode=0), + ) + drv = DockerDriver(manifest=manifest, runner=fake_runner.runner) + drv.start() + compose_ups = [c for c in fake_runner.calls if c[:3] == ["docker", "compose", "-f"]] + assert any("up" in c for c in compose_ups) + + +def test_docker_driver_start_falls_back_to_docker_run_when_compose_absent(manifest, fake_runner): + # docker compose version → nonzero → fall back + fake_runner.rule( + prefix=["docker", "compose", "version"], + result=CommandResult(returncode=1), + ) + drv = DockerDriver(manifest=manifest, runner=fake_runner.runner) + drv.start() + runs = [c for c in fake_runner.calls if c[:2] == ["docker", "run"]] + assert len(runs) == 1 + run = runs[0] + # Container name + image from manifest + assert manifest.container in run + assert manifest.image_ref() in run + # Bind mount from manifest fields + bm = manifest.bind_mount + expected_bind = f"{bm.host}:{bm.container}:{bm.mode}" + assert expected_bind in run + # Secondary volumes + for vol in manifest.run_args.volumes: + assert f"{vol.name}:{vol.target}" in run + # working_dir from run_args + assert manifest.run_args.working_dir in run + + +# ── stop / restart / reset ─────────────────────────────────────────── + + +def test_docker_driver_stop_uses_compose_down_when_available(manifest, fake_runner): + fake_runner.rule( + prefix=["docker", "compose", "version"], + result=CommandResult(returncode=0), + ) + drv = DockerDriver(manifest=manifest, runner=fake_runner.runner) + drv.stop() + downs = [c for c in fake_runner.calls if c[:3] == ["docker", "compose", "-f"] and "down" in c] + assert len(downs) == 1 + + +def test_docker_driver_stop_falls_back_to_docker_stop(manifest, fake_runner): + fake_runner.rule( + prefix=["docker", "compose", "version"], + result=CommandResult(returncode=1), + ) + drv = DockerDriver(manifest=manifest, runner=fake_runner.runner) + drv.stop() + stops = [c for c in fake_runner.calls if c[:2] == ["docker", "stop"]] + assert stops and manifest.container in stops[0] + + +def test_docker_driver_reset_refuses_without_confirm(manifest, fake_runner, capsys): + drv = DockerDriver(manifest=manifest, runner=fake_runner.runner) + rc = drv.reset(confirm=False) + out = capsys.readouterr().out + assert rc != 0 + assert "destructive" in out.lower() + assert "--confirm" in out + + +def test_docker_driver_reset_with_confirm_drops_volume_via_compose(manifest, fake_runner): + fake_runner.rule( + prefix=["docker", "compose", "version"], + result=CommandResult(returncode=0), + ) + drv = DockerDriver(manifest=manifest, runner=fake_runner.runner) + rc = drv.reset(confirm=True) + assert rc == 0 + downs = [ + c + for c in fake_runner.calls + if c[:3] == ["docker", "compose", "-f"] and "down" in c and "-v" in c + ] + assert len(downs) == 1 + + +def test_docker_driver_reset_with_confirm_falls_back_to_volume_rm(manifest, fake_runner): + fake_runner.rule( + prefix=["docker", "compose", "version"], + result=CommandResult(returncode=1), + ) + drv = DockerDriver(manifest=manifest, runner=fake_runner.runner) + rc = drv.reset(confirm=True) + assert rc == 0 + vol_rms = [c for c in fake_runner.calls if c[:3] == ["docker", "volume", "rm"]] + # One volume rm per declared named volume + assert len(vol_rms) == len(manifest.run_args.volumes) + + +# ── exec / shell / logs / version ──────────────────────────────────── + + +def test_docker_driver_exec_wraps_in_bash_lc(manifest, fake_runner): + drv = DockerDriver(manifest=manifest, runner=fake_runner.runner) + drv.exec("write 1+2,!") + execs = [c for c in fake_runner.calls if c[:2] == ["docker", "exec"]] + assert execs + cmd = execs[0] + assert manifest.container in cmd + assert "bash" in cmd + assert "-lc" in cmd + # The M command is shlex-quoted inside the bash -lc script + inner = cmd[-1] + assert "%XCMD" in inner + assert "write 1+2,!" in inner + + +def test_docker_driver_shell_uses_docker_exec_it(manifest, fake_runner): + drv = DockerDriver(manifest=manifest, runner=fake_runner.runner) + drv.shell() + shells = [c for c in fake_runner.calls if c[:3] == ["docker", "exec", "-it"] and "bash" in c] + assert shells + + +def test_docker_driver_logs_follow_flag(manifest, fake_runner): + drv = DockerDriver(manifest=manifest, runner=fake_runner.runner) + drv.logs(follow=True) + follow_calls = [c for c in fake_runner.calls if c[:2] == ["docker", "logs"] and "--follow" in c] + assert follow_calls + + +def test_docker_driver_version_prints_manifest_summary(manifest, fake_runner, capsys): + drv = DockerDriver(manifest=manifest, runner=fake_runner.runner) + rc = drv.version() + out = capsys.readouterr().out + assert rc == 0 + assert manifest.image_ref() in out + assert manifest.ydb_version in out + assert "protocol" in out + + +# ── Entry-point group seam ─────────────────────────────────────────── + + +def test_entry_point_group_name_is_locked(): + """The group name is part of the PLUGIN_API_VERSION contract.""" + from m_cli.engine_driver import ENGINE_DRIVER_ENTRY_POINT_GROUP + + assert ENGINE_DRIVER_ENTRY_POINT_GROUP == "m_cli_engines" + + +def test_discover_drivers_returns_list_without_builtin(): + """Built-in DockerDriver is NOT enumerated — only out-of-tree drivers.""" + from m_cli.engine_driver import discover_drivers + + drivers = discover_drivers() + # The list may be empty (no out-of-tree drivers installed); the + # built-in must never appear in it. + assert all(d.name != "docker" for d in drivers) diff --git a/tests/test_engine_manifest.py b/tests/test_engine_manifest.py index 9ca356a..5b5336d 100644 --- a/tests/test_engine_manifest.py +++ b/tests/test_engine_manifest.py @@ -61,6 +61,18 @@ def test_manifest_container_name_matches_existing_DockerEngine_default(): assert m.container == DockerEngine.container +def test_manifest_bind_mount_container_matches_DockerEngine_default(): + # Same drift gate but for bind_root — manifest's bind_mount.container + # must match DockerEngine's default bind_root. Otherwise a fresh + # DockerEngine() builds routine paths that don't exist inside the + # actual running container. + from m_cli.engine import DockerEngine + + m = load_engine_manifest() + eng = DockerEngine() + assert str(eng.bind_root) == m.bind_mount.container + + def test_manifest_bind_mount_is_typed(): m = load_engine_manifest() assert isinstance(m.bind_mount, BindMount) diff --git a/tests/test_engine_transports.py b/tests/test_engine_transports.py index 580414a..b1d4daa 100644 --- a/tests/test_engine_transports.py +++ b/tests/test_engine_transports.py @@ -114,6 +114,34 @@ def test_docker_engine_stage_routines_translates_to_bind_mount() -> None: assert "/work" in stage +def test_docker_engine_default_bind_root_is_m_work() -> None: + """Default bind_root tracks the shared-mount cutover (Phase 2).""" + eng = DockerEngine() + assert eng.bind_root == Path("/m-work") + assert eng.container == "m-test-engine" + + +def test_docker_engine_stage_routines_maps_host_m_work_subdir(tmp_path) -> None: + """Project under host /m-work maps 1:1 to container /m-work. + + Uses tmp_path with a fake /m-work structure to avoid depending on + real host /m-work being populated. The engine only cares about the + string-shape mapping; project_root walks the parent chain looking + for a project marker (e.g. pyproject.toml), so we add a marker. + """ + # Build a fake project under a fake /m-work prefix + fake_m_work = tmp_path / "m-work" + fake_repo = fake_m_work / "m-cli" + (fake_repo / "src").mkdir(parents=True) + (fake_repo / "pyproject.toml").write_text("[project]\nname = 'fake'\n") + # Without monkeypatching _HOST_SHARED_ROOT this test only exercises + # the fallback branch; the value-shape (in-container path under + # bind_root) is the real assertion here. + eng = DockerEngine(bind_root=Path("/m-work")) + stage = eng.stage_routines(fake_repo / "src") + assert "/m-work" in stage + + # ── SSHEngine (Connection alias for backward compat) ───────────────