feat: declarative policy engine, denial explanation, and dry-run mode

CHANGELOG.md

@@ -7,6 +7,76 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Added
+- Declarative policy engine (`DeclarativePolicyEngine`) that loads rules from YAML or TOML files.
+  Rules are evaluated top-down with first-match-wins semantics; supports `safety_class`, `sensitivity`,
+  `roles`, `attributes`, and `min_justification` match conditions. (#42)
+- Policy denial explanation: `ExplainingPolicyEngine` protocol plus `DefaultPolicyEngine.explain()` and
+  `DeclarativePolicyEngine.explain()` implementations return a structured `DenialExplanation` with a
+  `FailedCondition` list for every failing check (no short-circuit), a `remediation` list, and a
+  human-readable `narrative`. (#48)
+- Dry-run invocation mode: `kernel.invoke(..., dry_run=True)` verifies the token and resolves the
+  execution plan without calling the driver. Returns `DryRunResult` with the resolved `driver_id`,
+  `operation`, `response_mode`, and an `estimated_cost` tier (`low`/`medium`/`high`). (#43)
+- `Kernel.explain_denial()` convenience method that calls the policy engine's `explain()` for a given
+  `CapabilityRequest` and `Principal` without requiring a token. Raises `AgentKernelError` when the
+  configured engine does not implement `explain()`.
+- New public types exported from `agent_kernel`: `DeclarativePolicyEngine`, `ExplainingPolicyEngine`,
+  `PolicyEngine`, `PolicyMatch`, `PolicyRule`, `DenialExplanation`, `FailedCondition`, `DryRunResult`,
+  `PolicyConfigError`.
+- `policy` optional extra (`pip install weaver-kernel[policy]`) pulls in `pyyaml` and `tomli` (Python 3.10).
+- Example policy files in `examples/policies/` (YAML and TOML formats).
+
+### Changed
+- `PolicyEngine` protocol no longer requires `explain()`. Engines that need to support
+  `Kernel.explain_denial()` should implement the new `ExplainingPolicyEngine` protocol. Built-in
+  engines satisfy both. This avoids a breaking typing change for downstream implementers.
+- `DeclarativePolicyEngine` now defers `yaml` and `tomllib`/`tomli` imports into the corresponding
+  loaders, so `import agent_kernel` works without the `policy` extra installed. Calling
+  `from_yaml`/`from_toml` without the parser surfaces a `PolicyConfigError` with an install hint.
+- `Kernel.invoke(dry_run=True)` resolves `operation` the same way drivers do
+  (`args.get("operation", capability_id)`) so `DryRunResult.operation` matches what a driver would
+  actually receive — instead of `capability.impl.operation`, which can diverge.
+- `Kernel.invoke(dry_run=True)` mirrors the Firewall's admin-only gate for `raw` mode: non-admin
+  principals see their requested `raw` mode downgraded to `summary` in `DryRunResult`, matching
+  what they would actually get at real-invoke time. Prevents probing for raw availability.
+
+### Documentation
+- `docs/architecture.md` now describes `PolicyEngine` / `ExplainingPolicyEngine` protocols,
+  `DefaultPolicyEngine` and `DeclarativePolicyEngine` (with policy-DSL semantics), and dry-run
+  mode (admin gate, operation resolution rule). Closes the canonical "Components & API
+  reference" gap flagged in audit.
+- `docs/capabilities.md` adds a "Dry-run mode" section (semantics, the three parity rules,
+  no-side-effects guarantee), a "Declarative policies" section (loaders, match conditions,
+  optional-extra behaviour), and a "Denial explanations" section. Closes the affected-files
+  gap from issue #43.
+
+### Fixed
+- `DeclarativePolicyEngine._parse_rule()` now validates the types of `roles`, `attributes`,
+  `min_justification`, and `constraints` in policy files and raises `PolicyConfigError` with a
+  precise message instead of silently producing misbehaving rules or raising at evaluation time.
+- `DeclarativePolicyEngine.explain()` now correctly reports explicit deny rules that fully match
+  (previously fell through to the misleading `no_matching_rule` fallback and dropped the rule's
+  reason). Partial-match deny rules are now skipped so the explanation focuses on actionable allow
+  rules instead of suggesting changes that would only trigger the deny.
+- Example policy files (`examples/policies/default.{yaml,toml}`) now use the correct `default` key
+  (was `default_action`, which the parser silently ignored), express PII-with-tenant as an allow
+  rule paired with default-deny (the previous deny rule was inverted under first-match-wins), and
+  order the `allow-secrets-service` rule before the deny rule (the deny was previously unreachable).
+- `Kernel.explain_denial()` docstring no longer contradicts itself ("never raises" vs.
+  `CapabilityNotFound`).
+- `DryRunResult.budget_remaining` docstring no longer references the unimplemented `BudgetManager`;
+  the field is documented as reserved for a future cross-invocation budget mechanism.
+- `drivers/mcp.py` adds an explicit `_McpError: type[BaseException] | None` annotation so mypy
+  `--strict` remains happy across the try/except import branches.
+
+### Tests
+- `tests/test_policy.py` adds `test_declarative_replicates_default_policy_decisions` — a
+  comparative test asserting that `DeclarativePolicyEngine` and `DefaultPolicyEngine` produce
+  the same allow/deny outcomes across a curated scenario matrix (READ × non-sensitive / PII /
+  PCI / SECRETS, WRITE/DESTRUCTIVE with and without required roles and justification). Closes
+  issue #42's "comparative test" acceptance criterion.
+
 ## [0.5.0] - 2026-04-12
 
 ### Added

docs/agent-context/invariants.md

@@ -64,6 +64,19 @@ tag is **silently ignored** — capabilities tagged with it pass policy without
 
 **Rule:** When adding a `SensitivityTag`, always add a matching policy rule and test.
 
+### Dry-run response-mode parity
+`Kernel.invoke(dry_run=True)` reports the response mode the caller would actually
+get at real-invoke time. The Firewall downgrades `raw` to `summary` for non-admin
+principals (`firewall/transform.py:108`), so dry-run must mirror that downgrade —
+otherwise a non-admin caller can probe/assume raw-mode availability they will never
+actually receive. The same applies to `operation`: dry-run resolves it the same way
+drivers do (`args.get("operation", capability_id)`), so what the caller sees in
+`DryRunResult` matches what a driver would receive.
+
+**Rule:** Any code path that reports a response mode or driver operation back to the
+caller must apply the same admin gate / resolution rule the real-invoke path uses,
+including dry-run, mock, and test paths.
+
 ## Safe vs. unsafe changes
 
 | Safe | Unsafe |

docs/architecture.md

@@ -27,23 +27,42 @@ graph TD
 ## Components
 
 ### Kernel
-The central orchestrator. Wires all components together and exposes five methods:
+The central orchestrator. Wires all components together and exposes:
 - `request_capabilities(goal)` — discover relevant capabilities
 - `grant_capability(request, principal, justification)` — policy check + token issuance
-- `invoke(token, principal, args, response_mode)` — execute + firewall + trace
+- `invoke(token, principal, args, response_mode, dry_run=False)` — execute + firewall + trace, or short-circuit before driver dispatch when `dry_run=True`
 - `expand(handle, query)` — paginate/filter stored results
 - `explain(action_id)` — retrieve audit trace
+- `explain_denial(request, principal, justification)` — return a structured `DenialExplanation` instead of raising `PolicyDenied`
 
 ### CapabilityRegistry
 A flat dict of `Capability` objects indexed by `capability_id`. Provides keyword-based search (no LLM, no vector DB — purely token overlap scoring).
 
 ### PolicyEngine
-The `DefaultPolicyEngine` implements role-based rules:
-1. **READ** — always allowed
-2. **WRITE** — requires `justification ≥ 15 chars` + role `writer|admin`
-3. **DESTRUCTIVE** — requires role `admin`
-4. **PII/PCI** — requires `tenant` attribute; enforces `allowed_fields` unless `pii_reader`
-5. **max_rows** — 50 (user), 500 (service)
+Two protocols and two built-in engines:
+
+- **`PolicyEngine`** (protocol) — single required method: `evaluate(request, capability, principal, justification) -> PolicyDecision`.
+- **`ExplainingPolicyEngine`** (protocol, extends `PolicyEngine`) — adds `explain(...) -> DenialExplanation`. Only engines that implement this protocol can be used with `Kernel.explain_denial`; otherwise that call raises `AgentKernelError` with a clear message. Splitting the contract keeps existing downstream `PolicyEngine` implementers backward-compatible.
+
+Both built-in engines satisfy `ExplainingPolicyEngine`:
+
+- **`DefaultPolicyEngine`** — hardcoded role-based rules:
+  1. **READ** — always allowed
+  2. **WRITE** — requires `justification ≥ 15 chars` + role `writer|admin`
+  3. **DESTRUCTIVE** — requires role `admin` + `justification ≥ 15 chars`
+  4. **PII/PCI** — requires `tenant` attribute; enforces `allowed_fields` unless `pii_reader`
+  5. **SECRETS** — requires role `admin|secrets_reader` + `justification ≥ 15 chars`
+  6. **max_rows** — 50 (user), 500 (service)
+  7. **Rate limiting** — sliding-window per `(principal_id, capability_id)` (60 READ / 10 WRITE / 2 DESTRUCTIVE per 60s; service role gets 10×)
+- **`DeclarativePolicyEngine`** — loads rules from a YAML or TOML file (or a plain dict). Supports `safety_class`, `sensitivity`, `roles`, `attributes`, and `min_justification` match conditions; `allow`/`deny` actions; per-rule `constraints` merged into the resulting `PolicyDecision`; configurable `default` action. Rules are evaluated top-down with first-match-wins. `pyyaml` and `tomli` are optional dependencies — `import agent_kernel` works without them; calling `from_yaml`/`from_toml` without the parser raises `PolicyConfigError` with an install hint.
+
+#### Denial explanations
+
+`PolicyEngine.explain()` (when available) returns a structured `DenialExplanation` with `denied`, `rule_name`, a `failed_conditions: list[FailedCondition]` describing each missing condition with `required`/`actual`/`suggestion`, a `remediation` list, and a human-readable `narrative`. Engines collect all failing conditions (no short-circuit) so callers get the full picture. For `DeclarativePolicyEngine`, an explicit deny rule that fully matches is reported as the cause; partial-match deny rules are skipped during explanation so the surfaced advice is actionable rather than self-defeating.
+
+#### Dry-run mode
+
+`Kernel.invoke(dry_run=True)` verifies the token and resolves the route plan but **never calls the driver**. It returns a `DryRunResult` with the resolved `driver_id`, the same `operation` a driver would receive (`args.get("operation", capability_id)`), the request constraints, the effective `response_mode` (Firewall's admin-only gate is mirrored: non-admin `raw` is downgraded to `summary`), and a coarse `estimated_cost` tier based on `SafetyClass`. Token verification still raises `TokenExpired` / `TokenInvalid` / `TokenScopeError` in dry-run, so the mode is safe as a policy/route sanity check. See [`docs/capabilities.md`](capabilities.md#dry-run-mode) for usage and [`docs/agent-context/invariants.md`](agent-context/invariants.md) for the parity rule with the real-invoke path.
 
 ### TokenProvider (HMAC)
 Issues HMAC-SHA256 signed tokens. Each token is bound to `principal_id + capability_id + constraints`. Verification checks: expiry → signature → principal → capability.

docs/capabilities.md

@@ -47,3 +47,123 @@ Capability(
     ...
 )
 ```
+
+## Dry-run mode
+
+`Kernel.invoke(..., dry_run=True)` verifies the token and resolves the route
+plan but **never calls the driver**. Use it to validate that a principal can
+invoke a capability, inspect what a driver *would* receive, or run policy
+checks in CI without live tool backends.
+
+```python
+result = await kernel.invoke(
+    token,
+    principal=principal,
+    args={"operation": "billing.list_invoices", "max_rows": 5},
+    response_mode="summary",
+    dry_run=True,
+)
+# result: DryRunResult(
+#   capability_id="billing.list_invoices",
+#   principal_id="user-001",
+#   policy_decision=PolicyDecision(allowed=True, ...),
+#   driver_id="billing",
+#   operation="billing.list_invoices",
+#   resolved_args={"operation": "billing.list_invoices", "max_rows": 5},
+#   response_mode="summary",
+#   budget_remaining=None,
+#   estimated_cost="low",
+# )
+```
+
+Three rules govern dry-run behaviour — keep them in sync with the real-invoke
+path if you change either:
+
+1. **Token verification still runs.** Expired, revoked, or scope-mismatched
+   tokens raise `TokenExpired` / `TokenRevoked` / `TokenInvalid` /
+   `TokenScopeError` exactly as they would at real-invoke. Policy is *not*
+   re-evaluated at invoke time — the granting policy decision is encoded in
+   the token at `grant_capability`.
+2. **Operation resolution mirrors drivers.** `DryRunResult.operation` is
+   computed the same way every driver computes it:
+   `str(args.get("operation", capability_id))`. Always use `args["operation"]`
+   when you need a fixed operation; otherwise the dry-run operation is the
+   capability ID, matching what the driver would see.
+3. **Raw-mode admin gate mirrors the Firewall.** Non-admin principals never
+   get `response_mode="raw"` at real-invoke (the Firewall downgrades it to
+   `"summary"` — see `firewall/transform.py`). Dry-run downgrades the same
+   way, so non-admin callers cannot probe for raw-mode availability via
+   `DryRunResult`.
+
+The driver's `execute()` is never called in dry-run, so the mode is free of
+side effects regardless of driver type (`InMemoryDriver`, `HTTPDriver`,
+`MCPDriver`). `DryRunResult.budget_remaining` is currently always `None`; the
+field is reserved for a future cross-invocation budget mechanism.
+
+## Declarative policies
+
+`DeclarativePolicyEngine` is an alternative to `DefaultPolicyEngine` that
+loads rules from a YAML or TOML file (or a plain dict). Rules are evaluated
+top-down, first-match-wins; if no rule matches, the policy's `default` action
+applies (`"deny"` unless overridden).
+
+```python
+from pathlib import Path
+from agent_kernel import DeclarativePolicyEngine, Kernel
+
+# YAML or TOML — both formats are interchangeable.
+policy = DeclarativePolicyEngine.from_yaml(Path("examples/policies/default.yaml"))
+
+# Or build entirely in-memory:
+policy = DeclarativePolicyEngine.from_dict({
+    "default": "deny",
+    "rules": [
+        {"name": "allow-read", "action": "allow",
+         "match": {"safety_class": ["READ"], "sensitivity": ["NONE"]}},
+        # ...
+    ],
+})
+
+kernel = Kernel(registry=registry, policy=policy)
+```
+
+A rule's `match` block supports `safety_class`, `sensitivity`, `roles`
+(ANY-of), `attributes` (ALL-of, with `"*"` meaning "attribute must be
+present"), and `min_justification` (minimum stripped length). On `allow`, the
+rule's `constraints` are merged into the resulting `PolicyDecision`. On
+`deny`, `reason` is embedded in the raised `PolicyDenied`.
+
+The DSL has no negation/missing-attribute operator today, so a policy that
+should deny "when an attribute is missing" should be expressed as an allow
+rule requiring the attribute paired with `default: deny`. See
+[`examples/policies/default.yaml`](../examples/policies/default.yaml) for a
+worked example.
+
+`pyyaml` and `tomli` are **optional** — they live behind the `[policy]`
+extra. `import agent_kernel` always works; calling `from_yaml` / `from_toml`
+without the parser installed raises `PolicyConfigError` with an install hint.
+
+## Denial explanations
+
+When a capability call is denied, `Kernel.explain_denial(request, principal,
+justification="")` returns a structured `DenialExplanation` describing
+**every** unmet condition (not just the first one), so the caller can see the
+full remediation path:
+
+```python
+explanation = kernel.explain_denial(
+    CapabilityRequest(capability_id="billing.update_invoice", goal="..."),
+    principal,
+    justification="too short",
+)
+# explanation.denied == True
+# explanation.rule_name == "write-min_justification"
+# explanation.failed_conditions == [FailedCondition(condition="roles", required=[...]), ...]
+# explanation.remediation == ["Add 'writer' or 'admin' role to ...", "Provide ..."]
+# explanation.narrative == "Request for 'billing.update_invoice' by '...' would be denied: ..."
+```
+
+Both built-in engines support `explain()`. If you bring a custom policy
+engine that implements only `PolicyEngine.evaluate`, `explain_denial` raises
+`AgentKernelError` with guidance — implement the `ExplainingPolicyEngine`
+protocol to enable structured explanations.

examples/policies/default.toml

@@ -0,0 +1,68 @@
+# Example declarative policy for agent-kernel (TOML format)
+# Load with: DeclarativePolicyEngine.from_toml(Path("examples/policies/default.toml"))
+#
+# Equivalent to default.yaml — TOML and YAML formats are interchangeable.
+#
+# Note: the current DSL has no negation / missing-attribute operator. Policies
+# that should deny "when an attribute is missing" must be expressed as allow
+# rules requiring the attribute, paired with `default = "deny"`.
+
+default = "deny"
+
+# Allow read-only operations on non-sensitive data.
+[[rules]]
+name = "allow-read-nonsensitive"
+action = "allow"
+[rules.match]
+safety_class = ["READ"]
+sensitivity = ["NONE"]
+
+# Allow PII access only when the principal carries a tenant attribute.
+# PII requests without a tenant fall through to default-deny.
+[[rules]]
+name = "allow-pii-with-tenant"
+action = "allow"
+[rules.match]
+sensitivity = ["PII"]
+[rules.match.attributes]
+tenant = "*"   # "*" means: attribute must be present with any value
+
+# Allow write operations for editors and admins with a justification.
+[[rules]]
+name = "allow-write-editors"
+action = "allow"
+[rules.match]
+safety_class = ["WRITE"]
+sensitivity = ["NONE"]
+roles = ["editor", "admin"]
+min_justification = 20
+
+# Allow destructive operations for admins only with a strong justification.
+[[rules]]
+name = "allow-destructive-admin"
+action = "allow"
+[rules.match]
+safety_class = ["DESTRUCTIVE"]
+roles = ["admin"]
+min_justification = 50
+
+# Allow service principals to read secrets with a justification.
+# IMPORTANT: this allow rule must precede any deny-secrets rule —
+# first-match-wins means an earlier unconditional deny would shadow it.
+[[rules]]
+name = "allow-secrets-service"
+action = "allow"
+[rules.match]
+sensitivity = ["SECRETS"]
+roles = ["service"]
+min_justification = 10
+
+# Any SECRETS request that wasn't already allowed is explicitly denied so
+# the caller gets a descriptive PolicyDenied reason. Functionally equivalent
+# to default-deny but provides clearer feedback.
+[[rules]]
+name = "deny-secrets-non-service"
+action = "deny"
+reason = "SECRETS access is restricted to service-role principals"
+[rules.match]
+sensitivity = ["SECRETS"]