Skip to content

docs(skills): ban post-hoc math on query audit JSON#10

Merged
alexskatell merged 1 commit into
mainfrom
feat/skills-no-posthoc-math
May 13, 2026
Merged

docs(skills): ban post-hoc math on query audit JSON#10
alexskatell merged 1 commit into
mainfrom
feat/skills-no-posthoc-math

Conversation

@alexskatell
Copy link
Copy Markdown
Contributor

Summary

  • Adds a new banned item in the standard pipeline audit contract for both bundled skills (hermes and claude-code): no python3 heredocs, jq sums, awk totals, or bash arithmetic over the already-returned query audit JSON.
  • Adds pitfall #14 to the Hermes long-form skill capturing the failure mode and the recovery path (re-issue as query sql and disclose).
  • Bumps the Hermes skill from 1.5.0 to 1.6.0.

Why

Recent agent traces showed query doctor + query audit running cleanly, then the agent opening a python3 - <<'PY' vals=[...] PY heredoc to compute averages/totals over activity.by_stage and deals rollups before answering. The audit payload already carries those rollups:

  • activity.total_messages, activity.by_stage[*]
  • deals.open_count, deals.open_value_total
  • snapshot.avg_days_in_stage, value totals
  • movement.advances / regresses / stalls

Wrapping the audit response in a Python/jq/bash math step is the same anti-pattern as wrapping the CLI itself — it just moves the violation one step past the CLI boundary. If a question genuinely needs a number not in the payload, the agent should issue topline --agent query sql --sql ... and disclose it as non-standard analytics.

This is a straight port of pitfall #21 from the canonical topline-os-crm-audits skill (v2.7.0) into the bundled repo.

Test plan

  • Re-run a "what activity happened this week in the qualified pipeline?" trace and confirm no python3/jq/awk/bash math step after query audit.
  • Confirm the agent either cites payload fields directly or escalates to one extra query sql with disclosure when a non-default metric is asked for.

The audit payload already carries activity totals/by-stage, deals
rollups, movement counts, snapshot avg_days_in_stage, and freshness.
Wrapping the response in python3 heredocs, jq sums, awk totals, or bash
arithmetic to compute averages/totals is the same "I rejected the
contract" signal as wrapping the CLI itself — it just moves the
violation one step past the CLI boundary.

If a question genuinely needs a number not in the payload (e.g. p95
instead of avg), express it as `topline --agent query sql` and disclose
it as non-standard analytics.

Bumps Hermes skill 1.5.0 -> 1.6.0.
@alexskatell alexskatell merged commit 01cf443 into main May 13, 2026
1 check passed
@alexskatell alexskatell deleted the feat/skills-no-posthoc-math branch May 13, 2026 22:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant