diff --git a/README.md b/README.md index f2c4538..ffe5bfc 100644 --- a/README.md +++ b/README.md @@ -30,6 +30,22 @@ cd os-cli go build ./cmd/topline ``` +Hermes-aware install (recommended for any agent that loads CLI env from +`~/.hermes/.env` or `~/.hermes/profiles/*/.env`): + +```bash +git clone https://github.com/Topline-com/os-cli.git +cd os-cli +./scripts/install-local.sh +``` + +The script builds `~/.local/bin/topline-bin` and writes a wrapper at +`~/.local/bin/topline` that loads only the `TOPLINE_*` env keys needed by REST +and SQL commands (`TOPLINE_PIT`, `TOPLINE_LOCATION_ID`, `TOPLINE_BRAND_NAME`, +`TOPLINE_BASE_URL`, `TOPLINE_QUERY_TOKEN`, `TOPLINE_QUERY_BASE_URL`, +`TOPLINE_MCP_ACCESS_TOKEN`, `TOPLINE_MCP_TOKEN`). It does not print or persist +secrets — only those env-var keys are written into the wrapper. + ## Auth Set the same environment variables used by the MCP: @@ -94,6 +110,7 @@ only need the open count/value/stage breakdown. Warehouse SQL/query API: ```bash +topline --agent query doctor topline --agent query schema topline --agent query explain --tables opportunities,pipeline_stages,messages topline --agent query sql --sql ' @@ -104,6 +121,14 @@ topline --agent query sql --sql ' ' ``` +`query doctor` is the readiness probe agents should run before deciding SQL vs +REST. It reports whether `TOPLINE_QUERY_TOKEN` is present, whether the token is +a (rejected) raw PIT, whether the hosted schema endpoint is reachable, and +whether the warehouse exposes the core analytics tables (`contacts`, +`opportunities`, `messages`, `pipelines`, `pipeline_stages`, `call_events`, +`appointments`, `conversations`). Missing tables are surfaced as os-mcp +coverage bugs — not as a reason to silently fall back to REST. + `query` delegates to the hosted `Topline-com/os-mcp` SQL surface (`/query/api/*`): schema/catalog discovery, table explanation, and safe `SELECT` / `WITH ... SELECT` execution. The worker enforces the same read-only diff --git a/docs/examples.md b/docs/examples.md index f2ac286..835798f 100644 --- a/docs/examples.md +++ b/docs/examples.md @@ -25,6 +25,11 @@ Use `--skip-activity` for a fast snapshot-only count/value/stage breakdown. ```bash export TOPLINE_QUERY_TOKEN="signed_connection_token_from_/connect" +# Confirm SQL is reachable before fanning out REST calls. doctor reports +# token presence, raw-PIT rejection, schema reachability, and which expected +# warehouse tables are synced. +topline --agent query doctor + topline --agent query schema topline --agent query explain --tables opportunities,pipeline_stages,messages diff --git a/docs/plans/2026-05-13-sql-first-retrieval-improvements.md b/docs/plans/2026-05-13-sql-first-retrieval-improvements.md new file mode 100644 index 0000000..a4db56b --- /dev/null +++ b/docs/plans/2026-05-13-sql-first-retrieval-improvements.md @@ -0,0 +1,478 @@ +# SQL-First Topline OS Retrieval Improvements Plan + +> **For Hermes:** Use subagent-driven-development skill to implement this plan task-by-task. + +**Goal:** Make Topline OS CRM questions feel like deterministic, SQL-first retrieval instead of clunky REST pagination, one-off SQL construction, and agent memory of table/field names. + +**Architecture:** Keep hosted `Topline-com/os-mcp` warehouse SQL as the analytics source. Improve the local Go CLI so agents call stable high-level report/query commands that compile to safe SQL templates, include freshness/evidence metadata, and fall back to REST only when SQL is unavailable or freshness is unacceptable. Keep writes/mutations outside this path. + +**Tech Stack:** Go CLI (`/Users/oc/workspace/os-cli`), hosted os-mcp HTTP query API, SQLite SQL dialect, Go tests with `httptest`, GitHub Actions, Hermes skills. + +--- + +## Current Findings + +- GitHub PR #3, `Add hosted warehouse query commands`, is merged and green. +- Local checkout is still on `feat/query-api-cli`; local `main` is behind `origin/main`. +- `go test ./...` passes locally on the current checkout. +- The local wrapper at `/Users/oc/.local/bin/topline` did not load `TOPLINE_QUERY_TOKEN` from Hermes `.env` files. This would make new Discord-thread tests fail SQL even if the token exists in profile env. The wrapper has been locally patched to allow `TOPLINE_QUERY_TOKEN`, `TOPLINE_QUERY_BASE_URL`, `TOPLINE_MCP_ACCESS_TOKEN`, and `TOPLINE_MCP_TOKEN`. +- No full query token should be committed, printed, or stored in skills/docs. Persist it to profile `.env` only with explicit approval. +- There are unrelated untracked plan docs in `docs/plans/`; do not commit them unless Alex explicitly wants them included. + +## Product Diagnosis + +The current CLI is directionally right but still clunky because: + +1. Agents still have to decide when to use SQL, discover table names, and hand-write SQL. +2. The SQL surface is exposed as generic `topline query sql`, not as sales-native commands. +3. Output is raw query JSON, not an operator answer with facts, evidence, freshness, and caveats. +4. Pipeline names/stages/users still require manual lookup or remembered IDs. +5. There is no `doctor`/readiness command that proves: token present, hosted SQL reachable, schema available, and expected tables synced. +6. Local/GitHub state can drift after PRs merge, so agents may use an old branch or binary. + +## Non-Negotiables + +- SQL first for analytics: counts, rollups, joins, movement, funnel, contact activity. +- REST first only for live writes, exact operational object reads, and SQL freshness gaps. +- No raw PIT for SQL. SQL requires connection-bound query token. +- No secrets in docs, commits, skills, issue bodies, test fixtures, or terminal output. +- No LLM embedded inside the CLI for v1. The CLI should be deterministic. Let the agent choose high-level commands; the CLI compiles those commands to vetted SQL templates. +- Report activity and opportunity movement separately. + +--- + +## Phase 0: Local Baseline and Release Hygiene + +### Task 0.1: Sync local repo to merged GitHub state + +**Objective:** Start all future work from `origin/main`, not the already-merged feature branch. + +**Files:** none. + +**Steps:** + +```bash +cd /Users/oc/workspace/os-cli +git fetch origin --prune +git switch main +git pull --ff-only origin main +git status --short +``` + +**Expected:** `main` points at the merge commit for PR #3. Untracked docs may remain; do not stage them accidentally. + +### Task 0.2: Install current binary locally + +**Objective:** Ensure `/Users/oc/.local/bin/topline-bin` matches merged `origin/main`. + +**Files:** none. + +**Steps:** + +```bash +cd /Users/oc/workspace/os-cli +go test ./... +go build -o /Users/oc/.local/bin/topline-bin ./cmd/topline +/opt/homebrew/bin/topline --agent query help +``` + +**Expected:** tests pass; help prints query commands. + +### Task 0.3: Confirm query-token wiring without printing secrets + +**Objective:** Verify the wrapper can load query-token keys from profile `.env` files. + +**Files:** +- Already locally patched: `/Users/oc/.local/bin/topline` + +**Steps:** + +```bash +tmp=$(mktemp -d) +cat > "$tmp/.env" <<'EOF' +TOPLINE_PIT=pit-test +TOPLINE_LOCATION_ID=loc-test +TOPLINE_QUERY_TOKEN=signed-query-token-test +EOF +HERMES_HOME="$tmp" /opt/homebrew/bin/topline --agent query schema --url http://127.0.0.1:9 >/tmp/out 2>/tmp/err || true +! grep -q 'TOPLINE_QUERY_TOKEN is required' /tmp/err +rm -rf "$tmp" /tmp/out /tmp/err +``` + +**Expected:** command fails by connection/auth, not by missing token. + +### Task 0.4: Persist the live query token only after approval + +**Objective:** Store `TOPLINE_QUERY_TOKEN` in the relevant Hermes profile env files so new Discord threads work. + +**Files:** +- `/Users/oc/.hermes/.env` +- `/Users/oc/.hermes/profiles/sales_agent/.env` +- optionally `/Users/oc/.hermes/profiles/marketing_agent/.env` and `/Users/oc/.hermes/profiles/cfo_agent/.env` + +**Steps:** + +```bash +# Use an editor or a script that never prints the value. +# Add exactly: +# TOPLINE_QUERY_TOKEN= +``` + +**Expected:** + +```bash +HERMES_HOME=/Users/oc/.hermes/profiles/sales_agent \ + /opt/homebrew/bin/topline --agent query schema >/tmp/schema.json +python3 - <<'PY' +import json +json.load(open('/tmp/schema.json')) +print('schema_ok') +PY +rm -f /tmp/schema.json +``` + +--- + +## Phase 1: Add `topline query doctor` + +### Task 1.1: Add query readiness tests + +**Objective:** Prove agents can run one command to diagnose SQL readiness. + +**Files:** +- Modify: `internal/commands/query_test.go` +- Modify: `internal/commands/query.go` + +**Test cases:** + +- Missing token returns `queryTokenPresent: false` and human guidance. +- Raw `pit-` token returns `rawPitRejected: true`. +- Valid fake token against `httptest.Server` calls `/query/api/get-overview`. +- Response includes table count and expected table presence when server returns schema. + +**Command target:** + +```bash +topline --agent query doctor +``` + +**Output shape:** + +```json +{ + "queryTokenPresent": true, + "baseUrl": "https://os-mcp.topline.com", + "schemaReachable": true, + "tableCount": 12, + "expectedTables": { + "contacts": true, + "opportunities": true, + "messages": true, + "pipeline_stages": true + }, + "recommendation": "SQL analytics ready" +} +``` + +### Task 1.2: Implement `doctor` + +**Objective:** Make readiness self-evident before agents choose SQL or REST. + +**Implementation notes:** + +- Do not print token values. +- Reuse `topline.LoadQueryConfig()` but provide a mode that reports missing token instead of returning only an error. +- Call schema endpoint only if token looks valid. +- Keep output JSON and PII-safe. + +**Verification:** + +```bash +go test ./internal/commands -run Query +go test ./... +``` + +--- + +## Phase 2: Add SQL Template Registry + +### Task 2.1: Create a deterministic query-template package + +**Objective:** Stop making agents hand-write common SQL from memory. + +**Files:** +- Create: `internal/queries/templates.go` +- Create: `internal/queries/templates_test.go` + +**Template metadata:** + +```go +type Template struct { + Name string + Description string + Tables []string + Params []Param + SQL string +} +``` + +**Initial templates:** + +- `pipeline.activity_by_channel` +- `pipeline.stage_value_snapshot` +- `pipeline.movement` +- `pipeline.active_deals` +- `contact.activity_rollup` +- `owner.pipeline_snapshot` + +### Task 2.2: Add safe parameter binding/rendering + +**Objective:** Allow parameterized templates without SQL injection or quoting mistakes. + +**Rules:** + +- Parameters are values only, never identifiers. +- Pipeline/stage/user identifiers must be resolved before template rendering. +- Strings are single-quoted and escaped. +- Dates are ISO strings computed by the CLI from human windows like `this-week-et`. + +**Verification:** template tests assert exact SQL output and reject unknown params. + +### Task 2.3: Add template listing and execution commands + +**Commands:** + +```bash +topline --agent query templates +topline --agent query template pipeline.activity_by_channel \ + --param pipeline_id=CLUy1QapsrEeBiNrmQiL \ + --param since=2026-05-11T00:00:00-04:00 +``` + +**Output:** include `template`, `params`, `sql`, and query result. + +--- + +## Phase 3: Add Sales-Native Report Commands + +### Task 3.1: Add pipeline resolver + +**Objective:** Let agents use names like `Sales - Flex - Qualified` without remembering IDs. + +**Files:** +- Create: `internal/reports/resolver.go` +- Create: `internal/reports/resolver_test.go` + +**Behavior:** + +- SQL path: query `pipelines` + `pipeline_stages` and fuzzy-match names. +- REST fallback: use `opportunities pipelines` shape. +- Return ambiguity errors with candidate names, not silent wrong matches. + +### Task 3.2: Add `activity rollup` + +**Command:** + +```bash +topline --agent activity rollup \ + --pipeline "Sales - Flex - Qualified" \ + --since this-week-et \ + --group-by channel,direction +``` + +**Objective:** One command for the exact Discord prompt: “What activity happened this week in our qualified pipeline?” + +**Output shape:** + +```json +{ + "answerType": "pipeline_activity_rollup", + "pipeline": { "id": "...", "name": "Sales - Flex - Qualified" }, + "window": { "since": "...", "timezone": "America/New_York" }, + "activity": [ + { "type": "Email", "direction": "outbound", "touches": 5, "contactsTouched": 4 } + ], + "movement": { + "stageMoves": 0, + "opportunityUpdates": 0 + }, + "evidence": { "source": "warehouse_sql", "sqlTemplates": ["pipeline.activity_by_channel", "pipeline.movement"] }, + "fallbackUsed": false +} +``` + +### Task 3.3: Add `pipeline snapshot` + +**Command:** + +```bash +topline --agent pipeline snapshot --pipeline "Sales - Flex - Qualified" --status open +``` + +**Returns:** open count/value by stage plus warehouse freshness metadata. + +### Task 3.4: Add `pipeline movement` + +**Command:** + +```bash +topline --agent pipeline movement --pipeline "Sales - Flex - Qualified" --since this-week-et +``` + +**Returns:** created, updated, stage-changed, status-changed, won/lost counts, explicitly all-status when proving negatives. + +### Task 3.5: Add REST fallback but mark it clearly + +**Rule:** if `query doctor` fails, report commands can call existing `pipeline audit`, but output must include: + +```json +"fallbackUsed": true, +"fallbackReason": "TOPLINE_QUERY_TOKEN missing" +``` + +--- + +## Phase 4: Agent-Shaped Answer Packets + +### Task 4.1: Standardize report packet schema + +**Objective:** Make every report easy for Paul/Francis/Bernard to summarize consistently. + +**Fields:** + +- `toplineAnswer` +- `facts` +- `movement` +- `activity` +- `hygieneFlags` +- `nextActions` +- `evidence` +- `freshness` +- `warnings` +- `rawRows` optional behind `--include-rows` + +### Task 4.2: Add freshness metadata + +**Objective:** Prevent SQL-vs-live confusion. + +**Best path:** expose sync timestamps from os-mcp schema/overview if available. If not currently available, open a companion os-mcp issue/PR to add it to `/query/api/get-overview`. + +**CLI behavior:** + +- If freshness is present, include it. +- If absent, include `freshness.known=false` and warn that SQL is synced warehouse data. + +--- + +## Phase 5: GitHub Repo Improvements + +### Task 5.1: Commit local wrapper/install support to repo + +**Objective:** Avoid one-off local wrapper drift. + +**Files:** +- Create: `scripts/install-local.sh` +- Modify: `README.md` +- Modify: `docs/examples.md` + +**Script responsibilities:** + +- Build `topline-bin`. +- Install wrapper to `~/.local/bin/topline`. +- Wrapper allowlist includes all `TOPLINE_*` keys needed by REST and SQL. +- Does not create or print secrets. + +### Task 5.2: Add docs for SQL-first reporting commands + +**Files:** +- Modify: `README.md` +- Modify: `docs/examples.md` +- Create: `docs/sql-first-reporting.md` + +**Docs should include:** + +- SQL vs REST decision rule. +- `query doctor`. +- `activity rollup` examples. +- Query token setup without leaking a token. +- Freshness caveat. + +### Task 5.3: Update bundled skills after code lands + +**Files:** +- Modify: `skills/hermes/SKILL.md` +- Modify: `skills/claude-code/SKILL.md` + +**Also sync Hermes active skills:** + +- `/Users/oc/.hermes/skills/topline-stack/topline-os-cli` +- `/Users/oc/.hermes/skills/topline-stack/topline-os-crm-audits` +- `/Users/oc/.hermes/profiles/sales_agent/skills/topline-stack` +- `/Users/oc/.hermes/profiles/marketing_agent/skills/topline-stack` + +--- + +## Phase 6: PR Strategy + +### PR A: Local readiness and `query doctor` + +- Base: `main` +- Branch: `feat/query-doctor` +- Scope: local install script, wrapper allowlist, `query doctor`, docs. +- Tests: `go test ./...`. + +### PR B: Query template registry + +- Base: `main` after PR A merges. +- Branch: `feat/query-templates` +- Scope: `internal/queries`, `query templates`, `query template`. +- Tests: template rendering, query client httptest. + +### PR C: Sales-native report commands + +- Base: `main` after PR B merges. +- Branch: `feat/sql-first-reports` +- Scope: `activity rollup`, `pipeline snapshot`, `pipeline movement`, resolver, output packets. +- Tests: golden JSON report outputs. + +### PR D: os-mcp freshness/catalog improvements, if needed + +- Repo: `/Users/oc/workspace/os-mcp` +- Scope: add sync freshness metadata to `/query/api/get-overview` and MCP `topline_describe_schema` if not already present. +- Tests: Worker/unit tests in os-mcp. + +--- + +## Final Acceptance Test + +In a fresh Discord sales thread, ask: + +> What activity happened this week in our qualified pipeline? + +Expected agent behavior: + +1. Load Topline OS skills. +2. Run `topline --agent query doctor` or directly use SQL-ready command if already proven. +3. Run high-level SQL-first command, ideally: + +```bash +topline --agent activity rollup --pipeline "Sales - Flex - Qualified" --since this-week-et +``` + +4. Answer with: + +- Activity by channel/direction. +- Number of contacts/opportunities touched. +- Stage/status movement separately. +- Open pipeline value/stage snapshot if requested. +- Freshness/caveat line. +- No REST fan-out unless SQL is unavailable. + +## Done Criteria + +- Local binary and wrapper are synced and SQL-capable. +- `topline --agent query doctor` gives a clear ready/not-ready result. +- Common CRM questions use high-level report commands instead of bespoke SQL. +- Report outputs are compact, evidence-backed, and Discord-friendly. +- GitHub repo has merged PRs with tests and docs. +- Active Hermes/Paul/Bernard skills guide agents to the new commands. diff --git a/internal/commands/query.go b/internal/commands/query.go index 1aa9e14..46b20fd 100644 --- a/internal/commands/query.go +++ b/internal/commands/query.go @@ -24,6 +24,9 @@ func runQueryCommand(args []string, stdout io.Writer, globals globalOptions) err if err != nil { return err } + if subcommand == "doctor" { + return runQueryDoctor(flags, stdout, globals) + } cfg, err := topline.LoadQueryConfig() if err != nil { return err @@ -76,6 +79,7 @@ func printQueryHelp(w io.Writer) { _, _ = fmt.Fprintln(w, "Topline SQL/query commands") _, _ = fmt.Fprintln(w, "") _, _ = fmt.Fprintln(w, "Usage:") + _, _ = fmt.Fprintln(w, " topline query doctor") _, _ = fmt.Fprintln(w, " topline query schema") _, _ = fmt.Fprintln(w, " topline query catalog") _, _ = fmt.Fprintln(w, " topline query explain --tables contacts,opportunities") @@ -86,6 +90,117 @@ func printQueryHelp(w io.Writer) { _, _ = fmt.Fprintln(w, " TOPLINE_QUERY_BASE_URL Defaults to https://os-mcp.topline.com") } +// expectedQueryTables are the warehouse tables every SQL-first agent flow assumes +// are reachable. `query doctor` reports presence so agents can decide SQL vs REST +// without guessing. +var expectedQueryTables = []string{ + "contacts", + "opportunities", + "messages", + "pipelines", + "pipeline_stages", + "call_events", + "appointments", + "conversations", +} + +type queryDoctorReport struct { + QueryTokenPresent bool `json:"queryTokenPresent"` + TokenSourceEnvVar string `json:"tokenSourceEnvVar,omitempty"` + RawPITRejected bool `json:"rawPitRejected"` + BaseURL string `json:"baseUrl"` + SchemaReachable bool `json:"schemaReachable"` + SchemaError string `json:"schemaError,omitempty"` + TableCount int `json:"tableCount"` + ExpectedTables map[string]bool `json:"expectedTables"` + MissingTables []string `json:"missingTables,omitempty"` + Recommendation string `json:"recommendation"` +} + +func runQueryDoctor(flags map[string]string, stdout io.Writer, globals globalOptions) error { + status := topline.InspectQueryEnv() + if flags["url"] != "" { + status.BaseURL = flags["url"] + } + + report := queryDoctorReport{ + QueryTokenPresent: status.TokenPresent, + TokenSourceEnvVar: status.SourceEnvVar, + RawPITRejected: status.RawPITToken, + BaseURL: status.BaseURL, + ExpectedTables: map[string]bool{}, + } + + for _, t := range expectedQueryTables { + report.ExpectedTables[t] = false + } + + switch { + case !status.TokenPresent: + report.Recommendation = "Set TOPLINE_QUERY_TOKEN to a connection-bound token from https://os-mcp.topline.com/connect, or fall back to REST commands (e.g. `topline pipeline audit`) until SQL is wired." + return output.WriteJSON(stdout, report, globals.MaskPII) + case status.RawPITToken: + report.Recommendation = "TOPLINE_QUERY_TOKEN looks like a raw PIT (pit-...). SQL surface requires a connection-bound token; generate one at https://os-mcp.topline.com/connect." + return output.WriteJSON(stdout, report, globals.MaskPII) + } + + client := topline.NewQueryClient(topline.QueryConfig{BaseURL: status.BaseURL, Token: status.Token}) + ctx := context.Background() + + var schema any + if err := client.Get(ctx, "/query/api/get-overview", nil, &schema); err != nil { + report.SchemaReachable = false + report.SchemaError = err.Error() + report.Recommendation = "SQL endpoint unreachable. Verify TOPLINE_QUERY_BASE_URL and that the connection-bound token is still valid; fall back to REST until resolved." + return output.WriteJSON(stdout, report, globals.MaskPII) + } + + report.SchemaReachable = true + tables := extractTableNames(schema) + report.TableCount = len(tables) + for _, name := range tables { + if _, ok := report.ExpectedTables[name]; ok { + report.ExpectedTables[name] = true + } + } + for _, name := range expectedQueryTables { + if !report.ExpectedTables[name] { + report.MissingTables = append(report.MissingTables, name) + } + } + if len(report.MissingTables) == 0 { + report.Recommendation = "SQL analytics ready. Prefer warehouse SQL for pipeline/activity questions; REST only for live writes." + } else { + report.Recommendation = fmt.Sprintf("SQL reachable but missing expected tables (%s). Treat the gap as an os-mcp coverage bug rather than silently falling back to REST.", strings.Join(report.MissingTables, ", ")) + } + return output.WriteJSON(stdout, report, globals.MaskPII) +} + +func extractTableNames(schema any) []string { + root, ok := schema.(map[string]any) + if !ok { + return nil + } + raw, ok := root["tables"].([]any) + if !ok { + return nil + } + out := make([]string, 0, len(raw)) + for _, entry := range raw { + switch t := entry.(type) { + case string: + if t = strings.TrimSpace(t); t != "" { + out = append(out, t) + } + case map[string]any: + if name, ok := t["name"].(string); ok && name != "" { + out = append(out, name) + } + } + } + return out +} + func splitCSV(s string) []string { parts := strings.Split(s, ",") out := make([]string, 0, len(parts)) diff --git a/internal/commands/query_test.go b/internal/commands/query_test.go index 9c84390..f7dc5b6 100644 --- a/internal/commands/query_test.go +++ b/internal/commands/query_test.go @@ -119,3 +119,125 @@ func TestQueryRejectsRawPITToken(t *testing.T) { t.Fatalf("unexpected error: %v", err) } } + +func decodeDoctor(t *testing.T, raw string) map[string]any { + t.Helper() + var out map[string]any + if err := json.Unmarshal([]byte(raw), &out); err != nil { + t.Fatalf("decode doctor output: %v\nraw: %s", err, raw) + } + return out +} + +func TestQueryDoctorReportsMissingToken(t *testing.T) { + t.Setenv("TOPLINE_QUERY_TOKEN", "") + t.Setenv("TOPLINE_MCP_ACCESS_TOKEN", "") + t.Setenv("TOPLINE_MCP_TOKEN", "") + t.Setenv("TOPLINE_QUERY_BASE_URL", "") + + var stdout bytes.Buffer + if err := Execute([]string{"query", "doctor"}, &stdout, io.Discard); err != nil { + t.Fatalf("doctor returned error: %v", err) + } + out := decodeDoctor(t, stdout.String()) + if got, _ := out["queryTokenPresent"].(bool); got { + t.Fatalf("queryTokenPresent = true, want false") + } + if got, _ := out["schemaReachable"].(bool); got { + t.Fatalf("schemaReachable should be false when token is missing") + } + rec, _ := out["recommendation"].(string) + if !strings.Contains(rec, "TOPLINE_QUERY_TOKEN") { + t.Fatalf("recommendation should name the env var; got %q", rec) + } +} + +func TestQueryDoctorRejectsRawPIT(t *testing.T) { + t.Setenv("TOPLINE_QUERY_TOKEN", "pit-example-token") + t.Setenv("TOPLINE_QUERY_BASE_URL", "http://127.0.0.1:1") + + var stdout bytes.Buffer + if err := Execute([]string{"query", "doctor"}, &stdout, io.Discard); err != nil { + t.Fatalf("doctor returned error: %v", err) + } + out := decodeDoctor(t, stdout.String()) + if got, _ := out["rawPitRejected"].(bool); !got { + t.Fatalf("rawPitRejected should be true for pit-* tokens") + } + if got, _ := out["schemaReachable"].(bool); got { + t.Fatalf("schemaReachable should be false when rejecting raw PIT") + } +} + +func TestQueryDoctorReportsTablePresence(t *testing.T) { + t.Setenv("TOPLINE_QUERY_TOKEN", "signed-query-token") + + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.URL.Path != "/query/api/get-overview" { + t.Fatalf("unexpected path %q", r.URL.Path) + } + w.Header().Set("Content-Type", "application/json") + _, _ = w.Write([]byte(`{"tables":[ + {"name":"contacts","row_count":10}, + {"name":"opportunities","row_count":20}, + {"name":"messages","row_count":30}, + {"name":"pipelines","row_count":4}, + {"name":"pipeline_stages","row_count":12}, + {"name":"call_events","row_count":5}, + {"name":"appointments","row_count":3}, + {"name":"conversations","row_count":7} + ]}`)) + })) + defer server.Close() + t.Setenv("TOPLINE_QUERY_BASE_URL", server.URL) + + var stdout bytes.Buffer + if err := Execute([]string{"query", "doctor"}, &stdout, io.Discard); err != nil { + t.Fatalf("doctor returned error: %v", err) + } + out := decodeDoctor(t, stdout.String()) + if got, _ := out["schemaReachable"].(bool); !got { + t.Fatalf("schemaReachable should be true; got %v", out) + } + if n, _ := out["tableCount"].(float64); int(n) != 8 { + t.Fatalf("tableCount = %v, want 8", n) + } + expected, _ := out["expectedTables"].(map[string]any) + for _, table := range []string{"contacts", "opportunities", "messages", "pipeline_stages"} { + if got, _ := expected[table].(bool); !got { + t.Fatalf("expectedTables[%q] = %v, want true", table, expected[table]) + } + } + rec, _ := out["recommendation"].(string) + if !strings.Contains(rec, "ready") { + t.Fatalf("recommendation should report ready; got %q", rec) + } +} + +func TestQueryDoctorFlagsMissingTables(t *testing.T) { + t.Setenv("TOPLINE_QUERY_TOKEN", "signed-query-token") + + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + w.Header().Set("Content-Type", "application/json") + _, _ = w.Write([]byte(`{"tables":[ + {"name":"contacts","row_count":10}, + {"name":"opportunities","row_count":20} + ]}`)) + })) + defer server.Close() + t.Setenv("TOPLINE_QUERY_BASE_URL", server.URL) + + var stdout bytes.Buffer + if err := Execute([]string{"query", "doctor"}, &stdout, io.Discard); err != nil { + t.Fatalf("doctor returned error: %v", err) + } + out := decodeDoctor(t, stdout.String()) + missing, _ := out["missingTables"].([]any) + if len(missing) == 0 { + t.Fatalf("expected missingTables to include core analytics tables") + } + rec, _ := out["recommendation"].(string) + if !strings.Contains(rec, "coverage") { + t.Fatalf("recommendation should call out coverage gap; got %q", rec) + } +} diff --git a/internal/topline/query_client.go b/internal/topline/query_client.go index 2f9f451..b20e8b9 100644 --- a/internal/topline/query_client.go +++ b/internal/topline/query_client.go @@ -36,18 +36,45 @@ func (e *QueryAPIError) Error() string { return fmt.Sprintf("Topline query API error %d: %s", e.StatusCode, e.Message) } -func LoadQueryConfig() (QueryConfig, error) { - cfg := QueryConfig{ +// QueryEnvStatus reports what the CLI sees in the environment for the SQL/query +// surface without erroring. Use this for `query doctor`; use LoadQueryConfig for +// commands that must hard-fail when the token is missing or unusable. +type QueryEnvStatus struct { + BaseURL string + Token string + TokenPresent bool + RawPITToken bool + SourceEnvVar string +} + +func InspectQueryEnv() QueryEnvStatus { + status := QueryEnvStatus{ BaseURL: strings.TrimSpace(os.Getenv("TOPLINE_QUERY_BASE_URL")), - Token: firstNonEmptyEnv("TOPLINE_QUERY_TOKEN", "TOPLINE_MCP_ACCESS_TOKEN", "TOPLINE_MCP_TOKEN"), } - if cfg.BaseURL == "" { - cfg.BaseURL = DefaultQueryBaseURL + if status.BaseURL == "" { + status.BaseURL = DefaultQueryBaseURL + } + for _, key := range []string{"TOPLINE_QUERY_TOKEN", "TOPLINE_MCP_ACCESS_TOKEN", "TOPLINE_MCP_TOKEN"} { + if v := strings.TrimSpace(os.Getenv(key)); v != "" { + status.Token = v + status.TokenPresent = true + status.SourceEnvVar = key + break + } + } + if status.TokenPresent && strings.HasPrefix(status.Token, "pit-") { + status.RawPITToken = true } - if cfg.Token == "" { + return status +} + +func LoadQueryConfig() (QueryConfig, error) { + status := InspectQueryEnv() + cfg := QueryConfig{BaseURL: status.BaseURL, Token: status.Token} + if !status.TokenPresent { return cfg, errors.New("TOPLINE_QUERY_TOKEN is required for SQL/query commands; generate a connection-bound token at https://os-mcp.topline.com/connect or set TOPLINE_MCP_ACCESS_TOKEN") } - if strings.HasPrefix(cfg.Token, "pit-") { + if status.RawPITToken { return cfg, errors.New("TOPLINE_QUERY_TOKEN must be a connection-bound MCP/query token, not a raw PIT; generate one at https://os-mcp.topline.com/connect") } return cfg, nil diff --git a/scripts/install-local.sh b/scripts/install-local.sh new file mode 100755 index 0000000..05cbe97 --- /dev/null +++ b/scripts/install-local.sh @@ -0,0 +1,76 @@ +#!/usr/bin/env bash +# install-local.sh — Build and install the Topline OS CLI for local use. +# +# Produces two files: +# ~/.local/bin/topline-bin the compiled Go binary +# ~/.local/bin/topline a Hermes-friendly wrapper that loads +# TOPLINE_* keys from Hermes .env files +# +# Idempotent: rerun any time to refresh the binary or wrapper. +# Never prints secrets; only the env-var KEYS are written into the wrapper. + +set -euo pipefail + +repo_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +install_dir="${TOPLINE_INSTALL_DIR:-$HOME/.local/bin}" +binary_path="$install_dir/topline-bin" +wrapper_path="$install_dir/topline" +hermes_home="${HERMES_HOME_DEFAULT:-$HOME/.hermes}" + +mkdir -p "$install_dir" + +echo "==> Building topline-bin from $repo_root" +(cd "$repo_root" && go build -o "$binary_path" ./cmd/topline) +chmod +x "$binary_path" +echo " installed: $binary_path" + +echo "==> Writing wrapper $wrapper_path" +cat > "$wrapper_path" <