|
| 1 | +# LLX TestQL Integration |
| 2 | + |
| 3 | +This guide documents how **llx** bridges **TestQL** DSL validation with **planfile** ticket generation, upsert, and multi-backend synchronization. |
| 4 | + |
| 5 | +## Overview |
| 6 | + |
| 7 | +TestQL scenarios describe expected API behavior in a declarative DSL. When a scenario fails, llx can: |
| 8 | + |
| 9 | +1. **Validate** the scenario via the TestQL CLI. |
| 10 | +2. **Generate** planfile tickets for each failure. |
| 11 | +3. **Upsert** tickets into `planfile.yaml` with identity-aware deduplication. |
| 12 | +4. **Sync** tickets to `TODO.md` and configured integrations (GitHub, GitLab, Jira). |
| 13 | + |
| 14 | +All of this is available both as a standalone CLI command and as a workflow step. |
| 15 | + |
| 16 | +--- |
| 17 | + |
| 18 | +## `llx plan testql` — Standalone Command |
| 19 | + |
| 20 | +Run a TestQL scenario and optionally create/sync tickets in one shot. |
| 21 | + |
| 22 | +```bash |
| 23 | +llx plan testql SCENARIO [OPTIONS] |
| 24 | +``` |
| 25 | + |
| 26 | +### Arguments |
| 27 | + |
| 28 | +- `SCENARIO` — Path to the `.testql.toon.yaml` scenario file. |
| 29 | + |
| 30 | +### Options |
| 31 | + |
| 32 | +| Option | Default | Description | |
| 33 | +|--------|---------|-------------| |
| 34 | +| `--strategy`, `-s` | `planfile.yaml` | Target planfile YAML | |
| 35 | +| `--project`, `-p` | `.` | Project root directory | |
| 36 | +| `--url` | `http://localhost:8101` | TestQL service base URL | |
| 37 | +| `--dry-run`, `-d` | `False` | Parse/validate only; do not execute | |
| 38 | +| `--create-tickets` / `--no-create-tickets` | `True` | Create planfile tickets for failures | |
| 39 | +| `--sync` / `--no-sync` | `True` | Sync tickets to TODO.md and integrations | |
| 40 | +| `--max-tickets` | `25` | Max tickets generated from one run | |
| 41 | +| `--testql-bin` | `testql` | TestQL CLI executable name/path | |
| 42 | +| `--testql-repo-path` | `/home/tom/github/oqlos/testql` | Fallback local repo path | |
| 43 | +| `--output-yaml`, `-o` | — | Save full payload to a YAML file | |
| 44 | + |
| 45 | +### Examples |
| 46 | + |
| 47 | +```bash |
| 48 | +# Basic validation + ticket generation + sync |
| 49 | +llx plan testql testql-scenarios/api-smoke.testql.toon.yaml |
| 50 | + |
| 51 | +# Dry-run: validate without creating tickets |
| 52 | +llx plan testql testql-scenarios/api-smoke.testql.toon.yaml --dry-run |
| 53 | + |
| 54 | +# Custom project and strategy |
| 55 | +llx plan testql scenarios/smoke.testql.toon.yaml -p ./my-api -s my-api/planfile.yaml |
| 56 | + |
| 57 | +# Save results for CI inspection |
| 58 | +llx plan testql scenarios/smoke.testql.toon.yaml -o testql-results.yaml |
| 59 | +``` |
| 60 | + |
| 61 | +### Output |
| 62 | + |
| 63 | +The command prints a summary to stderr and writes a YAML payload to stdout: |
| 64 | + |
| 65 | +```yaml |
| 66 | +scenario: scenarios/smoke.testql.toon.yaml |
| 67 | +strategy: planfile.yaml |
| 68 | +project: /home/tom/projects/my-api |
| 69 | +dry_run: false |
| 70 | +validation: |
| 71 | + ok: false |
| 72 | + passed: 2 |
| 73 | + failed: 1 |
| 74 | + exit_code: 1 |
| 75 | + errors: |
| 76 | + - "step 'health-check': status mismatch" |
| 77 | +tickets: |
| 78 | + generated: 1 |
| 79 | + created: 1 |
| 80 | + skipped: 0 |
| 81 | + created_ticket_ids: |
| 82 | + - TQL-a1b2c3d4e5 |
| 83 | + sync: |
| 84 | + sync_order: |
| 85 | + - markdown |
| 86 | + - github |
| 87 | + integrations: |
| 88 | + - integration: markdown |
| 89 | + created: 1 |
| 90 | + updated: 0 |
| 91 | + skipped: 0 |
| 92 | + failed: 0 |
| 93 | + - integration: github |
| 94 | + created: 1 |
| 95 | + updated: 0 |
| 96 | + skipped: 0 |
| 97 | + failed: 0 |
| 98 | +``` |
| 99 | +
|
| 100 | +If validation fails (`ok: false`), the CLI exits with code `1`. |
| 101 | + |
| 102 | +--- |
| 103 | + |
| 104 | +## Workflow Step — `testql` |
| 105 | + |
| 106 | +In llx workflow YAML, use the `testql` step to embed TestQL validation inside a pipeline. |
| 107 | + |
| 108 | +```yaml |
| 109 | +steps: |
| 110 | + - name: validate-api |
| 111 | + type: testql |
| 112 | + params: |
| 113 | + scenario: testql-scenarios/api-smoke.testql.toon.yaml |
| 114 | + strategy: planfile.yaml |
| 115 | + project: . |
| 116 | + url: http://localhost:8101 |
| 117 | + dry_run: false |
| 118 | + create_tickets: true |
| 119 | + sync_targets: true |
| 120 | + max_tickets: 25 |
| 121 | + testql_bin: testql |
| 122 | + testql_repo_path: /home/tom/github/oqlos/testql |
| 123 | + fail_on_failure: true |
| 124 | +``` |
| 125 | + |
| 126 | +### Parameters |
| 127 | + |
| 128 | +| Parameter | Default | Description | |
| 129 | +|-----------|---------|-------------| |
| 130 | +| `scenario` | **required** | Path to `.testql.toon.yaml` | |
| 131 | +| `strategy` | `planfile.yaml` | Target planfile YAML | |
| 132 | +| `project` | `.` | Project root directory | |
| 133 | +| `url` | `http://localhost:8101` | TestQL service URL | |
| 134 | +| `dry_run` | `false` | Parse/validate only | |
| 135 | +| `create_tickets` | `true` | Create tickets for failures | |
| 136 | +| `sync_targets` | `true` | Sync to TODO.md and integrations | |
| 137 | +| `max_tickets` | `25` | Ticket cap per run | |
| 138 | +| `testql_bin` | `testql` | TestQL CLI executable | |
| 139 | +| `testql_repo_path` | `/home/tom/github/oqlos/testql` | Fallback repo path | |
| 140 | +| `fail_on_failure` | `true` | Mark step as failed when checks fail | |
| 141 | + |
| 142 | +--- |
| 143 | + |
| 144 | +## Pre-flight Freshness Validation (`llx plan run` & `llx plan validate`) |
| 145 | + |
| 146 | +Before executing tasks, `llx plan run` can run a **prefact-driven freshness check** to detect stale tickets and optionally skip or cancel them. |
| 147 | + |
| 148 | +### `llx plan run` Options |
| 149 | + |
| 150 | +| Option | Default | Description | |
| 151 | +|--------|---------|-------------| |
| 152 | +| `--validate` / `--no-validate` | `True` | Pre-flight: scan code and skip stale tickets | |
| 153 | +| `--cancel-stale` | `False` | Mark stale tickets as canceled in planfile | |
| 154 | +| `--prefact-yaml` | — | Explicit prefact.yaml for the scan | |
| 155 | +| `--prefact-bin` | — | Custom prefact executable name/path | |
| 156 | + |
| 157 | +### Examples |
| 158 | + |
| 159 | +```bash |
| 160 | +# Run with pre-flight freshness check (default) |
| 161 | +llx plan run planfile.yaml |
| 162 | +
|
| 163 | +# Cancel stale tickets before running |
| 164 | +llx plan run planfile.yaml --cancel-stale |
| 165 | +
|
| 166 | +# Disable pre-flight validation |
| 167 | +llx plan run planfile.yaml --no-validate |
| 168 | +``` |
| 169 | + |
| 170 | +### `llx plan validate` — Dedicated Freshness Command |
| 171 | + |
| 172 | +```bash |
| 173 | +# Validate ticket freshness and print markdown report |
| 174 | +llx plan validate planfile.yaml |
| 175 | +
|
| 176 | +# Cancel stale tickets in the planfile |
| 177 | +llx plan validate planfile.yaml --cancel-stale |
| 178 | +
|
| 179 | +# Exit non-zero when stale tickets are found |
| 180 | +llx plan validate planfile.yaml --fail-on-stale |
| 181 | +
|
| 182 | +# Require a successful prefact scan |
| 183 | +llx plan validate planfile.yaml --require-scan --fail-on-stale |
| 184 | +``` |
| 185 | + |
| 186 | +The freshness report contains: |
| 187 | + |
| 188 | +```yaml |
| 189 | +scan: |
| 190 | + backend: prefact |
| 191 | + ok: true |
| 192 | +stale_ticket_ids: |
| 193 | + - TICKET-42 |
| 194 | + - TICKET-99 |
| 195 | +stale: 2 |
| 196 | +current: 15 |
| 197 | +unknown: 3 |
| 198 | +``` |
| 199 | + |
| 200 | +--- |
| 201 | + |
| 202 | +## Identity-Aware Ticket Deduplication |
| 203 | + |
| 204 | +When upserting TestQL tickets into `planfile.yaml`, the system collects **identity keys** from each ticket to prevent duplicates across integrations: |
| 205 | + |
| 206 | +- `local_id` — ticket `id` or `ticket_id` |
| 207 | +- `external_id` — `external_id` or integration-specific IDs (`github_id`, `gitlab_id`, `jira_id`) |
| 208 | +- `external_key` — `external_key`, `key`, or integration-specific keys |
| 209 | +- `external_url` — `external_url`, `url`, `issue_url`, `ticket_url`, or integration-specific URLs |
| 210 | +- `source` and `external_refs` dictionaries |
| 211 | + |
| 212 | +If an incoming ticket shares any identity key with an existing task, it is **skipped** instead of recreated. |
| 213 | + |
| 214 | +--- |
| 215 | + |
| 216 | +## End-to-End Example |
| 217 | + |
| 218 | +```bash |
| 219 | +# 1. Start the API under test |
| 220 | +python -m my_api & |
| 221 | +
|
| 222 | +# 2. Run TestQL validation, create tickets, sync to GitHub |
| 223 | +llx plan testql testql-scenarios/api-smoke.testql.toon.yaml --strategy planfile.yaml --sync |
| 224 | +
|
| 225 | +# 3. Execute planfile tasks, skipping stale tickets and canceling them |
| 226 | +llx plan run planfile.yaml --cancel-stale --format markdown |
| 227 | +
|
| 228 | +# 4. Validate remaining tickets are still fresh |
| 229 | +llx plan validate planfile.yaml --fail-on-stale |
| 230 | +``` |
| 231 | + |
| 232 | +--- |
| 233 | + |
| 234 | +**LLX TestQL Integration** — Validate APIs, generate tickets, and keep your backlog in sync. 🚀 |
0 commit comments