Skip to content

test: add testing framework with Vitest Workers pool and Playwright#9

Open
nshen wants to merge 2 commits intodevfrom
feat/testing-framework
Open

test: add testing framework with Vitest Workers pool and Playwright#9
nshen wants to merge 2 commits intodevfrom
feat/testing-framework

Conversation

@nshen
Copy link
Copy Markdown
Owner

@nshen nshen commented Apr 7, 2026

Summary

  • apps/server: Vitest + @cloudflare/vitest-pool-workers v0.14 — integration tests running in real miniflare Workers runtime with D1/KV bindings
  • apps/web: Playwright E2E framework with Chromium + auto dev server startup
  • docs/testing.md: testing strategy overview for all workspaces

Smoke tests included

  • GET / — hello message
  • POST /rpc/planet/list — 8 planets via oRPC
  • Todos CRUD — D1 create + list

Reference

Based on MizuFinancial/affiliate testing approach using @cloudflare/vitest-pool-workers for real Workers runtime tests instead of mocking cloudflare:workers.

Test plan

  • pnpm test — 3 tests passed
  • pnpm test:e2e — requires running dev server

@nshen nshen changed the base branch from main to dev April 7, 2026 04:41
@nshen nshen force-pushed the feat/testing-framework branch 2 times, most recently from 94d1993 to 1cfd7d9 Compare April 7, 2026 04:48
Set up the foundational testing infrastructure for the monorepo:

- apps/server: Vitest + @cloudflare/vitest-pool-workers (v0.14) for
  integration tests running in real miniflare Workers runtime with D1/KV
- apps/web: Playwright for E2E tests against dev server
- docs/testing.md: testing strategy overview for all workspaces

Includes smoke tests verifying Hono routes, oRPC planet API, and
D1 CRUD operations (todos create + list).
@nshen nshen force-pushed the feat/testing-framework branch from 1cfd7d9 to 273e50e Compare April 7, 2026 04:52
- Initialize Playwright Test Agents with agent definitions in
  .claude/agents/ for Claude Code discovery
- Add playwright-test MCP server to .mcp.json with --config pointing
  to apps/web/playwright.config.ts
- Add seed.spec.ts and specs/ directory for test plans
- Update docs/testing.md with agent workflow documentation
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a unified testing setup across the monorepo: Workers-runtime integration tests for apps/server (Vitest + Cloudflare workers pool) and E2E tests for apps/web (Playwright), plus supporting docs and MCP agent configs.

Changes:

  • Introduces Vitest Workers-runtime integration testing for apps/server (miniflare D1/KV + migration setup + smoke/CRUD tests).
  • Adds Playwright E2E testing for apps/web (config, scripts, smoke spec, seed placeholder).
  • Updates workspace tooling/docs: dependency catalogs + lockfile, root scripts, testing strategy doc, and Playwright Test Agents MCP configuration.

Reviewed changes

Copilot reviewed 16 out of 17 changed files in this pull request and generated 9 comments.

Show a summary per file
File Description
pnpm-workspace.yaml Adds testing-related dependencies to the shared catalog.
pnpm-lock.yaml Locks new test tooling deps (Vitest, Playwright, Cloudflare pool) and transitive updates.
package.json Adds root test / test:e2e scripts and pnpm.onlyBuiltDependencies allowlist.
docs/testing.md Rewrites testing strategy doc to reflect per-workspace approach and commands.
apps/web/specs/README.md Adds placeholder README for test plan specs directory.
apps/web/playwright.config.ts Adds Playwright runner config and dev-server auto-start.
apps/web/package.json Adds test:e2e script and Playwright devDependency.
apps/web/e2e/smoke.spec.ts Adds basic E2E smoke coverage (page title + hello API).
apps/web/e2e/seed.spec.ts Adds a seed placeholder spec intended for generated tests.
apps/server/vitest.config.ts Adds Vitest + @cloudflare/vitest-pool-workers config with D1/KV bindings and migrations.
apps/server/tests/setup.ts Adds D1 migration application in test setup.
apps/server/tests/server.test.ts Adds server smoke tests + D1-backed todos CRUD integration test.
apps/server/package.json Adds test script and devDependencies for Vitest + Cloudflare pool.
.mcp.json Adds playwright-test MCP server wiring for Playwright Test Agents.
.claude/agents/playwright-test-planner.md Adds planner agent definition for Playwright Test Agents workflow.
.claude/agents/playwright-test-healer.md Adds healer agent definition for Playwright Test Agents workflow.
.claude/agents/playwright-test-generator.md Adds generator agent definition for Playwright Test Agents workflow.
Files not reviewed (1)
  • pnpm-lock.yaml: Language not supported

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread package.json
"pnpm": {
"onlyBuiltDependencies": [
"esbuild",
"workerd"
Copy link

Copilot AI Apr 16, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pnpm.onlyBuiltDependencies is an allowlist for dependencies that may run install/build scripts. Playwright relies on install scripts to download browser binaries; since playwright / @playwright/test aren’t listed here, installs may skip that step and playwright test will fail at runtime. Add the relevant Playwright package(s) to the allowlist (or remove this restriction if not needed).

Suggested change
"workerd"
"workerd",
"@playwright/test",
"playwright"

Copilot uses AI. Check for mistakes.
],
webServer: {
command: 'pnpm run dev',
cwd: '../../',
Copy link

Copilot AI Apr 16, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

webServer.command: 'pnpm run dev' with cwd: '../../' starts the root recursive dev script (all workspaces), which is heavier than needed for E2E and can introduce port conflicts / unrelated logs. Prefer starting only the web app dev server (e.g. running the apps/web dev script via pnpm --filter @nn-stack/web dev or equivalent).

Suggested change
cwd: '../../',
cwd: '.',

Copilot uses AI. Check for mistakes.
Comment thread apps/web/e2e/seed.spec.ts
Comment on lines +1 to +4
import { test, expect } from '@playwright/test';

test.describe('Test group', () => {
test('seed', async ({ page }) => {
Copy link

Copilot AI Apr 16, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This placeholder seed test will be executed by Playwright (it matches *.spec.ts) but doesn’t perform any setup and includes unused imports. If the intent is a shared seed helper for generated tests, consider moving it to a non-test module (e.g. e2e/seed.ts) or marking it test.skip() so it doesn’t run in CI.

Suggested change
import { test, expect } from '@playwright/test';
test.describe('Test group', () => {
test('seed', async ({ page }) => {
import { test } from '@playwright/test';
test.describe('Test group', () => {
test.skip('seed', async () => {

Copilot uses AI. Check for mistakes.
Comment on lines +10 to +43
const migrationsPath = path.resolve(
__dirname,
'../../packages/db/migrations',
);
const migrations = await readD1Migrations(migrationsPath);

return {
plugins: [
cloudflareTest({
main: './src/index.ts',
miniflare: {
compatibilityDate: '2025-01-01',
compatibilityFlags: ['nodejs_compat'],
bindings: {
CORS_ORIGIN: 'http://localhost:3000',
TEST_MIGRATIONS: migrations,
},
d1Databases: {
DB: {
id: 'test-db',
},
},
kvNamespaces: {
KV: {
id: 'test-kv',
},
},
},
}),
],
test: {
setupFiles: ['./tests/setup.ts'],
},
};
Copy link

Copilot AI Apr 16, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This config file is indented with tabs, but the repo standard is 2-space indentation per biome.json. Please format the file to match the configured style.

Suggested change
const migrationsPath = path.resolve(
__dirname,
'../../packages/db/migrations',
);
const migrations = await readD1Migrations(migrationsPath);
return {
plugins: [
cloudflareTest({
main: './src/index.ts',
miniflare: {
compatibilityDate: '2025-01-01',
compatibilityFlags: ['nodejs_compat'],
bindings: {
CORS_ORIGIN: 'http://localhost:3000',
TEST_MIGRATIONS: migrations,
},
d1Databases: {
DB: {
id: 'test-db',
},
},
kvNamespaces: {
KV: {
id: 'test-kv',
},
},
},
}),
],
test: {
setupFiles: ['./tests/setup.ts'],
},
};
const migrationsPath = path.resolve(
__dirname,
'../../packages/db/migrations',
);
const migrations = await readD1Migrations(migrationsPath);
return {
plugins: [
cloudflareTest({
main: './src/index.ts',
miniflare: {
compatibilityDate: '2025-01-01',
compatibilityFlags: ['nodejs_compat'],
bindings: {
CORS_ORIGIN: 'http://localhost:3000',
TEST_MIGRATIONS: migrations,
},
d1Databases: {
DB: {
id: 'test-db',
},
},
kvNamespaces: {
KV: {
id: 'test-kv',
},
},
},
}),
],
test: {
setupFiles: ['./tests/setup.ts'],
},
};

Copilot uses AI. Check for mistakes.
Comment thread package.json
"destroy:prod": "pnpm run -r destroy:prod",
"deploy": "pnpm run -r deploy",
"destroy": "pnpm run -r destroy",
"test": "pnpm -r test",
Copy link

Copilot AI Apr 16, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The root test script runs recursively across all workspaces, but several packages (e.g. packages/api, packages/db, apps/tanstack, apps/web) do not define a test script. pnpm -r test will error with ...NO_SCRIPT in that case. Consider using pnpm -r --if-present test or filtering to just the workspaces that actually have integration tests (e.g. apps/server).

Suggested change
"test": "pnpm -r test",
"test": "pnpm -r --if-present test",

Copilot uses AI. Check for mistakes.
Comment on lines +4 to +23
testDir: './e2e',
timeout: 30000,
retries: 1,
expect: { timeout: 10000 },
use: {
baseURL: 'http://localhost:3000',
headless: true,
video: 'retain-on-failure',
trace: 'retain-on-failure',
},
projects: [
{ name: 'chromium', use: { browserName: 'chromium' } },
],
webServer: {
command: 'pnpm run dev',
cwd: '../../',
url: 'http://localhost:3000',
reuseExistingServer: true,
timeout: 60000,
},
Copy link

Copilot AI Apr 16, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This file uses tab indentation, but the repo Biome config enforces 2-space indentation (biome.jsonformatter.indentStyle: "space", indentWidth: 2). Running biome format (or adjusting indentation) will prevent formatting churn and CI style failures.

Suggested change
testDir: './e2e',
timeout: 30000,
retries: 1,
expect: { timeout: 10000 },
use: {
baseURL: 'http://localhost:3000',
headless: true,
video: 'retain-on-failure',
trace: 'retain-on-failure',
},
projects: [
{ name: 'chromium', use: { browserName: 'chromium' } },
],
webServer: {
command: 'pnpm run dev',
cwd: '../../',
url: 'http://localhost:3000',
reuseExistingServer: true,
timeout: 60000,
},
testDir: './e2e',
timeout: 30000,
retries: 1,
expect: { timeout: 10000 },
use: {
baseURL: 'http://localhost:3000',
headless: true,
video: 'retain-on-failure',
trace: 'retain-on-failure',
},
projects: [
{ name: 'chromium', use: { browserName: 'chromium' } },
],
webServer: {
command: 'pnpm run dev',
cwd: '../../',
url: 'http://localhost:3000',
reuseExistingServer: true,
timeout: 60000,
},

Copilot uses AI. Check for mistakes.
Comment on lines +4 to +15
test('homepage loads', async ({ page }) => {
await page.goto('/');
await expect(page).toHaveTitle(/nn-stack|Next/i);
});

test('hello API route returns JSON', async ({ request }) => {
const res = await request.get('/api/hello');
expect(res.ok()).toBeTruthy();
const body = await res.json();
expect(body.message).toBe('Hello from Next.js on Cloudflare Workers!');
expect(body.timestamp).toBeDefined();
});
Copy link

Copilot AI Apr 16, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This file is indented with tabs, which conflicts with the repo’s Biome formatter settings (2 spaces). Please run the formatter or adjust indentation to avoid style CI issues.

Suggested change
test('homepage loads', async ({ page }) => {
await page.goto('/');
await expect(page).toHaveTitle(/nn-stack|Next/i);
});
test('hello API route returns JSON', async ({ request }) => {
const res = await request.get('/api/hello');
expect(res.ok()).toBeTruthy();
const body = await res.json();
expect(body.message).toBe('Hello from Next.js on Cloudflare Workers!');
expect(body.timestamp).toBeDefined();
});
test('homepage loads', async ({ page }) => {
await page.goto('/');
await expect(page).toHaveTitle(/nn-stack|Next/i);
});
test('hello API route returns JSON', async ({ request }) => {
const res = await request.get('/api/hello');
expect(res.ok()).toBeTruthy();
const body = await res.json();
expect(body.message).toBe('Hello from Next.js on Cloudflare Workers!');
expect(body.timestamp).toBeDefined();
});

Copilot uses AI. Check for mistakes.
expect(await res.text()).toBe('Hello nn stack server!');
});

it('GET /rpc/planet/list returns 8 planets', async () => {
Copy link

Copilot AI Apr 16, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The test title says GET /rpc/planet/list..., but the rpc() helper always issues a POST request. Please update the description to match the actual HTTP method to avoid confusion when reading test output.

Suggested change
it('GET /rpc/planet/list returns 8 planets', async () => {
it('POST /rpc/planet/list returns 8 planets', async () => {

Copilot uses AI. Check for mistakes.
Comment thread .mcp.json
Comment on lines +7 to +14
"playwright-test": {
"command": "npx",
"args": [
"playwright",
"run-test-mcp-server",
"--config",
"apps/web/playwright.config.ts"
]
Copy link

Copilot AI Apr 16, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new playwright-test MCP server is started via npx playwright ..., which may download and run a different Playwright version than the one pinned in the workspace (or fail if the bin isn’t available at the repo root). To keep tooling reproducible, consider invoking the locally installed Playwright via pnpm exec (optionally with --filter @nn-stack/web) instead of npx playwright.

Copilot uses AI. Check for mistakes.
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 16 out of 17 changed files in this pull request and generated 7 comments.

Files not reviewed (1)
  • pnpm-lock.yaml: Language not supported

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread package.json
"destroy:prod": "pnpm run -r destroy:prod",
"deploy": "pnpm run -r deploy",
"destroy": "pnpm run -r destroy",
"test": "pnpm -r test",
Copy link

Copilot AI Apr 22, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pnpm -r test will fail because most workspaces do not define a test script (only apps/server does). Use pnpm -r --if-present test or narrow the command (e.g., filter to @nn-stack/server) so pnpm test reliably runs in this repo.

Suggested change
"test": "pnpm -r test",
"test": "pnpm -r --if-present test",

Copilot uses AI. Check for mistakes.
Comment on lines +17 to +22
webServer: {
command: 'pnpm run dev',
cwd: '../../',
url: 'http://localhost:3000',
reuseExistingServer: true,
timeout: 60000,
Copy link

Copilot AI Apr 22, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Playwright webServer.command runs pnpm run dev from the repo root, which (per root package.json) starts all workspace dev servers via pnpm -r dev. This can slow E2E runs and may introduce port conflicts/unrelated failures; consider starting only the web app (and any required deps) via pnpm --filter @nn-stack/web dev or using the existing root dev:web script instead.

Copilot uses AI. Check for mistakes.
Comment thread docs/testing.md
Comment on lines +5 to +12
| Workspace | Tool | Type | Notes |
|-----------|------|------|-------|
| `apps/server` | Vitest + `@cloudflare/vitest-pool-workers` | Integration | Real D1/KV via miniflare, `import app` + `app.fetch()` |
| `apps/web` | Playwright | E2E | Against running dev server |
| `packages/api` | None | — | Covered by `apps/server` tests (see below) |
| `packages/db` | None | — | Schema-only, no runtime logic |
| `packages/ui` | None | — | Shadcn components, covered by E2E |
| `packages/config` | None | — | tsconfig only |
Copy link

Copilot AI Apr 22, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The table markup uses || at the start of each row, which renders as an extra empty column in Markdown. Use single pipes (|) for the header and rows so the table renders correctly.

Copilot uses AI. Check for mistakes.
Comment thread docs/testing.md

```bash
npx playwright init-agents --loop=claude
pnpm test # all integration tests
Copy link

Copilot AI Apr 22, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The documented pnpm test command here assumes it runs “all integration tests”, but the root test script currently runs pnpm -r test and will fail for workspaces without a test script unless --if-present/filters are used. Update this section to match the actual root scripts/behavior.

Suggested change
pnpm test # all integration tests
pnpm test # runs the root recursive test script (`pnpm -r test`); may fail in workspaces without a `test` script

Copilot uses AI. Check for mistakes.
Comment thread apps/web/e2e/seed.spec.ts
Comment on lines +1 to +6
import { test, expect } from '@playwright/test';

test.describe('Test group', () => {
test('seed', async ({ page }) => {
// generate code here.
});
Copy link

Copilot AI Apr 22, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

seed.spec.ts currently contains a placeholder test that does nothing (and a placeholder comment). If this file is meant to bootstrap state for generated tests, implement the seeding/setup or mark the test as skipped with an explanation; otherwise it becomes a misleading always-green test in CI.

Suggested change
import { test, expect } from '@playwright/test';
test.describe('Test group', () => {
test('seed', async ({ page }) => {
// generate code here.
});
import { test } from '@playwright/test';
test.describe('Test group', () => {
test.skip(
'seed',
'Pending implementation: this file is reserved for E2E seed/setup logic and should not be an always-green placeholder test.',
async ({ page }) => {
// Intentionally skipped until real seed/setup steps are implemented.
}
);

Copilot uses AI. Check for mistakes.
Comment thread pnpm-lock.yaml
Comment on lines +6078 to +6086
wrangler@4.80.0:
resolution: {integrity: sha512-2ZKF7uPeOZy65BGk3YfvqBCPo/xH1MrAlMmH9mVP+tCNBrTUMnwOHSj1HrZHgR8LttkAqhko0fGz+I4ax1rzyQ==}
engines: {node: '>=20.3.0'}
hasBin: true
peerDependencies:
'@cloudflare/workers-types': ^4.20260401.1
peerDependenciesMeta:
'@cloudflare/workers-types':
optional: true
Copy link

Copilot AI Apr 22, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

wrangler@4.80.0 declares a peer dependency on @cloudflare/workers-types ^4.20260401.1, but this lockfile installs it with @cloudflare/workers-types@4.20260317.1. This mismatch can cause peer warnings (or failures if strict peers are enabled later); consider bumping @cloudflare/workers-types in the catalog/lock to satisfy Wrangler’s peer range or pin Wrangler to a version compatible with the current workers-types.

Suggested change
wrangler@4.80.0:
resolution: {integrity: sha512-2ZKF7uPeOZy65BGk3YfvqBCPo/xH1MrAlMmH9mVP+tCNBrTUMnwOHSj1HrZHgR8LttkAqhko0fGz+I4ax1rzyQ==}
engines: {node: '>=20.3.0'}
hasBin: true
peerDependencies:
'@cloudflare/workers-types': ^4.20260401.1
peerDependenciesMeta:
'@cloudflare/workers-types':
optional: true
# Removed incompatible wrangler@4.80.0 lock entry; pin to wrangler@4.78.0
# so the locked @cloudflare/workers-types@4.20260317.1 satisfies Wrangler's peer range.

Copilot uses AI. Check for mistakes.
Comment on lines +10 to +15
TEST_MIGRATIONS: unknown[];
}
}
}

await applyD1Migrations(env.DB, env.TEST_MIGRATIONS as Parameters<typeof applyD1Migrations>[1]);
Copy link

Copilot AI Apr 22, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TEST_MIGRATIONS is typed as unknown[] but then cast to Parameters<typeof applyD1Migrations>[1] at the call site. Prefer typing TEST_MIGRATIONS directly as Parameters<typeof applyD1Migrations>[1] (or the specific migration type) to avoid the unsafe cast.

Suggested change
TEST_MIGRATIONS: unknown[];
}
}
}
await applyD1Migrations(env.DB, env.TEST_MIGRATIONS as Parameters<typeof applyD1Migrations>[1]);
TEST_MIGRATIONS: Parameters<typeof applyD1Migrations>[1];
}
}
}
await applyD1Migrations(env.DB, env.TEST_MIGRATIONS);

Copilot uses AI. Check for mistakes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants