Skip to content

testing-guidelines.md

Latest commit

 

History

History
320 lines (223 loc) · 6.71 KB

testing-guidelines.md

File metadata and controls

320 lines (223 loc) · 6.71 KB

Testing Principles

We deliberately test at the edges of the system this gives us:

  • Freedom to refactor
  • Confidence to optimise
  • Stable, long-lived tests
  • Reduced coupling to implementation details

We write high-level, black-box integration tests that focus only on observable outcomes (behaviour not implementation details).

If a refactor breaks a test without changing observable behaviour, the test was too coupled.

Test Projects

We maintain two test suites:

./Tests        → CLI tests
./Tests.E2e    → Blazor Web UI tests (Playwright)

All tests must pass before a PR can be merged into main.

If something is worth testing manually, it’s worth automating.

When adding features:

  • Add tests with them
  • If tests are missing, add them
  • If behaviour changes, update tests intentionally

When fixing bugs:

  • Reproduce manually
  • Reproduce repeatably with a failing test
  • Make the test pass

Web UI (E2E) Tests

We use Playwright to test the Blazor Server app.

These tests are:

  • Slow(ish)
  • Primarily happy-path + critical flows

Avoid:

  • Testing styling details
  • Over-asserting UI minutiae

Running E2E Tests

You must build the Docker image before running:

cd RackPeek
docker build -t rackpeek:ci -f RackPeek.Web/Dockerfile .

cd Tests.E2e
dotnet tool install --global Microsoft.Playwright.CLI
dotnet build

playwright install
dotnet test

Rebuild the image whenever the Web project changes.

Page Object Model (POM)

Each page/component has a POM (Page Object Model) abstraction.

The POM:

  • Encapsulates selectors
  • Encapsulates browser interactions
  • Exposes intent-level methods (AddDesktopAsync, GotoHardwareAsync)
  • Shields tests from UI churn

Tests should read like workflows — not like browser scripts.

Example:

[Fact]
public async Task User_Can_Add_And_Delete_Desktop()
{
    var (context, page) = await CreatePageAsync();
    var resourceName = $"e2e-ap-{Guid.NewGuid():N}"[..16];

    try
    {
        await page.GotoAsync(fixture.BaseUrl);

        var layout = new MainLayoutPom(page);
        await layout.AssertLoadedAsync();
        await layout.GotoHardwareAsync();

        var hardwarePage = new HardwareTreePom(page);
        await hardwarePage.AssertLoadedAsync();
        await hardwarePage.GotoDesktopsListAsync();

        var listPage = new DesktopsListPom(page);
        await listPage.AssertLoadedAsync();
        await listPage.AddDesktopAsync(resourceName);
        await listPage.AssertDesktopExists(resourceName);
        await listPage.DeleteDesktopAsync(resourceName);
        await listPage.AssertDesktopDoesNotExist(resourceName);
    }
    catch (Exception)
    {
        _output.WriteLine("TEST FAILED — Capturing diagnostics");
        _output.WriteLine($"Current URL: {page.Url}");

        var html = await page.ContentAsync();
        _output.WriteLine("==== DOM SNAPSHOT START ====");
        _output.WriteLine(html);
        _output.WriteLine("==== DOM SNAPSHOT END ====");

        throw;
    }
    finally
    {
        await context.CloseAsync();
    }
}

Good E2E Test Traits

  • Single responsibility
  • Independent (no ordering dependencies)
  • Idempotent
  • Generates unique test data
  • Cleans up after itself
  • Fails with useful diagnostics

Debugging E2E Tests

You may temporarily modify:

Tests.E2e/infra/PlaywrightFixture.cs

To debug visually:

Browser = await _playwright.Chromium.LaunchAsync(new BrowserTypeLaunchOptions
{
    Headless = false,
    SlowMo = 500,
    Args = new[]
    {
        "--disable-dev-shm-usage",
        "--no-sandbox"
    }
});

⚠ Before committing, revert to CI-safe settings:

Browser = await _playwright.Chromium.LaunchAsync(new BrowserTypeLaunchOptions
{
    Headless = true,
    Args = new[]
    {
        "--disable-dev-shm-usage",
        "--no-sandbox"
    }
});

CI must always run headless.

CLI Tests

CLI tests are faster and more precise.

They behave more like unit/integration hybrids (intragrationtests if you like):

  • Validate both happy + unhappy paths
  • Assert command output
  • Assert YAML written to disk
  • Avoid UI overhead

Run them with:

cd Tests
dotnet test

Example:

[Fact]
public async Task servers_tree_cli_workflow_test()
{
    await File.WriteAllTextAsync(Path.Combine(fs.Root, "config.yaml"), "");

    var (output, yaml) = await ExecuteAsync("servers", "add", "node01");
    Assert.Equal("Server 'node01' added.\n", output);
    Assert.Contains("name: node01", yaml);

    (output, yaml) = await ExecuteAsync("systems", "add", "host01");
    Assert.Equal("System 'host01' added.\n", output);

    (output, yaml) = await ExecuteAsync("systems", "add", "host02");
    Assert.Equal("System 'host02' added.\n", output);

    (output, yaml) = await ExecuteAsync("systems", "add", "host03");
    Assert.Equal("System 'host03' added.\n", output);

    (output, yaml) = await ExecuteAsync(
        "systems", "set", "host01",
        "--runs-on", "node01"
    );
    Assert.Equal("System 'host01' updated.\n", output);

    (output, yaml) = await ExecuteAsync("services", "add", "immich");
    Assert.Equal("Service 'immich' added.\n", output);

    (output, yaml) = await ExecuteAsync("servers", "tree", "node01");
    Assert.Equal("""
                 node01
                 ├── System: host01
                 │   ├── Service: immich
                 │   └── Service: paperless
                 ├── System: host02
                 └── System: host03

                 """, output);
}

CLI Testing Best Practices

  • Assert exact output where meaningful
  • Validate file side effects
  • Test invalid arguments and failure modes
  • Avoid brittle whitespace assertions unless intentional
  • Keep tests deterministic
  • Avoid shared filesystem state

General Testing Standards

1. Behaviour First

Test what the user observes — not how we implement it.

2. Prefer Integration Over Micro-Mocking

We value confidence over isolation purity.

3. Fast Feedback

  • CLI tests should be fast
  • E2E tests should be minimal but meaningful

4. Tests Are Documentation

A good test explains:

  • What the feature does
  • How it’s expected to behave
  • What regressions look like

5. Stability > Coverage %

High-value tests matter more than coverage numbers.

6. No Flaky Tests

If a test is flaky:

  • Fix it immediately
  • Or remove it
  • Flaky tests erode trust

Definition of Done

A feature is complete when:

  • Behaviour is implemented
  • Tests exist
  • Tests pass locally
  • Tests pass in CI
  • Edge cases are covered
  • No debug flags remain enabled

We optimise for:

  • Confidence
  • Refactorability
  • Clarity
  • Long-term maintainability

Tests are a first-class citizen of this project.