Add mutation testing with StrykerJS to evaluate test suite quality

[mostronatorcoder]

What is Mutation Testing?

Mutation testing is a technique that evaluates the quality of your test suite by introducing small changes (mutations) to the source code and checking if the tests detect them.

How it works

1. A mutation tool takes your source code and creates small variations called mutants — for example:

   • Changing > to >=
   • Replacing true with false
   • Removing a function call
   • Changing + to -
   • Replacing a string with an empty string

2. For each mutant, the tool runs your test suite

3. If the tests fail → the mutant was killed ✅ (tests caught the change)

4. If the tests pass → the mutant survived ❌ (tests missed the change)

A high mutation score (% of killed mutants) means your tests are effective at catching bugs. A low score means there are code paths where bugs could hide undetected.

Why it matters

Code coverage only tells you which lines were executed during tests — not whether the tests actually verify correct behavior. A test that calls a function but never asserts its return value gives 100% coverage but catches 0% of bugs.

Mutation testing answers a deeper question: "If a bug were introduced here, would our tests catch it?"

Concrete example from this codebase

Consider util/index.ts — functions like getBtcFiatPrice(), numberFormat(), getCurrency(), etc. A mutation tool might:

Original	Mutant	What it tests
if (amount > 0)	if (amount >= 0)	Does the test check the zero boundary?
return rate * amount	return rate / amount	Does the test verify the calculation result?
user.banned = true	user.banned = false	Does the test assert the ban actually applied?
order.status = 'ACTIVE'	order.status = ''	Does the test check the status value?

If any of these mutants survive, it means a real bug in that code path would also go undetected.

Current Test Coverage

The project has 5 test files covering 70+ source files:

Test file	What it covers
tests/bot/bot.spec.ts	Bot commands
tests/bot/validation.spec.ts	Input validation
tests/ln/lightning.spec.ts	Lightning operations
tests/monitoring.spec.ts	Monitoring module
tests/util/index.spec.ts	Utility functions

This is a good starting point, but mutation testing would reveal which of these tests are truly effective at catching bugs vs. just executing code paths.

Recommended Tool: StrykerJS

StrykerJS is the most mature mutation testing framework for JavaScript/TypeScript projects. It supports:

• ✅ TypeScript (first-class support)
• ✅ Mocha test runner (our current setup)
• ✅ Incremental mode (only test changed files)
• ✅ HTML reports with detailed per-file mutation scores
• ✅ CI integration with dashboard
• ✅ Configurable mutators (choose which mutations to apply)

Alternatives Considered

Tool	Verdict
StrykerJS	✅ Best choice — mature, supports TS + Mocha, active community
mutode	❌ Abandoned, last release 2019
mutation-testing-elements	❌ Report viewer only, not a test runner

Implementation Plan

Phase 1: Local Setup

1. Install StrykerJS:

   npm install --save-dev @stryker-mutator/core @stryker-mutator/mocha-runner @stryker-mutator/typescript-checker

2. Create stryker.config.json:

   {
     "$schema": "./node_modules/@stryker-mutator/core/schema/stryker-schema.json",
     "packageManager": "npm",
     "testRunner": "mocha",
     "checkers": ["typescript"],
     "tsconfigFile": "tsconfig.test.json",
     "coverageAnalysis": "perTest",
     "mutate": [
       "bot/**/*.ts",
       "util/**/*.ts",
       "ln/**/*.ts",
       "models/**/*.ts",
       "!**/*.spec.ts",
       "!**/mocks/**"
     ],
     "mochaOptions": {
       "spec": ["dist/tests/**/*.spec.js"],
       "config": ".mocharc.yml"
     },
     "thresholds": {
       "high": 80,
       "low": 60,
       "break": null
     },
     "reporters": ["html", "clear-text", "progress"],
     "timeoutMS": 60000,
     "concurrency": 2
   }

3. Add npm script:

   {
     "scripts": {
       "mutation-test": "stryker run",
       "mutation-test:util": "stryker run --mutate 'util/**/*.ts'"
     }
   }

Phase 2: CI Workflow (Non-Blocking)

Create .github/workflows/mutation-testing.yaml:

name: Mutation Testing

on:
  push:
    branches: [main]
  schedule:
    - cron: '0 6 * * 1'  # Weekly on Monday 6:00 UTC
  workflow_dispatch:

jobs:
  mutation-test:
    runs-on: ubuntu-latest
    container:
      image: 'ubuntu:24.04'
    steps:
      - uses: actions/checkout@v4

      - name: Setup Node.js
        uses: actions/setup-node@v4
        with:
          node-version: '20'

      - name: Setup MongoDB
        run: |
          apt update && apt install -y sudo curl gnupg
          sudo rm -f /etc/ssl/certs/ca-bundle.crt
          sudo apt reinstall -y ca-certificates
          sudo update-ca-certificates
          curl -fsSL https://pgp.mongodb.com/server-6.0.asc | \
            sudo gpg -o /usr/share/keyrings/mongodb-server-6.0.gpg --dearmor
          echo "deb [ arch=amd64,arm64 signed-by=/usr/share/keyrings/mongodb-server-6.0.gpg ] https://repo.mongodb.org/apt/ubuntu jammy/mongodb-org/6.0 multiverse" | \
            sudo tee /etc/apt/sources.list.d/mongodb-org-6.0.list
          sudo apt-get update && sudo apt-get install -y mongodb-org
          sudo mongod --quiet --config /etc/mongod.conf &
          sleep 10
          mongosh --eval 'db.getSiblingDB("lnp2pbot").createCollection("mycollection")'

      - run: npm ci
      - run: npx tsc

      - name: Run mutation tests
        continue-on-error: true  # Non-blocking initially
        env:
          DB_USER: ''
          DB_PASS: ''
          DB_HOST: '127.0.0.1'
          DB_PORT: '27017'
          DB_NAME: 'lnp2pbot'
        run: npx stryker run

      - name: Upload mutation report
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: mutation-report
          path: reports/mutation/
          retention-days: 30

Phase 3: Incremental Improvement

Once the baseline is established:

1. Identify modules with lowest mutation scores
2. Write targeted tests for surviving mutants
3. Gradually increase the thresholds.break value to enforce minimum mutation score
4. Add incremental mutation testing to PRs (only mutate changed files)

Suggested Starting Point

Start with util/ — it has the most isolated, pure functions that are easiest to test thoroughly:

• util/index.ts — price calculations, formatting, validation
• ln/ — Lightning payment logic (critical path)
• bot/validation.ts — input validation (already has tests)

These modules have clear inputs/outputs and are the most impactful to verify.

Benefits Summary

Benefit	Impact
Find weak tests	Discover tests that execute code without verifying behavior
Prevent regressions	Ensure critical logic (payments, orders, disputes) is properly guarded
Guide test writing	Know exactly which code paths need better tests
Confidence in refactoring	Mutation score tells you if your safety net has holes
Complement coverage	Coverage = breadth, mutation score = depth

References

• StrykerJS Documentation
• Mutation Testing on Wikipedia
• MostroP2P/mostro#618 — Same initiative for the Rust codebase
• grunch/choke#37 — Same initiative for the Flutter codebase