Mutation Testing

yoyo uses cargo-mutants to assess test quality. Mutation testing works by making small changes (mutants) to the source code — flipping conditions, replacing return values, removing function bodies — and checking whether any test catches each change.

If a mutant survives (no test fails), it means that line of code isn't actually tested.

Baseline

As of Day 9, yoyo has 1004 total mutants across its source files. This number grows as features are added. The mutation testing setup uses a 20% maximum survival rate threshold — if more than 20% of tested mutants survive, the check fails.

Install cargo-mutants

cargo install cargo-mutants

Quick start with the threshold script

The easiest way to run mutation testing is with the threshold script:

# Run with default 20% threshold
./scripts/run_mutants.sh

# Run with a stricter threshold
./scripts/run_mutants.sh --threshold 10

# Just count mutants without running them
./scripts/run_mutants.sh --list

# Test mutants in a specific file only
./scripts/run_mutants.sh --file src/format.rs

The script:

1. Runs cargo mutants on the project
2. Counts caught vs survived mutants
3. Calculates the survival rate
4. Exits with code 1 if the rate exceeds the threshold
5. Prints surviving mutants on failure so you know what to fix

This makes it easy for maintainers to run locally and could be added to CI by the project owner.

Run mutation testing directly

From the project root:

# Run all mutants (this takes a while — several minutes)
cargo mutants

# Show only the surviving mutants (uncaught mutations)
cargo mutants -- --survived

# Run mutants for a specific file
cargo mutants -f src/format.rs

# Run mutants for a specific function
cargo mutants -F "format_cost"

Read the results

After a run, cargo-mutants creates a mutants.out/ directory with detailed results:

# Summary
cat mutants.out/caught.txt     # mutants killed by tests ✓
cat mutants.out/survived.txt   # mutants NOT caught — test gaps!
cat mutants.out/timeout.txt    # mutants that caused infinite loops
cat mutants.out/unviable.txt   # mutants that didn't compile

Focus on survived.txt — each line is a mutation that no test catches. These are the weak spots.

Configuration

The mutants.toml file in the project root excludes known-acceptable mutants:

• Cosmetic functions — ANSI color codes, banner printing, help text
• Interactive I/O — functions that read stdin or require a terminal
• Async API calls — prompt execution that needs a live Anthropic API

These exclusions keep mutation testing focused on logic that should be tested. If you add a new feature with testable logic, make sure it's not excluded.

Writing targeted tests

When you find a surviving mutant:

1. Read what the mutation does (e.g., "replace < with <= in format_cost")
2. Write a test that specifically catches that boundary condition
3. Re-run cargo mutants -F "function_name" to verify the mutant is now caught

Example workflow:

# Find surviving mutants
cargo mutants 2>&1 | grep "SURVIVED"

# Write a test to kill the mutant, then verify
cargo mutants -F "format_cost"

Threshold script for CI

The scripts/run_mutants.sh script is designed to be CI-friendly:

# In a CI pipeline or pre-merge check:
./scripts/run_mutants.sh --threshold 20

# Exit codes:
#   0 = survival rate within threshold (PASS)
#   1 = survival rate exceeds threshold (FAIL)

The project owner can add this to CI workflows when ready. For now, contributors should run it locally before submitting PRs that add new logic.

When to run

Mutation testing is slow — it builds and tests your code once per mutant. Run it:

• After adding a new feature, to verify test coverage
• Before a release, as a quality check
• When you suspect the test suite has gaps
• On specific files with --file to keep it fast during development

Notes for CI integration

The scripts/run_mutants.sh script and mutants.toml config are ready for a human maintainer to wire into CI. A few things to know:

• Git-dependent tests: Some tests (e.g. test_git_branch_returns_something_in_repo, test_build_project_tree_runs, test_get_staged_diff_runs) gracefully handle running outside a git repo. cargo-mutants copies source to a temp directory without .git/, so these tests skip git-specific assertions when not in a repo.
• Exclusions are reasonable: The mutants.toml excludes cosmetic/display functions (ANSI colors, banners), interactive I/O (stdin, terminal), and async API calls (needs live Anthropic key). These can't be meaningfully unit-tested.
• The script cannot be added to .github/workflows/ by the agent (safety rules), but it exits with code 0/1 and is designed for CI use.