Recipe: Expand input coverage with property-based testing and fuzzing

[moodmosaic]

Input coverage = how well we test the range of inputs, not just which lines or branches we hit. For example:

fn is_even(n: u32) -> bool {
    n % 2 == 0
}

Here, testing only n = 2 gives 100% line coverage but misses 0, 1, u32::MAX, etc. That’s low input coverage. We want to hit edge cases, boundaries, and weird values.

By pairing property-based testing (proptest via cargo test) with coverage-guided fuzzing (libFuzzer via cargo +nightly fuzz), we can explore code paths with broader, more meaningful input variation.

[moodmosaic]

This recipe shows how to expand input coverage using property-based testing and fuzzing in the stacks-core codebase. It builds on work I did with @BowTiedRadone, whose fuzzing contributions were especially helpful.

---

Property-Based Testing with proptest

Property-based testing generates many inputs automatically and checks that properties hold across all of them. Instead of writing individual test cases, you write properties that should always be true. For a broader overview, see this post on property-based testing in Rust and Go.

Basic proptest Example

Here's how to convert a traditional unit test to a property-based test, taken from the representations tests:

use proptest::prelude::*;

// Traditional test - limited input coverage.
#[test]
fn test_clarity_name_specific_case() {
    let name = ClarityName::try_from("hello".to_string()).unwrap();
    assert_eq!(name.as_str(), "hello");
}

// Property-based test - broad input coverage.
#[test]
fn prop_clarity_name_roundtrip() {
    proptest!(|(s in any_valid_clarity_name())| {
        let name = ClarityName::try_from(s.clone()).unwrap();
        prop_assert_eq!(name.as_str(), s);

        let mut buf = Vec::new();
        name.consensus_serialize(&mut buf).unwrap();
        let back = ClarityName::consensus_deserialize(&mut buf.as_slice()).unwrap();
        prop_assert_eq!(back, name);
    });
}

The any_valid_clarity_name() strategy generates thousands of valid clarity names automatically, testing the roundtrip property across all of them.

Notes:

• Prefer generators that encode constraints over prop_assume! to avoid high discard rates. Anchor generators to real validators (e.g., regex constants) to prevent drift.
• Add an explicit assertion that the generator matches the regex literal in code to catch accidental changes:
  assert_eq!(CLARITY_NAME_REGEX_STRING, EXPECTED_REGEX_STR);
• Keep properties pure and fast (no I/O). Target a few ms per case so you can run them in cargo test.
• Add a negative strategy alongside the positive one (e.g., any_invalid_clarity_name) to catch off-by-one and grammar blind spots.

---

Custom Strategies

Create custom generators for your domain types:

fn any_valid_clarity_name() -> impl Strategy<Value = String> {
    let letter_names = string_regex(&format!(
        "[a-zA-Z][a-zA-Z0-9_!?+<>=/*-]{{0,{}}}",
        (MAX_STRING_LEN as usize).saturating_sub(1)
    ))
    .unwrap();

    let single_ops = prop_oneof![
        Just("-".to_string()),
        Just("+".to_string()),
        Just("=".to_string()),
        Just("/".to_string()),
        Just("*".to_string()),
    ];

    let comparison_ops = prop_oneof![
        Just("<".to_string()),
        Just(">".to_string()),
        Just("<=".to_string()),
        Just(">=".to_string()),
    ];

    prop_oneof![letter_names, single_ops, comparison_ops]
}

This strategy systematically covers all valid patterns in the Clarity name grammar.

Notes:

• Also define any_invalid_clarity_name() (starts with digit, disallowed chars, too long) and assert it fails parsing. This dual-strategy pattern found real issues in production.
• Prefer prop_map for boundary sizes to avoid rejects; e.g., generate MAX_STRING_LEN +/- 1 explicitly when testing limits.

---

Model-Based State Machine Testing with madhouse-rs

While property-based testing with proptest excels at testing pure functions and stateless properties, many systems in stacks-core involve complex state machines where the sequence of operations matters. madhouse-rs fills this gap by providing model-based testing for stateful systems.

Instead of generating individual inputs, madhouse-rs generates sequences of commands that represent state transitions. Each command encapsulates:

• Preconditions: When the command can be applied (check()).
• State changes: How it modifies the system (apply()).
• Invariants: What should remain true after execution.

Setting Up madhouse-rs

Add madhouse-rs to your dependencies:

[dev-dependencies]
madhouse = { git = "https://github.com/stacks-network/madhouse-rs.git", tag = "0.2.0" }
proptest = "1.6.*"

Command Pattern

The core abstraction is the Command trait, which represents a single state transition:

use madhouse::prelude::*;
use proptest::prelude::*;
use std::sync::Arc;

// Define your system state.
#[derive(Debug, Default)]
struct MinerState {
    current_block_height: u64,
    pending_commits: Vec<BlockCommit>,
    active_miners: Vec<MinerId>,
}

impl State for MinerState {}

// Define test context for parameterizing command generation.
#[derive(Debug, Clone)]
struct MinerContext {
    max_miners: usize,
    block_time_ms: u64,
}

impl TestContext for MinerContext {}

// Define a command to submit a block commit.
struct SubmitBlockCommitCommand {
    miner_id: MinerId,
    burn_fee: u64,
}

impl Command<MinerState, MinerContext> for SubmitBlockCommitCommand {
    fn check(&self, state: &MinerState) -> bool {
        // Only allow commits from active miners.
        state.active_miners.contains(&self.miner_id)
    }

    fn apply(&self, state: &mut MinerState) {
        state.pending_commits.push(BlockCommit {
            miner_id: self.miner_id,
            burn_fee: self.burn_fee,
            block_height: state.current_block_height,
        });
        
        // Invariant: Pending commits should never exceed active miners.
        assert!(state.pending_commits.len() <= state.active_miners.len());
    }

    fn label(&self) -> String {
        format!("SUBMIT_COMMIT(miner={}, fee={})", self.miner_id, self.burn_fee)
    }

    fn build(ctx: Arc<MinerContext>) -> impl Strategy<Value = CommandWrapper<MinerState, MinerContext>> {
        (0..ctx.max_miners, 1000..10000u64).prop_map(|(miner_id, fee)| {
            CommandWrapper::new(SubmitBlockCommitCommand { 
                miner_id: MinerId(miner_id), 
                burn_fee: fee 
            })
        })
    }
}

Running Scenarios

Use the scenario! macro to define test scenarios that exercise sequences of commands:

#[test]
fn test_miner_competition_scenario() {
    let ctx = Arc::new(MinerContext {
        max_miners: 3,
        block_time_ms: 10000,
    });

    scenario![
        ctx,
        InitializeMinersCommand,
        SubmitBlockCommitCommand,
        MineBitcoinBlockCommand,
        ProcessSortitionCommand,
        VerifyWinnerCommand
    ];
}

Two execution modes:

• Deterministic mode (default): Commands run in the specified order with proptest-generated parameters.
• Random mode (MADHOUSE=1): Commands are selected randomly, testing different interleaving patterns.

Integration with stacks-core

The signer coordination tests show madhouse-rs testing complex reorg scenarios:

fn allow_reorg_within_first_proposal_burn_block_timing_secs_scenario() {
    if env::var("BITCOIND_TEST") != Ok("1".into()) {
        return;
    }

    let num_signers = 5;
    let num_transfer_txs = 3;
    let test_context = Arc::new(SignerTestContext::new(num_signers, num_transfer_txs));

    scenario![
        test_context,
        SkipCommitOpMiner2,
        BootToEpoch3,
        SkipCommitOpMiner1,
        PauseStacksMining,
        MineBitcoinBlock,
        VerifyMiner1WonSortition,
        SubmitBlockCommitMiner2,
        ResumeStacksMining,
        WaitForTenureChangeBlockFromMiner1,
        MineBitcoinBlock,
        VerifyMiner2WonSortition,
        VerifyLastSortitionWinnerReorged,
        WaitForTenureChangeBlockFromMiner2,
        ShutdownMiners
    ];
}

This scenario tests a specific reorg timing by orchestrating multiple miners and verifying that sortition winners can be properly reorged under certain conditions.

Command Design Patterns

Concrete vs Generated Commands:

scenario![
    ctx,
    // Generated command - parameters vary across runs.
    SubmitBlockCommitCommand,
    
    // Concrete command - fixed parameters.
    (MineBitcoinBlockCommand { target_height: 100 }),
    
    // Mixed approach.
    ProcessSortitionCommand,
    (VerifyWinnerCommand { expected_miner: MinerId(1) })
];

Conditional Commands:

impl Command<MinerState, MinerContext> for ProcessSortitionCommand {
    fn check(&self, state: &MinerState) -> bool {
        // Only process sortition if we have pending commits.
        !state.pending_commits.is_empty()
    }

    fn apply(&self, state: &mut MinerState) {
        // Simulate sortition logic.
        if let Some(winner) = select_sortition_winner(&state.pending_commits) {
            state.current_block_height += 1;
            state.pending_commits.clear();
        }
    }
    
    // ... rest of implementation.
}

Notes:

• Keep check() fast and pure - it's called frequently during command filtering.
• Put invariants in apply() as assertions. Failed invariants will trigger test shrinking.
• Use descriptive label() strings for debugging failed scenarios.
• Prefer smaller, composable commands over large monolithic ones.

Test Shrinking

When a scenario fails, madhouse-rs automatically shrinks the command sequence to find minimal failing cases:

# Enable shrinking to find minimal failure.
PROPTEST_MAX_SHRINK_ITERS=50 cargo test mining_scenario

# Use random mode to explore different orderings.
MADHOUSE=1 PROPTEST_MAX_SHRINK_ITERS=10 cargo test mining_scenario

The block assembly tests demonstrate shrinking in practice - a complex failure with 15 commands might shrink down to just 3 commands that reproduce the same bug.

Performance Considerations

Stateful tests can be expensive. Optimize for fast iteration:

#[derive(Debug, Clone)]
struct TestContext {
    use_mock_bitcoin: bool,
    fast_mode: bool,
}

impl Command<MinerState, TestContext> for MineBitcoinBlockCommand {
    fn apply(&self, state: &mut MinerState) {
        if self.ctx.use_mock_bitcoin {
            // Fast mock implementation.
            state.current_block_height += 1;
        } else {
            // Full integration with bitcoind.
            mine_real_bitcoin_block();
        }
    }
}

Notes:

• Use in-memory databases and mock external services for fast iteration.
• Keep the state representation minimal - only model what's necessary for testing.
• Consider splitting expensive integration scenarios from fast unit-level state machine tests.

Comparison with Traditional Property Testing

Aspect	proptest	madhouse-rs
Focus	Individual inputs, pure functions	Command sequences, stateful systems
Generation	Random inputs	Random command sequences + Random inputs
Assertions	Properties over outputs	Invariants over state transitions
Use cases	Parsers, codecs, algorithms	Consensus protocols, state machines
Shrinking	Input minimization	Command sequence minimization

Both approaches complement each other in a comprehensive testing strategy. Use proptest for testing individual components and madhouse-rs for testing how those components interact over time.

---

This positions madhouse-rs as a natural evolution from basic property testing to more sophisticated stateful testing, which fits well with the document's progression from simple to complex testing techniques.

---

Coverage-Guided Fuzzing with libFuzzer

libFuzzer uses coverage feedback to guide input generation toward unexplored code paths. It's especially good at finding parsing bugs and crashes. For more background, see this introduction to coverage-guided fuzzing in Rust and Go.

Setting Up libFuzzer

Add a fuzz target in clarity/fuzz/fuzz_targets/:

#![no_main]

use libfuzzer_sys::fuzz_target;

fuzz_target!(|s: &str| {
    let _ = clarity::vm::execute_v2(s);
});

This raw string harness feeds arbitrary strings to the Clarity VM to find parsing crashes.

Notes:

• Start with raw-bytes targets for parsers; add a dictionary for language-like inputs to jumpstart deeper paths.
• Use -use_value_profile=1 and a corpus/ of small seeds to improve path discovery.
• Keep the harness minimal: avoid filtering unless it saves lots of cycles (e.g., skip empty or all-whitespace inputs).
• List targets with cargo +nightly fuzz list. Keep each target single purpose (one API surface per bin).

---

Running libFuzzer

# Build the fuzz target.
cargo +nightly fuzz build

# Run the fuzzer (quick).
cargo +nightly fuzz run execute_v2_raw -- -print_final_stats=1

# With seeds and artifacts.
cargo +nightly fuzz run execute_v2_raw corpus/ \
  -- -artifact_prefix=artifacts/ -use_value_profile=1

# Generate coverage report.
cargo +nightly fuzz coverage execute_v2_raw

Notes:

• In stackslib/fuzz, use ./report-coverage.sh <target> to generate HTML + text summaries; it wraps llvm-cov and filters noise.

---

Structured Fuzzing with Arbitrary

For more sophisticated fuzzing, use the arbitrary crate to generate structured inputs:

use libfuzzer_sys::arbitrary::{Arbitrary, Unstructured};

#[derive(Debug)]
struct Input<'a> {
    height: usize,
    epoch: StacksEpochId,
    tx_data: &'a [u8],
}

impl<'a> Arbitrary<'a> for Input<'a> {
    fn arbitrary(u: &mut Unstructured<'a>) -> libfuzzer_sys::arbitrary::Result<Self> {
        let height = u.arbitrary::<u32>()? as usize;
        let epoch = *u.choose(&[
            StacksEpochId::Epoch20,
            StacksEpochId::Epoch21,
            StacksEpochId::Epoch30,
        ])?;
        let tx_data = u.bytes(u.len())?;
        Ok(Self { height, epoch, tx_data })
    }
}

fuzz_target!(|input: Input| {
    if let Ok(tx) = deserialize::<Transaction>(input.tx_data) {
        let parser = BitcoinBlockParser::new(
            BitcoinNetworkType::Mainnet,
            MagicBytes::default(),
        );
        let _ = parser.parse_tx(&tx, input.height, input.epoch);
    }
});

This structured harness generates valid transaction structures while still fuzzing the raw transaction bytes.

Notes:

• This mirrors the bitcoin_blocks_parser_parse_tx_raw approach: mix structured params with raw payload to reach deep logic fast.
• Reuse the same Arbitrary model in proptest when helpful, but split into raw-bytes and structured targets if discard rates climb.

---

AFL++ for Out-of-Process Fuzzing

AFL++ runs the target in a separate process, making it better at catching certain classes of bugs and providing different coverage heuristics than libFuzzer.

Setting Up AFL++

Create a separate AFL++ project in fuzz-clarity-cli/:

[dependencies]
afl = "0.15"
stackslib = { path = "../stackslib" }

[profile.release]
panic = "abort"  # Critical for crash detection in non-persistent mode.

Notes:

• Use persistent mode (afl::fuzz!) to avoid per-input fork/exec cost.
• Ensure panic = "abort" so panics become crashes; otherwise AFL will treat them as clean exits.
• Add other speed knobs in release profile (lto = true, codegen-units = 1) for throughput.
• Provide a clarity.dict with language tokens to guide mutations.

---

AFL++ Harness

use afl::fuzz;
use blockstack_lib::clarity_cli as clarity;

fn main() {
    fuzz!(|data: &[u8]| {
        let content = match std::str::from_utf8(data) {
            Ok(s) => s,
            Err(_) => return,
        };

        if content.trim().is_empty() {
            return;
        }

        // Use tmpfs for performance.
        let test_db = format!("/dev/shm/fuzz_test_{}.db", timestamp());
        let temp_contract = format!("/dev/shm/fuzz_contract_{}.clar", timestamp());

        // Initialize and test.
        let init_args = vec!["".to_string(), "initialize".to_string(),
            test_db.clone()];
        let (init_code, _) = clarity::invoke_command(&init_args[0],
            &init_args[1..]);
        if init_code != 0 {
            return;
        }

        std::fs::write(&temp_contract, content).ok()?;

        let contract_id = format!(
            "ST1HTBVD3JG9C05J7HBJTHGR0GGW7KXW28M5JS8QE.fuzz-{}",
            timestamp()
        );
        let launch_args = vec!["".to_string(), "launch".to_string(),
            contract_id, temp_contract.clone(), test_db.clone()];

        let _ = clarity::invoke_command(&launch_args[0], &launch_args[1..]);

        // Cleanup.
        let _ = std::fs::remove_file(&temp_contract);
        let _ = std::fs::remove_dir_all(&test_db);
    });
}

Notes:

• Keep inputs short at first (-m none -t 1000+) and rely on dictionary-guided growth.
• Parallelize with -M/-S workers; set AFL_TMPDIR=/dev/shm for faster I/O.

---

Running AFL++

# System setup (once).
cargo install afl
cargo afl system-config

# Build.
cargo afl build --release

# Single core.
cargo afl fuzz -i corpus/ -o out -x clarity.dict \
  ./target/release/fuzz-clarity-cli

# Multi-core (recommended).
cargo afl fuzz -i corpus/ -o out -M main -x clarity.dict \
  ./target/release/fuzz-clarity-cli
cargo afl fuzz -i corpus/ -o out -S worker1 -x clarity.dict \
  ./target/release/fuzz-clarity-cli

The clarity.dict file contains language keywords to guide mutations:

"define-public"
"define-private"
"define-read-only"
"let"
"begin"
"if"
"and"
"or"
"+="
"-="

---

Shared Arbitrary Models

You can share arbitrary::Arbitrary implementations between proptest and libFuzzer to ensure consistent generation logic. This hybrid approach is part of a larger trend where fuzzing meets property testing to combine the strengths of both.

use arbitrary::Arbitrary;

#[derive(Arbitrary, Debug)]
struct TransactionInput {
    amount: u64,
    recipient: String,
}

// Use with proptest.
proptest! {
    #[test]
    fn prop_transaction_serialization(tx in any::<TransactionInput>()) {
        let serialized = serialize(&tx).unwrap();
        let deserialized = deserialize(&serialized).unwrap();
        prop_assert_eq!(tx, deserialized);
    }
}

// Use with libFuzzer.
fuzz_target!(|tx: TransactionInput| {
    let _ = process_transaction(tx);
});

Notes:

• Pros: reuse generation logic, ensure consistency between tools.
• Cons: higher discard rates in proptest, less fuzzing depth in libFuzzer.
• If discards rise, split targets: keep a raw-bytes + dictionary target for parser fronts, and a structured Arbitrary target for deeper logic.

---

Mocking for Speed

Fuzzing works best when tests run fast. Mock I/O, networking, and database operations using mockall. For a deeper dive, see this post on mocking I/O to go faster:

use mockall::*;

#[automock]
trait BlockDownloader {
    fn download(&mut self, header: &BlockHeader) -> Result<Block, Error>;
}

#[test]
fn test_sync_with_mock_downloader() {
    let mut mock_downloader = MockBlockDownloader::new();
    mock_downloader
        .expect_download()
        .returning(|h| Ok(Block::stub_for_header(h)));

    let mut syncer = BlockSyncer::new(mock_downloader);
    let result = syncer.sync_range(1, 10);
    assert!(result.is_ok());
}

Notes:

• Traits with associated types need explicit glue; consider small wrapper traits for testability.
• Prefer static, closure-based returns for mock hot paths to keep cycles cheap in fuzz targets.

---

Practical Examples from stacks-core

Converting Unit Tests to Property Tests

The representations module conversion shows systematic conversion from unit tests to property-based tests:

• Before: Test specific values like "hello", "test-name".
• After: Generate all valid patterns and test invariants like roundtrip serialization. Also add invalid generators to ensure rejects.

---

Ephemeral Data Structure Testing

#[test]
fn prop_ephemeral_tip_height_matches_current() {
    proptest!(|(n in 1usize..=12)| {
        // Set up n blocks in a test MARF.
        let mut marf_ephemeral = setup_ephemeral_marf(n);

        // Invariant: ephemeral tip height equals current height.
        let height = marf_ephemeral.get_current_block_height();
        let open_height = marf_ephemeral.get_open_chain_tip_height();
        prop_assert_eq!(height, open_height);
    });
}

Notes:

• Keep setup bounded and deterministic. Use small n ranges so shrinking stays fast and failures are reproducible.

---

Raw Bytes Fuzzing

The execute_v2_raw target feeds arbitrary strings to the Clarity parser to find edge cases in the grammar implementation.

Notes:

• Pair the raw target with a clarity.dict to accelerate reaching the analyzer/interpreter.
• Measure coverage with cargo +nightly fuzz coverage execute_v2_raw.

---

Tips and Gotchas

libFuzzer Coverage Reports

cd stackslib/fuzz
./report-coverage.sh bitcoin_blocks_parser_parse_tx_raw
open coverage/bitcoin_blocks_parser_parse_tx_raw/html/index.html

Notes:

• The script validates toolchain, runs corpus replay, demangles, and filters std/Cargo noise, giving actionable stats.

---

AFL++ Performance Optimizations

• Use panic = "abort" in Cargo.toml - essential for crash detection. See Panics Are Not Crashes and this discussion for why this is important.
• Store temporary files in /dev/shm (tmpfs) for speed.
• Use persistent mode with afl::fuzz!.
• Provide a dictionary file for domain-specific mutations.
• Run multi-core fuzzing for better coverage.

Notes:

• Keep inputs short initially (-m none -t 1000+) and let AFL grow them.
• Set AFL_TMPDIR=/dev/shm to avoid slow disks.

---

Debugging Fuzz Failures

# Reproduce the crash.
echo "crash_input" | ./target/debug/fuzz_target

# Debug with GDB.
gdb --args ./target/debug/fuzz_target crash_file

Notes:

• Prefer running the minimized artifact produced by the fuzzer.
• For libFuzzer, check artifacts/ and re-run with -runs=1.

---

Resource Management

fuzz_target!(|data: &[u8]| {
    let temp_file = create_temp_file(data);

    // Test logic here.
    let _ = process_file(&temp_file);

    // Always cleanup.
    let _ = std::fs::remove_file(&temp_file);
});

Notes:

• For CLI fuzzing, prefer in-memory DB paths under /dev/shm and reuse allocations where possible.

---

Discard Rates

Monitor proptest discard rates. High discards indicate your input generators are too restrictive:

proptest! {
    // This will discard ~99% of inputs.
    #![proptest_config(ProptestConfig::with_cases(1000))]
    #[test]
    fn bad_generator(x in 1u32..1000000u32) {
        prop_assume!(x % 12345 == 0);
        // ...
    }
}

Better to build the constraint into the generator:

fn multiples_of_12345() -> impl Strategy<Value = u32> {
    (1u32..81).prop_map(|x| x * 12345)
}

Notes:

• Keep prop_assume! rare and simple; complex predicates hinder shrinking and slow failures.

---

This recipe provides the foundation for systematic input coverage testing in stacks-core. Start with property-based testing for fast iteration, add libFuzzer for coverage-guided exploration, and use AFL++ for deep out-of-process testing.