Fuzz Testing

What is fuzz testing? Fuzzing is a testing technique that injects random pieces of data to a software function to uncover crashes and vulnerabilities. It helps improving code security and reliability, since it can trigger edge cases that went unnoticed during unit testing.

How does it work? Fuzz testing relies on a fuzzing engine, a library that runs your code in a loop, injecting different inputs at each iteration. The fuzzing engine will instrument your code to measure coverage, and use this information to drive the generation of samples. Most of the samples will contain malformed input, and will test your code’s tolerance to ill-formed inputs.

Which kind of errors does fuzzing detect? The fuzzing engine will monitor your code for crashes. Fuzzing is often used with the address and undefined sanitizers. In short, fuzzing will make sure that your code doesn’t crash, leak or incur in undefined behavior, regardless of how malformed the input is. A lot of vulnerabilities in C++ code are related to the former kind of errors, so fuzzing can make your code more secure.

Should I use it? Fuzz testing is specially relevant for libraries that process potentially untrusted, user-controlled input, like network data. Libraries that implement parsers, decoders or network protocols usually benefit from fuzz testing.

Which Boost libraries use it? Libraries like Boost.Json, Boost.Url and Boost.Mysql use this technique - if you’re about to implement it in your library, have a look at what these libraries do.

Should I still write unit tests? Yes. Absolutely. Fuzzing does not replace unit tests, but complements them. Unit tests verify that your code produces the intended results by providing known inputs and running assertions on the outputs. In fuzz testing, inputs are generated randomly by the fuzzing engine, so no assertions are usually run on the outputs - fuzzing will only monitor for crashes and memory errors.

How can I add fuzzing to my library? We recommend using LibFuzzer, since it’s the easiest fuzzing engine to use, and the one that other Boost libraries use. You can use other fuzzing engines if you prefer.

LibFuzzer Basics

Quoting documentation, "LibFuzzer is an in-process, coverage-guided, evolutionary fuzzing engine". LibFuzzer will run your code multiple times with different, random inputs. It will instrument your code to measure coverage, and will attempt to generate inputs that maximize it, effectively trying to discover new paths in your code.

LibFuzzer is included in clang, so you don’t need to install anything to get started.

Let’s say we want to fuzz a function that parses JSON data, like parse_json(string_view input). We will create a source file with the following code:

#include <string_view>
#include <your/parsing/function.hpp>

extern "C" int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size)
{
    // The range [data, data+size) contains the data generated by the fuzzer
    std::string_view input_data (reinterpret_cast<const char*>(data), size);
    parse_json(input_data);
    return 0;
}

We can build a fuzzer executable by adding -fsanitize=fuzzer to clang’s compile and link flags. This will automatically link LibFuzzer to your code. It’s advised to also enable the address and undefined sanitizers, which increases the range of errors detected by the fuzzer. We recommend building in release mode with debug symbols enabled, so crashes are symbolized correctly.

From the command line:

clang++ -g -O3 -fsanitize=fuzzer,address,undefined -o fuzzer fuzzer.cpp

As a Jamfile target:

exe fuzzer : fuzzer.cpp : requirements
    <debug-symbols>on
    <optimization>speed
    <address-sanitizer>norecover
    <undefined-sanitizer>norecover
    <cxxflags>-fsanitize=fuzzer
    <linkflags>-fsanitize=fuzzer
;

Or as a CMake target:

add_executable(fuzzer fuzzer.cpp)
target_compile_options(
    fuzzer
    PRIVATE
    -fsanitize=fuzzer,address,undefined
    -fno-sanitize-recover=address,undefined
    -g
    -O3
)
target_link_options(
    fuzzer
    PRIVATE
    -fsanitize=fuzzer,address,undefined
    -fno-sanitize-recover=address,undefined
)

Note that you must not define a main function - LibFuzzer will do it for you. The LLVMFuzzerTestOneInput function will be invoked repeatedly, with different input ranges.

You can run your fuzzer with no arguments, which will fuzz until you stop it with Ctrl+C. The executable will print a lot of messages to stdout. This section contains a reference to what they mean, if you’re curious.

To run the fuzzer for a limited period of time (for example, 30 seconds), use:

./fuzzer -max_total_time=30

Corpus

A corpus is a collection of input samples to be used by the fuzzer. LibFuzzer uses these samples to create random mutations to use as new inputs. If a newly created sample triggers extra coverage, this sample is stored in the corpus.

Until now, we’ve been running our fuzzer without an initial corpus. The fuzzer will try random inputs, without any guidance, and will generate a corpus. Doing this is not advisable, though, since it reduces the effectiveness of your fuzzing - the fuzzer may fail to find some relevant inputs.

We always advise to provide an initial corpus (often called a seed corpus) to the fuzzer, to provide some guidance. The seed corpus should contain a variety of valid and invalid samples. You can reuse samples from your unit tests. In our JSON example, we could create a seedcorpus directory and copy all JSON files we use for unit testing.

Assuming that your seed corpus resides in your-lib/test/fuzzing/seedcorpus, we can run the fuzzer like this:

./fuzzer /tmp/corpus your-lib/test/fuzzing/seedcorpus -max_total_time=30

The two positional arguments are understood as corpus directories. The first one is an empty directory, and the second one is our seed corpus. The fuzzer will use the first corpus directory we provide (/tmp/corpus in our case) to write all the samples it finds relevant. Using separate directories allows us to keep the seed corpus clean, since it may reside in source control.

When running your fuzzer as part of your CI builds, you’ll likely want to persist this new corpus to make the newly generated samples available to subsequent fuzzer runs. This section digs deeper on running fuzzers during CI builds.

Verifying the Effectiveness of your Fuzzer

Once you’ve written a fuzzer and run it with an adequate corpus, you should have a look at the code coverage that your fuzzer triggered. This will help you verify that your fuzzing code is correct and that your corpus is in shape. The authors have found cases where some paths were missed due to errors in the seed corpus samples. Better check!

We recommend to use clang’s source-based coverage for this task. To get coverage info, you should build your fuzzer with the -fprofile-instr-generate and -fcoverage-mapping compile and link flags, and then run the fuzzer normally. This will create a default.profraw file in your current directory, containing raw coverage data.

To visualize your coverage, run:

llvm-profdata merge -sparse default.profraw -o fuzzer.profdata (1)
llvm-cov show path/to/fuzzer -instr-profile=fuzzer.profdata (2)

1	Converts from the raw profile format emitted by the binary to something `llvm-cov` can understand. This command can be used to merge several coverage files from different runs, too.
2	Prints a report with line coverage for your fuzzer and any headers it uses. Replace `path/to/fuzzer` with the path to your compiled fuzzer. `llvm-cov` requires it to properly understand coverage data.

This may generate a lot of output. You can use the -sources argument to scope which files are presented. Pay attention to the header path printed by the above command, since Boost creates symlinks for headers. For example, if you’re in the Boost super-project root, you can scope the report to Boost.Json headers by running:

llvm-cov show path/to/fuzzer -instr-profile=fuzzer.profdata -sources=boost/json/

Corpus Minimization

As we’ve mentioned, it’s advisable to persist the corpus generated by your fuzzer between runs. However, it can become very big as new samples are added. Before saving the corpus, we recommend performing corpus minimization.

This process is run by the same fuzzer executable we’ve been using. It will run the different samples in your corpus and discard "repeated" ones, based on the code paths they trigger.

To run corpus minimization, use the -merge=1 flag:

./fuzzer /tmp/mincorpus /tmp/corpus -merge=1

This will minimize the samples in /tmp/corpus, writing the results to /tmp/mincorpus. Note that no actual fuzzing is performed by this command.

Handling Crashes

If your fuzzer finds an input that makes your code crash, it will report the error and exit immediately, creating a file named crash-<id> containing the sample that caused the problem. Similarly, if an input takes too long to process, or a memory leak is found, a file timeout-<id> or leak-<id> will be written.

When a crash is detected, you should save the offending sample to source control, reproduce the crash, and fix your code. During regression testing, you should make your fuzzer run that specific sample, to verify that the crash doesn’t happen again.

You can make your fuzzer run a single sample by specifying it as a positional command-line argument. For example, if the sample that caused the crash is your-lib/test/fuzzing/old_crashes/crash-abc:

./fuzzer your-lib/test/fuzzing/old_crashes/crash-abc

This will run your fuzzer only with crash-abc. It will not perform actual fuzzing.

Running the Fuzzer in CIs

Your fuzzer won’t be really useful unless you run it continiously. CI platforms are a good way to achieve this. We recommend using GitHub Actions for fuzzing jobs, although other platforms with similar functionality should work, too.

Your fuzzing CI job should, at least:

Attempt to restore corpus samples from previous runs.
Build the fuzzers.
Run them with any old crash samples, to prevent regressions.
Run the actual fuzzing for some time. Most libraries run each fuzzer for 30 seconds.
Minimize the corpus generated by the previous step.
Persist the minimized corpus so that it can be used by subsequent CI runs.
Archive any crashes, timeouts and leaks, so you can recover them later.

If you’re using GitHub actions, corpus persistance can be achieved using the cache action. Building the fuzzers should be part of your B2 or CMake builds. You can use Boost.MySQL’s Jamfile as inspiration. It’s a good practice to run the fuzzers both nightly and on push/pull request events.

Best Practices for Writing Fuzzers

It is advisable to keep your fuzzers as targetted as possible. For example, if you have functions to parse JSON and BSON (binary JSON) files, you should write two different fuzzers, instead of a single one that invokes one or the other based on the input.

Your fuzzing code should be as efficient as possible. The faster it is, the more iterations the fuzzer will do, and the better the results. Avoid logging, cubic or greater complexity, and anything else that may slow down your code.

Try to avoid any randomness in your code. LibFuzzer works best with deterministic functions - that is, functions that, for a certain input, take always the same code paths.

Aside from the raw input data, you may need some extra input to configure your parsing function. For example, a JSON parser may be configured to allow comments or not. You may use part of the raw input data to configure flags like this and boost your coverage.

Boost Examples

Boost.Mysql fuzzes all its message deserialization routines. Fuzzers are located under test/fuzzing. The seed corpus is composed of multiple binary files, compressed and stored in the same directory. Fuzzers are built and run from test/fuzzing/Jamfile. Targets in this directory are built using b2 from the fuzz.yml GitHub Actions workflow.
Boost.Json fuzzes its JSON parsing functions. Fuzzers are stored under fuzzing/. The seed corpus is generated dynamically, by copying all JSON files used for unit testing. Fuzzers are built and run from fuzzing/Jamfile. Targets in this directory are built using b2 from the run_fuzzer.yml GitHub Actions workflow.
Boost.Url is similar to to JSON, but doesn’t use a seed corpus.