-->

Fuzzy Testing in Go

Posted by : at

Category : guides   go


Fuzzing is being released as part of the standard library in Go 1.18. It is a pretty cool way of locating bugs in your code that you never thought of. I know many bugs have been found in the standard library in Go by people using third-party fuzzers.

Fuzzing will be part of the regular testing library since it is a kind of test. It can also be used together with the other utilities from testing which is nice.

In this article, we are going to learn how to use the new fuzzer, and fuzz an HTTP handler that we have built.

Before we being you need to make sure you are running at least Go 1.18, you can see how to install it in my Go 1.18 walkthrough covering the changes in 1.18.

Setting up the Fuzz

Adding corpus seeds to the fuzzer to allow it to generate data based on it. Image by Percy Bolmér. Gopher by Takuya Ueda, Original Go Gopher by Renée French (CC BY 3.0)

To begin fuzzing I created a new project and initialize a module that we will work inside. We also need a file suffixed with _test.go, in my case, I init a module called Fuzzy and a main_test.go

mkdir fuzzy
go mod init programmingpercy.tech/fuzzy
touch main_test.go

Before we fuzz, we need something to fuzz. Let us create a handler that calculates the highest value amongst a slice of ints.  We will introduce a bug in the handler to see how we can find it by fuzzing, the bug will be that any result equals 50 will cause it to fail.

main.go - Simple HTTP handler with a known bug

Whats All The Fuzz About

Time to start learning how to use the Fuzzer.

If you are familiar with the Go testing this will probably be very easy. A test in Go is defined by creating a function prefixed with Test and accepting the input parameter t *testing.T.

The same pattern is used for Fuzzing, but instead prefixed with Fuzz and accepting the new f *testing.F.

Defining the Fuzzer is easy, simply use *testing.F as input instead of *testing.T

The first thing we need to do is to provide the testing.F with a Seed Corpus which you should consider example data. This is the data that the fuzzer will use and modify into new inputs that are tried.

The seed should reflect how the input to your function should look as much as possible to get the best results of the fuzzer.

Adding seeds is done by f.Add() which accepts the following data types.

  • string, []byte, rune
  • int,int8,int16,int32,int64
  • uint,uint8,unit16,uint32,uint64
  • float32,float64
  • bool

Note that you cant add multiple data types, if you first run f.Add("hello") and after that f.Add(10) it will panic.

mismatched types in corpus entry: [int], want [string]

Wait, but my function requires multiple input parameters? - Angry Anti Fuzzer

If you require multiple input parameters, f.Add() accepts a variable amount of inputs. So you can add many example data inputs, just make sure the example seeds match the same order as your function input parameters.

Fuzzing multiple input parameters

Now that we know how to add seeds, let us create example data for the handler and Add it. We will want to add []byte with JSON data as we are fuzzing an HTTP handler.

We will begin by setting up an HTTP server that hosts the Handler we are fuzzing, and create some example slices to provide to the Seed Corpus. We then marshal the slices into proper JSON and add them to the fuzzer.

Hosting the HTTP handler and adding Seed Corpus.

It is time to implement the Fuzz Target.

To start fuzzing we must use f.Fuzz() which accepts a function as input. That function is the fuzz target, and should check for errors prepare the data, and trigger the function that we are fuzzing.

The input function to Fuzz has to accept testing.T as the first parameter, followed by the input data types added to the corpus, in the same order. In our case we only need to pass testing.T, []byte since we only added a []byte to the seed. But if you added more inputs they need to be declared here as well.

Example of how multiple inputs would look

Let us add the Fuzz target. We will want to do the following

  1. Post the data to our handler
  2. Check the response status
  3. Check that the response is a int.

You might think that we would want to check the returned value if it is proper, and that is something that you could do if you have another function that you know works. But the most time you will not know the expected return because you don’t know what input the fuzzer is creating.

First draft of the Fuzzer

It is finally time to run the fuzzer. To start the fuzzer we run the regular go test command, but add the --fuzz=Fuzz flag, the Flag value will be used as a prefix and start all methods starting with Fuzz. You can turn it into what you want to match only certain Method signatures.

For instance, to start fuzzing only our FuzzTestHTTPHandler you can run the following.

go test --fuzz=FuzzTestHTTPHandler

This is useful when you have multiple Fuzzing functions in the future.

Fuzzing is a bit different from regular tests, the default behavior is to run forever until failures occur, so you either need to cancel the fuzzer or wait until an error. There is a third option, adding the -fuzztime flag will cancel after the set time.

So to run for 10 seconds you would run

go test --fuzz=Fuzz -fuzztime=10s

Make sure you run the fuzzer and wait until it does fail. You should see an output similar to mine, but since the output is generated it might not be the same.

--- FAIL: FuzzTestHTTPHandler (0.51s)
    --- FAIL: FuzzTestHTTPHandler (0.00s)
        main_test.go:80: Expected status code 200, got 400
        main_test.go:87: Error: invalid character 'j' looking for beginning of value
Failing input written to testdata\fuzz\FuzzTestHTTPHandler\582528ddfad69eb57775199a43e0f9fd5c94bba343ce7bb6724d4ebafe311ed4
    To re-run:
    go test -run=FuzzTestHTTPHandler/582528ddfad69eb57775199a43e0f9fd5c94bba343ce7bb6724d4ebafe311ed4

When an error occurs during the fuzzer, it will cancel and write the input parameters that caused the failure into a file. All failures are written to testdata\fuzz\NameOfYourFuzz\inputID.

In this case, it seems that the JSON marshaling has failed, let us view the contents of the fuzzer file.

go test fuzz v1
[]byte("0")

The first row is information about the fuzzer and what version it was. The second row is the generated payload that we send to the HTTP handler.  This time it failed when the fuzzer sent a raw 0, which is fine because that is not proper JSON.

This is not uncommon to experience when you write your first fuzzer, so let us make the fuzzer data even better. We only want to test when the payload is actual JSON because we do already know any bad JSON payload will fail, and should.

You can skip payloads that are not correct using t.Skip("reason of skipping") and this is useful when fuzzing. We will use the json.Valid function to make sure we only pass in proper JSON to the handler. Now only checking if json.Valid is not enough, “0” is valid JSON. We can however try Unmarshalling the data to a ValuesRequest struct, and skip any payload that fails. This is a good trick to filter out badly generated payloads.

A final fuzzer with payload validation to make sure only real data is tested

Run that fuzzer with

go test -v --fuzz=Fuzz

Let it run for a while, eventually, you should get a new failure file written. That file will show you the detected bug, any value of 50 causes a failure.

go test fuzz v1
[]byte("{\"vAlues\":[50]}")

Conclusion

Right, you can now write your fuzzers. The example we used might seem ridiculous because you knew the value == 50 bug was present by a simple if statement.

However, writing fuzzers for your HTTP handlers or any other methods is a great way to detect hard-to-find bugs.

Fuzzers usually can find bugs that your unit tests miss because unit tests often only contain values that you inputted could predict would create failures.

As you can see in the example, the code to write a fuzzer for an HTTP handler is not that much, and once you have done it a few times it can be pretty quick to implement new ones, leaving your code stack more robust and bug-free.

What are you waiting for, go out there and fuzz off!

If you want to learn more about the changes in Go 1.18, you can read Learning Generics in Go.

Learning Generics in Go »


About Programming Percy
Programming Percy

Hey, I'm Percy Bolmér. I'm a software engineer, writer and tech enthusiast. I've been professionally building software for 7+ years, and absoluetly love sharing any ideas I come across.

Email : programmingpercy@gmail.com

Website : https://programmingpercy.tech

About Percy Bolmér

Hey, I'm Percy Bolmér. I'm a software engineer, writer and tech enthusiast. I've been professionally building software for 7+ years, and absoluetly love sharing any ideas I come across.

Useful Links