Fuzzing Tests in Go
Fuzzing is a technique where you automagically generate input values for your functions to find bugs
by Percy Bolmér, January 31, 2022
Fuzzing is being released as part of the standard library in Go 1.18. It is a pretty cool way of locating bugs in your code that you never thought of. I know many bugs have been found in the standard library in Go by people using third-party fuzzers.
Fuzzing will be part of the regular testing library since it is a kind of test. It can also be used together with the other utilities from testing which is nice.
In this article, we are going to learn how to use the new fuzzer, and fuzz an HTTP handler that we have built.
Before we being you need to make sure you are running at least Go 1.18, you can see how to install it in my walkthrough of the changes in 1.18, Go 1.18 Comes With Many Amazing Changes
To begin fuzzing I created a new project and initialize a module that we will work inside. We also need a file suffixed with _test.go, in my case, I init a module called Fuzzy and a main_test.go
go mod init programmingpercy.tech/fuzzy
Before we fuzz, we need something to fuzz. Let us create a handler that calculates the highest value amongst a slice of ints.
We will introduce a bug in the handler to see how we can find it by fuzzing, the bug will be that any result equals 50 will cause it to fail.
Time to start learning how to use the Fuzzer.
If you are familiar with the Go testing this will probably be very easy. A test in Go is defined by creating a function prefixed with Test and accepting the input parameter t *testing.T.
The same pattern is used for Fuzzing, but instead prefixed with Fuzz and accepting the new f *testing.F.
The first thing we need to do is to provide the testing.F with a Seed Corpus which you should consider example data. This is the data that the fuzzer will use and modify into new inputs that are tried.
The seed should reflect how the input to your function should look as much as possible to get the best results of the fuzzer.
Adding seeds is done by f.Add() which accepts the following data types.
- string, byte, rune
Note that you cant add multiple data types, if you first run f.Add(“hello”) and after that f.Add(10) it will panic.
mismatched types in corpus entry: [int], want [string]
Wait, but my function requires multiple input parameters? — Angry Anti Fuzzer
If you require multiple input parameters, f.Add() accepts a variable amount of inputs. So you can add many example data inputs, just make sure the example seeds match the same order as your function input parameters.
Now that we know how to add seeds, let us create example data for the handler and Add it. We will want to add byte with JSON data as we are fuzzing an HTTP handler.
We will begin by setting up an HTTP server that hosts the Handler we are fuzzing, and create some example slices to provide to the Seed Corpus. We then marshal the slices into proper JSON and add them to the fuzzer.
It is time to implement the Fuzz Target.
To start fuzzing we must use f.Fuzz() which accepts a function as input. That function is the fuzz target, and should check for errors prepare the data, and trigger the function that we are fuzzing.
The input function to Fuzz has to accept testing.T as the first parameter, followed by the input data types added to the corpus,in the same order.
In our case we only need to pass testing.T, byte since we only added a byte to the seed. But if you added more inputs they need to be declared here as well.
Let us add the Fuzz target. We will want to do the following
- Post the data to our handler
- Check the response status
- Check that the response is a int.
You might think that we would want to check the returned value if it is proper, and that is something that you could do if you have another function that you know works. But the most time you will not know the expected return because you don’t know what input the fuzzer is creating.
It is finally time to run the fuzzer. To start the fuzzer we run the regular go test command, but add the –fuzz=Fuzz flag, the Flag value will be used as a prefix and start all methods starting with Fuzz. You can turn it into what you want to match only certain Method signatures.
For instance, to start fuzzing only our FuzzTestHTTPHandler you can run the following.
go test --fuzz=FuzzTestHTTPHandler
This is useful when you have multiple Fuzzing functions in the future.
Fuzzing is a bit different from regular tests, the default behavior is to run forever until failures occur, so you either need to cancel the fuzzer or wait until an error. There is a third option, adding the -fuzztime flag will cancel after the set time.
So to run for 10 seconds you would run
go test --fuzz=Fuzz -fuzztime=10s
Make sure you run the fuzzer and wait until it does fail. You should see an output similar to mine, but since the output is generated it might not be the same.
--- FAIL: FuzzTestHTTPHandler (0.51s)
--- FAIL: FuzzTestHTTPHandler (0.00s)
main_test.go:80: Expected status code 200, got 400
main_test.go:87: Error: invalid character 'j' looking for beginning of value
Failing input written to testdatafuzzFuzzTestHTTPHandler582528ddfad69eb57775199a43e0f9fd5c94bba343ce7bb6724d4ebafe311ed4
go test -run=FuzzTestHTTPHandler/582528ddfad69eb57775199a43e0f9fd5c94bba343ce7bb6724d4ebafe311ed4
When an error occurs during the fuzzer, it will cancel and write the input parameters that caused the failure into a file. All failures are written to testdata\fuzz\NameOfYourFuzz\inputID.
In this case, it seems that the JSON marshaling has failed, let us view the contents of the fuzzer file.
go test fuzz v1
The first row is information about the fuzzer and what version it was. The second row is the generated payload that we send to the HTTP handler.
This time it failed when the fuzzer sent a raw 0, which is fine because that is not proper JSON.
This is not uncommon to experience when you write your first fuzzer, so let us make the fuzzer data even better. We only want to test when the payload is actual JSON because we do already know any bad JSON payload will fail, and should.
You can skip payloads that are not correct using t.Skip(“reason of skipping”) and this is useful when fuzzing. We will use the json.Valid function to make sure we only pass in proper JSON to the handler.
Now only checking if json.Valid is not enough, “0” is valid JSON. We can however try Unmarshalling the data to a ValuesRequest struct, and skip any payload that fails. This is a good trick to filter out badly generated payloads.
Run that fuzzer with
go test -v --fuzz=Fuzz
Let it run for a while, eventually, you should get a new failure file written. That file will show you the detected bug, any value of 50 causes a failure.
go test fuzz v1
Right, you can now write your fuzzers. The example we used might seem ridiculous because you knew the value == 50 bug was present by a simple if statement. However, writing fuzzers for your HTTP handlers or any other methods is a great way to detect hard-to-find bugs.
Fuzzers usually can find bugs that your unit tests miss because unit tests often only contain values that you inputted could predict would create failures.
As you can see in the example, the code to write a fuzzer for an HTTP handler is not that much, and once you have done it a few times it can be pretty quick to implement new ones, leaving your code stack more robust and bug-free.
What are you waiting for, go out there and fuzz off!
If you want to learn more about the changes in Go 1.18, you can read about how to use the new generics in my other article.
Feel free to reach out to me on any of my social media below!
If you enjoyed my writing, please support future articles by buying me an Coffee