Fuzzing Tests in Go

Fuzzing is a technique where you automagically generate input values for your functions to find bugs

by Percy Bolmér, January 31, 2022

Fuzzing is being released as part of the standard library in Go 1.18. It is a pretty cool way of locating bugs in your code that you never thought of. I know many bugs have been found in the standard library in Go by people using third-party fuzzers.

Fuzzing will be part of the regular testing library since it is a kind of test. It can also be used together with the other utilities from testing which is nice.

In this article, we are going to learn how to use the new fuzzer, and fuzz an HTTP handler that we have built.

Before we being you need to make sure you are running at least Go 1.18, you can see how to install it in my walkthrough of the changes in 1.18, Go 1.18 Comes With Many Amazing Changes

Setting up the Fuzz

To begin fuzzing I created a new project and initialize a module that we will work inside. We also need a file suffixed with _test.go, in my case, I init a module called Fuzzy and a main_test.go

mkdir fuzzy
go mod init programmingpercy.tech/fuzzy
touch main_test.go

Before we fuzz, we need something to fuzz. Let us create a handler that calculates the highest value amongst a slice of ints.

We will introduce a bug in the handler to see how we can find it by fuzzing, the bug will be that any result equals 50 will cause it to fail.

type ValuesRequest struct {
	Values []int `json:"values"`
}

func CalculateHighest(w http.ResponseWriter, r *http.Request) {
	// Declare a valuerequest
	var vr ValuesRequest

	// Decode and respond with error incase it fails
	err := json.NewDecoder(r.Body).Decode(&vr)
	if err != nil {
		http.Error(w, err.Error(), http.StatusBadRequest)
		return
	}

	var high int
	// Range all values
	for _, value := range vr.Values {
		// Check if value is higher than high
		if value > high {
			// If so, set high to value
			high = value
		}
	}

	// Return high
	if high == 50 {
		w.WriteHeader(http.StatusInternalServerError)
		w.Write([]byte("Something went wrong"))
	}
	fmt.Fprintf(w, "%d", high)
}

Whats All The Fuzz About

Time to start learning how to use the Fuzzer.

If you are familiar with the Go testing this will probably be very easy. A test in Go is defined by creating a function prefixed with Test and accepting the input parameter t *testing.T.

The same pattern is used for Fuzzing, but instead prefixed with Fuzz and accepting the new f *testing.F.

// How to define a Unit Test
func TestCalculateHighest(t *testing.T){
}
// To define a Fuzz, we use Fuzz and testing.F
func FuzzTestHTTPHandler(f *testing.F) {
}

The first thing we need to do is to provide the testing.F with a Seed Corpus which you should consider example data. This is the data that the fuzzer will use and modify into new inputs that are tried.

The seed should reflect how the input to your function should look as much as possible to get the best results of the fuzzer.

Adding seeds is done by f.Add() which accepts the following data types.

• string, []byte, rune
• int,int8,int16,int32,int64
• uint,uint8,unit16,uint32,uint64
• float32,float64
• bool

Note that you cant add multiple data types, if you first run f.Add(“hello”) and after that f.Add(10) it will panic.

mismatched types in corpus entry: [int], want [string]

Wait, but my function requires multiple input parameters? — Angry Anti Fuzzer

If you require multiple input parameters, f.Add() accepts a variable amount of inputs. So you can add many example data inputs, just make sure the example seeds match the same order as your function input parameters.

func MultipleInputs(a, b int, name string) {
    // ... fancy code goes here 
}

func FuzzMultipleInputs(f *testing.F) {
  // We can add Multiple Seeds, but it has to be the same order as the input parameters for MultipleInputs
  f.Add(10,20,"John the Ripper")
  f.Fuzz(func(t *testing.T,a int,b int,name string){
      MultipleInputs(a,b,name)      
  })
}

Now that we know how to add seeds, let us create example data for the handler and Add it. We will want to add []byte with JSON data as we are fuzzing an HTTP handler.

We will begin by setting up an HTTP server that hosts the Handler we are fuzzing, and create some example slices to provide to the Seed Corpus. We then marshal the slices into proper JSON and add them to the fuzzer.

func FuzzTestHTTPHandler(f *testing.F) {
	// Create a new server hosting our calculate func
	srv := httptest.NewServer(http.HandlerFunc(CalculateHighest))
	defer srv.Close()

	// Create example values for the fuzzer
	testCases := []ValuesRequest{
		ValuesRequest{[]int{1, 2, 3, 4, 5, 6, 7, 8, 9, 10}},
		ValuesRequest{[]int{10, 9, 8, 7, 6, 5, 4, 3, 2, 1}},
		ValuesRequest{[]int{-50, -9, -8, -7, -6, -5, -4, -3, -2, -1}},
		ValuesRequest{[]int{1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20}},
		ValuesRequest{[]int{10, 20, 30, 40, 50, 60, 70, 80, 90, 100, 110, 120, 130, 140, 150, 160, 170, 180, 190, 200}},
	}

	// Add Corpus Seeds
	for _, testCase := range testCases {
		// Skip error, very bad practice
		data, _ := json.Marshal(testCase)
		// Add JSON data as Corpus
		f.Add(data)

	}
 }

It is time to implement the Fuzz Target.

To start fuzzing we must use f.Fuzz() which accepts a function as input. That function is the fuzz target, and should check for errors prepare the data, and trigger the function that we are fuzzing.

The input function to Fuzz has to accept testing.T as the first parameter, followed by the input data types added to the corpus,in the same order.

In our case we only need to pass testing.T, []byte since we only added a []byte to the seed. But if you added more inputs they need to be declared here as well.

func MultipleInputs(a, b int, name string) {
    // ... fancy code goes here 
}

func FuzzMultipleInputs(f *testing.F) {
  // We can add Multiple Seeds, but it has to be the same order as the input parameters for MultipleInputs
  f.Add(10,20,"John the Ripper")
  f.Fuzz(func(t *testing.T,a int,b int,name string){
      MultipleInputs(a,b,name)      
  })
}

Let us add the Fuzz target. We will want to do the following

1. Post the data to our handler
2. Check the response status
3. Check that the response is a int.

You might think that we would want to check the returned value if it is proper, and that is something that you could do if you have another function that you know works. But the most time you will not know the expected return because you don’t know what input the fuzzer is creating.

func FuzzTestHTTPHandler(f *testing.F) {
	// Create a new server hosting our calculate func
	srv := httptest.NewServer(http.HandlerFunc(CalculateHighest))
	defer srv.Close()

	// Add Corpus Seeds
	testCases := []ValuesRequest{
		ValuesRequest{[]int{1, 2, 3, 4, 5, 6, 7, 8, 9, 10}},
		ValuesRequest{[]int{10, 9, 8, 7, 6, 5, 4, 3, 2, 1}},
		ValuesRequest{[]int{-50, -9, -8, -7, -6, -5, -4, -3, -2, -1}},
		ValuesRequest{[]int{1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20}},
		ValuesRequest{[]int{10, 20, 30, 40, 50, 60, 70, 80, 90, 100, 110, 120, 130, 140, 150, 160, 170, 180, 190, 200}},
	}

	for _, testCase := range testCases {
		// Skip error, very bad practice
		data, _ := json.Marshal(testCase)
		// Add JSON data as Corpus
		f.Add(data)
	}
	// Start fuzzing
	f.Fuzz(func(t *testing.T, data []byte) {
		// Create a new request
		// using DefaultClient, should not be done in prod
		resp, err := http.DefaultClient.Post(srv.URL, "application/json", bytes.NewBuffer(data))
		if err != nil {
			t.Errorf("Error: %v", err)
		}

		if resp.StatusCode != http.StatusOK {
			t.Errorf("Expected status code %d, got %d", http.StatusOK, resp.StatusCode)
		}
		// Umarshal data
		var response int
		err = json.NewDecoder(resp.Body).Decode(&response)
		if err != nil {
			t.Errorf("Error: %v", err)
		}
	})
}

It is finally time to run the fuzzer. To start the fuzzer we run the regular go test command, but add the –fuzz=Fuzz flag, the Flag value will be used as a prefix and start all methods starting with Fuzz. You can turn it into what you want to match only certain Method signatures.

For instance, to start fuzzing only our FuzzTestHTTPHandler you can run the following.

go test --fuzz=FuzzTestHTTPHandler

This is useful when you have multiple Fuzzing functions in the future.

Fuzzing is a bit different from regular tests, the default behavior is to run forever until failures occur, so you either need to cancel the fuzzer or wait until an error. There is a third option, adding the -fuzztime flag will cancel after the set time.

So to run for 10 seconds you would run

go test --fuzz=Fuzz -fuzztime=10s

Make sure you run the fuzzer and wait until it does fail. You should see an output similar to mine, but since the output is generated it might not be the same.

--- FAIL: FuzzTestHTTPHandler (0.51s)
    --- FAIL: FuzzTestHTTPHandler (0.00s)
        main_test.go:80: Expected status code 200, got 400
        main_test.go:87: Error: invalid character 'j' looking for beginning of value
Failing input written to testdatafuzzFuzzTestHTTPHandler582528ddfad69eb57775199a43e0f9fd5c94bba343ce7bb6724d4ebafe311ed4
    To re-run:
    go test -run=FuzzTestHTTPHandler/582528ddfad69eb57775199a43e0f9fd5c94bba343ce7bb6724d4ebafe311ed4

When an error occurs during the fuzzer, it will cancel and write the input parameters that caused the failure into a file. All failures are written to testdata\fuzz\NameOfYourFuzz\inputID.

In this case, it seems that the JSON marshaling has failed, let us view the contents of the fuzzer file.

go test fuzz v1
[]byte("0")

The first row is information about the fuzzer and what version it was. The second row is the generated payload that we send to the HTTP handler.

This time it failed when the fuzzer sent a raw 0, which is fine because that is not proper JSON.

This is not uncommon to experience when you write your first fuzzer, so let us make the fuzzer data even better. We only want to test when the payload is actual JSON because we do already know any bad JSON payload will fail, and should.

You can skip payloads that are not correct using t.Skip(“reason of skipping”) and this is useful when fuzzing. We will use the json.Valid function to make sure we only pass in proper JSON to the handler.

Now only checking if json.Valid is not enough, “0” is valid JSON. We can however try Unmarshalling the data to a ValuesRequest struct, and skip any payload that fails. This is a good trick to filter out badly generated payloads.

func FuzzTestHTTPHandler(f *testing.F) {
	// Create a new server hosting our calculate func
	srv := httptest.NewServer(http.HandlerFunc(CalculateHighest))
	defer srv.Close()

	// Create example values for the fuzzer
	testCases := []ValuesRequest{
		ValuesRequest{[]int{1, 2, 3, 4, 5, 6, 7, 8, 9, 10}},
		ValuesRequest{[]int{10, 9, 8, 7, 6, 5, 4, 3, 2, 1}},
		ValuesRequest{[]int{-50, -9, -8, -7, -6, -5, -4, -3, -2, -1}},
		ValuesRequest{[]int{1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20}},
		ValuesRequest{[]int{10, 20, 30, 40, 50, 60, 70, 80, 90, 100, 110, 120, 130, 140, 150, 160, 170, 180, 190, 200}},
	}

	// Add Corpus Seeds
	for _, testCase := range testCases {
		// Skip error, very bad practice
		data, _ := json.Marshal(testCase)
		// Add JSON data as Corpus
		f.Add(data)

	}

	// Start fuzzing
	f.Fuzz(func(t *testing.T, data []byte) {
		// Validate that it is proper JSON before sending
		if !json.Valid(data) {
			t.Skip("invalid JSON")
		}
		// Make sure it is a real ValuesRequest formatted payload by marshalling into
		vr := ValuesRequest{}
		err := json.Unmarshal(data, &vr)
		if err != nil {
			t.Skip("Only correct requests are intresting")
		}
		// Create a new request
		// using DefaultClient, should not be done in prod
		resp, err := http.DefaultClient.Post(srv.URL, "application/json", bytes.NewBuffer(data))
		if err != nil {
			t.Errorf("Error: %v", err)
		}
		// Check status Code
		if resp.StatusCode != http.StatusOK {
			t.Errorf("Expected status code %d, got %d", http.StatusOK, resp.StatusCode)
		}

		// Umarshal data
		var response int
		err = json.NewDecoder(resp.Body).Decode(&response)
		if err != nil {
			t.Errorf("Error: %v", err)
		}

	})
}

Run that fuzzer with

go test -v --fuzz=Fuzz

Let it run for a while, eventually, you should get a new failure file written. That file will show you the detected bug, any value of 50 causes a failure.

go test fuzz v1
[]byte("{"vAlues":[50]}")

Conclusion

Right, you can now write your fuzzers. The example we used might seem ridiculous because you knew the value == 50 bug was present by a simple if statement. However, writing fuzzers for your HTTP handlers or any other methods is a great way to detect hard-to-find bugs.

Fuzzers usually can find bugs that your unit tests miss because unit tests often only contain values that you inputted could predict would create failures.

As you can see in the example, the code to write a fuzzer for an HTTP handler is not that much, and once you have done it a few times it can be pretty quick to implement new ones, leaving your code stack more robust and bug-free.

What are you waiting for, go out there and fuzz off!

If you want to learn more about the changes in Go 1.18, you can read about how to use the new generics in my other article.

Learning Generics in Go

January 31, 2022

Generics is released in Go 1.18 and it is time to learn how to leverage this new feature

Read more

Feel free to reach out to me on any of my social media below!