Reviewing GitHub Copilot by Building a Deck of Cards

Testing the Copilot by building a Deck of Cards with it in Go

by Percy Bolmér, December 22, 2021

Not long ago GitHub Copilot was released for beta access. You have to sign up for access, at their GitHub. I signed up, not expecting to get an invite. Turns out I was lucky and got one. I’ve been actively using it during the development of my projects for a few months now and wanted to share my insights on it.

What is GitHub Copilot?

Copilot is a plugin for your editor, that enables an AI-trained model to help you with code suggestions, code generation, and tips. The AI is powered by OpenAI Codex. The supported editors are currently VS Code, Jetbrains, and NeoVim.

There has been a lot of debate around Copilot, seeing as GitHub (Microsoft) trained the model on licensed code. This fact has raised a lot of discussions about if the generated code is illegal to use due to breaking licenses. Some licenses state that you are free to use the code, but not redistribute it.

Many articles out there say don’t use it, but I am going to go against the stream. This is probably one of the coolest extensions I’ve ever seen. It will speed up development by miles, it also helps immensely during the development process. You should try it out as soon as you get a chance.

Not only is it good, but it is also fast and comes up with suggestions that match the current codebase. It analyses your code and seems to magically know what you want to code, and matches your current codebase.

How does that work you ask? Well here comes the kicker, your code is sent to the model (one of the discussed topics) so that It can analyze the context of your code. This is needed to come up with relevant and good suggestions.

This is also one of the reasons why I haven’t started using it at work yet, I don’t want that corporate code flying around, and It is still unclear to me what parts of the code are shared and not.

To review the Copilot we will code a playing card deck while walking through the features of the Copilot one by one.

Code suggestions

One of the major features of the Copilot is the ability to give code suggestions. Most IDEs can suggest the variable or function use, but what differs here is that Copilot actually can suggest full code blocks.

Code suggestions come in two ways, inline suggestions which appear grayed out code after your current location as you code. Copilot does not only suggest completing your words like most IDEs, it can suggest new fields based on your context. The snippet below shows how I use Copilot inline suggestions to create the Deck for our game. You press the tab to accept the suggestion.

See how we created the Deck and Card types almost purely by pressing Tab to accept the suggestions.

If you don’t like an inline suggestion, you can always toggle to the next suggestion by skipping to the next one, this is done depending on your OS.

By pressing alt + ], or in my case (swedish), I press alt + å to toggle the next solution.

There might be times when the suggestion is not correct, you might want a more advanced deck. You can ask Copilot to show you 10 different suggestions by pressing ctrl + enter which will bring up full code suggestions. Instead of line-by-line suggestions, this will suggest multi-line solutions. Not all suggestions are always what you want, but I find them a useful way of finding inspiration on how to solve the problem.

Now we have a complete Deck in a matter of seconds, and along with it the functions to shuffle, discard and draw from it.

Let’s add some logic to our main function to see if this does what we want. Let’s see if we can shuffle and grab a few cards from the deck and print them. It is a very easy task, but even now we can grab assistance from Copilot. Look at how I simply tab my way through the main function. The Copilot can suggest these things because it is constantly using your current code bases context.

Notice how I accept most changes in the gif, but sometimes I manually type the first letters to make the Copilot figure out what I want. It’s when you find that collaboration with the Copilot that you realize that the name is amazing. It is not an automated code writer that you can purely rely on, but it does code with you. You press the tab a few times, then realize it’s on the wrong track, give it a gentle nudge and go back to tabs.

Running the program now will print 5 cards, however, it will always print the same 5 cards, we have encountered a small problem with relying too much on generated code. While it servers well as an example, we cant rely on it to solve the problem.

[{Spades Six} {Spades King} {Spades Nine} {Clubs Ace} {Hearts Two}]

Unit tests

Now we have a Deck of cards that we can play with, right? Let’s be honest, when generating code, you might become lazy and take a quick look at it and think Looks good!

One important thing that I always push to my team is unit tests, and especially now in this example when we have code that has been generated.

Let’s write some unit tests to make sure the Deck works as expected, and let’s make sure to let the Copilot help us do that as well. Writing unit tests is one of those things I’ve noticed many developers put in the tech debt compartment due to stress in low on time. That is a big risk as it tends to leave buggy code out there, and making changes will get harder because you don’t know what you break when applying fixes.

I will begin by making sure the shuffle works and that the Deck is of the correct size. So let us create a main_test.go and make sure.

I begin by verifying that my Deck is of the correct size, notice how I have to toggle between different suggestions before finding one that fits my needs. Still a time save, however.

Since running the main function and seeing the same output I suspect that the Shuffle is not random. So let’s generate tests for that as well.

Let’s take a look at it, to see how it works and then write a simple test.

func (d Deck) Shuffle() {
    for i := range d {
        j := rand.Intn(len(d) - 1)
        d[i], d[j] = d[j], d[i]
    }
}

It loops through the whole deck, switching places on the current index with a randomly generated number using the math/rand package. Let us see if Copilot can help us create a test.

Both tests pass, the length of correct and the two decks are not the same. But running the main function will always trigger the same output, must be the draw function? Let’s review that function.

func (d Deck) Draw(n int) []Card {
    return d[:n]
}

Not much that can go wrong there.

I’m writing down this debug process because it is super important to highlight. Copilot is amazing and has helped us get a play deck in no time, but it is a double-edged sword. The bug is quite obvious for a developer with experience with the math/rand package. The code we have looks like it is doing what we want at first glance, but if the developer has the little experience it is easy to be fooled.

The math/rand package uses a global source that manages what random numbers to generate, and if this source is not set using a Seed we will always get the same “randomly” generated numbers, making the Deck always shuffle and hand us the same cards.

Yes, the unit tests created for the shuffle pass, because the first deck and the second deck are not the same, however, if we run the test multiple times and print them you will see that the decks are always identical.

To fix this bug we can change the Seed to prevent people from being able to predict what cards to draw, pretty important if you are creating an online casino.

func (d Deck) Shuffle() {
    rand.Seed(time.Now().UnixNano())
    for i := range d {
        j := rand.Intn(len(d) - 1)
        d[i], d[j] = d[j], d[i]
    }
}

Comment suggestions

Another cool thing about the Copilot is that you can get suggestions based on the context of your comments. Write a comment about what you want to do and the Copilot will comply. For instance, I like Table Driven unit tests more so let’s tell that to the Copilot for the Draw functions tests.

See how easy that was? For some reason, Copilot made it a comment, but other than having to remove that, I had a full table-driven unit test with an example of 5 cards drawn from a small comment.

Sure, It was not perfect and I modified it into the following gist, but 95% of the test was generated.

// table driven unit tests for Deck.Draw
func TestDeck_Draw(t *testing.T) {
	type args struct {
		n int
	}
	tests := []struct {
		name string
		d    Deck 	
		args args
		want []Card
	}{
		{
			name: "Draw 5 cards from a deck",
			d:    NewDeck(),
			args: args{n: 5},
			want: []Card{
				{
					Suit: "Spades",
					Rank: "Ace",
				},
				{
					Suit: "Spades",
					Rank: "Two",
				},
				{
					Suit: "Spades",
					Rank: "Three",
				},
				{
					Suit: "Spades",
					Rank: "Four",
				},
				{
					Suit: "Spades",
					Rank: "Five",
				},
			},
		},
	}
	for _, tt := range tests {
		t.Run(tt.name, func(t *testing.T) {
			if got := tt.d.Draw(tt.args.n); !reflect.DeepEqual(got, tt.want) {
				t.Errorf("Deck.Draw() = %v, want %v", got, tt.want)
			}
		})
	}
}

Again, everything seems to be working as expected. But reviewing the generated code for the draw we will see a small bug again.

func (d Deck) Draw(n int) []Card {
    return d[:n]
}

Drawing a card simply returns the latest index, it does not Discard the drawn cards. This means we will get the same cards over and over which is not something we want.

Let’s move to that function and see if Copilot can fix its own mistake.

Well, that was easy. Another way would be to use a comment telling Copilot what we want to do.

Conclusion and Review

Copilot is an amazing product, it will help very much. However there is a big BUT, you cant rely on it to solve your issues for you. You still have to have an understanding of what is going on and review the code, as demonstrated in the examples above.

Don’t forget to write tests for the generated code (Copilot helps you when doing this), as it can appear to do what you want but contain issues as seen in the Unit tests chapter.

My biggest PROS are the following

• Copilot is great for helping out with “boilerplate” code. It is very helpful when coding to see these suggestions, they might not always be on point with what you want, but most of it is still usable.
• It is also a great guiding light when traversing topics you are unfamiliar with and need inspiration, usually, I google, but a lot of that googling can be replaced by the Copilot.

However there are some CONs, as I mentioned before, it is a double-edged sword.

• High chances of buggy code unless the developer is aware of what’s happening and takes the time to understand what the Copilot is creating for you. The Shuffle function is a great example, it looks correct but a big hidden flaw was present all along.
• You can lose knowledge of your codebase if too much of it is generated by the Copilot
• Still, a lot of development time for me goes into the architecture and figuring out the solution rather than typing code.
• The risk of trusting it too much is high, it makes very good assumptions, and most of the time it does the right thing. That is something that can make the developer sluggish in mind and “tab” in faulty code.

Things I haven’t yet tried is the following

• Corporate Code — I don’t know how code is treated fully, so I don’t want to expose it. Also unsure about the legality. It would be fun to try it out on a bigger codebase to see how it handles context detection.

That is it for today, be sure to sign up for the Copilot and give it a spin. If you want more tips about Visual Studio Code I suggest reading my earlier article.

7 VS Code Tricks Every Developer Needs To Know

August 22, 2021

A handpicked set of amazing features in VS Code

Read more

Take care and thanks for reading, feel free to reach out using my social media handles found below!