Test and benchmark your code in go

When I started writing my first programs in Go, I noticed the tooling ships out of the box with test and benchmark features. You simply follow some naming conventions when it comes to file names. You import a reference to the "testing" package which is kind of part of the language. Aaaand… Ready set, and of you Go with writing some tests and benchmarks in Go. In my earlier blog post I briefly touched writing a test in Go already. I recommend reading this blogpost whenever you are a real newby with Go.

The conventions

In Go you put your tests in the same folder as the code. The file containing the tests will be postfixed with _test.go. Don't worry, it won't be shipped in you compiled binary. So lets start with an example folder structure for a Go project to get an idea how that would look like.

terminal

$ tree my-awesome-project
my-awesome-project
├── api
│   ├── handlers.go
│   ├── handlers_test.go
│   ├── routes.go
│   └── routes_test.go
├── main.go
├── main_test.go
├── storage.go
└── storage_test.go
 
1 directory, 8 files

At first I had to get used to this kind of structure as I have always been used to put tests in different packages in Java or different assemblies in c# or a test folder in Javascript projects. In the beginning I was kind of resisting to this, but once I started to write more and more code I figured this helps you when doing TDD. In general tests and code are more close to each other and therefore enforced me to practition TDD even more strictly then I already did with even less effort of navigating folders and files and stuff. Above folder structure shows one main package with code in main.go and storage.go. We also have one api package which contains routes and handlers for those routes.

For developers working recently on ReactJS web development you might also have noticed some have started applying same folder structures to have the tests in the same folder as the component.

Writing tests

In Go all tests go inside a file named similar to the file you are testing. So if you want to test code in main.go, then you write the tests in main_test.go. In the test file we will always need to import the testing package. Then there is the following signature for every test you write, func TestXxx(*testing.T), where the first X MUST be Uppercase.

main_test.go

package main
 
import "testing"
 
func TestSomething(t *testing.T) {
  expected := "Something Awesome Happened!"
  something := Something("Awesome")
  if something != expected {
    t.Errorf("Expected %s, got %s", expected, something)
  }
}

Benchmarking

Go also has builtin Benchmarking support. This Benchmarking feature is also part of the testing package. A Benchmark has a similar signarture as tests, func BenchmarkXxx(*testing.B), where the first X MUST also be uppercase.

main_test.go

func BenchmarkSomething(b *testing.B) {
  for i := 0 ; i < b.N ; i++ {
    Something("Awesome")
  }
}

Example

In this example we are going to define a struct which we would like to use in our API as a DTO to respond with some JSON. Let's approach this in a TDD manner. I usually start of by creating the files which will contain my code.

api/models.go
api/models_test.go

First I will layout some tests to describe what I would like to achieve.

api/models_test.go

package api
 
import "testing"
 
var awesomeJSON = []byte(`{
  "id": "123456789",
  "message": "Total awesomeness",
  "score": 9.99,
  "confirmed": true
}`)
 
func TestAwesomeToJSON(t *testing.T) {
  awesome := NewAwesome("123456789", "Total awesomeness", 9.99, true)
 
  testJSON, err := awesome.ToJSON()
 
  if err != nil {
    t.Error("Failed to create json from awesome")
  }
 
  if string(testJSON) != string(awesomeJSON) {
    t.Errorf("JSON output\n%s\nis not as expected\n%s", testJSON, awesomeJSON)
  }
}
 
func TestAwesomeFromJSON(t *testing.T) {
  awesome := NewAwesomeFromJSON(awesomeJSON)
 
  if awesome == nil {
    t.Error("Unmarshalling json into awesome failed")
  }
 
  if awesome.Id != "123456789" {
    t.Error("Awesome Id does not match expected value")
  }
 
  if awesome.Message != "Total awesomeness" {
    t.Error("Awesome Message does not match expected value")
  }
 
  if awesome.Score != 9.99 {
    t.Error("Awesome Id does not match expected value")
  }
 
  if !awesome.Confirmed {
    t.Error("Awesome Confirmed does not match expected value")
  }
}

With the test in place I can now define the struct, Contructor function and the instance method on my struct.

api/models.go

package api
 
import "encoding/json"
 
type Awesome struct {
  Id        string  `json:"id"`
  Message   string  `json:"message"`
  Score     float64 `json:"score"`
  Confirmed bool    `json:"confirmed"`
}
 
func NewAwesome(id string, message string, score float64, confirmed bool) *Awesome {
  return &Awesome{
    id,
    message,
    score,
    confirmed,
  }
}
 
func NewAwesomeFromJSON(jsonData []byte) *Awesome {
  var awesome *Awesome
  err := json.Unmarshal(jsonData, &awesome)
  if err != nil {
    return nil
  }
  return awesome
}
 
func(a *Awesome) ToJSON() ([]byte, error) {
  return json.Marshal(a)
}

Now we are ready to run our test to see if the code works and gives expected output.

terminal

$ go test ./api
--- FAIL: TestAwesomeToJSON (0.00s)
    models_test.go:24: JSON output
        {"id":"123456789","message":"Total awesomeness","score":9.99,"confirmed":true}
        is not as expected
        {
          "id": "123456789",
          "message": "Total awesomeness",
          "score": 9.99,
          "confirmed": true
        }
FAIL
FAIL    test-go/api     0.296s

As you can see one of our tests is failing. We expected nicely formatted JSON but we got the most compact form of JSON without any formatting. Now we have 2 options. Change our requirement (The Test) or we change the implementation to have the JSON outputted with some formatting. At the end of this blogpost I also provide you a solution to fix the failing test. Before that I would like to zoom in a bit on how to benchmark your code.

Now lets also add a Benchmark to test the performance of our implementation.

api/models_test.go

func BenchmarkAwesomeFromJSON(b *testing.B) {
  for i := 0; i < b.N; i++ {
    NewAwesomeFromJSON(awesomeJSON)
  }
}
 
func BenchmarkAwesomeToJSON(b *testing.B) {
  for i := 0; i < b.N; i++ {
    awesome := NewAwesome("123456789", "Total awesomeness", 9.99, true)
    awesome.ToJSON()
  }
}

Now let us run our Benchmark. In case you still have the failing test case you will notice the benchmark doesn't run. No need to benchmark broken code right. So to bypass that in case you didn't take previous challenge lets skip this failing test using t.Skip().

terminal

$ go test -bench=. ./api
goos: windows
goarch: amd64
pkg: github.com/marcofranssen/test-go/api
BenchmarkAwesomeFromJSON-8        500000              3736 ns/op
BenchmarkAwesomeToJSON-8         1000000              1189 ns/op
PASS
ok      github.com/marcofranssen/test-go/api    3.408s

The -8 in this benchmark means that the benchmark ran with GOMAXPROCS=8, in our case it doesn't really make a difference probably if we would run with only one CPU proc as we don't use any parallelism in our code.

In case you didn't take previous challenge I want to challenge you again. Lets add support to have formatted and unformatted JSON and see how the performance is for both implementations by adding another Benchmark for formatted JSON.

Once ready continue reading, I'll provide you a solution as well for the challenge.

In following example I implemented the full example code in main.go so you can easily test it from the playground.

main.go

package main
 
import (
  "encoding/json"
  "fmt"
)
 
type Awesome struct {
  Id        string  `json:"id"`
  Message   string  `json:"message"`
  Score     float64 `json:"score"`
  Confirmed bool    `json:"confirmed"`
}
 
func NewAwesome(id string, message string, score float64, confirmed bool) *Awesome {
  return &Awesome{
    id,
    message,
    score,
    confirmed,
  }
}
 
func NewAwesomeFromJSON(jsonData []byte) *Awesome {
  var awesome *Awesome
  err := json.Unmarshal(jsonData, &awesome)
  if err != nil {
    return nil
  }
  return awesome
}
 
func(a *Awesome) ToJSON(pretty bool) ([]byte, error) {
  if pretty {
    return json.MarshalIndent(a, "", "  ")
  } else {
    return json.Marshal(a)
  }
}
 
func main() {
  awesome := NewAwesome("123456789", "Total awesomeness", 9.99, true)
  awesomeJSON, _ := awesome.ToJSON(false)
  fmt.Printf("%s\n", awesomeJSON)
  moreAwesomeJSON, _ := awesome.ToJSON(true)
  fmt.Printf("%s\n", moreAwesomeJSON)
}

playground

I also updated the Benchmark and test.

api/models_test.go

func TestAwesomeToJSON(t *testing.T) {
  t.Skip()
  awesome := NewAwesome("123456789", "Total awesomeness", 9.99, true)
 
  testJSON, err := awesome.ToJSON(true)
 
  if err != nil {
    t.Error("Failed to create json from awesome")
  }
 
  if string(testJSON) != string(awesomeJSON) {
    t.Errorf("JSON output\n%s\nis not as expected\n%s", testJSON, awesomeJSON)
  }
}
 
func BenchmarkAwesomeToJSON(b *testing.B) {
  for i := 0; i < b.N; i++ {
    awesome := NewAwesome("123456789", "Total awesomeness", 9.99, true)
    awesome.ToJSON(false)
  }
}
 
func BenchmarkAwesomeToJSONPretty(b *testing.B) {
  for i := 0; i < b.N; i++ {
    awesome := NewAwesome("123456789", "Total awesomeness", 9.99, true)
    awesome.ToJSON(true)
  }
}

Lets run the updated benchmarks so we can compare the performance of both implementations.

terminal

$ go test -bench=. ./api
goos: windows
goarch: amd64
pkg: github.com/marcofranssen/test-go/api
BenchmarkAwesomeFromJSON-8                500000              2685 ns/op
BenchmarkAwesomeToJSON-8                 2000000               862 ns/op
BenchmarkAwesomeToJSONPretty-8            500000              2308 ns/op
PASS
ok      github.com/marcofranssen/test-go/api    5.409s

As you notice the performance of the nicely formatted json including newlines and 2 spaces of indentation is a factor 4 slower then the unformatted json.

Bonus

Below I will show you a few more options you can use for running tests and benchmarks.

Test

Following will run test with code coverage.

terminal

$ go test -covermode=atomic -coverprofile=coverage.out ./...
?       github.com/marcofranssen/test-go        [no test files]
ok      github.com/marcofranssen/test-go/api    0.259s  coverage: 77.8% of statements

-covermode=atomic is the safest option in multithreaded environments. ./... will run the tests for all packages in the solution.

Following will run the tests using GOMAXPROCS=1 and GOMAXPROCS=4.

terminal

$ go test -cpu 1,4 ./...
ok      github.com/marcofranssen/test-go        0.286s

Asserts

As you noticed there is no assertion library included in the standard testing package. In case you want to have assert functions like you are used to in other programming languages, you need to install a third party package. E.g.: testify.

This allows you to write the assertions in following style.

main.go

package main
 
import (
  "testing"
  "github.com/stretchr/testify/assert"
)
 
func TestSomething(t *testing.T) {
  assert := assert.New(t)
 
  var a string = "Hello"
  var b string = "Hello"
 
  assert.Equal(a, b, "The two words should be the same.")
}

Testify comes also with other features like mocking etc.

Benchmark

Following will run the benchmarks for 15 seconds. The default is 1s. We will run the benchmark twice for every cpu configuration we specify. And we will only run these benchmarks for the tests that match the regex. Using . as regex would run all benchmarks.

terminal

$ go test -bench=^BenchmarkAwesomeToJSON.* -benchtime 15s -count 2 -cpu 1,4 ./api
goos: windows
goarch: amd64
BenchmarkAwesomeToJSON                  30000000               870 ns/op
BenchmarkAwesomeToJSON                  30000000               855 ns/op
BenchmarkAwesomeToJSON-4                30000000               859 ns/op
BenchmarkAwesomeToJSON-4                30000000               851 ns/op
BenchmarkAwesomeToJSONPretty            10000000              2476 ns/op
BenchmarkAwesomeToJSONPretty            10000000              2446 ns/op
BenchmarkAwesomeToJSONPretty-4          10000000              2468 ns/op
BenchmarkAwesomeToJSONPretty-4          10000000              2399 ns/op
PASS
ok      github.com/marcofranssen/test-go/api    214.339s

With these options on more complicated scenarios you can make sure there is a more reliable report as you run the benchmark for a longer amount of time and twice. You also notice it doesn't really make a difference if you run it using one or multiple cpus as already predicted earlier. You also notice now the benchmark for 4 cpus is postfixed with -4. Also note that the pretty formatting now only shows up as a factor 3 slower.

For more options you could check go test --help. There are for example options to create memory profiles and cpu profiles which allow you to analyze the performance. Looking forward to your feedback, as there is never a limit to what a person can learn.

The full solution can be downloaded here.