Manage Go tools via Go modules
Marco Franssen /
6 min read • 1196 words
In this blog I will cover how I'm managing and versioning the tools my Go projects depend on. Go Modules are available since Go 1.11. Using Go Modules you can manage the dependencies for your project. You can compare it to NPM in Nodejs projects or Maven in Java project or Nuget in .NET projects.
In general Go Modules are used to manage your compile time dependencies. However in my projects I also like to manage the tools required for Continuous Integration in my projects. To ensure all developers have same versions of tools installed and to ensure my CI server (Jenkins, Travis, CircleCI) can install and use the same version of the tools. I found a way using Go Modules, by default you will have some issue with go mod tidy if you would just manually add the tools as dependencies to your go.mod file.
Initialize
To start with Go Modules we first have to initialize a new Go Module. We do that by creating a new folder and run the go mod init command.
$ mkdir my-project
$ cd my-project
$ go mod init github.com/marcofranssen/my-project
go: creating new go.mod: module github.com/marcofranssen/my-projectThis will result in following go.mod file.
$ cat go.mod
module github.com/marcofranssen/my-project
go 1.12Add your tools as dependency
Now we want to add our tools as dependency to our go.mod. In general you would think to just add them using go get, which works at first sight perfectly fine. See below example.
$ go get -u github.com/goreleaser/goreleaser
$ go get -u golang.org/x/lint/golint
$ cat go.mod
module github.com/marcofranssen/my-project
go 1.12
require (
golang.org/x/crypto v0.0.0-20191001170739-f9e2070545dc // indirect
golang.org/x/lint v0.0.0-20190930215403-16217165b5de // indirect
golang.org/x/net v0.0.0-20191002035440-2ec189313ef0 // indirect
golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e // indirect
golang.org/x/sys v0.0.0-20191002091554-b397fe3ad8ed // indirect
golang.org/x/text v0.3.2 // indirect
golang.org/x/tools v0.0.0-20191001184121-329c8d646ebe // indirect
)As you can see it added indirect dependencies, and you will notice the goreleaser dependencies are not there at all as the second go get has removed them. Indirect means there is none of your own code which has a direct dependency on the module, which is correct as these are tools I would use and not dependencies of my own to be written code. You will also see there is a go.sum file generated, which I will skip for now as it is not relevant for this explanation.
Now there is also the go mod tidy command which cleans your dependencies etc. This is a recommended command to run before you make a release to ensure all dependencies are cleaned and accurate with the real needs of your code. So lets run that command now and check what happens.
$ go mod tidy
$ cat go.mod
module github.com/marcofranssen/my-project
go 1.12As you can see all the dependencies are again removed from the go.mod file and the go.sum file is cleaned up as well. This happens because there is no dependency in any .go file.
TL;DR
To ensure my tool dependencies are not removed and can leverage the Go Modules, I create a file tools.go. In this file I will list all my tool dependencies using an import statement.
// +build tools
package main
import (
_ "github.com/fullstorydev/grpcui/cmd/grpcui"
_ "github.com/golang/protobuf/protoc-gen-go"
_ "github.com/golangci/golangci-lint/cmd/golangci-lint"
_ "github.com/goreleaser/goreleaser"
_ "github.com/spf13/cobra/cobra"
_ "github.com/tebeka/go2xunit"
_ "golang.org/x/lint/golint"
_ "golang.org/x/perf/cmd/benchstat"
_ "golang.org/x/tools/cmd/stringer"
)As you can notice I have also added a build constraint a.k.a. build tag comment in the top of the file, to ensure it is not compiled into the binary, when running go build. Now with this file in place I can very easily install all my tools using a simple bash command. go install will make all the tools available in your $GO_WORKSPACE/bin folder. Normally this folder is available in your PATH so you can use the binaries in any folder of your choice.
$ cat tools.go | grep _ | awk -F'"' '{print $2}' | xargs -tI % go install %
go install github.com/fullstorydev/grpcui/cmd/grpcui
go: finding github.com/fullstorydev/grpcui/cmd/grpcui latest
go: finding github.com/fullstorydev/grpcui/cmd latest
go: downloading golang.org/x/net v0.0.0-20190522155817-f3200d17e092
go: extracting golang.org/x/net v0.0.0-20190522155817-f3200d17e092
go install github.com/golang/protobuf/protoc-gen-go
go install github.com/golangci/golangci-lint/cmd/golangci-lint
go: finding github.com/golangci/golangci-lint/cmd/golangci-lint latest
go: finding github.com/golangci/golangci-lint/cmd latest
go: downloading golang.org/x/tools v0.0.0-20190912215617-3720d1ec3678
go: extracting golang.org/x/tools v0.0.0-20190912215617-3720d1ec3678
go: downloading github.com/securego/gosec v0.0.0-20190912120752-140048b2a218
go: downloading github.com/matoous/godox v0.0.0-20190910121045-032ad8106c86
go: extracting github.com/securego/gosec v0.0.0-20190912120752-140048b2a218
go: extracting github.com/matoous/godox v0.0.0-20190910121045-032ad8106c86
go install github.com/goreleaser/goreleaser
go: downloading github.com/goreleaser/goreleaser v0.118.2
go: extracting github.com/goreleaser/goreleaser v0.118.2
......
.....
....
.......Above bash script will read all the lines starting with an _ from the file and it will strip the " before it passes them to go install. As you can see now the go.mod and go.sum files are updated. Also notice the dependencies are not cleared anymore when running go mod tidy.
$ cat go.mod
module github.com/marcofranssen/my-project
go 1.12
require (
github.com/fullstorydev/grpcui v0.2.1
github.com/golang/protobuf v1.3.2
github.com/golangci/golangci-lint v1.19.1
github.com/goreleaser/goreleaser v0.118.2
github.com/spf13/cobra v0.0.5
github.com/tebeka/go2xunit v1.4.10
golang.org/x/lint v0.0.0-20190409202823-959b441ac422
golang.org/x/perf v0.0.0-20190823172224-ecb187b06eb0
golang.org/x/tools v0.0.0-20190912215617-3720d1ec3678
)
$ go mod tidy
$ cat go.mod
module github.com/marcofranssen/my-project
go 1.12
require (
github.com/fullstorydev/grpcui v0.2.1
github.com/golang/protobuf v1.3.2
github.com/golangci/golangci-lint v1.19.1
github.com/goreleaser/goreleaser v0.118.2
github.com/spf13/cobra v0.0.5
github.com/tebeka/go2xunit v1.4.10
golang.org/x/lint v0.0.0-20190409202823-959b441ac422
golang.org/x/perf v0.0.0-20190823172224-ecb187b06eb0
golang.org/x/tools v0.0.0-20190912215617-3720d1ec3678
)Also notice the dependencies are no longer indirect as you now have code which depends on the given module. Don't worry, due to the build constraint this code will not be compiled into the binary when running go build ., unless you ofcourse provide a build argument with the tools constraint.
Now I can imagine you don't want to type this command all the time, so the next thing I do in my projects is adding a Makefile including a install-tools task.
download:
@echo Download go.mod dependencies
@go mod download
install-tools: download
@echo Installing tools from tools.go
@cat tools.go | grep _ | awk -F'"' '{print $$2}' | xargs -tI % go install %This now allows me to run simply make install-tools.
$ make install-tools
Download go.mod dependencies
Installing tools from tools.go
go install github.com/fullstorydev/grpcui/cmd/grpcui
go install github.com/golang/protobuf/protoc-gen-go
go install github.com/golangci/golangci-lint/cmd/golangci-lint
go install github.com/goreleaser/goreleaser
go install github.com/spf13/cobra/cobra
go install github.com/tebeka/go2xunit
go install golang.org/x/lint/golint
go install golang.org/x/perf/cmd/benchstat
go install golang.org/x/tools/cmd/stringerNow in your Makefile you can add more tasks for compiling, testing, benchmarking and running your application, so you have less manual commands to type in your project.
Summarized you will now have following files in your folder, which you could now start committing in your repo before you continue setting up your project and adding the code.
$ tree my-project
my-project
├── Makefile
├── go.mod
├── go.sum
└── tools.go
0 directories, 4 filesReferences
Thanks you for your attention! Also consider to share this with your friends and colleagues on social media and leave me a comment below. See you next time.
