Loading...
Manage Go tools via Go modules
Marco Franssen
6 min read•1020 words
In this post, I'll show how I manage and version the tools my Go projects depend on. Go modules have been available since Go 1.11 and let you manage project dependencies, much like npm in Node.js, Maven in Java, or NuGet in .NET.
Go modules generally manage compile-time dependencies. In my projects, I also want to manage the tools used for continuous integration, so developers and CI servers such as Jenkins, Travis, or CircleCI use the same versions. I've found a way to do this with Go modules. Simply adding tools to go.mod isn't enough, though: go mod tidy can remove them.
Initialize
To get started with Go modules, we'll initialize a module by creating a folder and running go mod init.
$ mkdir my-project
$ cd my-project
$ go mod init github.com/marcofranssen/my-project
go: creating new go.mod: module github.com/marcofranssen/my-projectThis creates the following go.mod file.
$ cat go.mod
module github.com/marcofranssen/my-project
go 1.12Add your tools as dependencies
Now we want to add our tools as dependencies in go.mod. You might expect to do this with go get, which appears to work at first. Let's try it.
$ go get -u github.com/goreleaser/goreleaser
$ go get -u golang.org/x/lint/golint
$ cat go.mod
module github.com/marcofranssen/my-project
go 1.12
require (
golang.org/x/crypto v0.0.0-20191001170739-f9e2070545dc // indirect
golang.org/x/lint v0.0.0-20190930215403-16217165b5de // indirect
golang.org/x/net v0.0.0-20191002035440-2ec189313ef0 // indirect
golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e // indirect
golang.org/x/sys v0.0.0-20191002091554-b397fe3ad8ed // indirect
golang.org/x/text v0.3.2 // indirect
golang.org/x/tools v0.0.0-20191001184121-329c8d646ebe // indirect
)The command adds indirect dependencies, but notice that the GoReleaser dependencies aren't there: the second go get removed them. "Indirect" means our own code doesn't directly depend on the module, which makes sense because these are tools we use rather than dependencies of the application code. A go.sum file is also generated, but I'll skip that for now because it isn't relevant to this explanation.
The go mod tidy command cleans up dependencies. It's a useful command to run before a release to make sure the dependencies match what your code actually needs. Let's run it and see what happens.
$ go mod tidy
$ cat go.mod
module github.com/marcofranssen/my-project
go 1.12All the dependencies have been removed from go.mod again, and go.sum has been cleaned up as well. This happens because no .go file depends on those modules.
TL;DR
To keep my tool dependencies managed by Go modules without having them removed, I create a file named tools.go. In it, I list each tool dependency with an import statement.
// +build tools
package main
import (
_ "github.com/fullstorydev/grpcui/cmd/grpcui"
_ "github.com/golang/protobuf/protoc-gen-go"
_ "github.com/golangci/golangci-lint/cmd/golangci-lint"
_ "github.com/goreleaser/goreleaser"
_ "github.com/spf13/cobra/cobra"
_ "github.com/tebeka/go2xunit"
_ "golang.org/x/lint/golint"
_ "golang.org/x/perf/cmd/benchstat"
_ "golang.org/x/tools/cmd/stringer"
)Notice the build constraint, also called a build tag, at the top of the file. It keeps this file out of the binary when running go build. With the file in place, I can install all my tools using a simple Bash command. go install makes them available in your $GO_WORKSPACE/bin folder. If that folder is in your PATH, you can run the binaries from any directory.
$ cat tools.go | grep _ | awk -F'"' '{print $2}' | xargs -tI % go install %
go install github.com/fullstorydev/grpcui/cmd/grpcui
go: finding github.com/fullstorydev/grpcui/cmd/grpcui latest
go: finding github.com/fullstorydev/grpcui/cmd latest
go: downloading golang.org/x/net v0.0.0-20190522155817-f3200d17e092
go: extracting golang.org/x/net v0.0.0-20190522155817-f3200d17e092
go install github.com/golang/protobuf/protoc-gen-go
go install github.com/golangci/golangci-lint/cmd/golangci-lint
go: finding github.com/golangci/golangci-lint/cmd/golangci-lint latest
go: finding github.com/golangci/golangci-lint/cmd latest
go: downloading golang.org/x/tools v0.0.0-20190912215617-3720d1ec3678
go: extracting golang.org/x/tools v0.0.0-20190912215617-3720d1ec3678
go: downloading github.com/securego/gosec v0.0.0-20190912120752-140048b2a218
go: downloading github.com/matoous/godox v0.0.0-20190910121045-032ad8106c86
go: extracting github.com/securego/gosec v0.0.0-20190912120752-140048b2a218
go: extracting github.com/matoous/godox v0.0.0-20190910121045-032ad8106c86
go install github.com/goreleaser/goreleaser
go: downloading github.com/goreleaser/goreleaser v0.118.2
go: extracting github.com/goreleaser/goreleaser v0.118.2
......
.....
....
.......The Bash pipeline above finds lines containing _, extracts the import paths between the " characters, and passes each path to go install. The go.mod and go.sum files are updated, and go mod tidy no longer removes the tool dependencies.
$ cat go.mod
module github.com/marcofranssen/my-project
go 1.12
require (
github.com/fullstorydev/grpcui v0.2.1
github.com/golang/protobuf v1.3.2
github.com/golangci/golangci-lint v1.19.1
github.com/goreleaser/goreleaser v0.118.2
github.com/spf13/cobra v0.0.5
github.com/tebeka/go2xunit v1.4.10
golang.org/x/lint v0.0.0-20190409202823-959b441ac422
golang.org/x/perf v0.0.0-20190823172224-ecb187b06eb0
golang.org/x/tools v0.0.0-20190912215617-3720d1ec3678
)
$ go mod tidy
$ cat go.mod
module github.com/marcofranssen/my-project
go 1.12
require (
github.com/fullstorydev/grpcui v0.2.1
github.com/golang/protobuf v1.3.2
github.com/golangci/golangci-lint v1.19.1
github.com/goreleaser/goreleaser v0.118.2
github.com/spf13/cobra v0.0.5
github.com/tebeka/go2xunit v1.4.10
golang.org/x/lint v0.0.0-20190409202823-959b441ac422
golang.org/x/perf v0.0.0-20190823172224-ecb187b06eb0
golang.org/x/tools v0.0.0-20190912215617-3720d1ec3678
)Notice that the dependencies are no longer indirect, because we now have code that depends on the modules. Thanks to the build constraint, this code won't be compiled into the binary when running go build ., unless you explicitly enable the tools build tag.
I imagine you don't want to type that command every time. The next thing I do is add a Makefile with an install-tools task.
download:
@echo Download go.mod dependencies
@go mod download
install-tools: download
@echo Installing tools from tools.go
@cat tools.go | grep _ | awk -F'"' '{print $$2}' | xargs -tI % go install %Now I can simply run make install-tools.
$ make install-tools
Download go.mod dependencies
Installing tools from tools.go
go install github.com/fullstorydev/grpcui/cmd/grpcui
go install github.com/golang/protobuf/protoc-gen-go
go install github.com/golangci/golangci-lint/cmd/golangci-lint
go install github.com/goreleaser/goreleaser
go install github.com/spf13/cobra/cobra
go install github.com/tebeka/go2xunit
go install golang.org/x/lint/golint
go install golang.org/x/perf/cmd/benchstat
go install golang.org/x/tools/cmd/stringerYou can add more Makefile tasks for compiling, testing, benchmarking, and running your application, leaving fewer commands to type manually.
You now have the following files in your folder, ready to commit before continuing with the project setup and application code.
$ tree my-project
my-project
├── Makefile
├── go.mod
├── go.sum
└── tools.go
0 directories, 4 filesReferences
Thanks for your attention! Consider sharing this post with your friends and colleagues on social media, and leave me a comment below. See you next time.
