Blog.

Manage Go tools via Go modules

MF

Marco Franssen /

6 min read1196 words

Cover Image for Manage Go tools via Go modules

In this blog I will cover how I'm managing and versioning the tools my Go projects depend on. Go Modules are available since Go 1.11. Using Go Modules you can manage the dependencies for your project. You can compare it to NPM in Nodejs projects or Maven in Java project or Nuget in .NET projects.

In general Go Modules are used to manage your compile time dependencies. However in my projects I also like to manage the tools required for Continuous Integration in my projects. To ensure all developers have same versions of tools installed and to ensure my CI server (Jenkins, Travis, CircleCI) can install and use the same version of the tools. I found a way using Go Modules, by default you will have some issue with go mod tidy if you would just manually add the tools as dependencies to your go.mod file.

Initialize

To start with Go Modules we first have to initialize a new Go Module. We do that by creating a new folder and run the go mod init command.

$ mkdir my-project
$ cd my-project
$ go mod init github.com/marcofranssen/my-project
go: creating new go.mod: module github.com/marcofranssen/my-project

This will result in following go.mod file.

$ cat go.mod
module github.com/marcofranssen/my-project

go 1.12

Add your tools as dependency

Now we want to add our tools as dependency to our go.mod. In general you would think to just add them using go get, which works at first sight perfectly fine. See below example.

$ go get -u github.com/goreleaser/goreleaser
$ go get -u golang.org/x/lint/golint
$ cat go.mod
module github.com/marcofranssen/my-project

go 1.12

require (
        golang.org/x/crypto v0.0.0-20191001170739-f9e2070545dc // indirect
        golang.org/x/lint v0.0.0-20190930215403-16217165b5de // indirect
        golang.org/x/net v0.0.0-20191002035440-2ec189313ef0 // indirect
        golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e // indirect
        golang.org/x/sys v0.0.0-20191002091554-b397fe3ad8ed // indirect
        golang.org/x/text v0.3.2 // indirect
        golang.org/x/tools v0.0.0-20191001184121-329c8d646ebe // indirect
)

As you can see it added indirect dependencies, and you will notice the goreleaser dependencies are not there at all as the second go get has removed them. Indirect means there is none of your own code which has a direct dependency on the module, which is correct as these are tools I would use and not dependencies of my own to be written code. You will also see there is a go.sum file generated, which I will skip for now as it is not relevant for this explanation.

Now there is also the go mod tidy command which cleans your dependencies etc. This is a recommended command to run before you make a release to ensure all dependencies are cleaned and accurate with the real needs of your code. So lets run that command now and check what happens.

$ go mod tidy
$ cat go.mod
module github.com/marcofranssen/my-project

go 1.12

As you can see all the dependencies are again removed from the go.mod file and the go.sum file is cleaned up as well. This happens because there is no dependency in any .go file.

TL;DR

To ensure my tool dependencies are not removed and can leverage the Go Modules, I create a file tools.go. In this file I will list all my tool dependencies using an import statement.

// +build tools

package main

import (
    _ "github.com/fullstorydev/grpcui/cmd/grpcui"
    _ "github.com/golang/protobuf/protoc-gen-go"
    _ "github.com/golangci/golangci-lint/cmd/golangci-lint"
    _ "github.com/goreleaser/goreleaser"
    _ "github.com/spf13/cobra/cobra"
    _ "github.com/tebeka/go2xunit"
    _ "golang.org/x/lint/golint"
    _ "golang.org/x/perf/cmd/benchstat"
    _ "golang.org/x/tools/cmd/stringer"
)

As you can notice I have also added a build constraint a.k.a. build tag comment in the top of the file, to ensure it is not compiled into the binary, when running go build. Now with this file in place I can very easily install all my tools using a simple bash command. go install will make all the tools available in your $GO_WORKSPACE/bin folder. Normally this folder is available in your PATH so you can use the binaries in any folder of your choice.

$ cat tools.go | grep _ | awk -F'"' '{print $2}' | xargs -tI % go install %
go install github.com/fullstorydev/grpcui/cmd/grpcui
go: finding github.com/fullstorydev/grpcui/cmd/grpcui latest
go: finding github.com/fullstorydev/grpcui/cmd latest
go: downloading golang.org/x/net v0.0.0-20190522155817-f3200d17e092
go: extracting golang.org/x/net v0.0.0-20190522155817-f3200d17e092
go install github.com/golang/protobuf/protoc-gen-go
go install github.com/golangci/golangci-lint/cmd/golangci-lint
go: finding github.com/golangci/golangci-lint/cmd/golangci-lint latest
go: finding github.com/golangci/golangci-lint/cmd latest
go: downloading golang.org/x/tools v0.0.0-20190912215617-3720d1ec3678
go: extracting golang.org/x/tools v0.0.0-20190912215617-3720d1ec3678
go: downloading github.com/securego/gosec v0.0.0-20190912120752-140048b2a218
go: downloading github.com/matoous/godox v0.0.0-20190910121045-032ad8106c86
go: extracting github.com/securego/gosec v0.0.0-20190912120752-140048b2a218
go: extracting github.com/matoous/godox v0.0.0-20190910121045-032ad8106c86
go install github.com/goreleaser/goreleaser
go: downloading github.com/goreleaser/goreleaser v0.118.2
go: extracting github.com/goreleaser/goreleaser v0.118.2
......
.....
....
.......

Above bash script will read all the lines starting with an _ from the file and it will strip the " before it passes them to go install. As you can see now the go.mod and go.sum files are updated. Also notice the dependencies are not cleared anymore when running go mod tidy.

$ cat go.mod
module github.com/marcofranssen/my-project

go 1.12

require (
        github.com/fullstorydev/grpcui v0.2.1
        github.com/golang/protobuf v1.3.2
        github.com/golangci/golangci-lint v1.19.1
        github.com/goreleaser/goreleaser v0.118.2
        github.com/spf13/cobra v0.0.5
        github.com/tebeka/go2xunit v1.4.10
        golang.org/x/lint v0.0.0-20190409202823-959b441ac422
        golang.org/x/perf v0.0.0-20190823172224-ecb187b06eb0
        golang.org/x/tools v0.0.0-20190912215617-3720d1ec3678
)

$ go mod tidy
$ cat go.mod
module github.com/marcofranssen/my-project

go 1.12

require (
        github.com/fullstorydev/grpcui v0.2.1
        github.com/golang/protobuf v1.3.2
        github.com/golangci/golangci-lint v1.19.1
        github.com/goreleaser/goreleaser v0.118.2
        github.com/spf13/cobra v0.0.5
        github.com/tebeka/go2xunit v1.4.10
        golang.org/x/lint v0.0.0-20190409202823-959b441ac422
        golang.org/x/perf v0.0.0-20190823172224-ecb187b06eb0
        golang.org/x/tools v0.0.0-20190912215617-3720d1ec3678
)

Also notice the dependencies are no longer indirect as you now have code which depends on the given module. Don't worry, due to the build constraint this code will not be compiled into the binary when running go build ., unless you ofcourse provide a build argument with the tools constraint.

Now I can imagine you don't want to type this command all the time, so the next thing I do in my projects is adding a Makefile including a install-tools task.

download:
  @echo Download go.mod dependencies
  @go mod download

install-tools: download
  @echo Installing tools from tools.go
  @cat tools.go | grep _ | awk -F'"' '{print $$2}' | xargs -tI % go install %

This now allows me to run simply make install-tools.

$ make install-tools
Download go.mod dependencies
Installing tools from tools.go
go install github.com/fullstorydev/grpcui/cmd/grpcui
go install github.com/golang/protobuf/protoc-gen-go
go install github.com/golangci/golangci-lint/cmd/golangci-lint
go install github.com/goreleaser/goreleaser
go install github.com/spf13/cobra/cobra
go install github.com/tebeka/go2xunit
go install golang.org/x/lint/golint
go install golang.org/x/perf/cmd/benchstat
go install golang.org/x/tools/cmd/stringer

Now in your Makefile you can add more tasks for compiling, testing, benchmarking and running your application, so you have less manual commands to type in your project.

Summarized you will now have following files in your folder, which you could now start committing in your repo before you continue setting up your project and adding the code.

$ tree my-project
my-project
├── Makefile
├── go.mod
├── go.sum
└── tools.go

0 directories, 4 files

References

Thanks you for your attention! Also consider to share this with your friends and colleagues on social media and leave me a comment below. See you next time.

You have disabled cookies. To leave me a comment please allow cookies at functionality level.

More Stories

Cover Image for Secure 2FA SSH and PGP using Krypton

Secure 2FA SSH and PGP using Krypton

MF

Marco Franssen /

In this blogpost I want to show you how easy it is to setup SSH and PGP key securily without even having them on your laptop. Instead we will have those keys on our mobile device. Yes, I hear you thinking…. Wutt, but, but, but, whaat! No worries bear with me, I walk you through it an will even explain you some magic behind the Krypton commands which we are about to use, so you will have a fully transparant understanding on Krypton. First of all you shouldn't worry about the safety of your keys…

Cover Image for Howto Secure Shell easily from the terminal

Howto Secure Shell easily from the terminal

MF

Marco Franssen /

I see many struggle when it comes to using Secure Shell in a comfortable way. Many are installing unneeded applications like Putty on Windows for example. Just like I did 4 years ago. Over the years I have been working a lot on servers where there was no GUI available and learned a lot doing that. I would like to share my tips and tricks so you can also be empowered by just sticking to the terminal on your OS or simply using Git Bash on Windows. What is SSH The SSH protocol (also referred to a…

Cover Image for Install fresh Raspbian image on your Raspberry Pi - part 2

Install fresh Raspbian image on your Raspberry Pi - part 2

MF

Marco Franssen /

In the previous blog of this series I explained the basics of getting a fresh installation of Raspbian on your Rapberry Pi including SSH access and configuration of a static IP for your Wifi. In this blog we are going to have a look at tweaking it a little further to get a better commandline experience and have some more tooling available to operate your Raspberry Pi. Admin experience and tooling On any server I will most likely always work with Git and Vim. I have many of my bash scripts and…

Cover Image for Install fresh Raspbian image on your Raspberry Pi - part 1

Install fresh Raspbian image on your Raspberry Pi - part 1

MF

Marco Franssen /

In my last blog I shown you how to upgrade from Raspbian Stretch to Raspbian Buster. As the whole upgrade went smooth it seems there are still some issues with Raspbian Buster. The one which blocked me completely was the ability to run Kodi. Kodi was not able to start due to a GUI error. Therefore I decided to switch back to Raspbian Stretch until Raspbian Buster packages catch up. Prerequisuites Before you start a fresh install make sure you have an ethernet cable at hand, as we will need it…