Go Modules has been the default since Go 1.13. One of microservices I am responsible for uses Golang 1.13 currently. The oldest Go version used by the application was 1.9 and at the time, dep was the most obvious choice for dependency management.

There are three interesting points characterising the app:

1. It runs inside Docker container on production.
2. All continuous integration checks (fmt, vet and tests) are ran inside containers as well.
3. It relies on an external Go package hosted on a private GitHub repository. Let’s name it go-utils.

dep and private GitHub repositories

To support the above points we maintain two similar but different files:

1. Dockerfile,
2. Dockerfile-test.

At the same time, we treat security seriously👮. Due to that, during creation of the files, we had to find a secure solution for fetching Go external packages from private GitHub repository within Docker images. Fortunately, some clever solutions were proposed by smart people on the Internet. We decided to take usage of GitHub personal access token:

ARG GITHUB_TOKEN
git config --global url."https://${GITHUB_TOKEN}:x-oauth-ba[email protected]/AirHelp/".insteadOf "https://github.com/AirHelp/"

The GITHUB_TOKEN environment variable is injected to Docker image dynamically on CI server’s agent. On local machine a developer can use her/his personal access token.

Using git config --global url.<base>.insteadOf trick allowed as to inject the token as a part of requests made to GitHub by dep to fetch private repository stored within our organization.

Thanks to the above solution, we were able to fetch the external Go package from the private repository and build the application’s binary within a Docker image successfully. We were done…

…unfortunately, we were not. You may already know that using ARG and ENV leaves traces. Everybody who has access to the image can obtain secret values without effort, e.g. by using the docker history command. That’s a security concern ⚠️.

Multi-stage builds to the rescue

Luckily, some solutions to the concern had already been proposed as well. One of them is called multi-stage builds. To keep it short, you can build multiple images (steps) using a single Dockerfile and only history of the last one (the one at the bottom of the file) will be outputted by docker history command. Moreover, the size of the final image is equal to the size of the latest one 💯.

The final version of our Dockerfile-test file used for running CI checks looked like below (some lines were removed to hide sensitive data):

# We use multi-stage build to avoid storing GitHub token inside Docker image
FROM golang:1.9.3 AS golang-build

WORKDIR /go/src/github.com/AirHelp/app

ARG GITHUB_TOKEN

RUN apk --no-cache add git gcc musl-dev && \
    git config --global url."https://${GITHUB_TOKEN}:[email protected]/AirHelp/".insteadOf "ssh://[email protected]/AirHelp/" && \
    go get -u github.com/golang/dep/cmd/dep && \

# Copy the Gopkg.toml and Gopkg.lock to WORKDIR
COPY Gopkg.toml Gopkg.lock ./

# Install the dependencies without checking for go code
RUN dep ensure -v -vendor-only

COPY . .

RUN CGO_ENABLED=1 GOOS=linux go build --ldflags '-extldflags "-static"' -installsuffix cgo -o app.

FROM golang:1.9.3

WORKDIR /usr/src/app

RUN go get github.com/onsi/ginkgo/ginkgo && \
    go get github.com/onsi/gomega

COPY --from=golang-build /go/src/github.com/AirHelp/app .

As a result, we had a secure production-ready image with an ability to fetch external Go package from our private repository.

Migration to Go Modules

No so long ago, the go-utils package the app relies on was adjusted to use Go Modules for dependency management. To standardise our approach across Golang projects we decided to migrate the app as well. Especially, because Golang 1.13 (finally!) provides official support for private modules.

The process of migration from dep to Go Modules has already been described in many places, so we wrote down a plan:

1. To have go.mod and go.sum files we will execute go mod init inside app’s directory.
2. We will run go mod tidy command.
3. We will remove Gopkg.toml and Gopkg.lock files.
4. We will decouple app’s Dockerfiles from dep tool by removing go get -u github.com/golang/dep/cmd/dep line.
5. We will replace COPY Gopkg.toml Gopkg.lock ./ line in the dockerfiles by COPY go.mod go.sum ./
6. We will replace RUN dep ensure -v -vendor-only line in the dockerfiles by go mod download.

Quite a straightforward set of instructions. Such straightforward that we failed at the second step ¯\_(ツ)_/¯.

verifying github.com/AirHelp/[email protected]/go.mod: github.com/AirHelp/[email protected]/go.mod: reading https://sum.golang.org/lookup/github.com/!air!help/[email protected]: 410 Gone

In the above log, we spotted the name of our private module. According to the official Golang 1.13 release note, the GOPRIVATE environment variable can be used for marking modules not available publicly. We quickly adjusted the second point:

2. We will run GOPRIVATE=github.com/AirHelp/go-utils go mod tidy command.

It worked like a charm 🎉. We had to reflect the above change also in the dockerfiles, so we extended the plan with one action point:

7. We will set GOPRIVATE variable inside the dockerfiles by putting ENV GOPRIVATE github.com/AirHelp/go-utils line before go mod install line to mark the private module there as well.

After executing the above steps we managed to build app’s binary within Docker container (Dockerfile file). At the same time, we didn’t manage to run tests (Dockerfile-test file).

go: github.com/AirHelp/[email protected]: reading github.com/AirHelp/go-utils/go.mod at revision v1.0.0: unknown revision v1.0.0
make: *** [vet] Error 1

The error is almost identical like the one we encountered a while ago. The only difference is that this one was raised during vet command execution. Trying to find the root cause we spent some time googling for similar issues. We found a lot of issues and blog posts describing usage of Go Modules inside Docker images and containers. Only building a binary was described to the letter, though. Is it because nobody runs tests inside containers? I dunno.

The missing piece

After starring at the Dockerfile-test I tracked a line that made me wonder:

COPY --from=golang-build /go/src/github.com/AirHelp/app .

What does the above line do? It copies app’s directory from the previous step (image). That makes sense, we need all the files for running tests. What about Go Modules, though? Where are they located? After quick googling, it occurred that the default location for them is $GOPATH/pkg/mod.

The modules are installed in the first step. The second one has no access to it. To solve the issue we had to add another point to the migration list:

8. We will copy the downloaded Go Modules to the second stage to have them available there: COPY --from=golang-build $GOPATH/pkg/mod $GOPATH/pkg/mod

The final Dockerfile-test adjusted to support Go Modules:

# We use multi-stage build to avoid storing GitHub token inside Docker image
FROM golang:1.13.6-alpine3.11 AS golang-build

WORKDIR /go/src/github.com/AirHelp/app

# GitHub token for downloading private dependencies
ARG GITHUB_TOKEN

RUN apk add --no-cache git ca-certificates && \
    git config --global url."https://${GITHUB_TOKEN}:[email protected]/AirHelp/".insteadOf "https://github.com/AirHelp/"

# Required to access go mod from private AirHelp repo
ENV GOPRIVATE github.com/AirHelp/go-utils

# Copy the go.mod and go.sum files to WORKDIR
COPY go.mod go.sum ./

# Install the dependencies without checking for go code
RUN go mod download

COPY . .

FROM golang:1.13.6-alpine3.11

WORKDIR /go/src/github.com/AirHelp/app

RUN go get github.com/onsi/ginkgo/ginkgo && \
    go get github.com/onsi/gomega

COPY --from=golang-build /go/src/github.com/AirHelp/app .

# Copy go modules fetched by go mod download command in the previous stage
COPY --from=golang-build $GOPATH/pkg/mod $GOPATH/pkg/mod

Summary

We have had official support for marking and fetching modules which are not publicly available since Golang 1.13. Thanks to that, we were able to start migration process of our dockerized internal Go packages and applications from dep to Go modules. After some initial obstacles, we have an eight-steps list written down which simply works.

If you consider a similar migration, I hope you will find the above list helpful 🙂