proposal: cmd/go: allow GOPRIVATE to provide source repository URI

[ohir]

Proposal: Allow GOPRIVATE to provide source repository URI

Last updated: 2021/04/24

Rationale:

In current state of Go modules' tooling, working with private repositories without setting up a private proxy infrastructure serving modules over https is possible but cumbersome; and still is confined to git VCS only.

For git repositories it is usually done like that:

go env -w GOPRIVATE=example.com
git config --global url."git@example.com:".insteadOf "https://example.com"

This configuration works, but it has to be global (per machine/user). To my knowledge no other VCS allows for such workaround.

Related:

#40189 : add a sub command to manage authentication

Proposal:

Allow GOPRIVATE to point to the source repository for a given module path matching pattern. Let user provide vcs type, transport, and credentials.

GOPRIVATE=pattern["["vcs+transport://[userid[:password]@]host[:port][/path]"]"][,...]

GOPRIVATE=example.com/[fossil+ssh://internal.example.com]
GOPRIVATE=example.com/[git+ssh://git@internal.example.com]
GOPRIVATE=example.com/[fossil+https://internal.example.com]
GOPRIVATE=example.com/[git+https://testing.example.com/test]

If pattern ends with slash, allow URI to be constructed using parts of matched module path represented by five percent-letter variables:

   matched.pattern/path/to/module
                           ^^^^^^ %z end element of module path
                   ^^^^           %s start element after the match
                   ^^^^^^^^^^^^^^ %p part after the match
   ^^^^^^^^^^^^^^^                %m matched by (excluding ending slash) 
   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ %w whole module path

example.com[git+file:///repos/%p.git]          # get from bare repos archive
example.com[fossil+file:///repos/%s-%z.fossil]    # or a bag of user fossils
github.com/myuser/[git+file:///gopath/src/%w/] # use old GOPATH as source
example.com/[git+ssh://%s@citest.example.com:2222/%p.git] # per CI domain

Reserved characters of RFC 3986 that should be literal part of produced URI must be given in percent-encoding form. Literal characters ":", "[", "]", "", and "@" can also be provided after a backslash character.

---

edits:

• 2021/04/24 add reference to #‍40189; add "reserved characters encoding" note.

[seankhliao]

Note it has always been possible to use your.module/path.vcs as your canonical module path, directly specifying both the module root and the VCS system to use to retrieve it

[ohir]

Note to the readers: (answering doubts raised at irc discussion) This proposal is just about simplifying private repository setup for solo developers and small teams. As per docs GOPRIVATE affects GONOPROXY and GONOSUMDB. Ie. modules still are "downloaded" (here copied) to the GOMODCACHE as if coming from any other source.

This proposal is entirely separate to #44347.

Also, the "keep GOPATH for private repositories" attitude will better be served by #‍44649 proposal.

[ianlancetaylor]

CC @bcmills @jayconrod

[rsc]

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

[marwan-at-work]

@ohir I think #40189 and this issue are two different concerns. This issue seems to be about making GOPRIVATE be smarter when authenticating with a VCS backend. While #40189 is about having a sub command that manages authenticating with a GOPROXY.

Although I agree that authenticating with private Go Modules has always been a point of friction, the syntax here seems a bit overloaded and potentially even harder for people to configure correctly.

Go, keeping a small surface area in how it interacts with VCS systems like git is a good thing in my opinion so that it's up to the user to choose the authentication mechanism (git has many ways to do so such as .netrc, config files, and credentials stores). So Go shouldn't be concerned there.

I think for that reason, a GOPROXY is a better answer which means only one server has to configure their VCS credentials correctly and everybody else wouldn't even need to have a VCS program installed on their machine to fetch private Go Modules.

However, the current state of things means that the authentication mechanism with a GOPROXY is almost as cumbersome as the authentication mechanism with a VCS sub system, hence the reason for #40189

Thanks

[ohir]

@marwan-at-work
Thank you for your insights.

the syntax here seems a bit overloaded and potentially even harder for people to configure correctly.

This is pretty much (except for vcs+ part) syntax used by most CLI net tools (wget, curl, ftp, even git itself). Anyway, I got an answer I asked for, thank you :)

[rsc]

GOPRIVATE is really for encoding one specific piece of data. Trying to add more data to the encoding does not seem like it is the right approach. It seems like there's a more general problem here to solve, but I'm not sure what the exact problem statement is and how much to generalize it.

/cc @matloob @jayconrod @bcmills

[ohir]

@rsc

for encoding one specific piece of data [...]

This specific piece of data is enough to opt-out of public, but not enough to work in private.

what the exact problem statement is.

Setting up private repositories is cumbersome for small enterprises and for solo developers. Going private needs either a fragile and error-prone configuration dispersed among at least three env variables (GOPRIVATE GOPROXY GONOPROXY), and possibly local vcs configuration — or it needs a private goproxy server like Athens to be configured and accessible.

This proposal conceives that GOPRIVATE should be the single place where small teams and solo developers can setup all things "private" — without need to think about GOPROXY order and providing external authentication tools (as proposed in #‍40189).

Then tools can access vcs repository and add missing private modules to GOMODCACHE /cache/downloads obeying goproxy protocol.