How to handle dependencies

[markose]

Hello,

I have another question.
I've created a package for launchpad, but the build fails, because of missing dependencies. Where do I have to specify external go modules that I include so it can be fetched during build step?

Regards,
Marko

[markose]

This is the error message in buildlog:

dpkg-buildpackage: info: source package dbsync2
dpkg-buildpackage: info: source version 1.2
dpkg-buildpackage: info: source distribution bionic
 dpkg-source --before-build dbsync2-1.2
dpkg-buildpackage: info: host architecture amd64
 fakeroot debian/rules clean
dh clean --buildsystem=golang --with=golang
   dh_auto_clean -O--buildsystem=golang
   dh_clean -O--buildsystem=golang
 debian/rules build
dh build --buildsystem=golang --with=golang
   dh_update_autotools_config -O--buildsystem=golang
   dh_autoreconf -O--buildsystem=golang
   dh_auto_configure -O--buildsystem=golang
   dh_auto_build -O--buildsystem=golang
	cd obj-x86_64-linux-gnu && go install -gcflags=\"-trimpath=/<<PKGBUILDDIR>>/obj-x86_64-linux-gnu/src\" -asmflags=\"-trimpath=/<<PKGBUILDDIR>>/obj-x86_64-linux-gnu/src\" -v -p 4 bitbucket.org/modima/dbsync2 bitbucket.org/modima/dbsync2/database bitbucket.org/modima/dbsync2/ttlcache
src/bitbucket.org/modima/dbsync2/main.go:23:2: cannot find package "github.com/knadh/koanf" in any of:
	/usr/lib/go-1.10/src/github.com/knadh/koanf (from $GOROOT)
	/<<PKGBUILDDIR>>/obj-x86_64-linux-gnu/src/github.com/knadh/koanf (from $GOPATH)
src/bitbucket.org/modima/dbsync2/main.go:24:2: cannot find package "github.com/knadh/koanf/providers/env" in any of:
	/usr/lib/go-1.10/src/github.com/knadh/koanf/providers/env (from $GOROOT)
	/<<PKGBUILDDIR>>/obj-x86_64-linux-gnu/src/github.com/knadh/koanf/providers/env (from $GOPATH)
src/bitbucket.org/modima/dbsync2/main.go:25:2: cannot find package "github.com/knadh/koanf/providers/posflag" in any of:
	/usr/lib/go-1.10/src/github.com/knadh/koanf/providers/posflag (from $GOROOT)
	/<<PKGBUILDDIR>>/obj-x86_64-linux-gnu/src/github.com/knadh/koanf/providers/posflag (from $GOPATH)
src/bitbucket.org/modima/dbsync2/database/connection.go:19:2: cannot find package "github.com/op/go-logging" in any of:
	/usr/lib/go-1.10/src/github.com/op/go-logging (from $GOROOT)
	/<<PKGBUILDDIR>>/obj-x86_64-linux-gnu/src/github.com/op/go-logging (from $GOPATH)
src/bitbucket.org/modima/dbsync2/main.go:27:2: cannot find package "github.com/spf13/pflag" in any of:
	/usr/lib/go-1.10/src/github.com/spf13/pflag (from $GOROOT)
	/<<PKGBUILDDIR>>/obj-x86_64-linux-gnu/src/github.com/spf13/pflag (from $GOPATH)
dh_auto_build: cd obj-x86_64-linux-gnu && go install -gcflags=\"-trimpath=/<<PKGBUILDDIR>>/obj-x86_64-linux-gnu/src\" -asmflags=\"-trimpath=/<<PKGBUILDDIR>>/obj-x86_64-linux-gnu/src\" -v -p 4 bitbucket.org/modima/dbsync2 bitbucket.org/modima/dbsync2/database bitbucket.org/modima/dbsync2/ttlcache returned exit code 1
debian/rules:4: recipe for target 'build' failed
make: *** [build] Error 1
dpkg-buildpackage: error: debian/rules build subprocess returned exit status 2

[ThreeFx]

The Debian way is to prepare the dependencies as packages as well. So you'll have to identify the dependencies and prepare packages for them as well. Once you have created those you can include them either by installing them in your build environment.

It should all be described here.

[zhsj]

The dh-make-golang is intended to make a Debian package in the official Debian archive, which has our own packaging policy, like the dependencies should be packaged first.

If you just want to upload package to Launchpad, you can use other methods, which can be very flexible.

[anthonyfok]

Hi @markose, are you Marko Seidenglanz of https://bitbucket.org/modima/dbsync2 ? The name dbsync2 looks familiar for some reason. :-)

I couldn't find a place to file an issue at your Bitbucket repo, so I'll write to you here:

1. For Ubuntu PPA, feel free to it anyway you like, though I noticed that dbsync2 is missing only two dependencies in Debian sid (hence Ubuntu eoan or focal):

2020/02/24 13:10:18 Build-Dependency "github.com/knadh/koanf" is not yet available in Debian, or has not yet been converted to use XS-Go-Import-Path in debian/control
2020/02/24 13:10:18 Build-Dependency "github.com/op/go-logging" is not yet available in Debian, or has not yet been converted to use XS-Go-Import-Path in debian/control

and github.com/knadh/koanf itself depends on github.com/rhnvrm/simples3

If dbsync2 is generally useful, I think it is worthwhile to get it into Debian, as it is only 4 packages including dbsync2 to get into Debian, so that dbsync2 would eventually flow into Ubuntu universe and other Debian or Ubuntu derivatives. If you'd like that, we could work together to get dbsync2 into Debian. What do you think?

2. Please consider prepending a "v" to your tags, as in changing e.g. 1.2.7 to v1.2.7. It seems that Go modules only recognize the v1.2.7 form as semantic version. For example:

$ GO111MODULE=on go get bitbucket.org/modima/[email protected]
go: finding bitbucket.org v1.2.7
go: finding bitbucket.org/modima/dbsync2 v1.2.7
go: finding bitbucket.org/modima v1.2.7
go: finding bitbucket.org/modima/dbsync2 v1.2.7
go get bitbucket.org/modima/[email protected]: bitbucket.org/modima/[email protected]: invalid version: unknown revision v1.2.7

and it stops there without actually getting anything.

How about using @1.2.7?

$ GO111MODULE=on go get bitbucket.org/modima/[email protected]
go: finding bitbucket.org 1.2.7
go: finding bitbucket.org/modima 1.2.7
go: finding bitbucket.org/modima/dbsync2 1.2.7
go: downloading bitbucket.org/modima/dbsync2 v0.0.0-20200123095626-eaf8f02e09be
go: extracting bitbucket.org/modima/dbsync2 v0.0.0-20200123095626-eaf8f02e09be
...

so it does dbsync2 and its dependencies, but notice how Go completely ignores the supposed 1.2.7 version and went on to use its own pseudo-version v0.0.0-20200123095626-eaf8f02e09be.

Recent versions of dh-make-golang do warn about this to:

2020/02/24 13:10:03 Determining upstream version number
2020/02/24 13:10:03 Found latest tag "1.2.7"
2020/02/24 13:10:03 WARNING: Latest tag "1.2.7" is not a valid SemVer version
2020/02/24 13:10:03 INFO: master is ahead of "1.2.7" by 1 commits
2020/02/24 13:10:03 Package version is "1.2.7"

[francislavoie]

@zhsj is there any way around that at all re packaging all individual dependencies?

I'm looking into packaging for https://github.com/caddyserver/caddy and requiring all dependencies to be packaged individually is too big of a maintenance burden.

Could we set up a separate git repo where we use go mod vendor to store all the code the project depends on for purposes of packaging?

For the time being, we're going to release .debs built with GoReleaser/NFPM via https://gemfury.com with the binary built during our CI workflow.

We'd love to get Caddy in official debian repos, but we need help to do it.

[zhsj]

@zhsj is there any way around that at all re packaging all individual dependencies?

I'm looking into packaging for https://github.com/caddyserver/caddy and requiring all dependencies to be packaged individually is too big of a maintenance burden.

Could we set up a separate git repo where we use go mod vendor to store all the code the project depends on for purposes of packaging?

For the time being, we're going to release .debs built with GoReleaser/NFPM via https://gemfury.com with the binary built during our CI workflow.

We'd love to get Caddy in official debian repos, but we need help to do it.

If you want to package caddy for the official repo, yes, to package every dependency separately is desired.

To bundle all the dependencies in one source package is possible, but that's not the thing we want in official repo.
ref: https://wiki.debian.org/EmbeddedCopies

[zhsj]

@francislavoie if I read your profile correctly, you're part of caddy upstream. I think you may interested in https://bugs.debian.org/810890 and http://bugs.debian.org/954793

And, this repo's issue is not watched by most Debian developers. The debian-go(at)lists.debian.org list is better place to seek helps.

[francislavoie]

Thanks for those links - we were aware that people have attempted to get Caddy in official distro in the past, but as far as I know, they never contacted the Caddy project.

Also as an aside, personally, I'm not a fan of mailing lists for communication, there are much better methods in the year 2020. Hence why I asked here first.

So from what I understand, we'd need to validate that every one of the dependencies listed in our go.sum is a separate debian package? The dependency tree is pretty massive, I don't think that's a realistic requirement here. Is there really no simpler way to go about this?

Ultimately the only thing Caddy needs to distribute is the statically compiled binary, a default config, a default index.html landing page, and the systemd service files.

[zhsj]

Thanks for those links - we were aware that people have attempted to get Caddy in official distro in the past, but as far as I know, they never contacted the Caddy project.

Also as an aside, personally, I'm not a fan of mailing lists for communication, there are much better methods in the year 2020. Hence why I asked here first.

But it's off-topic here. And you need to live with mailing list if you want to work with Debian people...

So from what I understand, we'd need to validate that every one of the dependencies listed in our go.sum is a separate debian package? The dependency tree is pretty massive, I don't think that's a realistic requirement here. Is there really no simpler way to go about this?

There're alread many dependencies which have been packaged, like the golang.org/x/*, cloud.google.com/go, many github.com/xx.

I bet you don't want to review these libraries' license and copyright again in the vendor directory. Just to mention that it's required to review every single file's license in the source package.

Ultimately the only thing Caddy needs to distribute is the statically compiled binary, a default config, a default index.html landing page, and the systemd service files.

Yes, I'm well aware of it, and I was a caddy user :)