Remove packages from erlang.mk by default; package indexes as plugins; + better hex.pm integration

[zuiderkwast]

Maybe new users will get old version by mistake. The example in the used in the README contains DEPS = cowboy which will give them 1.0.4. The index file for cowboy contains:

pkg_cowboy_repo = https://github.com/ninenines/cowboy
pkg_cowboy_commit = 1.0.4

Either the examples could include an explicit version for each dep or the index could be updated. I don't know which is better. Personally, I don't use the index.

[essen]

The intent is to remove the index but there are no good alternatives. On the other hand at this point the Cowboy commit can safely be set back to master since nobody uses 1.x in new projects anymore.

[zuiderkwast]

Using github URIs with tag or hash is an alternative. The examples in the README and maybe also the erlang.mk guide might influence or mislead new users. Maybe some of them assume that the index is updated and that Cowboy 1.0.4 is a recent version....

[essen]

Just need to set dep_cowboy_commit = 2.8.0 for example. But problem is keeping this updated anyway.

[zuiderkwast]

Yes I know. Automatically updating the index to the latest tag of each project using some script perhaps, but if you don't want to spend time on the index, perhaps just add dep_cowboy = git https://github.com/ninenines/cowboy 1.0.4 to the examples and a description "Remember to change the version to a recent version that you want to use" or so, so they know it's not updated.

[essen]

Sure but like I said, dep_cowboy_commit = 2.8.0 is enough, no need to set the URI again it's in the index. We just override the version.

[zuiderkwast]

#908

You can regard the PR as an example or suggestion. Just by showing how to specify a version explicitly may make the users think a bit and choose a version.

[essen]

OK so the plan is to do work around both #902 and the package index. Group breaking changes with potentially breaking changes together, even though #902 work will not be enabled by default.

In the case of the package index I will add proper support for Hex via https://github.com/hexpm/hex_core and then when not providing a version number Erlang.mk will first look into its own package index, and then if it finds nothing it will look into Hex. Maybe we can make the lookup order configurable as well.

At the same time I will remove most packages from the Erlang.mk index. As a first pass I will remove:

• The packages I maintain personally can stay in the index (with the version set back to master)
• The packages that are in Hex can be removed from the Erlang.mk index
• The packages that people are maintaining here and are otherwise in Hex can be left here and the maintainer be informed
• The packages that are obviously broken (not maintained and don't work in newer OTP versions for example) can be removed

[essen]

For publishing this is what RabbitMQ currently does:

• https://github.com/rabbitmq/rabbitmq-common/blob/master/mk/rabbitmq-hexpm.mk
• https://github.com/rabbitmq/hexpm-cli

We should stay compatible as much as possible.

[essen]

Useful links:

• https://github.com/hexpm/hex_core/blob/master/src/hex_api_release.erl
• https://github.com/hexpm/hex_core/blob/master/test/hex_tarball_SUITE.erl
• https://hex.pm/docs/rebar3_publish
• https://github.com/tsloughter/rebar3_hex/blob/master/test/support/hex_api_model.erl

I'd much rather test against a real Hex repo though. I wonder if that's possible. Any pointer is helpful.

[essen]

Opened #910 with the initial work on this.

[zuiderkwast]

Register a Hex user, publish a test package using rebar and then update it using make, is that real enough? You can use Mix as well, to compare the results.

Otherwise, you could ask the hex maintainers of e.g. cowboy to add you as an owner. I guess they will approve. You can undo a publish within an hour or so, so it should be fine to test publishing. I have a package as well (mysql) but I almost don't remember how I did it. :-)

[essen]

I think it's possible to run https://github.com/hexpm/hexpm locally and test against that. That'll work out much better than doing stuff against real packages.

[essen]

I will soon drop the packages that are currently in Erlang.mk. I will move them to an external plugin that can be used for quick compatibility.

I will keep the package system, the only difference being that packages will be defined by external plugins. This will allow users to curate their own packages if necessary, without having to go through Erlang.mk itself. Decentralized for the win. Those external plugins will be defined as packages inside Erlang.mk itself for convenience, so you can use them in BUILD_DEPS and DEP_PLUGINS without having to define them.

At the same time I will create another external plugin that will contain my own list of curated packages. This will include the most recent Cowboy, Gun and other projects.

It is an extra step compared to today, but it should allow people to curate packages better than today.

[essen]

Plan is to do this sometimes next year.