Do not build compiler from source in postinstall

[cknitt]

Introduction

This is a follow-up to this discussion and poll on the forum: https://forum.rescript-lang.org/t/platform-support-and-prebuilt-binaries-postinstall-scripts/3443.

Currently, on postinstall of the rescript npm package, the install script checks if a prebuilt binary is available. If not, it will try to build from scratch:

• Ninja (from vendored sources)
• OCaml (from vendored sources)
• The ReScript compiler binaries

IMHO building from source in postinstall for “exotic” platforms (for which no prebuilt binaries are available at the moment) has the following disadvantages:

1. Installation of the rescript package takes a long time.
2. We can not be sure if it actually works. I do not know if anyone is regularly testing installations for e.g. FreeBSD or OpenBSD. In my experience, if something is not tested by CI, there is a high likelihood that it has already stopped working some time ago.
3. Even if still working in principle and even though only (I think) a C compiler, make and python are required as prerequisites for the postinstall script, there is a good chance that the build/installation may fail on some machines because of a local setup issue.
4. The install scripts bring a lot of complexity and restrictions to the project setup and prevent us from moving to a dune-only build setup.

My proposal

• Provide prebuilt binaries for Linux ARM. Linux ARM was the only additional platform that people expressed interest in in the forum poll (except for NixOS which should build from source anyway according to its philosophy).
• This would extend the list of platforms that we provide prebuilt binaries for to:
  • macOS x64
  • macOS ARM
  • Linux x64
  • Linux ARM
  • Windows x64
• For any platform not in this list, users are required to "bring their own binaries" (either built from source by themselves or provided by some third party). Then they would install ReScript e.g. as follows:

export RESCRIPT_PLATFORM_BIN_DIR="<path to my exotic rescript binaries>"
npm install rescript

• The postinstall script of the rescript npm package will then take the binaries from this directory after verifying that the versions match.

[Minnozz]

I use ReScript on NixOS but via yarn, so the ReScript package is not managed by NixOS. I currently use a custom yarn hook to patch the ReScript binary during yarn install, which runs patchElf to make it work on my system. I believe this would not be necessary if the shipped binary was statically linked.

Would it be possible to ship statically linked binaries?

[cknitt]

It may be possible, see the instructions in https://ocamlpro.com/blog/2021_09_02_generating_static_and_portable_executables_with_ocaml.

However,

• Someone needs to put in the time and effort to get this to work and integrated into the CI process. This will be quite a lot of work, and personally I would prefer to focus on other tasks.
• Following the above instructions, the binary would be linked against musl instead of glibc which might lead to subtle bugs / differences in behavior that we are not aware of now.

Would you be willing to invest the time to investigate this further?

[ryyppy]

We'd like to push this further because it would help us greatly cleaning up the repository from unnecessary build system complexity -> this would allow us to migrate the rest of the compiler codebase to dune.

Actionable items:

• Organize ARM based Linux server (running costs can be covered via the ReScript Association -> may ask hosting providers for sponsorship) -> What hosting service should we consider?
• Set up CI -> should be straight forward via GitHub actions (\cc @Ostera could you help us out with that?)
• Remove postinstall logic that will try to locally build the compiler -> in case where the binary test fails on install, instruct the user that this platform is not officially supported and that they have to locally build the compiler themselves (or report an issue if they think their platform is worth supporting)

[Minnozz]

@cknitt I'm afraid I don't have the expertise for making the binaries statically linked. I will probably make an attempt after the other build changes have been done, but I can't promise anything.

[cknitt]

@Minnozz I did some experiments based on my PR #5820 and the results looked promising.
I think I will be able to add support for statically linked binaries for 11.0 based on this. 🙂

[Minnozz]

@cknitt Awesome! Let me know if you want me to test something on NixOS.

[cknitt]

@Minnozz Just got CI to build an npm package with statically linked Linux binaries. It is based on the latest master.
If you'd like to try it, you can get it here: https://github.com/rescript-lang/rescript-compiler/actions/runs/3667800316

[Minnozz]

@cknitt The static binaries run without any patching on NixOS :)

[cknitt]

@Minnozz Great! 🎉 Thanks for testing!