Why is the LuaJIT vcpkg port so complicated?

[Chris Djali (AnyOldName3)]

The upstream build system is pretty straightforward - if using GCC or Clang, you run a makefile (potentially with some environment variables set), and if using MSVC, you run a batch file (again, potentially with some environment variables set). It figures out the host details based on some sensible defaults and the environment variables, and if the right environment variables are set to specify a different target, it figures out the target details based on sensible defaults and the environment variables. This seems to work for every platform that LuaJIT supports, and as an assembly-heavy project, it's not going to work anywhere that upstream doesn't support.

Without knowing the details as to why it doesn't, I'd have expected the vcpkg port to set the environment variables the upstream build system uses based on the triplet settings (and any general vcpkg settings, e.g. using /Z7 for Debug and Release with MSVC instead of /Zi for just Debug) and called either the Makefile or batch file from upstream once for Debug and once for Release.

Instead, the port jumps through an awful lot of hoops to set things up that the upstream build system would have dealt with itself before eventually calling into it anyway. This seems to lead to some problems, e.g. Windows Arm builds don't work, despite being supported upstream, and might be the root cause of other problems we've hit, but it makes investigating that much harder as it's a nuisance to try and find out exactly how the vcpkg port has been built differently to an upstream build.

The closest thing I can find to an answer is that maybe someone decided they had to have a configure script so they could call vcpkg_configure_make before vcpkg_build_make, like the docs suggest, even though upstream doesn't have a configure script as its build system is controlled by build-time environment variables. If so, that would be bureaucracy rather than a technical reason, and would suggest that a rule needs to change to fit the real world rather than making the port unnecessarily complicated.

[Sarah Sunday (ssunday)]

+1 Been having problems with the vcpkg luajit port for unknown reasons on macOS and debugging why has been very weedy. Haven't raised an issue as no root cause has been identified. Using a simpler custom port works (OpenMW/openmw-deps-build@ca07558 example) so it's something with the vcpkg luajit port specifically that's causing macOS run-time failures and...why is hard to figure out.

One salient example of how simply using vcpkg settings would make things simpler is the fact that this https://github.com/microsoft/vcpkg/blob/master/ports/luajit/003-do-not-set-macosx-deployment-target.patch could be replaced with just setting it to https://learn.microsoft.com/en-us/vcpkg/users/triplets#vcpkg_osx_deployment_target for example.

[Kai Pastor (dg0yt)]

You could take a look at https://github.com/microsoft/vcpkg/pulls?q=is%3Apr+luajit+is%3Aclosed+in%3Atitle.

The luajit port is complicated because its "pretty straightforward" build system is incompatible with vcpkg principle of "host binaries in the host triplet" and the resulting strict build order. Cf. #30608 (comment): The build needs a host tool to determine target properties to build the next host tool. In vcpkg, host artifacts cannot depend on target artifacts. Ignoring the vcpkg triplet separation isn't an option for ports in the curated registry.

#30608 is also the PR labeled "Use common build functions". The reason is not a bureaucracy exercise, but the fact that these functions are the primary, carefully maintained facility to determine tools, flags, and environment variables based on detected properties of the CMake toolchain. Most of the properties are passed to configure, not build, so the port got a trivial script to capture the configuration in a Makefile which takes care of the actual build time enviroment variables. This is a significant simplification with regard to portability and build types.

arm64-windows might simply work in a native build after a port update. However vcpkg CI doesn't offer native arm64 windows builds, so the success would need to be verified outside vcpkg CI. It might also be possible to add polyfill to help with arm64-on-x64 builds, such as preconfigured host/buildvm_arch.h.

VCPKG_OSX_DEPLOYMENT_TARGET is a triplet variable, and it is probably good to use it. However, it is not set in the official osx triplets here, so it cannot be used directly. It would take effect via the mentioned detection of CMake toolchain properties.

Good reporting of problems is an indication of interest in improvments. (Admittedly, I check issues and discussions only occassionally. To many bad reports for my free time.)

[Kai Pastor (dg0yt)]

I guess this can be selected as the "answer" in this discussion ("why?").

No new issues reported since #49565.

Many thanks to Andrew Tribick (@ajtribick) who regularly updates the port now.

[Chris Djali (AnyOldName3)]

I have looked at all of that before posting this, and I'm unconvinced that what you're saying matches reality. The current port looks more like theatre to give the impression of the goals you're describing being achieved rather than something that achieves those goals because the goals don't seem to have been achieved.

None of the cross-compilation attempts we've made have worked, even trivial ones like building x64-windows-static from x64-windows, so that makes it seem like the complexity that supports vcpkg's rules about cross-compilation is only disguising the fact that the port doesn't support cross-compilation, whereas forgoing that complexity and just emitting an error would at least be less misleading, without dropping support for anything that currently works.

As far as we can tell, some of the native triplets checked in CI do not produce a build that actually runs. Initially, we thought we were somehow using something incorrectly, but we've eliminated every possibility other than that the Mac build wouldn't work for anyone and hasn't been tested. Before #30608, the blame would have clearly been on the upstream build system, and we know that works, so it seems from our end like #30608 broke Macs.

The port is more fragile than the upstream build system so updating the port to support new things that work upstream isn't as simple as just changing the commit it refers to. It's calling into the upstream makefile, but with variables pre-set in undocumented and unsupported ways, so there's no reason why Mike Pall would ensure that they're still interpreted the same way and used for the same things. This makes it much harder for anyone who has problems with the vcpkg port to try and fix it themselves, or even do basic troubleshooting like working out if they're hitting an upstream problem that upstream LuaJIT has fixed, or something specific to how vcpkg is packaging it.

This kind of thing means that users of the vcpkg port (or people who want to become users of the port) don't have the tools to create good issue reports. There's not really a middle ground between I tried using the vcpkg port and now my application crashes but I don't know why and I have investigated the LuaJIT vcpkg port so thoroughly that I think I'm qualified to take over maintenance, and the journey from one to the other involves making stressed-out discussions posts.

I don't like being on the receiving end of this doesn't work and I don't know why reports, so I don't like submitting them. Otherwise, you'd probably have had several by now.

The reason is not a bureaucracy exercise, but the fact that these functions are the primary, carefully maintained facility to determine tools, flags, and environment variables based on detected properties of the CMake toolchain. Most of the properties are passed to configure, not build, so the port got a trivial script to capture the configuration in a Makefile which takes care of the actual build time enviroment variables.

What you're describing is an exercise in bureaucracy. Not every project has a build system with a distinct build and configure step, and vcpkg forcing that convention upon them is a decision made by a human rather than a fundamental technical limitation with how computers work.

Fundamentally, it seems to me like the only viable route to a maintainable LuaJIT port is to campaign for the vcpkg rules to be changed, or at least for an exception to be carved out for this package. I can't see how the current rules are doing this particular package more good than harm, and rules that mainly do harm are bad and should be abolished.

[Kai Pastor (dg0yt)]

Fundamentally, it seems to me like the only viable route to a maintainable LuaJIT port is to campaign for the vcpkg rules to be changed

I think this ship has sailed. Even when it blocked qt5 cross builds.

---

None of the cross-compilation attempts we've made have worked, even trivial ones like building x64-windows-static from x64-windows

So the build for this combination failed? It would love to see it fail in vcpkg CI then.

some of the native triplets checked in CI do not produce a build that actually runs.

So it fails at runtime? Which triplet? I often add test ports nowadays, and sometimes this includes running executables so that CI can catch even more problems.

This kind of thing means that users of the vcpkg port (or people who want to become users of the port) don't have the tools to create good issue reports.

What I mean by "good issue report" is to at least add the logs for failed builds when the console output tells you so, or to correctly report the affected versions. This is available to any developer. This is where an analysis and interaction can start.

I don't like being on the receiving end of this doesn't work and I don't know why reports, so I don't like submitting them. Otherwise, you'd probably have had several by now.

There aren't many reports for luajit, and most seem to have been resolved: https://github.com/microsoft/vcpkg/issues?q=is%3Aissue%20state%3Aopen%20luajit%20in%3Atitle

The lack of arm64-windows support is a standing issue.

The lack of updating is an implicit issue. I understand that this is related to the topic of this discussion. FTR there were a number of attempts to update the port which where closed quickly. OTOH a PR is a context even more inviting for support if given time for other voluntary contributors to join the effort. Nobody can be expected to always understand all the platforms and configuration which might break in vcpkg CI.

---

What you're describing is an exercise in bureaucracy. Not every project has a build system with a distinct build and configure step, and vcpkg forcing that convention upon them is a decision made by a human rather than a fundamental technical limitation with how computers work.

"Exercise in bureaucracy", "decision made by a human", "fundamental technical limiation" - I'm unable to draw a good conclusion from this paragraph.

There is not much choice when there are fundamental technical limitations, but there is a lot of room for trade-offs in absence of limitations.

For now, the triplet separation is a fundamental technical limitation in vcpkg. We are not forced to apply the configure/build pattern - it is a pragmatic choice. If you think this should be replaced, it would be good to show a convincing improvement.

---

Creating a trivial test port - a C or CXX CMake project using lua - would be a good start, in particular if the test port fails due to the claimed problems of the current approach. That test port would also identify/prevent regressions in possible improvements and future PRs.

[Chris Djali (AnyOldName3)]

So the build for this combination failed? It would love to see it fail in vcpkg CI then.

It was a couple of years ago, so I don't remember the details, but I think this one was something along the lines of putting both the host and target libs into the target triplet's directory, so not anything major, but enough to mean a project consuming it might end up needing an extra DLL or something. This wasn't a priority for us, so we didn't write anything down.

There was another thing that I was remembering as a problem with the LuaJIT port, but going through the chat logs it turned out to possibly be a red herring as none of the symptoms went away when making the host triplet match the target. It seems that none of the cross-compilation attempts we've made have worked was misleading as some of them were being killed by the general MacOS doesn't work problem we're hitting.

So it fails at runtime? Which triplet? I often add test ports nowadays, and sometimes this includes running executables so that CI can catch even more problems.

Yep. It crashes nearly immediately with both the default Arm MacOS one and a mildly-custom one with VCPKG_LIBRARY_LINKAGE=dynamic. I think this is probably reproducible with the standalone LuaJIT shell that gets installed as a tool, but we don't know how.

As this is for an unfunded open-source project, I've been trying to investigate this without access to a real Mac, so all the testing has been asking someone else to do tests, so the results I've got aren't especially organised.

What I mean by "good issue report" is to at least add the logs for failed builds when the console output tells you so, or to correctly report the affected versions. This is available to any developer. This is where an analysis and interaction can start.

The build itself didn't fail, all we've got is

* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x14)
    frame #0: 0x0000000103af308c libluajit-5.1.2.1.0.dylib`___lldb_unnamed_symbol395 + 52
libluajit-5.1.2.1.0.dylib`___lldb_unnamed_symbol395:
->  0x103af308c <+52>: ldr    w10, [x28, #0xc]
    0x103af3090 <+56>: ldr    x2, [x17, #0x28]
    0x103af3094 <+60>: and    w9, w9, w10
    0x103af3098 <+64>: add    x9, x9, x9, lsl #1

from trying to run an application. I guess the return pointer's been messed up, because we don't even have a stack trace, let alone one with named symbols. I don't know what function we called or even whether it's ended up where it is because a Lua script did a function call or the application called something in LuaJIT's C API. We just know that it works fine if we build the same upstream LuaJIT commit as vcpkg does with the upstream build system and the compile and link flags end up (effectively) the same (except the different include and lib directories), so it has to be because of something the vcpkg port is doing differently.

[Sarah Sunday (ssunday)]

I raised a low-information issue of the problem: #49556

[Kai Pastor (dg0yt)]

FYI #49565 introduces a simple test port.
Unfortunately arm64-osx is currently not usable in vcpkg CI, and I don't have other access to this hardware.

[Sarah Sunday (ssunday)]

Thank you! I can perform tests if needed on arm64-osx.

Currently not usable—is that because it's using azure and not github actions and osx isn't available?

[Kai Pastor (dg0yt)]

There is some problem with the AZP osx image used by vcpkg CI.

[Kai Pastor (dg0yt)]

The PR now fixes the installation for Windows. The test passes for all x64 triplets. (It also passed locally for x64-linux-dynamic.)

To test locally, e.g. for osx, checkout that branch and do:

vcpkg install --overlay-ports scripts/test_ports 'vcpkg-ci-luajit[*]'

And as with every port, there are log files in buildtrees/<port>.

[Sarah Sunday (ssunday)]

Commented on your PR

[Chris Djali (AnyOldName3)]

I've got some build flags that are definitely different that might be causing the problems when using the same LuaJIT commit. If I pick a file, I'm seeing the upstream build system do:

gcc  -O2 -fomit-frame-pointer -Wall   -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -U_FORTIFY_SOURCE  -DLUA_MULTILIB=\"lib\" -fno-stack-protector -DLUAJIT_UNWIND_EXTERNAL   -c -o lj_assert.o lj_assert.c

whereas the vcpkg port does

/usr/bin/cc  -fPIC -g -DNDEBUG -fPIC -O3  -O2 -fomit-frame-pointer -Wall  -DNDEBUG -fPIC -O3 -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -U_FORTIFY_SOURCE  -DLUA_ROOT=\"/usr/local/share/vcpkg/installed/arm64-osx\" -DLUA_MULTILIB=\"lib\" -fno-stack-protector -DLUAJIT_UNWIND_EXTERNAL   -c -o lj_assert.o lj_assert.c

so obviously:

• GCC vs Clang. Obviously they're different compilers, but they're both supported upstream
• -fPIC. Upstream only uses this for dynamic builds, not static ones, and even then, only passes it when linking rather than when compiling.
• -g. Shouldn't affect the code, just whether there are symbols.
• -DNDEBUG. I guess maybe it might care about this.
• -O2 vs -O3. Upstream has a comment saying -O3 is slower with GCC 4, but otherwise doesn't specify any reason why it wouldn't work.
• -DLUA_ROOT=\"/usr/local/share/vcpkg/installed/arm64-osx\". Might cause portability problems, but I'd expect a printed error rather than an attempt to jump to 0x14.
• Arguments specified more than once. Shouldn't break anything, but isn't especially tidy either.

There are possibly other files with more differences, but they don't stand out in my diff compared to the files that get built in a different order or skipped altogether.

[Kai Pastor (dg0yt)]

• GCC vs Clang. Obviously they're different compilers, but they're both supported upstream

And AppleClang isn't pristine Clang.
In any case, the port must use the tools chosen by triplet and vcpkg toolchain.
(OT: For this reason some ports cannot be built on Windows because some package insist on clang for Windows.)

• -fPIC. Upstream only uses this for dynamic builds, not static ones, and even then, only passes it when linking rather than when compiling.

Enforced by vcpkg toolchain. There are always people putting static libs into their dynamic libs.
To take effect, it is needed during compilation. (OT: Using pic for compilation is sometimes suggested by the gcc linker when the code comes from assembler, and then people are confused.)

• -O2 vs -O3. Upstream has a comment saying -O3 is slower with GCC 4, but otherwise doesn't specify any reason why it wouldn't work.

And GCC 4 is not something I care much about in 2026.
(Such things can be adjusted in triplets per port.)

• -DLUA_ROOT=\"/usr/local/share/vcpkg/installed/arm64-osx\". Might cause portability problems, but I'd expect a printed error rather than an attempt to jump to 0x14.

This is just standard passing of strings via command line -D.
The real problem is that it is part of burning in "staging directories" which may not exist when a cached vcpkg artifact is restored in a different location, or after deploying your app to the user. This is something to discuss after tests are green in CI. (Hint: your app may need to set environment variables.)

[Chris Djali (AnyOldName3)]

I was specifically meaning portable between different machines as it's a hardcoded path not every machine would have (or even worse, something that some machines might have and which would force different .lua files to get loaded at runtime). It's not going to stop our application from accessing the files as we already add the right paths (and possibly even remove what's already there to ensure sandboxing).

I realised that I haven't compared the commands used during the make install phase or the generated luaconf.h, and in particular, the latter contains definitions that affect the ABI, so it's possible that the port's messing up how it's generated.

[Kai Pastor (dg0yt)]

I was specifically meaning portable between different machines as it's a hardcoded path not every machine would have

So it is the same thing I talked about. But this is a problem of luajit itself. The port just prevents a bad guess (and enables cross builds). Only on Windows, luajit is able to consider the path of the executable (!).

[Timofei Zhakov (rinrab)]

I'm currently working on a project where I also have a lot of heavy assembly file and my buildsytem of choice was plain make. It wins CMake in such scenarios, where portability and low-level control of build flow is important. Even though I think cmake is cool, it's just not applicable sometimes. Somehow I found this discussion and feel like it could be related.

I'm also trying to make a port for vcpkg. I'd say it's gets complicated to get something non-cmake-ish working with it. There is so much things to be worked around.

I think there should be some kind of official way to porting/supporting such kind of libraries.

[Kai Pastor (dg0yt)]

I'd say it's gets complicated to get something non-cmake-ish working with it.

It gets complicated to get something non-standardish working, across different platforms and with debug+release.

I think there should be some kind of official way to porting/supporting such kind of libraries.

make is fairly standard, but Makefiles are not. And then there is vcpkg with its range of platforms, debug+release installation, static library linkage as a CI requirement, and host triplet separation. There is a point where a little bit of configure is just the right glue between an "official way" and the individual package.