NFG: New FFI Generator

[snunez1]

@rpav mentioned in a post, and in the README, about the preprocessor handling being a huge hack, and a rewrite of the system may be in order. The timing may be a happy coincidence, as I was participating in another CFFI discussion with the CCL folks, who are embarking on their own version of a CFFI generator.

In that discussion, I suggested using c2ffi for theirs, based on resource constraints. CCL is only barely surviving on life support, and it didn't seem to me a wise use of scarce resources to reinvent the wheel, but you're comments have got me reconsidering the fundamental basis of that assumption.

In that discussion, the main reason for developing a new ffi generator were:

https://clang.llvm.org/docs/Tooling.html talks about the various interfaces one might use when looking for syntactic and semantic information about a program. They basically say, "use libclang unless there's some reason that you can't".
...
By contrast, because libclang's interface is stable, the version of the installed llvm doesn't matter much.

As well as a lack of ObjectiveC support. The basic architecture of c2ffi seems fine to me, but there is a lot hassle with clang/llvm to make it work.

Given your experience over the last few years developing and maintaining c2ffi, I wonder if you'd be able to comment? If you were going to rewrite c2ffi, what would you do differently this time around?

P.S. If you have the ability to move this into a discussion thread rather than an issue, it probably is better discussed there.

[rpav]

cpp

So the preprocessor thing will always be a huge hack, because the preprocessor is a huge hack: you've got arbitrary textual substitutions you need to somehow make sense of. There might be better heuristics about what to pick, but there haven't been many issues of "this doesn't capture a preprocessor constant we need." It works, and I'm not sure much better is possible.

(For instance, consider there can be #define FOO_SIZE (2*sizeof(Foo)), or #define HAS_FOO __weird_intrinsic_magic; simple parsing will never resolve these, but c2ffi will, and they're often the definitions you really want.)

libclang

Libclang never appeared to give sufficient information about the source; it's possible I either misinterpreted the docs or that's changed.

But generally, it's less than an hour to maintain per version. So why does it take me forever to update sometimes? See below.

ObjC etc

ObjectiveC support is limited because I have no reference; I don't know ObjC and I have nothing to test/verify/use it with. C++ support is limited and could be better, but this is mostly some additional info (I think public/private/protected still isn't there). I would certainly welcome patches for ObjC.

rewrite

When I think "rewriting," I don't mean tossing the source and starting over: there doesn't seem to be much reason, all the parts are working and in order, and do what would need done anyway.

I mean going back through all the source and making it cleaner, more portable C++. For instance, nlohmann/json, cxxopt instead of getopt, and generally less C-like and more modern C++ code. This makes working with it nicer when it comes to maintenance, and would probably make building with MSVC possible (as long as the clang libraries are available).

Of course there are always other features that would be nice, but these are mostly independent modular changes:

• Better json code, making it easier to extend; this is the worst at a glance and the least fun to work with. C doesn't change much but this might help C++/ObjC.
• Built-in filtering / treeshaking.

below

But here are the real actual issues:

• New versions of llvm/clang are annoyingly inconsistent to obtain: Which distro has them and when varies wildly, and they are often not what I'm using at any given time.
• Distro-built versions aren't always consistent/working, though this has been less of an issue recently.
• My personal dev setup hasn't been the greatest the past few years (though this will be solved after next week >:), so among other things building clang has been out of the question.
• Few of the users of c2ffi seem to be C++ devs and therefore contribs tend to be low. (And therefore doubly-thanks here to all those who have!)

For instance, right now, I couldn't work on the 18.x branch on Linux, as there doesn't appear to be any distro with packages for it. (Changes were minimal, so 17->18 porting would be easy enough, but it's another place the problem appears.)

What really needs done:

• Alpine + docker/podman, and/or AppImage: Alpine linux seems to have the most consistently up-to-date clang/llvm
• Testing/CI/distribution based around this

This basically solves the "wait until Arch has it" problem. It does require a bit of infrastructure building, but this is mostly editor/container configuration, but should minimize the biggest issue today, i.e. "when is c2ffi going to get updated."