rlibs retain reference to proc-macro dependencies - possibly unnecessary?

[adetaylor]

Summary

If a Rust .rlib project depends on a procedural macro crate, then the produced rlib will forever depend on the crate containing the proc macro, as shown by rustc -Z ls <path to rlib>. When creating downstream binaries, e.g. executables or staticlibs, rustc will require the location of the proc-macro to be provided as an -Ldependency argument.

I don't believe that the downstream rustc necessarily needs to know about the proc-macro. I have two bits of evidence:

• Sometimes the proc-macro is a different architecture from the binary (e.g. host vs target), so it obviously can't be used as part of the final binary.
• Using a linker directly it's possible to link an rlib into a functional executable without providing the location of the proc-macro on which it depends.

Why this might matter

Perhaps there's an opportunity for some dependency tree pruning here? The procedural macro would obviously be needed for initial compilation of the rlib, but may not be needed for downstream builds which depend on that rlib.

I can't think of a circumstance when this would allow more parallelism, but perhaps in some cases of pipelined compilation (rust-lang/cargo#6883) there might be some kind of saving?

And even if not, if these dependencies can be removed, it's presumably just fewer crates for rustc to search through and analyze so might marginally speed those downstream rustc invocations.

Unless of course the proc-macro is sometimes required by the linker steps, e.g. if LTO is enabled. In which case ignore this issue.

Test case

See https://github.com/adetaylor/repro-unexpected-macro-dependency.

This contains three ways of building the same code, in the clientc directory:

• build-ok-1.sh: this uses clang (or specifically lld) to link the rlibs directly into the final C executable. This is unsupported, but works. The linker command has no reference to the procedural macro, proving that it isn't necessary in the final linking step.
• build-ok-2.sh: this uses rustc to make all the Rust code into a staticlib then links that into the C executable. In this case, rustc needs to be told the location of the procedural macro.
• build-fails.sh: this also uses rustc to make all the Rust code into a staticlib then links that into the C executable. In this case, the rustc invocation isn't given the location of the macro, so building the staticlib fails.

In each case, we're building the final executable for ARM (Android), whilst the macro is built for the host OS (in my case x64). This of course proves that no part of the macro is actually linked into the final binary.

The dependency chain is thus:

[C executable] -> <staticlib_crate, omitted in build-ok-1.sh> -> [rlib_crate] -> [macro_crate]

Unfortunately, this means for the scripts to work, you'll need a cross-compiling clang toolchain. I used one from a handy copy of Chromium, but you can alter the scripts to point to an Android NDK or similar.

rustc 1.45.0-nightly (fa51f810e 2020-04-29)

[petrochenkov]

Proc macro crate is a dynamic library + metadata.

The dynamic library part is used for actually executing the macro during compilation of the rlib (via dlopen/dlsym assuming we are on Linux).
The metadata part is used for 1) reexporting macros from the rlib and 2) spans (including macro hygiene data).

As for reexports, the proc macro dependencies can be dropped if nothing was actually reexported. These optimization was recently removed in #69432.

As for spans, the proc macro dependencies can also be dropped if no spans from them are "reexported" through the rlib, but it's not a trivial optimization and it needs some design work (#69976 (comment)).

If proc macros were produced as two files (e.g. foo.so and foo.rmeta on Linux), then we would only need to keep foo.rmeta around.

[adetaylor]

Thanks for the reply and especially for the links to the existing PRs and issues from which I have learned a lot :)

@petrochenkov, I'm not sure it's especially useful for this issue to remain open, as it sounds like the extra dependencies here are a known issue and are hard to solve, so feel free to close this issue if you like.

[petrochenkov]

Yeah, closing as not immediately actionable.