Undefined reference to '__trampoline_setup' with the flang driver

[ssijaric-nv]

It seems like the default flang invocation on ARM doesn't link against libclang_rt.builtins, which contains the definition of __trampoline_setup. I also don't see '-z execstack' passed to the linker. Running 'flang -v t.f90' on the example program from issue #104798 results in:

FIRModule:(.text+0x30): undefined reference to `__trampoline_setup'

[llvmbot]

@llvm/issue-subscribers-flang-driver

Author: None (ssijaric-nv)

It seems like the default flang invocation on ARM doesn't link against libclang_rt.builtins, which contains the definition of __trampoline_setup. I also don't see '-z execstack' passed to the linker. Running 'flang -v t.f90' on the example program from issue #104798 results in:

FIRModule:(.text+0x30): undefined reference to `__trampoline_setup'

[pawosm-arm]

Indeed, default compilation fails at link time. To resolve the dependency on __trampoline_setup, you'd have to link against compiler-rt, e.g.:

$ flang -o t t.f90 -rtlib=compiler-rt

Of cousse, due to #104798 the compiled example will segfault. Yet what I've noticed is that it doesn't segfault when built with -O1 (still it needs explicit linking against compiler-rt):

$ flang -O1 -o t t.f90 -rtlib=compiler-rt

$ ./t
 Hello

$

What I also noticed is that the missing symbol and the segfault goes away when compiling with -O2:

$ flang -O2 -o t t.f90

$ ./t
 Hello

$

Keep in mind we're building compiler-rt as such:

    -DCOMPILER_RT_BUILD_BUILTINS=ON \
    -DCOMPILER_RT_USE_BUILTINS_LIBRARY=ON \
    -DCOMPILER_RT_EXCLUDE_ATOMIC_BUILTIN=OFF \
    -DCOMPILER_RT_BUILD_STANDALONE_LIBATOMIC=OFF \
    -DCOMPILER_RT_USE_ATOMIC_LIBRARY=ON

The default behavior for selecting rtlib can be specified when building LLVM with Flang with the following CMake setting:

      -DFLANG_DEFAULT_RTLIB=compiler-rt

Whether we will be using it as our default is something for us yet to decide. Whether this should be the official default in LLVM Flang is something we can discuss here.

[DavidTruby]

I was thinking about this when I hit this yesterday as well. I think at the very least we do need to give a sensible error if someone passes a nested function with a capture into a procedure pointer and compiler-rt isn't being used, so that the user knows what the fix is.
I'm not 100% sure what the implications would be for making compiler-rt the default in all cases would be.

[pawosm-arm]

Also, this would do if you don't wish to include entire compiler-rt: -Wl,--push-state -Wl,-as-needed -lclang_rt.builtins -Wl,--pop-state. Maybe we should make it a default in the frontend-driver when targeting AArch64?
NB, works with -static too, and there is only static libclang_rt.builtins.a shipped with our compiler, so shared/static applicability is not a problem here.

[DavidTruby]

Does it matter where on the link line that flag has to appear? I assume it has to be after the -lgcc_s flag that gets automatically added?

[pawosm-arm]

In the scenario above, the -lgcc flag (and -lgcc_eh/-lgcc_s depending on static/non-static builds) would be added at the end, so it's not perfect. But it can be re-arranged, these two invocations worked OK:

$ flang -static -O1 -o t t.f90 -Wl,--push-state -Wl,-as-needed -lgcc -lgcc_eh -lclang_rt.builtins -Wl,--pop-state

$ flang -O1 -o t t.f90 -Wl,--push-state -Wl,-as-needed -lgcc -lgcc_s -lclang_rt.builtins -Wl,--pop-state

I suppose this can be arranged in the front-end diver properly.

[DavidTruby]

This looks sensible to me I think. Is there any way we can verify that if we use that line we actually get the symbols from libgcc_s and not from clang-rt?

[pawosm-arm]

That's a tricky question, I don't recall any place where this could be tested in the LLVM tests.

[DavidTruby]

Even if we don't necessarily test for it; can you check locally whether the right function is being linked in? I'm not sure how I'd go about doing that.

[pawosm-arm]

Lemme do some experiment. As an example I'd pick a function present in both compiler-rt's builtins and libgcc, namely __aarch64_cas1_relax.

$ objdump --disassemble=__aarch64_cas1_relax aarch64-unknown-linux-gnu/libclang_rt.builtins.a
...
outline_atomic_cas1_1.S.o:     file format elf64-littleaarch64


Disassembly of section .text:

0000000000000000 <__aarch64_cas1_relax>:
   0:   90000010        adrp    x16, 0 <__aarch64_have_lse_atomics>
   4:   39400210        ldrb    w16, [x16]
   8:   34000070        cbz     w16, 14 <__aarch64_cas1_relax+0x14>
   c:   08a07c41        casb    w0, w1, [x2]
  10:   d65f03c0        ret
  14:   53001c10        uxtb    w16, w0
  18:   085f7c40        ldxrb   w0, [x2]
  1c:   6b10001f        cmp     w0, w16
  20:   54000061        b.ne    2c <__aarch64_cas1_relax+0x2c>  // b.any
  24:   08117c41        stxrb   w17, w1, [x2]
  28:   35ffff91        cbnz    w17, 18 <__aarch64_cas1_relax+0x18>
  2c:   d65f03c0        ret
...

$ objdump --disassemble=__aarch64_cas1_relax
In archive /lib/gcc/aarch64-linux-gnu/11/libgcc.a:

cas_1_1.o:     file format elf64-littleaarch64


Disassembly of section .text:

0000000000000000 <__aarch64_cas1_relax>:
   0:   d503245f        bti     c
   4:   90000010        adrp    x16, 0 <__aarch64_have_lse_atomics>
   8:   39400210        ldrb    w16, [x16]
   c:   34000070        cbz     w16, 18 <__aarch64_cas1_relax+0x18>
  10:   08a07c41        casb    w0, w1, [x2]
  14:   d65f03c0        ret
  18:   53001c10        uxtb    w16, w0
  1c:   085f7c40        ldxrb   w0, [x2]
  20:   6b10001f        cmp     w0, w16
  24:   54000061        b.ne    30 <__aarch64_cas1_relax+0x30>  // b.any
  28:   08117c41        stxrb   w17, w1, [x2]
  2c:   35ffff91        cbnz    w17, 1c <__aarch64_cas1_relax+0x1c>
  30:   d65f03c0        ret
...

Now, let's have example Fortran program, e.g. builtin.f90:

program builtin
  implicit none
  interface
    subroutine __aarch64_cas1_relax() bind(c)
      implicit none
    end subroutine
    subroutine __trampoline_setup() bind(c)
      implicit none
    end subroutine
  end interface

  call __aarch64_cas1_relax
  call __trampoline_setup
end program

Linking it would cause us trouble:

$ flang -o builtin builtin.f90
/usr/bin/ld: /tmp/builtin-91d5f4.o: in function `_QQmain':
FIRModule:(.text+0xc): undefined reference to `__trampoline_setup'
flang-21: error: linker command failed with exit code 1 (use -v to see invocation)

This can be resolved by using compiler-rt:

$ flang -o builtin builtin.f90 --rtlib=compiler-rt

$ nm builtin
0000000000000a80 t __aarch64_cas1_relax
...
0000000000000800 T _start
000000000000094c t __trampoline_setup

But the __aarch64_cas1_relax code comes from the compiler-rt library:

$ objdump --disassemble=__aarch64_cas1_relax builtin

builtin:     file format elf64-littleaarch64


Disassembly of section .init:

Disassembly of section .plt:

Disassembly of section .text:

0000000000000a80 <__aarch64_cas1_relax>:
 a80:   d0000090        adrp    x16, 12000 <__register_frame_info@Base>
 a84:   3942c210        ldrb    w16, [x16, #176]
 a88:   34000070        cbz     w16, a94 <__aarch64_cas1_relax+0x14>
 a8c:   08a07c41        casb    w0, w1, [x2]
 a90:   d65f03c0        ret
 a94:   53001c10        uxtb    w16, w0
 a98:   085f7c40        ldxrb   w0, [x2]
 a9c:   6b10001f        cmp     w0, w16
 aa0:   54000061        b.ne    aac <__aarch64_cas1_relax+0x2c>  // b.any
 aa4:   08117c41        stxrb   w17, w1, [x2]
 aa8:   35ffff91        cbnz    w17, a98 <__aarch64_cas1_relax+0x18>
 aac:   d65f03c0        ret

So let's force libgcc to be listed first for the linking:

$ flang -o builtin builtin.f90 -Wl,--push-state -Wl,-as-needed -lgcc -lgcc_s -lclang_rt.builtins -Wl,--pop-state

$ nm builtin
0000000000000950 t __aarch64_cas1_relax
...
0000000000000800 T _start
0000000000011048 D __TMC_END__
0000000000000984 t __trampoline_setup

Now the __aarch64_cas1_relax code should be as pulled in from libgcc:

$ objdump --disassemble=__aarch64_cas1_relax builtin

builtin:     file format elf64-littleaarch64


Disassembly of section .init:

Disassembly of section .plt:

Disassembly of section .text:

0000000000000950 <__aarch64_cas1_relax>:
 950:   d503245f        bti     c
 954:   b0000090        adrp    x16, 11000 <__gmon_start__@Base>
 958:   39412610        ldrb    w16, [x16, #73]
 95c:   34000070        cbz     w16, 968 <__aarch64_cas1_relax+0x18>
 960:   08a07c41        casb    w0, w1, [x2]
 964:   d65f03c0        ret
 968:   53001c10        uxtb    w16, w0
 96c:   085f7c40        ldxrb   w0, [x2]
 970:   6b10001f        cmp     w0, w16
 974:   54000061        b.ne    980 <__aarch64_cas1_relax+0x30>  // b.any
 978:   08117c41        stxrb   w17, w1, [x2]
 97c:   35ffff91        cbnz    w17, 96c <__aarch64_cas1_relax+0x1c>
 980:   d65f03c0        ret