ERR Failed to fully unwind, and some missing symbols in backtrace

[velvia]

Thank you for writing an awesome profiler tool.

I got the above error message on a Rust 1.62-compiled binary. Bytehound is installed using:

RUN wget --no-check-certificate https://github.com/koute/bytehound/releases/download/0.8.0/bytehound-x86_64-unknown-linux-gnu.tgz

0008 0228 ERR Failed to fully unwind; some of the stack traces will be incomplete. (Are you using a JIT compiler?)

Note that I have the following settings set:

  export MEMORY_PROFILER_LOG=warn
  export LD_PRELOAD=./libbytehound.so
  # See https://koute.github.io/bytehound/configuration.html
  export MEMORY_PROFILER_OUTPUT='/data/validator-bytehound-profile_%t_%p.dat'
  # Don't record "temporary" allocations - helps to reduce profile data
  export MEMORY_PROFILER_CULL_TEMPORARY_ALLOCATIONS=1
  export MEMORY_PROFILER_LOGFILE='/data/bytehound.log'

However, libbytehound is able to keep writing out the profile data.
I use LD_PRELOAD to run a release version of my program with no debug symbols. This is intentional, to keep the release image size down.
I then separately build a version with debug symbols (which swells to over 1.5GB for the final image !!) and use the -d option when starting byte hound to load it. (I have to run strip first often to reduce it in size)

The second problem is that when I load the byte hound, I find that stack traces are missing elements from my program itself, although it seems to have traces from my dependencies (for example, Tokio, Hyper/H2, RocksDB, etc. all show up).

Any pointers? I might be able to send over pointers to Docker images etc.

Many thanks for your help.

[velvia]

I have discovered that byte hound only seems to display full stack traces when I run the profiler with the program compiled with full debug symbols. This is a bit disappointing as I was hoping to load debug symbols later using -d, but that does not seem to work. I can confirm this as when I run with MEMORY_PROFILER_WRITE_BINARIES_TO_OUTPUT set to 0, and then run byte hound -d later to load debug symbols, I see nothing but hex addresses in the stack trace.

[koute]

The error you're seeing here means that it wasn't able to fully unwind the stack traces (so they'll be incomplete), but has nothing to do with whether the symbols are going to be decoded.

It's not really possible to tell in general what triggers it; normally it shouldn't happen. It could happen if you're compiling with panic=abort, you're using a libc which is not glibc, or if you're using some weird docker image where some of the system libraries were horribly mutilated to get their size down. None of these configurations are supported.

Loading the debug symbols separately should work to get the symbols decoded; if they don't it's either a bug, or perhaps the .note.gnu.build-id doesn't match for some reason.

If you insist on stripping your binary I'd suggest using strip --strip-debug which should strip most of the debug info but still leave the symbol table necessary to decode the symbols.

[velvia]

@koute we are using this for our image: FROM debian:bullseye-slim AS runtime .... I think it's fairly standard so I don't think "horribly mutilated system libraries" is true.....

[velvia]

I don't think we have panic=abort because we do see stack traces and almost all of them have std::panic::catch_unwind() in them (is that part of the profiling logic?)