Finding the right symbol fetcher

[slymas]

Hey,

Longer story incoming for context. TLDR: How to configure cpptrace for my project?

I have a C++ project built on Qt5, deployed to Linux and Windows using the same code-base. I built a quite different stack trace solution on the 2 platforms, because I was using boost/stacktrace on Linux for simplicity, and (since that's useless on Windows) I had a manually built StackWalk64+addr2line solution, which was okay but super ugly (as I am not well versed in this topic).

I recently found cpptrace, and I wanted to give it a go in unifying my stack tracing.

On Linux, it was a super simple cmake build and it worked perfectly with default settings.

However, on Windows I am having mixed results. Qt has its own project setup using "qmake": a twisted, cmake-like beast. In order to build depedencies, I am using MSYS2 (mingw64), which worked good so far. The root of my question comes from the fact that the default dwarf+dbghelp doesn't give me any symbols. I am building cpptrace with cmake -G "MinGW Makefiles" .. -DCMAKE_BUILD_TYPE=RelWithDebInfo -DCPPTRACE_BUILD_SHARED=On && mingw32-make -j, then linking in zlib1 from MSYS and dynamically linking cpptrace and its deps into my project. It compiles and runs but gives me stack lines like:

#6  0x7ffe7ead0ebd0000 in KiUserExceptionDispatcher at C:\WINDOWS\SYSTEM32\ntdll.dll
#7  0x140231e4d0000000 at C:\Path\To\my.exe

If I switch to addr2line (that my own home-brew was also using) by adding -DCPPTRACE_GET_SYMBOLS_WITH_ADDR2LINE=On, I get real symbols from my binary. Strangely, my homebrew was also fetching symbols from outside my binary (like Qt's own DLLs), while cpptrace doesn't get those symbols.

#6  0x7ffe7ead0ebd0000 in ?? at C:\WINDOWS\SYSTEM32\ntdll.dll:0
#7  0x140231e4d0000000 in letsCrash(int) at C:\Path\To/../another/path/to/source.cpp:87

The results are the same whether I configure Qt to put debug symbols into a separate file, or inside the main binary.

Can I configure cpptrace any better, or is this the limit for my project? Thanks in advance for any help, much appreciated! :)

[jeremy-rifkin]

Hi, thanks for your interest in the library! Unfortunately this may be an issue with cpptrace itself as opposed to a configuration problem. The ideal configuration for mingw is libdwarf+dbghelp, and if that isn't producing symbols that's concerning. One tricky thing here is that the mingw ecosystem is pretty fragmented so while I've tested locally and things work for me it might be broken in another mingw ecosystem. I'm also not familiar with QT or qmake. Would it be possible to send the full cmake output from when you built cpptrace?

[slymas]

Sure, I am happy to help if this can uncover an issue with cpptrace itself.

Just for the record, this is my MINGW64 and gcc versions in MINGW are:

$ uname -r
3.4.6.x86_64
$ g++ --version
g++.exe (Rev11, Built by MSYS2 project) 12.2.0

A default cmake call gives me:

$ cmake -G "MinGW Makefiles" .. -DCMAKE_BUILD_TYPE=RelWithDebInfo -DCPPTRACE_BUILD_SHARED=On
-- The C compiler identification is GNU 12.2.0
-- The CXX compiler identification is GNU 12.2.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: C:/Devenv/msys64/mingw64/bin/cc.exe - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: C:/Devenv/msys64/mingw64/bin/c++.exe - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Performing Test HAS_CXXABI
-- Performing Test HAS_CXXABI - Success
-- Performing Test HAS_STACKWALK
-- Performing Test HAS_STACKWALK - Success
-- Performing Test HAS_CXX_EXCEPTION_TYPE
-- Performing Test HAS_CXX_EXCEPTION_TYPE - Success
-- Cpptrace auto config: Using dbghelp for unwinding
-- Cpptrace auto config: Using libdwarf + dbghelp for symbols
-- Cpptrace auto config: Using cxxabi for demangling
-- The ASM compiler identification is GNU
-- Found assembler: C:/Devenv/msys64/mingw64/bin/cc.exe
-- ZSTD VERSION: 1.5.5
-- Performing Test C_FLAG_WALL
-- Performing Test C_FLAG_WALL - Success
-- Performing Test CXX_FLAG_WALL
-- Performing Test CXX_FLAG_WALL - Success
-- Performing Test C_FLAG_WEXTRA
-- Performing Test C_FLAG_WEXTRA - Success
-- Performing Test CXX_FLAG_WEXTRA
-- Performing Test CXX_FLAG_WEXTRA - Success
-- Performing Test C_FLAG_WUNDEF
-- Performing Test C_FLAG_WUNDEF - Success
-- Performing Test CXX_FLAG_WUNDEF
-- Performing Test CXX_FLAG_WUNDEF - Success
-- Performing Test C_FLAG_WSHADOW
-- Performing Test C_FLAG_WSHADOW - Success
-- Performing Test CXX_FLAG_WSHADOW
-- Performing Test CXX_FLAG_WSHADOW - Success
-- Performing Test C_FLAG_WCAST_ALIGN
-- Performing Test C_FLAG_WCAST_ALIGN - Success
-- Performing Test CXX_FLAG_WCAST_ALIGN
-- Performing Test CXX_FLAG_WCAST_ALIGN - Success
-- Performing Test C_FLAG_WCAST_QUAL
-- Performing Test C_FLAG_WCAST_QUAL - Success
-- Performing Test CXX_FLAG_WCAST_QUAL
-- Performing Test CXX_FLAG_WCAST_QUAL - Success
-- Performing Test C_FLAG_WSTRICT_PROTOTYPES
-- Performing Test C_FLAG_WSTRICT_PROTOTYPES - Success
-- Performing Test LD_FLAG_Z_NOEXECSTACK
-- Performing Test LD_FLAG_Z_NOEXECSTACK - Failed
-- Performing Test C_FLAG_QUNUSED_ARGUMENTS
-- Performing Test C_FLAG_QUNUSED_ARGUMENTS - Failed
-- Performing Test CXX_FLAG_QUNUSED_ARGUMENTS
-- Performing Test CXX_FLAG_QUNUSED_ARGUMENTS - Failed
-- Performing Test C_FLAG_WA_NOEXECSTACK
-- Performing Test C_FLAG_WA_NOEXECSTACK - Failed
-- Performing Test CXX_FLAG_WA_NOEXECSTACK
-- Performing Test CXX_FLAG_WA_NOEXECSTACK - Failed
-- CMAKE_INSTALL_PREFIX: C:/Program Files (x86)/cpptrace
-- CMAKE_INSTALL_LIBDIR: lib
-- ZSTD_LEGACY_SUPPORT not defined!
-- ZSTD_MULTITHREAD_SUPPORT is enabled
-- Building dwarfgen    ... OFF
-- Building dwarfexample... OFF
-- Building api tests   ...
-- Looking for sys/types.h
-- Looking for sys/types.h - found
-- Looking for sys/stat.h
-- Looking for sys/stat.h - found
-- Looking for stdint.h
-- Looking for stdint.h - found
-- Looking for unistd.h
-- Looking for unistd.h - found
-- Looking for stdafx.h
-- Looking for stdafx.h - not found
-- Looking for fcntl.h
-- Looking for fcntl.h - found
-- Performing Test HAVE_UINTPTR_T
-- Performing Test HAVE_UINTPTR_T - Success
-- Performing Test HAVE_INTPTR_T
-- Performing Test HAVE_INTPTR_T - Success
-- HAVE_UINTPTR_T 1: uintptr_t defined in stdint.h... YES
-- uintptr_t value considered NO
-- HAVE_INTPTR_T 1: intptr_t defined in stdint.h... YES
-- intptr_t value considered NO
-- Found ZLIB: C:/Devenv/msys64/mingw64/lib/libz.dll.a (found version "1.2.13")
-- CMAKE_SIZEOF_VOID_P ... 8
-- Compiler warning options... NO
-- Install prefix ... C:/Program Files (x86)/cpptrace
-- Configuring done (22.6s)
-- Generating done (0.1s)
-- Build files have been written to: C:/Path/to/cpptrace/build

Note that in my setup, I am doing a version of the "Local User Installation" process, where I don't do make install in the end, instead I link the cpptrace libs into my project. I didn't need the "Platforms logistics" for Windows either, because I put the cpptrace lib into my binary's path.

Here's a snippet from my qmake project. LIBS is qmake's way of setting up library paths and names for the linker and the binary, the unix and win32 blocks are the platform selectors, and the $$VAR are variables in the project:

LIBS += \
    -L$$CPPTRACE_BUILD_DIR \
    -L$$CPPTRACE_BUILD_DIR/_deps/libdwarf-build/src/lib/libdwarf \
    -L$$CPPTRACE_BUILD_DIR/_deps/zstd-build/lib

# WARNING: lib order matters!
unix {
    LIBS += \
        -lcpptrace -ldwarf -lz -lzstd -ldl
}

win32 {
# The msys64 folder is needed to provide zlib1.dll to cpptrace.
# Add `-ldwarf -lz -lzstd` if using default cpptrace config.
    LIBS += \
        -L$$_PRO_FILE_PWD_/../../msys64/mingw64/bin \
        -lcpptrace -ldbghelp
}

And one more additional info package, if I build only DBGHELP by adding -DCPPTRACE_GET_SYMBOLS_WITH_DBGHELP=On, then the symbols are empty:

#8 0x140231e620000000

And if I build only DWARF by adding -DCPPTRACE_GET_SYMBOLS_WITH_LIBDWARF=On, then the symbols are the same as with both DWARF and DBGHELP, namely just showing the exe name:

#8 0x140231e620000000 at C:\Path\To\my.exe

Hope this helps.

[jeremy-rifkin]

Thanks for the additional information. Mainly thinking out loud:

• This looks like msys2 mingw64 gcc 12.2 which is what I've used locally too
• libdwarf + dbghelp for symbols is right
• Zlib and zstd are both seemingly found, so it shouldn't be an issue with libdwarf not being able to decompress
• I think the qmake script looks fine
• The result from -DCPPTRACE_GET_SYMBOLS_WITH_DBGHELP=On makes sense to me, since presumably that symbol is in dwarf which dbghelp can't understand. As a side note I should update the lib to grab the exe path if all else fails for dbghelp.

If you have a moment to, do you think you could try calling cpptrace::absorb_trace_exceptions(false); and see if any errors from inside cpptrace are printed out?

Otherwise, I can try to setup a simple Qt project and see if that reproduces for me.

[slymas]

I added cpptrace::absorb_trace_exceptions(false); during init time, and when I crashed my app, it printed:

Error: "Cpptrace internal error: dwarf error 491 DW_DLE_PE_SECTION_SIZE_ERROR(491) Corrupt PE object. Section size too large."
Sending event 9QKeyEvent to object  (15KeyboardManager)

The first line looks like a smoking gun to me, but I don't understand where is it coming from.

The second line refers to QKeyEvent: an object sent by Qt whenever there are key presses, and KeyboardManager is my class processing key presses. FYI, I crash the app by inputting a numeric code, which makes the app call a function dereferencing a nullptr. This is how I test traces.

[jeremy-rifkin]

Thanks! That’s fantastic. I think this is likely a libdwarf bug. That error code is raised at a couple places in libdwarf’s PE loading code and a couple checks are heuristics-based, it appears it refuses to load a dwarf section larger than 200MB or a dwarf section with a compression ratio > 100, if I understand correctly. In a large qt binary maybe it’s feasible these could trigger. These are sensible heuristics but it’s frustrating they’re interfering with loading an ostensibly valid binary. I feel like I’ve seen this same issue before, I don’t remember where but it was also due to underestimating compression ratios. Could you open an issue upstream at https://github.com/davea42/libdwarf-code?

[slymas]

I filed davea42/libdwarf-code#240 linking back to this discussion.

[jeremy-rifkin]

Perfect thanks

[jeremy-rifkin]

Alright, thanks so much for your patience with this. The latest dev branch commit, ee6787f, includes the upstream fix and hopefully this will work for you!

[slymas]

A quick search revealed this error is created in https://github.com/davea42/libdwarf-code/blob/42367fc3195c0dd2cab059e15a87ed33f5f96e49/src/lib/libdwarf/dwarf_peread.c#L379.

I'll kindly wait for your help in figuring out how that traces back to cpptrace's usage

[slymas]

One last addition which may help: I get a warning during mingw32-make, which I ignored till now but since it's in symbols_with_libdwarf, it may be meaningful:

                 from C:\Path\to\cpptrace\src\symbols\symbols.hpp:9,
                 from C:\Path\to\cpptrace\src\symbols\symbols_with_libdwarf.cpp:4:
In constructor 'cpptrace::detail::optional<T, <anonymous> >::optional(cpptrace::detail::optional<T, <anonymous> >&&) [with T = unsigned int; typename std::enable_if<(! std::i
s_same<typename std::decay<_Tp>::type, void>::value), int>::type <anonymous> = 0]',
    inlined from 'cpptrace::detail::libdwarf::line_entry::line_entry(cpptrace::detail::libdwarf::line_entry&&)' at C:\Path\to\cpptrace\src\symbols\symbols_
with_libdwarf.cpp:58:12,
    inlined from 'void std::__new_allocator<_Tp>::construct(_Up*, _Args&& ...) [with _Up = cpptrace::detail::libdwarf::line_entry; _Args = {cpptrace::detail::libdwarf::line_e
ntry}; _Tp = cpptrace::detail::libdwarf::line_entry]' at C:/Devenv/msys64/mingw64/include/c++/12.2.0/bits/new_allocator.h:175:4,
    inlined from 'static void std::allocator_traits<std::allocator<_CharT> >::construct(allocator_type&, _Up*, _Args&& ...) [with _Up = cpptrace::detail::libdwarf::line_entry
; _Args = {cpptrace::detail::libdwarf::line_entry}; _Tp = cpptrace::detail::libdwarf::line_entry]' at C:/Devenv/msys64/mingw64/include/c++/12.2.0/bits/alloc_traits.h:516:17,
    inlined from 'std::vector<_Tp, _Alloc>::reference std::vector<_Tp, _Alloc>::emplace_back(_Args&& ...) [with _Args = {cpptrace::detail::libdwarf::line_entry}; _Tp = cpptra
ce::detail::libdwarf::line_entry; _Alloc = std::allocator<cpptrace::detail::libdwarf::line_entry>]' at C:/Devenv/msys64/mingw64/include/c++/12.2.0/bits/vector.tcc:117:30,
    inlined from 'void std::vector<_Tp, _Alloc>::push_back(value_type&&) [with _Tp = cpptrace::detail::libdwarf::line_entry; _Alloc = std::allocator<cpptrace::detail::libdwar
f::line_entry>]' at C:/Devenv/msys64/mingw64/include/c++/12.2.0/bits/stl_vector.h:1294:21,
    inlined from 'cpptrace::detail::optional<std::reference_wrapper<cpptrace::detail::libdwarf::line_table_info> > cpptrace::detail::libdwarf::dwarf_resolver::get_line_table(
const cpptrace::detail::libdwarf::die_object&)' at C:\Path\to\cpptrace\src\symbols\symbols_with_libdwarf.cpp:693:47:
C:/Path/to/cpptrace/src/utils/utils.hpp:226:84: warning: '<unnamed>.cpptrace::detail::libdwarf::line_entry::line_number.cpptrace::detail::optional<unsigned
 int>::<anonymous>.cpptrace::detail::optional<unsigned int>::<unnamed union>::uvalue' may be used uninitialized [-Wmaybe-uninitialized]
  226 |                 new (static_cast<void*>(std::addressof(uvalue))) T(std::move(other.uvalue));
      |                                                                              ~~~~~~^~~~~~
C:\Path\to\cpptrace\src\symbols\symbols_with_libdwarf.cpp: In member function 'cpptrace::detail::optional<std::reference_wrapper<cpptrace::detail::libdwarf
::line_table_info> > cpptrace::detail::libdwarf::dwarf_resolver::get_line_table(const cpptrace::detail::libdwarf::die_object&)':
C:\Path\to\cpptrace\src\symbols\symbols_with_libdwarf.cpp:693:47: note: '<anonymous>' declared here
  693 |                         line_entries.push_back({
      |                         ~~~~~~~~~~~~~~~~~~~~~~^~
  694 |                             low_addr,
      |                             ~~~~~~~~~
  695 |                             line
      |                             ~~~~
  696 |                         });
      |                         ~~

[jeremy-rifkin]

Thanks, I haven't seen this locally but I have seen sonar complaining about that optional class too. I think it's correct, but if two things are complaining maybe there's a legitimate issue.

[slymas]

If you want to start a Qt project, the online installer is here: https://www.qt.io/download-qt-installer-oss. I am sharing the link because it's not trivial to find. They are pushing their commercial pages mainly, and if you know where to click, you can find the "open source" installer for free use. You will need to make a free account to run the installer.

I am using Qt5, they released Qt6 recently, but switching over would require some time, so I stayed at the last release of Qt5. See the attached photos of what I have installed form their installer. After that's done, it takes 2 clicks to open one of their examples, build and execute it.

Hope this helps reproducing the issue!

[slymas]

I think something is broken on your dev branch, and I mean apart from having to change CMakeLists.txt to the newest commit on libdwarf (af6b8f2c380d06c2a6bc592f56592f733047ccab) to make libdwarf-populate succeed.

I rebuilt cpptrace by adding -DCPPTRACE_GET_SYMBOLS_WITH_LIBDWARF=On and NOT linking -ldwarf -lz -lzstd and it builds and creates a trace without symbol names. But it shouldn't even start without those libraries linked.

[slymas]

I think I found something. I had a hunch and rebuilt the binary with the debug info inside the exe, and now the default symbol resolution worked! So the issue now is cpptrace not looking up symbols in an external file linked from the main exe.

[jeremy-rifkin]

Thanks for the additional info. I'm not sure how anything worked without linking -ldwarf -lz -lzstd, I'd expect a linker error or at least a runtime error if doing lazy loading of shared symbols.

I had a hunch and rebuilt the binary with the debug info inside the exe, and now the default symbol resolution worked!

I'm glad to hear this at least. When the debug symbols weren't in the binary, where were they being put? Do you happen to know if it was using a split dwarf file, possibly with a .dwo extension?

[slymas]

Yeah, the linking issue is weird, it may be a symptom of cpptrace not finding any symbols to resolve so it doesn't load those libs even lazily?

Qt puts the symbols in a file my.exe.debug next to my.exe, and from PE file I uploaded for davea42/libdwarf-code#240 (comment), I can see there's a .gnu_debuglink section in my.exe.

---

On a similar note, but a different question, let's say I give someone my.exe only (with the symbols being in the separate .debug file). Then cpptrace makes a trace (that I write into a text file), it will obviously not have symbols, only addresses. Is there an easy way to parse that file back later (e.g. in a minimalistic source) and resolve the symbols, while in the possession of .debug file? I already have such a solution using addr2line, but it would be great to have cpptrace solution as well.

[jeremy-rifkin]

Yeah, the linking issue is weird, it may be a symptom of cpptrace not finding any symbols to resolve so it doesn't load those libs even lazily?

Libdwarf is what resolves the symbols, so at the very least it has to load libdwarf to know symbols can't be found. The one way it wouldn't go down this path is if something goes wrong with resolving offsets in the .exe file or looking up the module for an address, but if that were happening you wouldn't get symbols in the other case either. So I'm still confused 😅

.gnu_debuglink is helpful to know, I thought I made that work but I'll make sure.

Is there an easy way to parse that file back later (e.g. in a minimalistic source) and resolve the symbols, while in the possession of .debug file?

Yes! That should be possible with cpptrace. If you parse your trace file into an object_trace (which is just a vector of object_frames), then you can resolve on another machine when you have the debug symbols. One thing to note is that you'll probably want to record object addresses rather than the raw addresses (which is what cpptrace prints by default), since these will have undone any weirdness from things like address randomization.

[jeremy-rifkin]

Yeah, the linking issue is weird, it may be a symptom of cpptrace not finding any symbols to resolve so it doesn't load those libs even lazily?

Libdwarf is what resolves the symbols, so at the very least it has to load libdwarf to know symbols can't be found. The one way it wouldn't go down this path is if something goes wrong with resolving offsets in the .exe file or looking up the module for an address, but if that were happening you wouldn't get symbols in the other case either. So I'm still confused 😅

.gnu_debuglink is helpful to know, I thought I made that work but I'll make sure.

Is there an easy way to parse that file back later (e.g. in a minimalistic source) and resolve the symbols, while in the possession of .debug file?

Yes! That should be possible with cpptrace. If you parse your trace file into an object_trace (which is just a vector of object_frames), then you can resolve on another machine when you have the debug symbols. One thing to note is that you'll probably want to record object addresses rather than the raw addresses (which is what cpptrace prints by default), since these will have undone any weirdness from things like address randomization.

If it's helpful, the signal safe tracing demo for linux may serve as a good proof of concept as it sends object frames to another process which then resolves symbols for the original process. (the one difference being the use of safe_object_frames which can be resolved to a normal object_frame)

[slymas]

Thanks for the help with resolving symbols in a different process, I am sure I will be able to make that work. And I will wait for your findings about the issue when symbols are in a separate file. FYI, on Linux I am also putting symbols into a separate file (and it works perfectly), but Linux seems to be using ELF format for the binary, while Windows uses PE. The issue may lie in the difference there.

[slymas]

I changed my main binary to dump object traces (ss is a stringstream wrote into a file) based on your suggestion to use that instead of the default generate_trace() call which would write raw addresses.

cpptrace::generate_object_trace().resolve().print(ss, /*color=*/false);

Based on your safe tracing demo, I also put together a short program that can read back the file created by the main binary, parse out the hex addresses, and then... I don't know how to build object_frames from that and how to resolve using the main binary's debug file? In the snippet below I parse the hex address from the file into object_frame.object_address because that seemed like the best option, but do I need the other 2 fields of a object_frame? And then, how do I tell my parser program to find and load the main binary's debug file? My earlier addr2line example shows that addr2line has an argument for the symbolic binary, does cpptrace have an input for that?

Thanks for your continued help! :)

// Omitting includes and filenames for input/output.

// Trace lines should look like "#13 0x1234567890abcdef"
const std::regex frame_regex(R"(#\d+\s+(0x[0-9a-f]+))");

int main() {
    std::ifstream inputFile(inputFileName);
    std::ofstream outputFile(outputFileName);

    cpptrace::object_trace trace;
    std::string line;
    while(getline(inputFile, line)) {
        std::smatch address_match;
        if (!std::regex_search(line, address_match, frame_regex) || address_match.size() < 2) {
            continue;
        }

        std::string address = address_match[1].str();
        std::cout << "Found addr: " << address << std::endl;

        cpptrace::object_frame frame;
        frame.object_address = std::stoull(address);
        trace.frames.push_back(frame);
    }

    trace.resolve().print(outputFile, /*color=*/false);

    inputFile.close();
    outputFile.close();
}

[slymas]

I also tried parsing out the path to the binary or one of its libraries from the trace (printed by cpptrace after the 0x0 at portion in each frame), and setting that as frame.object_path but it didn't seem to help. I also tried running the parser in a folder where the main binary and its .debug file is present.

I did all this testing on Linux.

[jeremy-rifkin]

Sorry I should have clarified, and should probably document this better somewhere: The two fields you need to populate in the object_frame are object_path and object_address. The raw_address is the runtime address including ASLR and whatnot so this number is useless outside the invocation of the executable it comes from but the object path and object address will contain all the info needed to resolve the symbol from anywhere. In the context of addr2line, it's very much addr2line -e $object_path $object_address. How you serialize these two is up to you, pretty much anything should work.

[slymas]

Hi Jeremy, Could you look into the still open issue with the debug symbols in a separate file not found by cpptrace?

…

On Thu, 16 May 2024 at 05:03, Jeremy Rifkin ***@***.***> wrote: Sorry I should have clarified, and should probably document this better somewhere: The two fields you need to populate in the object_frame are object_path and object_address. The raw_address is the runtime address including ASLR and whatnot so this number is useless outside the invocation of the executable it comes from but the object path and object address will contain all the info needed to resolve the symbol from anywhere. In the context of addr2line, it's very much addr2line -e $object_path $object_address. How you serialize these two is up to you, pretty much anything should work. — Reply to this email directly, view it on GitHub <#118 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADZ2Q7WNBV6KPDXJ54JKOWTZCQOYLAVCNFSM6AAAAABHJX3CZGVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TINJSGY2DE> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[jeremy-rifkin]

I've been looking into it, I'm pretty sure it's just a matter of debug fission, which I'm working on adding support for. I've ran into a couple issues upstream in libdwarf which I have opened issues about.