IllegalArgumentException: Object id {id} not found in heap dump.

[tommyjackson]

Description

Anywhere in our app we try to dump the heap, the heap analysis fails with the stack trace below. This is making it impossible to use LeakCanary since we can't get any data from these heap dumps. We have been using LeakCanary for years and have only recently started seeing this issue, and I don't think there would be an issue on our side since all we do is bring in the dependency.

java.lang.IllegalArgumentException: Object id 336576656 not found in heap dump.
	at shark.HprofHeapGraph.findObjectById(HprofHeapGraph.kt:151)
	at shark.internal.PathFinder.findPathsFromGcRoots(PathFinder.kt:163)
	at shark.internal.PathFinder.findPathsFromGcRoots(PathFinder.kt:135)
	at shark.HeapAnalyzer.findLeaks(HeapAnalyzer.kt:275)
	at shark.HeapAnalyzer.analyzeGraph(HeapAnalyzer.kt:253)
	at shark.HeapAnalyzer.analyze$shark(HeapAnalyzer.kt:217)
	at shark.HeapAnalyzer.analyze(HeapAnalyzer.kt:166)
	at leakcanary.internal.AndroidDebugHeapAnalyzer.analyzeHeap(AndroidDebugHeapAnalyzer.kt:156)
	at leakcanary.internal.AndroidDebugHeapAnalyzer.runAnalysisBlocking(AndroidDebugHeapAnalyzer.kt:59)
	at leakcanary.internal.AndroidDebugHeapAnalyzer.runAnalysisBlocking$default(AndroidDebugHeapAnalyzer.kt:46)
	at leakcanary.internal.HeapAnalyzerWorker.doWork(HeapAnalyzerWorker.kt:18)
	at androidx.work.Worker$1.run(Worker.java:86)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1137)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:637)
	at java.lang.Thread.run(Thread.java:1012)
====================================
METADATA
                                                                                                    
Build.VERSION.SDK_INT: 33
Build.MANUFACTURER: Google
LeakCanary version: 2.12
Analysis duration: 9632 ms
Heap dump file path: /storage/emulated/0/Download/leakcanary-com.package.debug/2023-09-11_16-18-02_753.hprof
Heap dump timestamp: 1694467100244

Steps to Reproduce

hprof file: https://drive.google.com/file/d/1w1kwAmWlRB2PmzgaIDq2sRwCO9Sgsns2/view?usp=sharing

Expected behavior: [What you expect to happen]
I expect to be able to view the results from the analyzed heap dump.

Version Information

• LeakCanary version: 2.12
• Android OS version: 13
• Gradle version: 8.1.1

Additional Information

There isn't anything specific we do to reproduce it, it just happens whenever a heap is dumped, whether manually or automatically, and regardless of the device or emulator. Happy to provide anymore info if necessary.

[tommyjackson]

Spent some time bisecting to find the exact time LeakCanary stopped working and I found the root cause. We were bringing in the com.google.firebase:firebase-appindexing, and when we tried to add the shortcuts dependency we had to do it like this due to conflicts with the appindexing one:

implementation(libs.androidx.shortcuts) { 
    exclude group: "com.google.firebase", module: "firebase-appindexing" 
    exclude group: "com.google.android.gms", module: "play-services-appindex" 
}

Those exclude statements seem to be what broke LeakCanary for us, since even when removing the appindexing dependency altogether we still see the issue I described. Only removing the appindexing dependency and then the exlude parts from the shortcuts dependency did LeakCanary start to work successfully again.

So clearly there was a fixable issue on our end, but I'm not sure that there still isn't a bug in LeakCanary as I wouldn't have expected it to break due to excluding libs from a completely separate dependency. I'll leave it up to y'all to decide whether or not to close this.

[pyricau]

Thank you @tommyjackson for the details and the hprof!

I can reproduce locally with:

./shark-cli.sh -h /Users/py/Downloads/2023-09-11_16-18-02_753.hprof   analyze

====================================
HEAP ANALYSIS FAILED

You can report this failure at https://github.com/square/leakcanary/issues
Please provide the stacktrace, metadata and the heap dump file.
====================================
STACKTRACE

java.lang.IllegalArgumentException: Object id 336576656 not found in heap dump.
	at shark.HprofHeapGraph.findObjectById(HprofHeapGraph.kt:134)
	at shark.PrioritizingShortestPathFinder.findPathsFromGcRoots(PrioritizingShortestPathFinder.kt:180)
	at shark.PrioritizingShortestPathFinder.findShortestPathsFromGcRoots(PrioritizingShortestPathFinder.kt:152)
	at shark.RealLeakTracerFactory.findLeaks(RealLeakTracerFactory.kt:94)
	at shark.RealLeakTracerFactory.createFor$lambda$0(RealLeakTracerFactory.kt:81)
	at shark.HeapAnalyzer.analyze(HeapAnalyzer.kt:179)
	at shark.HeapAnalyzer.analyze(HeapAnalyzer.kt:67)
	at shark.AnalyzeCommand$Companion.analyze(AnalyzeCommand.kt:36)
	at shark.AnalyzeCommand.run(AnalyzeCommand.kt:16)
	at com.github.ajalt.clikt.parsers.Parser.parse(Parser.kt:139)
	at com.github.ajalt.clikt.parsers.Parser.parse(Parser.kt:14)
	at com.github.ajalt.clikt.core.CliktCommand.parse(CliktCommand.kt:215)
	at com.github.ajalt.clikt.core.CliktCommand.parse$default(CliktCommand.kt:212)
	at com.github.ajalt.clikt.core.CliktCommand.main(CliktCommand.kt:230)
	at com.github.ajalt.clikt.core.CliktCommand.main(CliktCommand.kt:253)
	at shark.MainKt.main(Main.kt:13)
====================================
METADATA

Build.VERSION.SDK_INT: -1
Build.MANUFACTURER: Unknown
LeakCanary version: Unknown
Analysis duration: 2104 ms
Heap dump file path: /Users/py/Downloads/2023-09-11_16-18-02_753.hprof
Heap dump timestamp: 1704191567845

My intuition here is that your setup led to a weird heap state, although that really shouldn't happen.

Let's make a change in LeakCanary to provide more details on the path we use to reach this unknown object id: TODO link PR

Here's the updated error:

java.lang.RuntimeException: Invalid object id reached through path:
┬───
│ GC Root: Thread object
│
├─ android.net.ConnectivityThread instance
│    ↓ Thread.contextClassLoader
├─ dalvik.system.PathClassLoader instance
│    ↓ ClassLoader.runtimeInternalObjects
├─ java.lang.Object[] array
│    ↓ Object[13501]
├─ androidx.core.google.shortcuts.builders.CapabilityBuilder[] class
│    ↓ static CapabilityBuilder.$class$componentType
╰→ UnknownObject336576656 instance

This is fascinating. androidx.core.google.shortcuts.builders.CapabilityBuilder[] is an array class, i.e. it's the class for creating arrays of androidx.core.google.shortcuts.builders.CapabilityBuilder.

It looks like the array class is malformed, it has a $class$componentType field that points to an object that doesn't exist. I opened up the heap dump with YourKit Java Profiler. This confirmed that:

• $class$componentType exists on object array classes and references the class. In this case, it should be a reference to androidx.core.google.shortcuts.builders.CapabilityBuilder
• While I can find the androidx.core.google.shortcuts.builders.CapabilityBuilder[] class in memory, I cannot find the androidx.core.google.shortcuts.builders.CapabilityBuilder class.

In other words, androidx.core.google.shortcuts.builders.CapabilityBuilder[] class is included in the heap dump and references a missing androidx.core.google.shortcuts.builders.CapabilityBuilder class.

The CapabilityBuilder class is defined here: https://cs.android.com/androidx/platform/frameworks/support/+/androidx-main:core/core-google-shortcuts/src/main/java/androidx/core/google/shortcuts/builders/CapabilityBuilder.java;drc=f965a68527c86919fef87a7705619d78b46d0139

As far as I can tell the only usage of the CapabilityBuilder[] class is here:

shortcutBuilder.setCapability(capabilityList.toArray(new CapabilityBuilder[0]));

https://cs.android.com/androidx/platform/frameworks/support/+/androidx-main:core/core-google-shortcuts/src/main/java/androidx/core/google/shortcuts/ShortcutInfoChangeListenerImpl.java;l=183;drc=f965a68527c86919fef87a7705619d78b46d0139

It's worth noting that the entire method likely does not ever actually load the CapabilityBuilder class, as List<CapabilityBuilder> loses the type information at runtime, and for (String capability : shortcut.getCategories()) { probably doesn't run. So could we possibly be in a place where we've loaded CapabilityBuilder[] without loading CapabilityBuilder ?

Note: the heap dump shows this is Android 13 / SDK 33, on a Google Pixel 4a, fingerprint "google/sunfish/sunfish:13/TQ3A.230805.001.A2/10385117:user/release-keys"

@tommyjackson by any chance, would you be able to look up your APKs from back then (or recreate a faulty one) and see if the dex files include CapabilityBuilder class as well as the CapabilityBuilder[] class?

Also, was this issue limited to specific versions of Android? Can you reproduce on recent versions of Android?

It'd be even more amazing if you could set up a small repro project since you've isolated the dependencies causing this issue.