Android native crash when running DeepLabV3Demo with vulkan backend in Android 16

[YuHayring]

🐛 Describe the bug

vkCreateComputePipelines return -3 and crashed in Android 16 and API 36 system Image in Android Studio.

It woks well in Android 14, 15

Steps:

1 build aar file including executorch.so with vulkan backend option.
2 generate deepLab3 pte file. (by vulkan partitioner)
3 clone demoapp (https://github.com/pytorch-labs/executorch-examples.git)
4 put .pte file into android assets dir.
5 Replace maven dependency with aar file in gradle dependency, add facebook (com.facebook.fbjni:fbjni:0.5.1 com.facebook.soloader:soloader:0.10.5 )dependency
6 build demo app (include a Module.forward function call from executorch)
7 run and click the button "Run"
8 it crashed

My aar file build step reference:

https://github.com/pytorch/executorch/blob/main/docs/source/using-executorch-android.md
https://github.com/pytorch/executorch/blob/main/docs/source/backends-vulkan.md

This is my crash stack:

com.facebook.jni.CppException: Exception raised from ComputePipeline at /Users/name/Desktop/Code/executorch/backends/vulkan/runtime/vk_api/Pipeline.cpp:319: vkCreateComputePipelines( device_, pipeline_cache, 1u, &compute_pipeline_create_info, nullptr, &handle_) returned -3
	at org.pytorch.executorch.NativePeer.forward(Native Method)
	at org.pytorch.executorch.Module.forward(Module.java:75)
	at org.pytorch.executorchexamples.dl3.ClassificationActivity.run(ClassificationActivity.java:80)
	at org.pytorch.executorchexamples.dl3.ClassificationActivity$1.onClick(ClassificationActivity.java:121)
	at android.view.View.performClick(View.java:8083)
	at android.view.View.performClickInternal(View.java:8060)
	at android.view.View.-$$Nest$mperformClickInternal(Unknown Source:0)
	at android.view.View$PerformClick.run(View.java:31549)
	at android.os.Handler.handleCallback(Handler.java:995)
	at android.os.Handler.dispatchMessage(Handler.java:103)
	at android.os.Looper.loopOnce(Looper.java:248)
	at android.os.Looper.loop(Looper.java:338)
	at android.app.ActivityThread.main(ActivityThread.java:9067)
	at java.lang.reflect.Method.invoke(Native Method)
	at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:593)
	at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:932)

Tested devices:

works well: Samsung Z Fold 6 Android 14,15
Samsung S21 Android14
Google Pixel 4 Latest Android13
Redmi Note 7(lavender) latest LineageOS18 Android11
official Google APIs arm64-v8a AVD API34 r14
official Google Play arm64-v8a AVD API35 r8

no working:
Pixel6 Android 16
official Google APIs arm64-v8a AVD API36

Extra:

This bug was found in April when using Android 16 beta4, and i reported it to google official:https://issuetracker.google.com/issues/412580391
Since they said they "received it and will provide more information in the future", there has been no response until now.
Even the beta has ended and Android 16 has been officially released, this problem still exists.

Versions

PyTorch version: 2.7.0
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: macOS 15.5 (arm64)
GCC version: Could not collect
Clang version: 16.0.0 (clang-1600.0.26.3)
CMake version: version 3.30.5
Libc version: N/A

Python version: 3.10.0 (default, Mar 3 2022, 03:54:28) [Clang 12.0.0 ] (64-bit runtime)
Python platform: macOS-15.5-arm64-arm-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Apple M3

Versions of relevant libraries:
[pip3] executorch==0.6.0a0+fc45b9b
[pip3] numpy==2.2.4
[pip3] torch==2.7.0
[pip3] torchao==0.10.0+git8b264ce1
[pip3] torchaudio==2.7.0
[pip3] torchsr==1.0.4
[pip3] torchvision==0.22.0
[conda] executorch 0.6.0a0+fc45b9b pypi_0 pypi
[conda] numpy 2.2.4 pypi_0 pypi
[conda] torch 2.7.0 pypi_0 pypi
[conda] torchao 0.10.0+git8b264ce1 pypi_0 pypi
[conda] torchaudio 2.7.0 pypi_0 pypi
[conda] torchsr 1.0.4 pypi_0 pypi
[conda] torchvision 0.22.0 pypi_0 pypi

cc @SS-JIA @manuelcandales @cbilgin @kirklandsign

[SS-JIA]

@YuHayring thanks for reporting this.

I don't have any devices on hand which have Android 16 installed. I will try to upgrade, and from there attempt to reproduce.

In the meantime, it would greatly be appreciated if you could add a logging line above this line, something like

std::cout << shader_.kernel_name

should suffice. I mostly want to get some info on which shader the compilation is failing.

I would also recommend that you re-try on executorch's main branch. I recently fixed some issues with some shaders which could potentially cause shader compilation errors.

Thanks in advance!

[YuHayring]

@YuHayring thanks for reporting this.

I don't have any devices on hand which have Android 16 installed. I will try to upgrade, and from there attempt to reproduce.

In the meantime, it would greatly be appreciated if you could add a logging line above this line, something like

std::cout << shader_.kernel_name

should suffice. I mostly want to get some info on which shader the compilation is failing.

I would also recommend that you re-try on executorch's main branch. I recently fixed some issues with some shaders which could potentially cause shader compilation errors.

Thanks in advance!

OK, thank you for your attention to this issue, but I'm quite busy recently, I will try to do these experiments earlier.

[YuHayring]

I took a moment to run a simple test with tag v0.7.0-rc2. When it call the function "mModule.forward" in DeepLabV3Demo , MainAtivity.java. This method simply executes indefinitely without returning, and there's no subsequent response. No useful output is found in the app console.
I'll try again later and provide more information when I have more time.

[YuHayring]

DeepLabV3 had no response yesterday when using android virtual device api 36, but work well on Physical device Pixel6.

This time i generate "inception_v3" in Pixel 6 Android 16 it cause new exception:

Process: org.pytorch.executorchexamples.dl3, PID: 24890
com.facebook.jni.CppException: Exception raised from add_concat_node at /Users/hayring/Desktop/Code/executorch/backends/vulkan/runtime/graph/ops/impl/Concat.cpp:72: (tensors->size() <= 3) is false! Currently only concatenation of <= 3 tensors is supported
    at org.pytorch.executorch.Module.executeNative(Native Method)
    at org.pytorch.executorch.Module.execute(Module.java:145)
    at org.pytorch.executorch.Module.forward(Module.java:128)
    at org.pytorch.executorchexamples.dl3.ClassificationActivity.run(ClassificationActivity.java:80)
    at org.pytorch.executorchexamples.dl3.ClassificationActivity$1.onClick(ClassificationActivity.java:121)
    at android.view.View.performClick(View.java:8083)
    at android.view.View.performClickInternal(View.java:8060)
    at android.view.View.-$$Nest$mperformClickInternal(Unknown Source:0)
    at android.view.View$PerformClick.run(View.java:31549)
    at android.os.Handler.handleCallback(Handler.java:995)
    at android.os.Handler.dispatchMessage(Handler.java:103)
    at android.os.Looper.loopOnce(Looper.java:248)
    at android.os.Looper.loop(Looper.java:338)
    at android.app.ActivityThread.main(ActivityThread.java:9067)
    at java.lang.reflect.Method.invoke(Native Method)
    at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:593)
    at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:932)

[YuHayring]

I tested InceptionV3, ResNet101, deeplabv3. It seems that only InceptionV3 causes crash.