Kernel oops and panic on Hamoa IOT EVK when booting meta-qcom nightly build

[mwasilew]

Trying to boot nightly from qualcomm-linux/meta-qcom@19a15a0 results in oops on one of the boards in the lab. The trace below:

[    8.686350] Run /init as init process
Starting systemd-udevd version 258.1
[    9.051981] EXT4-fs (nvme0n1p2): mounted filesystem 2a385980-fcd3-4a52-9d8d-3815e42f18e7 r/w with ordered data mode. Quota mode: none.
[    9.090159] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000000
[    9.099235] Mem abort info:
[    9.102112]   ESR = 0x0000000096000044
[    9.105976]   EC = 0x25: DABT (current EL), IL = 32 bits
[    9.111439]   SET = 0, FnV = 0
[    9.114586]   EA = 0, S1PTW = 0
[    9.117821]   FSC = 0x04: level 0 translation fault
[    9.122835] Data abort info:
[    9.125809]   ISV = 0, ISS = 0x00000044, ISS2 = 0x00000000
[    9.131451]   CM = 0, WnR = 1, TnD = 0, TagAccess = 0
[    9.136651]   GCS = 0, Overlay = 0, DirtyBit = 0, Xs = 0
[    9.142116] user pgtable: 4k pages, 48-bit VAs, pgdp=0000000883778000
[    9.148734] [0000000000000000] pgd=0000000000000000, p4d=0000000000000000
[    9.155710] Internal error: Oops: 0000000096000044 [#1]  SMP
[    9.161521] Modules linked in:
[    9.164666] CPU: 11 UID: 0 PID: 1 Comm: init Not tainted 6.18.0-00393-g27507852413b #1 PREEMPT 
[    9.173595] Hardware name: Qualcomm Technologies, Inc. Hamoa IoT EVK (DT)
[    9.180566] pstate: 01400005 (nzcv daif +PAN -UAO -TCO +DIT -SSBS BTYPE=--)
[    9.187710] pc : nvme_pci_prp_iter_next+0x88/0xb0
[    9.192549] lr : nvme_pci_prp_iter_next+0x70/0xb0
[    9.197386] sp : ffff80008007b400
[    9.200791] x29: ffff80008007b400 x28: 0000000000000020 x27: fffffdffed17c200
[    9.208114] x26: 0000000000000021 x25: ffff80008007b7a0 x24: ffff80008007b6b8
[    9.215446] x23: ffff00080e42b208 x22: ffff80008007b470 x21: 0000000000000000
[    9.222776] x20: ffff000806f440f0 x19: ffff000803e650c8 x18: 0000000000000000
[    9.230100] x17: ffff008f326a4000 x16: 0000000000000000 x15: 0000000000000000
[    9.237427] x14: 0000000000000000 x13: 0000000000000000 x12: 0000000000000000
[    9.244756] x11: 0000ffff9753ed28 x10: 0000ffff9753ecb0 x9 : 0000ffff9756e910
[    9.252085] x8 : 0000ffff9756f330 x7 : 0000ffff9756dcb8 x6 : 000000000003fe48
[    9.259412] x5 : ffff00005334e000 x4 : ffff000806f44000 x3 : ffff80008007b470
[    9.266741] x2 : 00000000d334d000 x1 : 0000000000000000 x0 : 0000000000000000
[    9.274072] Call trace:
[    9.276592]  nvme_pci_prp_iter_next+0x88/0xb0 (P)
[    9.281429]  nvme_prep_rq+0x280/0xb70
[    9.285195]  nvme_queue_rqs+0xb0/0x18c
[    9.289049]  blk_mq_dispatch_queue_requests.isra.0+0x108/0x124
[    9.295048]  blk_mq_flush_plug_list+0x90/0x190
[    9.299620]  __blk_flush_plug+0xe4/0x144
[    9.303659]  blk_finish_plug+0x2c/0x40
[    9.307511]  read_pages+0x18c/0x290
[    9.311100]  page_cache_ra_order+0x1d8/0x3b4
[    9.315495]  filemap_fault+0x4c0/0x9c0
[    9.319348]  __do_fault+0x3c/0x240
[    9.322849]  __handle_mm_fault+0x51c/0x1958
[    9.327151]  handle_mm_fault+0x8c/0x2a0
[    9.331099]  do_page_fault+0x1dc/0x760
[    9.334957]  do_translation_fault+0x5c/0xa0
[    9.339260]  do_mem_abort+0x40/0x90
[    9.342847]  el1_abort+0x44/0x68
[    9.346177]  el1h_64_sync_handler+0x50/0xac
[    9.350477]  el1h_64_sync+0x70/0x74
[    9.354067]  __arch_clear_user+0x30/0xb0 (P)
[    9.358458]  load_elf_binary+0x4d4/0x18dc
[    9.362584]  bprm_execve+0x280/0x4a4
[    9.366265]  do_execveat_common.isra.0+0x1a0/0x1e0
[    9.371191]  __arm64_sys_execve+0x40/0x50
[    9.375320]  invoke_syscall.constprop.0+0x40/0xf0
[    9.380159]  el0_svc_common.constprop.0+0x38/0xd8
[    9.384993]  do_el0_svc+0x1c/0x28
[    9.388408]  el0_svc+0x34/0xe8
[    9.391551]  el0t_64_sync_handler+0xa0/0xe4
[    9.395850]  el0t_64_sync+0x19c/0x1a0
[    9.399617] Code: f94013e3 f9405281 d37cec00 f9400062 (f8206822) 
[    9.405870] ---[ end trace 0000000000000000 ]---
[    9.420312] ------------[ cut here ]------------
[    9.425056] WARNING: CPU: 11 PID: 1 at /usr/src/kernel/kernel/exit.c:903 do_exit+0x6d0/0x8fc
[    9.433716] Modules linked in:
[    9.436861] CPU: 11 UID: 0 PID: 1 Comm: init Tainted: G      D             6.18.0-00393-g27507852413b #1 PREEMPT 
[    9.447388] Tainted: [D]=DIE
[    9.450351] Hardware name: Qualcomm Technologies, Inc. Hamoa IoT EVK (DT)
[    9.457322] pstate: 81400005 (Nzcv daif +PAN -UAO -TCO +DIT -SSBS BTYPE=--)
[    9.464468] pc : do_exit+0x6d0/0x8fc
[    9.468147] lr : make_task_dead+0x7c/0x1a0
[    9.472357] sp : ffff80008007b040
[    9.475773] x29: ffff80008007b070 x28: ffff000800928000 x27: fffffdffed17c200
[    9.483102] x26: 0000000000000021 x25: 0000000000000000 x24: 0000000000000000
[    9.490434] x23: ffffa081058a9ae8 x22: 0000000000000000 x21: ffffa08106387658
[    9.497756] x20: ffff000800928000 x19: 000000000000000b x18: 000000000000000a
[    9.505087] x17: 0000000000000000 x16: ffff8000803dbff0 x15: 4596b8406eb4f4f6
[    9.512418] x14: 836984320a8a41a1 x13: 9926d92c9b6ee2fa x12: 385164b9a44fcb1c
[    9.519747] x11: 04fbd9639926d92c x10: 9b6ee2fa385164b9 x9 : a44fcb1cc936f164
[    9.527074] x8 : 0000000000000000 x7 : 1658e649b7c6487f x6 : 000000000000000b
[    9.534403] x5 : ffff000800928000 x4 : ffff000800928000 x3 : 0000000000000000
[    9.541728] x2 : 000000000000270f x1 : 0000000000000000 x0 : ffff80008007b688
[    9.549050] Call trace:
[    9.551566]  do_exit+0x6d0/0x8fc (P)
[    9.555244]  make_task_dead+0x7c/0x1a0
[    9.559103]  die+0x144/0x15c
[    9.562073]  die_kernel_fault+0x4b8/0x4d4
[    9.566196]  __do_kernel_fault+0x1f8/0x200
[    9.570410]  do_page_fault+0x230/0x760
[    9.574262]  do_translation_fault+0x5c/0xa0
[    9.578563]  do_mem_abort+0x40/0x90
[    9.582151]  el1_abort+0x44/0x68
[    9.585469]  el1h_64_sync_handler+0x50/0xac
[    9.589772]  el1h_64_sync+0x70/0x74
[    9.593362]  nvme_pci_prp_iter_next+0x88/0xb0 (P)
[    9.598199]  nvme_prep_rq+0x280/0xb70
[    9.601964]  nvme_queue_rqs+0xb0/0x18c
[    9.605825]  blk_mq_dispatch_queue_requests.isra.0+0x108/0x124
[    9.611815]  blk_mq_flush_plug_list+0x90/0x190
[    9.616388]  __blk_flush_plug+0xe4/0x144
[    9.620424]  blk_finish_plug+0x2c/0x40
[    9.624277]  read_pages+0x18c/0x290
[    9.627864]  page_cache_ra_order+0x1d8/0x3b4
[    9.632251]  filemap_fault+0x4c0/0x9c0
[    9.636102]  __do_fault+0x3c/0x240
[    9.639604]  __handle_mm_fault+0x51c/0x1958
[    9.643903]  handle_mm_fault+0x8c/0x2a0
[    9.647852]  do_page_fault+0x1dc/0x760
[    9.651713]  do_translation_fault+0x5c/0xa0
[    9.656012]  do_mem_abort+0x40/0x90
[    9.659602]  el1_abort+0x44/0x68
[    9.662927]  el1h_64_sync_handler+0x50/0xac
[    9.667228]  el1h_64_sync+0x70/0x74
[    9.670814]  __arch_clear_user+0x30/0xb0 (P)
[    9.675209]  load_elf_binary+0x4d4/0x18dc
[    9.679333]  bprm_execve+0x280/0x4a4
[    9.683014]  do_execveat_common.isra.0+0x1a0/0x1e0
[    9.687935]  __arm64_sys_execve+0x40/0x50
[    9.692063]  invoke_syscall.constprop.0+0x40/0xf0
[    9.696899]  el0_svc_common.constprop.0+0x38/0xd8
[    9.701736]  do_el0_svc+0x1c/0x28
[    9.705146]  el0_svc+0x34/0xe8
[    9.708290]  el0t_64_sync_handler+0xa0/0xe4
[    9.712593]  el0t_64_sync+0x19c/0x1a0
[    9.716359] ---[ end trace 0000000000000000 ]---
[    9.721122] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b
[    9.728977] SMP: stopping secondary CPUs
[    9.733159] Kernel Offset: 0x208084fa0000 from 0xffff800080000000
[    9.739421] PHYS_OFFSET: 0xfff1000080000000
[    9.743726] CPU features: 0x020000,0000e000,427c62c1,557e7ea7
[    9.749628] Memory Limit: none
[    9.762945] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b ]---

Corresponding LAVA job: https://lava.infra.foundries.io/scheduler/job/131434#L1826

The build is using the following kernel revision:

tag: qcom-6.18.y-20251217

SRCREV ?= "27507852413bd8ba6205a95a4a1df15e62b88009"

[mwasilew]

ping. This problem happens roughly once in 3 boots, so it's really annoying. Could anyone take a look and suggest how to avoid it?

[shashim-quic]

• @nitinrawat123 who can help analyze this issue

[Pradeep-pvk]

This is a known issue caused by swiotlb_map() resetting the dma_skip_sync flag, which results in a NULL‑pointer dereference. As a workaround, we have raised a PR: qualcomm-linux/kernel-topics#550. You can pull the NVMe update from this PR to avoid the issue while the teams continue working on the proper fix.