Testing yanghaku's DRM patch for cache coherency on various graphics cards

[geerlingguy]

I wanted to consolidate some of my testing into a single issue, as I'd like to see how well the simple fix from #756 works for various graphics cards.

Test Setup

I have a Raspberry Pi 500+, and test graphics cards inside this JMT M.2 M-Key to OCuLink eGPU dock (model DOCK-OC4), using a chenyang SFF-8611 OCuLink cable. Photo of that part of the setup below:

For cards with HDMI and DisplayPort outputs, I plug HDMI directly into my 4K display. For cards with only DisplayPort, I plug DisplayPort directly into my 4K display.

I'll leave further details in the comments, but from a high level, here are cards that have been tested under Raspberry Pi OS rpi-6.15.y with that patch applied (click on the card name to go to the specific GitHub issue with more details on that card):

AMD (amdgpu)

Card	glmark2	vkmark	GravityMark	Notes
AMD Radeon RX 9070 XT 16GB	8151	14946	56944	Required newer version of Mesa
AMD Radeon RX 7900 XT 20GB	8011	14319	65809	Everything runs smoothly
AMD Radeon Pro W7700 16GB	7589	12929	40782	Everything runs smoothly
AMD Radeon RX 7600 8GB	8074	14790	27861	Everything runs smoothly
AMD Radeon RX 6700 XT 12GB	7841	14855	36455	GUI choppy until restarting lightdm
AMD Radeon RX 6500 XT 8GB	7668	14185	14865	GUI choppy until restarting lightdm
AMD Radeon RX 580 8GB	7613	12753	12103	Everything runs smoothly
AMD Radeon RX 460 4GB	7269	10089	5739	Everything runs smoothly

AMD (radeon)

Card	glmark2	vkmark	GravityMark	Notes
AMD R5 230 2GB	356	563	DNF	Only X11 works, see linked issue
AMD Radeon HD 7470 1GB	539	230	DNF	Only X11 works, see linked issue

Intel Arc

Card	glmark2	vkmark	GravityMark	Notes
Intel B580 Sparkle 12GB	5622	10133	55209	Artifacting, but better than before; see #695
Intel Arc A750	4223	12313	25462	Artifacting, but better than before; see #510

Nvidia RTX / GTX

Card	glmark2	vkmark	GravityMark	Notes
Nvidia RTX A4000	-	-	-	Get error dump on swiotlb_map
Nvidia GeForce GTX 750 Ti	-	-	-	Get error dump on swiotlb_map

Commands used (representative example):

# glmark2
$ DISPLAY=:0 glmark2-es2-wayland 
=======================================================
    glmark2 2023.01
=======================================================
    OpenGL Information
    GL_VENDOR:      AMD
    GL_RENDERER:    AMD Radeon RX 6500 XT (radeonsi, navi24, LLVM 15.0.6, DRM 3.63, 6.15.11-v8-16k+)
    GL_VERSION:     OpenGL ES 3.2 Mesa 24.2.8-1~bpo12+rpt4
    Surface Config: buf=32 r=8 g=8 b=8 a=8 depth=24 stencil=0 samples=0
    Surface Size:   800x600 windowed
=======================================================

# vkmark
$ DISPLAY=:0 build/src/vkmark --winsys-dir=build/src --data-dir=data
=======================================================
    vkmark 2025.01
=======================================================
    Vendor ID:      0x1002
    Device ID:      0x743F
    Device Name:    AMD Radeon RX 6500 XT (RADV NAVI24)
    Driver Version: 100671496
    Device UUID:    3ab6f656c2fadb58e7866ca2641bd4d6
=======================================================

# GravityMark
$ sudo ./GravityMark_1.88_arm64.run
# (Use defaults: 200k asteroids, 1600x900, windowed, Vulkan)

[RSC-Games]

I don't mind testing my 3 cards later on with the new patches this weekend
I have the Radeon HD 7870, GTX 1050 Ti, then a RTX 4060

[geerlingguy]

I haven't tried any Nvidia cards yet, I'm still assuming the deeper patchset is needed before those will work.

[geerlingguy]

Interestingly, I was testing on a fresh Pi OS install today, and found the 7900 XT was only getting 64817 on PCIe Gen 2 (the default). After bumping to Gen 3, I got 65809.

Though, running the test four more times, the scores ranged from about 64,700 to 65,800.

[6by9]

Seeing as the change is contained within CONFIG_DRM_TTM which isn't used by vc4 or v3d, the side effects of changing it on a Pi are very minimal.

For some reason the RP1 DSI, DPI, and VEC drivers are requesting it, but that may just be due to a bit too much copy/paste.

(I might dig out my NVidia GT710 quad HDMI output card tomorrow. Let's see how nouveau does with these changes)

[geerlingguy]

@6by9 That'd be great—and I will also try to test with nouveau at some point, just wanting to finish hammering out all the AMD cards I can get my hands on for testing right now.

I was also noticing on the RX 460 that @Coreforge and I both tested, his scores were mostly slightly higher than mine on GravityMark:

Could be there are some optimized paths in the amdgpu-driver-specific changes that aren't accounted for with the DRM-wide changes; but the difference is relatively small.

[RSC-Games]

Are you guys using the same card or is Coreforge using a slightly overclocked card? I know my specific card is factory overclocked 50 MHz over what the Radeon HD 7870 GHz edition is rated for (what's the base clock? Idk it's definitely not in the name of the card). Just a thought.

Also 6by9 I fully intend to test nouveau's codepaths later on both of my NVIDIA cards, since one of the main reasons I even came here (other than my long-standing fascination in Jeff's quest to get GPUs working on the Pi) is because I want to help get all 3 vendor's cards working. I won't rest until that's done... even if most of my time is currently eaten by my semester.

[geerlingguy]

@6by9 - Testing with two different generations of Nvidia GPUs, I get a similar kernel dump on each with the nouveau driver:

[   46.089143] nouveau 0001:01:00.0: enabling device (0000 -> 0002)
[   46.089176] nouveau 0001:01:00.0: NVIDIA GA104 (b74000a1)
[   46.234439] nouveau 0001:01:00.0: bios: version 94.04.57.00.08
[   46.235150] ------------[ cut here ]------------
[   46.235159] nouveau 0001:01:00.0: swiotlb addr 0x00000010fce00000+4096 overflow (mask ffffffff, bus limit ffffffffff).
[   46.235177] WARNING: CPU: 3 PID: 949 at kernel/dma/swiotlb.c:1594 swiotlb_map+0x2c0/0x2f0
[   46.235193] Modules linked in: nouveau(+) drm_gpuvm i2c_algo_bit drm_exec gpu_sched drm_display_helper drm_ttm_helper ttm drm_client_lib drm_kms_helper snd_seq_dummy snd_hrtimer snd_seq snd_seq_device snd_timer snd rfcomm algif_hash algif_skcipher af_alg bnep binfmt_misc brcmfmac_wcc spidev aes_ce_blk aes_ce_cipher ghash_ce rpi_hevc_dec hci_uart pisp_be btbcm v4l2_mem2mem videobuf2_dma_contig gf128mul bluetooth sha2_ce sha256_arm64 sha1_ce raspberrypi_hwmon sha1_generic brcmfmac brcmutil ecdh_generic ecc cfg80211 videobuf2_memops videobuf2_v4l2 spi_bcm2835 videodev gpio_keys rfkill videobuf2_common mc raspberrypi_gpiomem rp1_pio rp1_adc rp1_mailbox rp1_fw sg nvmem_rmem joydev hid_multitouch uio_pdrv_genirq uio drm i2c_dev ledtrig_pattern fuse drm_panel_orientation_quirks backlight dm_mod ip_tables x_tables ipv6
[   46.235294] CPU: 3 UID: 0 PID: 949 Comm: modprobe Not tainted 6.15.11-v8-4k-gpu+ #1 PREEMPT 
[   46.235299] Hardware name: Raspberry Pi 500 Rev 1.0 (DT)
[   46.235302] pstate: 60400009 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[   46.235305] pc : swiotlb_map+0x2c0/0x2f0
[   46.235308] lr : swiotlb_map+0x2c0/0x2f0
[   46.235310] sp : ffffffc0837b3630
[   46.235311] x29: ffffffc0837b3640 x28: ffffff8301bb4c00 x27: ffffffd08d667000
[   46.235316] x26: 000000108121b000 x25: fffffffec20486c0 x24: 0000000000001000
[   46.235321] x23: ffffff8003a5c0c8 x22: 0000000000000000 x21: 0000000000000000
[   46.235325] x20: 000000008121b000 x19: 000000108121b000 x18: 00000000ffffffff
[   46.235330] x17: 667265766f203639 x16: 30342b3030303030 x15: 6563663031303030
[   46.235334] x14: 3030307830207264 x13: 2e29666666666666 x12: 666666662074696d
[   46.235338] x11: 696c20737562202c x10: ffffffd08dd1eca8 x9 : ffffffd08c639948
[   46.235342] x8 : 00000000ffffefff x7 : ffffffd08dd1eca8 x6 : 80000000fffff000
[   46.235346] x5 : ffffff807ffdd5c8 x4 : 0000000000000000 x3 : 0000000000000000
[   46.235350] x2 : 0000000000000000 x1 : 0000000000000000 x0 : ffffff8081220000
[   46.235354] Call trace:
[   46.235356]  swiotlb_map+0x2c0/0x2f0 (P)
[   46.235360]  dma_map_page_attrs+0x1d8/0x368
[   46.235365]  nvkm_fb_ctor+0xc4/0x100 [nouveau]
[   46.235554]  gf100_fb_new_+0x68/0x98 [nouveau]
[   46.235697]  ga102_fb_new+0x6c/0x88 [nouveau]
[   46.235832]  nvkm_device_ctor+0xaec/0x36d8 [nouveau]
[   46.235968]  nvkm_device_pci_new+0xe0/0x2b8 [nouveau]
[   46.236105]  nouveau_drm_probe+0x4c/0x1e0 [nouveau]
[   46.236244]  local_pci_probe+0x48/0xb8
[   46.236253]  pci_device_probe+0xcc/0x1d8
[   46.236256]  really_probe+0xc4/0x2a8
[   46.236262]  __driver_probe_device+0x80/0x140
[   46.236266]  driver_probe_device+0x44/0x170
[   46.236270]  __driver_attach+0x9c/0x1b0
[   46.236274]  bus_for_each_dev+0x84/0xf0
[   46.236277]  driver_attach+0x2c/0x40
[   46.236281]  bus_add_driver+0xec/0x218
[   46.236285]  driver_register+0x68/0x138
[   46.236289]  __pci_register_driver+0x54/0x68
[   46.236292]  nouveau_drm_init+0x24c/0xff8 [nouveau]
[   46.236427]  do_one_initcall+0x64/0x288
[   46.236431]  do_init_module+0x60/0x230
[   46.236435]  load_module+0x1c70/0x1f08
[   46.236438]  __do_sys_init_module+0x188/0x1f8
[   46.236441]  __arm64_sys_init_module+0x24/0x38
[   46.236444]  invoke_syscall+0x50/0x120
[   46.236448]  el0_svc_common.constprop.0+0x48/0xf0
[   46.236452]  do_el0_svc+0x24/0x38
[   46.236455]  el0_svc+0x30/0xd0
[   46.236459]  el0t_64_sync_handler+0x10c/0x138
[   46.236462]  el0t_64_sync+0x198/0x1a0
[   46.236465] ---[ end trace 0000000000000000 ]---

I've added links to the two cards I've tested. I will also try the open source driver if I get some time.