Desktop freezes after driver for secondary GPU gets replaced

Desktop
,
1

Hi all!

I’ve run into a situation that made my Gnome desktop freeze in a reproducible manner. Last journal entries before the freeze showed some gnome-shell errors. I have found a workaround that fixes the problem for me, but getting your whole desktop frozen is not a nice experience.

I’d like to report this as a bug, but I just don’t know where to start. So, let me share a superficial description of the problem. If anyone here thinks I should report a bug, I’ll share more information in the bug report…

I’m using a laptop with dual GPUs: integrated Intel Raptor Lake-P and discrete NVIDIA AD107M. I have not been using the discrete Nvidia GPU much – my desktop and all apps run on the integrated Intel GPU. But recently I started tinkering with GPU intensive workloads in a VM. Its a Qemu VM (managed by Incus) and I’m using vfio to pass the discrete Nvidia GPU into the VM. All good so far, with graphics working both inside and outside the VM. But each time I stopped the VM, my whole desktop just froze. Had to trigger a hard shutdown through the power button.

I tried to dig deeper:

When the VM starts, Incus (or libvirt, qemu, etc) unloads the Nvidia GPU’s driver (i.e. the nouveau kernel module) and loads the vfio-pci driver instead. Then, when the VM stops, it does the opposite: it unloads the vfio-pci driver and reattaches the nouveau driver to the GPU. This creates new device files, like /dev/dri/card2 and /dev/dri/renderD130. It looks like gnome-shell watches for this kind of devices and tries to react when new ones appear. I suspect that this could be the root cause of my Gnome desktop freezing, because I get the following log events just before the freeze:

$ journalctl --boot -1 --no-pager | grep -iE "nouveau|vfio|drm|gpu"
[...]
Jn 02 17:04:34 deepthought kernel: [drm] Initialized nouveau 1.4.2 for 0000:01:00.0 on minor 0
Jun 02 17:04:34 deepthought kernel: nouveau 0000:01:00.0: [drm] No compatible format found
Jun 02 17:04:34 deepthought kernel: nouveau 0000:01:00.0: [drm] Cannot find any crtc or sizes
Jun 02 17:04:34 deepthought gnome-shell[6772]: Failed to hotplug secondary gpu '/dev/dri/renderD130': GDBus.Error:System.Error.ENODEV: No such device
Jun 02 17:04:34 deepthought gnome-shell[6772]: Added device '/dev/dri/card0' (nouveau) using non-atomic mode setting.
Jun 02 17:04:34 deepthought gnome-shell[6772]: Failed to ensure KMS FB ID on /dev/dri/card1: drmModeAddFB2WithModifiers failed: Inappropriate ioctl for device
Jun 02 17:04:35 deepthought gnome-shell[6772]: Failed to ensure KMS FB ID on /dev/dri/card1: drmModeAddFB2WithModifiers failed: Inappropriate ioctl for device
[...]

So far, I’ve only discussed this topic with an LLM assistant. And I have little knowledge of Gnome architecture and code-base. So, what do you think? Worth reporting this as a bug in Gnome? If so, which component?

I found similar log messages mentioned in mutter issues (e.g. #3177), but the details look different. If you think that my setup is just too weird, feel free to ignore this post.

Cheers,
Michael

2

Quick follow-up, regarding the workaround. This works for me, but your mileage may vary:

I’ve simply told Gnome to ignore the discrete Nvidia GPU. I did so by dropping the following udev rules in /lib/udev/rules.d/99-nouveau-noaccess.rules:

SUBSYSTEM=="drm", KERNEL=="card[0-9]*", DRIVERS=="nouveau", GROUP="root", MODE="0600", TAG-="uaccess", TAG-="seat", TAG-="master-of-seat"
SUBSYSTEM=="drm", KERNEL=="renderD*", DRIVERS=="nouveau", GROUP="root", MODE="0600", TAG-="uaccess", TAG-="xaccess-render"

I think that the seat tags are the most relevant part here.

In theory, block-listing the nouveau kernel module (and all other nvidia drivers) would also solve my problem. However, turns out that my nvidia GPU needs a driver to be loaded, just for basic power management. When no driver is loaded, the GPU just runs incredibly hot, even though not being in use at all.