I was excited to see that Mutter 3.34 can experimentally give itself a realtime priority. I’ve had a lot of situations where a rogue app or userspace process has managed to lock up my desktop system completely. After one of these I resolved to enable the rt-scheduler feature and see if it helped. And what do you know - it does!
I’m not sure if this has been fixed or not, but currently to use the rt-scheduler you must set the CAP_SYS_NICE capability in the binary, and it has the effect that the GVFS modules in GLib will stop working (in the sense that they will return a GDummyFile instead of a GFile object. It is a security feature, and seems to be fully on-purpose). Also storing extended attributes in files won’t work. With plain Gnome Shell it isn’t a problem, but if you install extensions that make use of GVFS (like Desktop Icons) you will have problems.
What I’ve noticed the rt-scheduler stuff has not fixed for me is massive I/O stalls. You can fairly trivially setup fio to run a bunch of sync I/O with 4k block size, 1gb files, and an iodepth of 4 to replicate the issue (using “seq-write” operations).
What does work, however, is ensuring that fio script runs in a user slice with MemoryLimit= to reduce how much page cache it steals.