Image Viewer (Loupe) uses glycin to load images. That’s a new library to decode images. It was developed initially for safer and better image loading in Image Viewer.
Gnome Shell uses Mutter to load images which in turn uses GdkPixbuf, what the previous image viewer (Eye of GNOME / eog) also used. Here is where the file gets loaded in Mutter: src/compositor/meta-background-image.c · main · GNOME / mutter · GitLab
IDK a lot about this but for the >8 bpc image problem the cause could be with how GdkPixbuf loads files, or if that supports >8 bpc images fine the cause could be with how Mutter displays such images. I’m not sure.
Perhaps one way to narrow down on that is to try those images also with the previous image viewer and compare (as that also uses GdkPixbuf to load images, same as Mutter uses). If that shows the same problem as in Gnome Shell, more likely it’s a GdkPixbuf issue. If that works fine same as the new Image Viewer (Loupe), more likely it’s a Mutter issue. The previous image viewer can still be installed on Fedora 40, the package name is eog.