On a clean and fully updated install of Fedora 44 with Gnome 50, Orca Mouse Review does not work. There is now no option to go back to X11 where it works. My understanding is that fixing this is blocked on a new protocol being implemented in UI toolkits and compositors, is that correct? Does anyone know if Mutter supports it yet?
There are two protocols involved: a DBus protocol for the compositor to
communicate with the screen reader and a Wayland protocol for applications
to communicate with the compositor. Mutter implements the former, but only
gtk 4 implements the latter currently.
I realized recently that I’d overlooked that mutter sends the pid of the
process with the window if the application does not implement the Wayland
protocol, so libatspi/orca might be able to look up the window that way.
Looking into that is on my list of things to do, hopefully before GNOME
50.2 (release is scheduled for two weeks from tomorrow).
I had a go with the latest at-spi2-core-2.60.4-1.fc45.x86_64. Incredible work, Firefox now works with mouse review again!
However everything is offset by a few centimetres from the actual mouse position. For example in the picture below, my mouse is located 2cm below, and 4cm to the right of, the “a new protocol” link from my original post. Nevertheless, Orca thinks my mouse is over that link, and reads it when I move the mouse around a little, or if I click. Near the top left of the screen it is very accurate and it degrades as I move towards the bottom right. I suspected scaling, hidpi, something like that, but even after changing “scale” in Display Settings from 125% to 100% there was no change.
Sorry for putting a bug report here but I thought it’s the simplest way to get your attention before it ships too widely. Happy to put it in a formal tracker too if that’s helpful.
It’s off in the Settings app too, although not as noticeable.
My guess is that the pointer coordinates that mutter is sending to
libatspi/orca aren’t matching the application’s notion of its coordinates.
From your description, it sounds like scaling might not be taken into
account. It would be helpful if you could file a work item at
GNOME / mutter · GitLab (I’m not 100% sure that that is the
correct place, but I would start there); I’m not sure who is reading this
thread, and it would make the issue easier to track. I’m also curious if
this happens with gtk 4 applications.
Thanks! I will open an issue there shortly. It seems there may be two issues at play. I tried turning scaling off and rebooting this time, and Firefox now works perfectly. However, everything in Settings, Nautilus, is offset by about 1.5cm in vertical direction only. So Firefox is broken by scaling while Settings has another problem. I guess that the Firefox bug is definitely Mutter’s fault while the Settings is uncertain. I do suspect that providing incorrect mouse review information is worse than nothing at all.
It gets stranger. I tried in Blanket, a GTK4 app. Everything is offset both vertically and horizontally by 0.5cm (scaling off) but the popup menu in Blanket (containing “hide inactive sounds” etc) works perfectly. I’ll put all this in the Mutter issue when I get round to it, mainly noting so I can remember.
Opened Accessible mouse pointer callback does not respect scaling (#4833) · Issues · GNOME / mutter · GitLab . Would the offset bug in Blanket and gnome-control-center need to be reported to gtk, rather than mutter? They certainly aren’t related to scaling.
I’ve been looking a bit deeper and I think there are at least four separate bugs here. I’ve set display scaling to 100% and rebooted but the following bugs remain:
- Nautilus is misaligned whether fullscreen or windowed. There is a consistent vertical offset of a few pixels throughout the application when maximised or about 30 pixels when windowed.
- A very simple GTK4 app works perfectly when maximised but is misaligned in windowed mode.
- Firefox is aligned correctly when fullscreen but misaligned when windowed. But it’s misaligned in the opposite direction to the GTK apps. What??? I feel like my computer is just making fun of me at this point.
And of course, Firefox remains misaligned when display scaling is active. I’m not even sure how to figure out which components are responsible for these bugs. I think there’s probably at least one in mutter, one in gtk and one in firefox, but honestly at this point who knows. I’ll try to get a proper dump of the relevant atspi trees along with the bounding boxes, and the cursor coordinates reported by mutter.
Okay, I think I’m onto something. On certain windows with a shadow, mutter reports the top of the shadow has a Y coordinate of 13 while the bottom of the shadow has a Y coordinate of 25. I think something (mutter?) is adding the shadow’s offset in the wrong direction (should be subtracting).
Here is a list of what mutter reports as the cursor offset for each visual pointer location. “Start of content” is determined by when my cursor changes from a resize operation to a click operation. All in windowed mode.
Windows where the shadow starts at 13 and content starts at 25:
- Firefox
- Nautilus (GTK4)
- Gnome-control-center (GTK4)
- Ptyxis (GTK4)
Windows where the shadow starts at 13 and the content starts at 23:
- Eye of Gnome (GTK3)
Windows where the shadow starts at 0 and the content starts at 12:
Windows where there is no shadow (but the notional shadow area starts at 0) and the content starts at 11:
- KDE Accessibility Inspector app
- CoMaps (Qt)
When in fullscreen, all the apps have a top coordinate of 0, as I would expect.
I had a look at the outline in Looking Glass and visually both the gtk-rs sample and Ptyxis have the same padding so I’m not sure what’s different.
I’m happy to keep looking into this but I don’t really know whose bug this is (or even what the expected behaviour is on a low level) or what else I should check.