On a clean and fully updated install of Fedora 44 with Gnome 50, Orca Mouse Review does not work. There is now no option to go back to X11 where it works. My understanding is that fixing this is blocked on a new protocol being implemented in UI toolkits and compositors, is that correct? Does anyone know if Mutter supports it yet?
There are two protocols involved: a DBus protocol for the compositor to
communicate with the screen reader and a Wayland protocol for applications
to communicate with the compositor. Mutter implements the former, but only
gtk 4 implements the latter currently.
I realized recently that I’d overlooked that mutter sends the pid of the
process with the window if the application does not implement the Wayland
protocol, so libatspi/orca might be able to look up the window that way.
Looking into that is on my list of things to do, hopefully before GNOME
50.2 (release is scheduled for two weeks from tomorrow).
I had a go with the latest at-spi2-core-2.60.4-1.fc45.x86_64. Incredible work, Firefox now works with mouse review again!
However everything is offset by a few centimetres from the actual mouse position. For example in the picture below, my mouse is located 2cm below, and 4cm to the right of, the “a new protocol” link from my original post. Nevertheless, Orca thinks my mouse is over that link, and reads it when I move the mouse around a little, or if I click. Near the top left of the screen it is very accurate and it degrades as I move towards the bottom right. I suspected scaling, hidpi, something like that, but even after changing “scale” in Display Settings from 125% to 100% there was no change.
Sorry for putting a bug report here but I thought it’s the simplest way to get your attention before it ships too widely. Happy to put it in a formal tracker too if that’s helpful.
It’s off in the Settings app too, although not as noticeable.
My guess is that the pointer coordinates that mutter is sending to
libatspi/orca aren’t matching the application’s notion of its coordinates.
From your description, it sounds like scaling might not be taken into
account. It would be helpful if you could file a work item at
GNOME / mutter · GitLab (I’m not 100% sure that that is the
correct place, but I would start there); I’m not sure who is reading this
thread, and it would make the issue easier to track. I’m also curious if
this happens with gtk 4 applications.
Thanks! I will open an issue there shortly. It seems there may be two issues at play. I tried turning scaling off and rebooting this time, and Firefox now works perfectly. However, everything in Settings, Nautilus, is offset by about 1.5cm in vertical direction only. So Firefox is broken by scaling while Settings has another problem. I guess that the Firefox bug is definitely Mutter’s fault while the Settings is uncertain. I do suspect that providing incorrect mouse review information is worse than nothing at all.
It gets stranger. I tried in Blanket, a GTK4 app. Everything is offset both vertically and horizontally by 0.5cm (scaling off) but the popup menu in Blanket (containing “hide inactive sounds” etc) works perfectly. I’ll put all this in the Mutter issue when I get round to it, mainly noting so I can remember.
Opened Accessible mouse pointer callback does not respect scaling (#4833) · Issues · GNOME / mutter · GitLab . Would the offset bug in Blanket and gnome-control-center need to be reported to gtk, rather than mutter? They certainly aren’t related to scaling.
I’ve been looking a bit deeper and I think there are at least four separate bugs here. I’ve set display scaling to 100% and rebooted but the following bugs remain:
- Nautilus is misaligned whether fullscreen or windowed. There is a consistent vertical offset of a few pixels throughout the application when maximised or about 30 pixels when windowed.
- A very simple GTK4 app works perfectly when maximised but is misaligned in windowed mode.
- Firefox is aligned correctly when fullscreen but misaligned when windowed. But it’s misaligned in the opposite direction to the GTK apps. What??? I feel like my computer is just making fun of me at this point.
And of course, Firefox remains misaligned when display scaling is active. I’m not even sure how to figure out which components are responsible for these bugs. I think there’s probably at least one in mutter, one in gtk and one in firefox, but honestly at this point who knows. I’ll try to get a proper dump of the relevant atspi trees along with the bounding boxes, and the cursor coordinates reported by mutter.
Okay, I think I’m onto something. On certain windows with a shadow, mutter reports the top of the shadow has a Y coordinate of 13 while the bottom of the shadow has a Y coordinate of 25. I think something (mutter?) is adding the shadow’s offset in the wrong direction (should be subtracting).
Here is a list of what mutter reports as the cursor offset for each visual pointer location. “Start of content” is determined by when my cursor changes from a resize operation to a click operation. All in windowed mode.
Windows where the shadow starts at 13 and content starts at 25:
- Firefox
- Nautilus (GTK4)
- Gnome-control-center (GTK4)
- Ptyxis (GTK4)
Windows where the shadow starts at 13 and the content starts at 23:
- Eye of Gnome (GTK3)
Windows where the shadow starts at 0 and the content starts at 12:
Windows where there is no shadow (but the notional shadow area starts at 0) and the content starts at 11:
- KDE Accessibility Inspector app
- CoMaps (Qt)
When in fullscreen, all the apps have a top coordinate of 0, as I would expect.
I had a look at the outline in Looking Glass and visually both the gtk-rs sample and Ptyxis have the same padding so I’m not sure what’s different.
I’m happy to keep looking into this but I don’t really know whose bug this is (or even what the expected behaviour is on a low level) or what else I should check.
So I kept looking at this and I think the attached patch “solves” most of the issues (excluding Firefox under display scaling - so I guess that wasn’t a mutter bug and maybe belongs in the Firefox tracker?). However, this is very much not the right approach. It feels wrong for Mutter to be accounting for decorations, and I think that ATK/GTK/other frameworks should be accounting for this when drawing the client-side window decoration in the first place (i.e. the root frame in the a11y tree should have a non-zero offset against the window). Especially as Mutter has no mechanism (that I saw) to be notified of changes to decoration that does not result in a resize of the surface, so this patch is fragile to application-side behaviour.
diff --git a/src/compositor/compositor.c b/src/compositor/compositor.c
index 4065c51473..0149021949 100644
--- a/src/compositor/compositor.c
+++ b/src/compositor/compositor.c
@@ -1723,6 +1723,10 @@ meta_compositor_query_pointer_a11y (MetaCompositor *compositor,
(int) rel_coords.y,
&rel_x, &rel_y);
+ // Translate to match the a11y tree - though maybe this should be handled by the client??
+ rel_x -= window->custom_frame_extents.left;
+ rel_y -= window->custom_frame_extents.top;
+
pid = meta_window_get_pid (window);
if (meta_window_get_a11y_properties (window,
This is presumably an underspecified part of the a11y tree because it has never been an issue until Wayland (decorations were server side) and it feels like asking applications to offset their entire a11y tree might not go down well with people with custom toolkits. I don’t really know what the answer is here but I think it needs collaboration from multiple parties, and I don’t know how to get them on board.
Thanks for tracking this down. It might be worth filing an issue against
gtk then. I’m not 100% sure where the fix belongs, but we need to decide
whether to patch the compositor or to adjust the toolkit at-spi/atk
implementations.
Created, linking for anyone who comes across this through search: Consider adding client shadow offset to a11y tree coordinates (#8242) · Issues · GNOME / gtk · GitLab and Clarification: client side decoration (#232) · Issues · GNOME / at-spi2-core · GitLab
Thanks for all the help by the way! Keep it up 
For anyone who needs a quick fix: either use the app in fullscreen or run this code:
flatpak override --filesystem='~/.config/gtk-4.0:ro'
echo '.background.csd{margin:0;box-shadow:0 0 0 transparent}' >> ~/.config/gtk-4.0/gtk.css
It should sort things out by disabling shadows. To get Firefox to work correctly you will also need to set scaling to 100% in Display Settings - but this will make everything on the display smaller.
This topic was automatically closed after 90 days. New replies are no longer allowed.