My Accessibility Stack and the future on Wayland

Desktop
1

Hi! Sorry for dropping a long winded essay in here with a brand new account, I’ve just not interacted with GNOME much before. But given that this is an ecosystem wide problem I figured it’s probably best to post here in addition to my home land of KDE.

So please take a look, as I have a lot of things to say about the current status of Wayland accessibility, and how I’m already no longer able to use GNOME, as X11 support was recently removed.

Let me know what you think, and if there’s anyone in particular I should talk to in the GNOME Foundation? Again, this isn’t my side of the pond, so I’m posting here in the hopes of casting a broad net and getting in front of the right pair of eyes.

2

Hello and welcome to the forum. Basically all current accessibility struggles are known. They are being worked on, but it takes time, coz no company really invests in accessibility

3

Its always a good idea to get feedback on accessibility especially on the wayland side, though I have a few nits:

The most basic task at hand-- inputting text into the system-- seems to simply be impossible in a truly “Wayland” way

input_method_v2 and text_input_v3 are already there and implemented in every major compositor sans GNOME, GNOME is a holdout in not implementing input_method_v2 but that is an issue with GNOME, Wayland already has all the protocol bits set up.

mouse positioning

There are pointer warp protocols implemented by all major compositors, would you mind expanding on what you mean?

screen reading, and so on and so forth… Wayland, as an ecosystem, supports NONE of this.

This is being improved right now with AccessKit, furthermore I can currently go into my setting, turn on Orca and it works. Why say that wayland supports none of this?

“There’s a way to do it in GNOME, but not KDE”, or “Yeah, wlroots implemented this ages ago” is not an answer to this problem.

Well then what do you think is? plenty of protocols have already been merged into upstream wayland-protocols, the fact that any particular compositor has failed to implement this-or-that protocol is not a failing of wayland-protocols, its a failing of the particular compositor devs. The Wayland developers don’t have the power to strongarm anyone into implementing something they don’t like.

I find the examples of wayland developers a bit weak, A mindset of “If you don’t want to contribute to upstream then why are you complaining :unamused_face:” has always been a part of FOSS development in general, so I don’t see that going away any time soon, nor are people eager to see it go despite how valid the complaints are, this is not just an issue with the culture of wayland-protocols, its an issue with FOSS development in general. Still, it is a valid point to make, even if its not unique to Wayland.

I find the example of what mclasen said to be a tad confusing though, for one he does not seem to be an authority on either the direction of GTK or wayland-protocols; he has some contributions and insight of course, but presenting him as a “GTK maintainer […] who refuses to engage with the discussion [and] refers to us as “accessibility maximalists”” feels to me like presenting his word as more important than it probably is.

Per the scare quotes around “Accessibility Maximalist” when I clicked on that link I was expecting to see a comment from a more prominent gtk/gnome developer going on some kind of rant and literally calling accessibility minded people “accessibility maximalists”. But as far as I can tell the thread seems completely innocuous:

  1. Justin Wheeler pings mclasen so he can provide insight on if any of the points listed above are being worked on upstream
  2. Mclasen responds with:

  3. I don’t feel like answering this point-by-point.

    Accessibility-focused people tend to come with maximalist lists of demands (“We need to have all these very intrusive capabilities, or the system will not be accessible at all”).

    Practical day-to-day improvements on accessibility of Wayland desktops happens in the Accessibility room on the gnome matrix instance.

    The work that is happening is on two tracks in parallel:

This, at least from how im reading it, is not calling accessibility folk “accessibility maximalists”, he is only complaining about how some people like to dump a big list of demands on him and then reprimand him if he has other ideas. A “Maximalist list” in this case is essentially just a synonym for a “big list”, he is not calling people “Accessibility Maximalists”. Its not a perfect comment, he probably should’ve just left that part out, but I feel like its being presented in a very negative light in the blog post.

But the biggest issue I have with this part is that the examples are all happening in places that are only tangentially related to wayland, these are not discussions happening in wayland-protocols, thats not to say discussion outside wayland-protocols is unimportant, but if you were going to give examples of wayland devs being silly and blocking progress I’d like examples from devs who actually hold some sway in protocol discussion or have even help prevent important a11y protocols from being accepted.

So when you say something like this:

Reading these two answers, many other threads like it, and the general state of wayland-protocols was and is super demoralizing to me, someone who believed that some extra communication was all we needed. What it tells me is that Wayland as an ecosystem demands our participation, yet makes said participation either impossible […] or requiring multiple years of full-time work to move the needle ever so slightly forward […]

I don’t think its a well supported conclusion, that isn’t to say that its wrong, its entirely possible that w-p really is exactly how you describe it, and from some of the other sentences in the post you seem to have come across examples that could show how uninviting wayland development is. I just don’t think the examples you chose from mclasen and Nate are enough to support this passage in what its saying.

4

Matthias Clasen is one of the GTK team developers, and his opinion carries weight.

He’s also been working on improving the accessibility of the toolkit, so his opinion should be heeded, especially when he says that accessibility on the Linux desktop is made of incremental changes, and that refusing to engage with upstreams until everything is implemented is not a way to make things better.

5

wayland-protocols is the wrong place to land this kind of accessibility infrastructure. This isn’t a “Wayland” problem, this is a “we don’t have an accessibility API” problem. The fact that accessibility tools have historically been able to rely on X11’s lack of security to achieve their ends doesn’t mean that we actually had proper support for this use-case.

I believe that accessibility tools do need hyper-privileged access to the desktop session. We’ve encountered this already: Orca uses two special D-Bus interfaces, org.freedesktop.a11y.KeyboardMonitor and org.freedesktop.a11y.PointerLocator to hook into the desktop’s input system to register keyboard shortcuts and locate the mouse for the purposes of Mouse Review. This is a specific, purpose-built API that we could extend with the necessary functionality for other accessibility tools.

Right now this is an informal cross-desktop spec, but I propose that we should formalize it by making it a portal. This means that access to the hyper-privileged accessibility interface will be gated behind a permissions prompt, and like all the proprietary platforms we’d be able to give the user control over what access their ATs have to the system. ATs would then also be distributable via Flathub.

This takes work, and that’s a resource we simply don’t have available on demand. If you know where to find people that can do this work, we could try to find funding and make it happen.

6

I’ll respond to this point by point.

input_method_v2 and text_input_v3 are already there and implemented in every major compositor sans GNOME, GNOME is a holdout in not implementing input_method_v2 but that is an issue with GNOME, Wayland already has all the protocol bits set up.

libei seems to be brand new and doesn’t quite work right, see my linked post about xdotool. And if a protocol isn’t available in the arguably biggest DE on Linux Desktop it’s basically not available at all.

There are pointer warp protocols implemented by all major compositors, would you mind expanding on what you mean?

Absolute mouse positioning, which from what I’ve read on mailing less is absolutely not supported and never will be. Wayland in general doesn’t like the idea of absolute positioning anything, see ext-zones.

This is being improved right now with AccessKit, furthermore I can currently go into my setting, turn on Orca and it works. Why say that wayland supports none of this?

“Screen reading” as in “observing the contents of the screen” This is for example required by gaze-ocr to view and interact with the contents of the screen, which I used to write this paragraph. As before the remote desktop portal is super new and doesn’t quite seem to work right. GNOME has put tremendous effort into making remote desktop work well on Wayland but e.g. I have never gotten the KDE RDP server to work and wlroots seems like it’s never supported it.

Well then what do you think is? plenty of protocols have already been merged into upstream wayland-protocols, the fact that any particular compositor has failed to implement this-or-that protocol is not a failing of wayland-protocols, its a failing of the particular compositor devs. The Wayland developers don’t have the power to strongarm anyone into implementing something they don’t like.

Although I was referring mostly to the idea that, for instance, you can use a bespoke GNOME Javascript extension to call internal methods to accomplish certain tasks (see e.g. the xdotool article, or this article briefly documenting the requirements for Talon) I think it absolutely is a failing of wayland-protocols as a concept that the fragmentation it creates makes expectations of protocol availability unsound even if you can manage to get a protocol merged. I’ve not the foggiest idea what the solution is, seems like people much smarter than me have spent years banging their heads against the wall at it.

I just don’t think the examples you chose from mclasen and Nate are enough to support this passage in what its saying.

I will accept this argument. I don’t think it’s productive to anyone to provide a dozen individual receipts for my overall perspective gained, but I could have picked better examples.

7

Sending this comment along to someone who might be able to help. Thank you.

9

FWIW, I don’t know the specifics but according to wayland.app, Mutter supports text_input_v3 and input_method_v2 is supported by neither Mutter nor Kwin.

10

Input method v2 is supported by Kwin and applications like fcitx5 use it, however the protocol will only be exposed to clients if they’re added to KDE’s input method editor settings. The application/method that wayland.app uses to detect protocol support across compositors does not handle this edge case.