Widget under the cursor

I am developing an application that can inspect the GUI of gnome application (inspect any application widget like windows accessibilityinsights not only one app).
I want to know in atk, is there any way to get the widget under the cursor?

Short answer is, it’s complicated.

If you already have a handle on the window over which the cursor is, you can fairly easily use atk_component_ref_accessible_at_point(). Note that this gets the direct child of the component on which you called it which contains the child¹, so to get the actual target object you might need to cal this in a loop until there’s no more child to be found.

If you don’t have a handle on the window already, and want a desktop-wide generic “what’s the accessible under the mouse whichever the window”, ATK alone won’t be able to answer you 100%, as this depends on several details it doesn’t have like window stacking, virtual desktops, and on Wayland, absolute coordinates.
You can have a look at how Orca and Accerciser try and solve this problem to get some inspiration. Orca is leveraging libwnck, but I’m not sure how reliable this all is on Wayland.

1: In the corner case where a child’s coordinates are not inside one of its parents, it’s unclear what this call should do, but Firefox will give you the child which is a parent of the object on which the coordinates actually lie (e.g. the object on the tree leading to the leaf object you’re likely looking for). So this means you cannot trust this call to always give you an intermediate child which will always contain the given coordinates. But all this is a corner case that only applies to very specific apps (like Firefox), as in the vast majority of cases children are contained inside the bounding box of their parents.

ok, thanks @cwendling , I’ve seen Orca, But i’m not quit clear about the python code

from gi.repository import Gdk

if i use C++, which dependency library should i reference to.

That would be libgdk, kind of GTK’s platform layer. It’s gonna be part of whichever GTK binding you’re using (plain C, gtkmm, etc.)

I want to know can I use GTK or GTKMM instead of ATK to get the widget under the cursor?

You can only get the widget under the cursor for your own application.

If you are trying to get the widget under the cursor for any random application, then you really can’t.

Windows accessibility api provide function to do this, is there any plan to support this?

Significant portions of the accessibility stack need to be redesigned for Wayland for a number of reasons, one of them is that libwnck does not work there. This has not happened yet and I have not seen any concrete plans. I would love to see it too but it is a large amount of work.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.