Querying top panel widget positions for automation?


Reposting my question here, since it looks like the mailing lists are almost dead.

Is there any way to programmatically lookup the position of widgets displayed in the Gnome top panel, as well as UI elements within those widgets?

I’m trying to build a UI test automation framework for some Gnome Shell panel widgets. I’d like to write a script that can click on a panel widget, confirm a popup appears, verify certain text exists in the popup, etc.

I maintain a Gnome Shell extension, and it’s become immensely expensive, not to develop, but to test and maintain. There are so many different distros with varying incompatible Gnome versions, and testing them all by hand is virtually impossible. Fixing a bug for one Gnome version introduces a bug in another version. It’s gotten to the point where I almost have to abandon the project because it’s too difficult to do basic integration and regression testing on Gnome, but I really want to find a solution.

Gnome is one of the most widely used desktop environments in the world, so I find it hard to believe no one else has encountered this problem before.

I’ve found some tools like xdotool and pyautogui for programmatically clicking elements. However, without knowing exactly where widgets are rendered, the script would be clicking blind. And even if I could reliably click a location, those tools don’t let me read DOM values for the UI elements at that location.

I’ve also found some tools like xwininfo that can query some window positions and do a little introspection, but this seems specific to X and treats the entire top panel as one huge window, with no ability to introspect any of the widgets inside of it.

I had hoped the Dogtail project might be helpful in this regard, but it seems to be dead with virtually no documentation. I was able to build it, but I couldn’t find any ability within it to introspect the top panel.

Are there any other tools or libraries I could potentially use? Is there any sort of undocumented Javascript API that would allow me to query Gnome Shell extension widget meta data and position information?

Does anyone have any alternate suggestions for how I might automate Gnome UI testing?


1 Like

Your best bet I think is probably to do this inside gnome-shell. AFAIK the Clutter.VirtualInputDevice seems to be stable and you should be able to get the coordinates from the widget itself.

If necessary to call from outside the process you could probably export a simple DBus interface, or if you already know the element’s expected location use the existing org.gnome.Mutter.RemoteDesktop interface to interact with them.

Is there any sort of undocumented Javascript API that would allow me to query Gnome Shell extension widget meta data and position information?

It would really help people to grapple with the fact that GJS is just JavaScript bindings for C libraries. It seems like a lot of folks go looking for magical JavaScript APIs that just don’t exist and inevitably end up frustrated. Look for the APIs you need on gjs-docs.gnome.org because most everything in GNOME Shell is just convenience wrappers and subclasses written in those APIs.

That’s the problem I’m trying to avoid. I’m sure if I spent months and months, I could built on a very fancy system in my extension to publish every bit of widget meta data I need. But I’m a volunteer, this isn’t the only project I’m working on, and I don’t have time for that level of development. However, I do have time to write some simple user story scripts, but only if Gnome supports this functionality already.

We have gnome-shell-perf-tool --perf=basic, but the --perf argument can only refer to modules in gnome-shell’s search path. Extending that to support external scripts would be a welcome addition, but I suspect that it’s a lot easier to wait for a port to EcmaScript modules (because there imports are expressed as file paths).

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.