Set location of cursor in extension to a fixed place for visually impaired people

[Janghou]

I’m trying to set my first steps in a Gnome Extension to help visually impaired people with a very narrow field of vision to find the cursor again by moving to it a fixed location. (top/left, or center)

Imagine someone with glaucoma, they are looking through a straw, so they have to scan the screen from left top to right bottom (or sometimes even a sentence character by character).

I’m able to bind a keyboard shortcut to read the cursor location, but I can’t find a way of moving it.

let [mouse_x, mouse_y, mask] = global.get_pointer();
global.set_pointer(0,0) // fails

I hacked already a workaround together with dotool, but I think that a Gnome Shell Extension is more secure and appropriate.

And honestly it were my first hours with a Gnome Extension, but I fail to see how proper debugging is done.

dbus-run-session -- gnome-shell --nested --wayland is fun, but I had a typo in ExtensionUtils.getSettings("org.gnome.shell.extensions.test"): no error message was shown.

And I found out the hard way that I should run each time glib-compile-schemas ~/.local/share/gnome-shell/extensions/test@example.com/schemas on a schema change.

And for example where can you find info about the global object?

Ubuntu 22.04 so shell 42

[andyholmes]

@Janghou

I’m able to bind a keyboard shortcut to read the cursor location, but I can’t find a way of moving it.

I would start by looking at how the keyboard emulates keypresses in keyboard.js, and do the same thing for the mouse.

@Janghou

And honestly it were my first hours with a Gnome Extension, but I fail to see how proper debugging is done.

dbus-run-session -- gnome-shell --nested --wayland is fun, but I had a typo in ExtensionUtils.getSettings("org.gnome.shell.extensions.test"): no error message was shown.

I’m not what the typo was, or what error message you were expecting. You’ll have to be more specific about what sort of debugging you’re looking for, or what you were expecting as output

@Janghou

And for example where can you find info about the global object?

You can find it under Shell.Global in the API documentation.

[Janghou]

Andy, thank you, that is a big help.

My fault about the debugging, my inexperience, I checked this morning, the info was there (typo "org.gnome.shell.extensions.tets") , I had to scroll some screens up in the terminal.

The output feels polluted by (a lot of) other messages, like gsconnect complaining: Address already in use (BTW thank you for writing that extension, I really like it.)

Apparently GSConnect doesn’t like running twice. Can installed extensions be disabled in a nested shell instance?

Anyhow it’s boils down to filtering the terminal output, which is a big help:

dbus-run-session -- gnome-shell --nested --wayland 2>&1 | | grep "test@example.com \| Shell-Message"

[Janghou]

I’m stuck again.

   in

 Main.wm.addKeybinding(
    "fix-mouse",
    settings,
    Meta.KeyBindingFlags.IGNORE_AUTOREPEAT,
    Shell.ActionMode.NORMAL | Shell.ActionMode.OVERVIEW,
    //     this._toggletest.bind(this)
    () => {
      let [mouse_x, mouse_y, mask] = global.get_pointer();
      console.log(mouse_x, mouse_y, global);

      this._seat = Clutter.get_default_backend().get_default_seat();

      this._virtualDevice = seat.create_virtual_device(
        Clutter.InputDeviceType.POINTER_DEVICE
      );
      this._virtualDevice.notify_absolute_motion(
        Clutter.get_current_event_time() * 1000,
        0,
        0
      );

      console.log("test");
    }
  );

Also I looked at this extension:

Release 5 · LeonMatthes/mousefollowsfocus

It does this:

  // pointer.warp(screen, 10, 10);
  // this._seat.warp_pointer(100, 1000);

Couldn’t get that working either.

Any tip?

[andyholmes]

Sorry, you’ll have to explain what problem you’re having, and what you expected to happen.

[Janghou]

Thank you Andy for your response.

TL;DR

a function to set the location of the cursor to a fixed place on a key-press

global.set_pointer(50%,50%) // (bogus code to work)

Maybe I did not explain myself properly but I try to help people with a very narrow field of vision. They often cant’f find the cursor. So I wanted to create an Gnome extension that moves the cursor to the center of the main monitor (or any configurable location) on a global shortcut.

Dead simple. Like I written before I have a sub-optimal solution with the external dotool.

When I lose track of the mouse myself I simply move it left, pick it up and move it left again, until it’s on left top, but it seems that older/disabled people are reluctant to move their mouse when the don’t see where it is.

( And I agree, every JS driven website can bind any event to a mouse movement nowadays.)

[fmuellner (Florian Müllner)]

@Janghou

this._seat.warp_pointer(100, 1000);

That’s supposed to work.

Note that to move the pointer to the center of the monitor, you’ll have to account for the offsets of the monitor itself:

    _warpPointer() {
        const {primaryMonitor} = Main.layoutManager;
        const seat = Clutter.get_default_backend().get_default_seat();
        
        seat.warp_pointer(
            primaryMonitor.x + Math.round(primaryMonitor.width / 2),
            primaryMonitor.y + Math.round(primaryMonitor.height/ 2));
    }

[Janghou]

Thank Florian, I will give it a try.

Pardon my ignorance but the more I read the less I understand.

Clutter is going to be retired, so to be future-proof I need to switch to GTK soon, but as it seems GTK4 does not support warping the pointer- because it’s confusing to the user (nonsense would I say, I’m just presenting the usecase here)- so I should use platform APIs such as XWarpPointer() directly, which sounds like it does not work on Wayland at all.

So although I can get my extension working in Ubuntu today, it’s doomed for tomorrow?

Or can I use XWarpPointer() (or any alternative) in GNOME 42+ Wayland right now.

[fmuellner (Florian Müllner)]

@Janghou

Clutter is going to be retired, so to be future-proof I need to switch to GTK soon

No, that is wrong.

The stand-alone Clutter library (for use in apps) is deprecated and shouldn’t be used anymore.

But that doesn’t apply to the fork of Clutter that is included in mutter and used by gnome-shell.

Furthermore, there are no plans on the GTK side to make it usable inside a compositor.

@Janghou

nonsense would I say, I’m just presenting the usecase here

But your use case is a global shortcut to move the pointer.

GTK is a toolkit for graphical apps, i.e. windows that contain contents and controls that allow the user to interact with it.

Your use case does not fit that at all, so it seems fair enough for GTK to not cover it.

(On a side note: Even if GTK had kept a “warp-pointer” abstraction, it doesn’t have API to register a global shortcut either. So your use case wouldn’t be covered anyway)

You want to effectively implement a compositor feature, and as the feature doesn’t exist out of the box, an extension is a good option.

[Janghou]

Thank you, my misunderstanding.

[Janghou]

@fmuellner

@Janghou

this._seat.warp_pointer(100, 1000);

That’s supposed to work.

Yes, but for some reason it does not in a nested shell.

• dbus-run-session – gnome-shell --nested --wayland

Still couldn’t get it working, so I tried it globally, and it is working, the warping the cursor part is just not working in a subshell, which I considered a test-environment. Why is that?

[fmuellner (Florian Müllner)]

The nested shell does not grab input, so the global keyboard shortcut is handled by the host (where it does nothing).

[Janghou]

Not sure about that, because I’m logging the cursor position just before warping, and the logging is shown in the terminal after I press the shortcut.