Hi everyone,
I’ve been building an open-source tool that lets ChatGPT control my Linux desktop with the mouse and keyboard. I wanted it to work with native applications, beyond what browser automation can do.
The latest demo is a music session in LMMS: opening the app, choosing instruments, entering notes, and playing the result. There’s also an earlier drawing demo in the README.
It uses XDG Desktop Portal for input and screen sharing, a GNOME Shell extension for window context, and a native helper to detect keyboard and touchpad activity. When I take over, automation pauses, then checks the current screen before continuing.
It’s an early preview targeting GNOME Shell 51 on Wayland. Most of the recent work has been on handling everyday interruptions - switching windows, changing workspaces, or touching the trackpad - without ending the session.
I’m sharing it here because I’d like feedback on the GNOME integration, particularly whether there are simpler ways to handle these transitions. Reports from anyone trying it on different hardware would also help.