Injecting arbitrary unicode characters

Sometimes, it is desirable for a remote desktop client to send unicode characters rather than keycodes for input injection. This is especially true for mobile clients, where the system input method may not resemble a physical keyboard at all for some languages, and there are many additional characters (e.g., emoji) available regardless of the language. Even if the user is just typing writing-system characters and the host is set to the same language, it’s still possible that characters don’t correspond 1:1 to a key code (e.g., if typing the character on the host would involve an IME).

Under X11, a remote desktop tool could accept arbitrary unicode characters by first checking to see if the desired character was already in the key map, and, if not, temporarily adding it to an unassigned key code, injecting the key, and then removing the temporary mapping. Indeed, the X11 implementation of NotifyKeyboardKeysym in Mutter will do just this, adding the key to the key map if needed on key down and removing it again on key up.

With the Wayland backend, I don’t see a similar option. With libei, it appears that it is only possible to send key codes. Further, it appears that it is the server, not the client, that determines the keymap for the virtual keyboard, so the client is limited to sending characters that are in the server-provided keymap. And while NotifyKeyboardKeysym is still available on Wayland, it appears that the Wayland implementation in Mutter will fail to inject the keysym if it’s not already in the key map, rather than temporarily allocating a key code for it like the X11 version.

I’m wondering if there’s an approach I’m missing, if it would make sense to add the temporary-mapping functionality to Wayland, or whether a completely different approach would be preferred (injecting characters via IBus, maybe?).

I see there’s an experimental text input Wayland protocol supported by Mutter. Perhaps an option would be to extend the remote desktop interface to allow the remote desktop tool to serve as an IME, opening up the possibility to inject text, and also seemlessly use a client-side IME.

cc @halfline, @jadahl, and @joantolo, who were involved in previous remote desktop discussion.

It seems like for now, our best bet for allowing the use of a client-side IME is probably to have the remote-desktop host process connect to IBus, register itself as an engine, and then set it self as the current engine while the remote user is composing text.

In the future, it would be nice to provide support for a remote desktop tool to provide IME functionality over the Remote Desktop API, as this would allow the compositor, which implements the Wayland text-input protocol, to broker IME functionality.

E.g., the compositor might keep the remote desktop tool apprised of the state of the input field, but still use the local IBus IME for key-code-based input (whether from a local or remote physical keyboard). On the other hand, if the remote desktop host is currently generating text input events, the compositor might choose to hide the local IME.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.