Python multiprocessing

[jeffb]

Do I have it right that it is not possible to use the Python multiprocessing module with Gtk? I get horrible errors (e.g., “gtk_text_view_validate_onscreen: assertion failed” and “Fatal IO error 11 (Resource temporarily unavailable) on X server”) whether I use spawn or fork as the start method.

The multiprocessing module makes it possible to run a function in a separate process, whereas Gio.Subprocess requires that I run a separate program. Getting results back from a separate program is more complicated. Does Gio provide a way to run functions as a subprocess which I have not found?

Do I have it right that the OS is free to run subprocesses created by Gio.Subprocess on separate cores?

[j_arun_mani]

I haven’t used multiprocessing but I had used in many cases threading. Of course, we need to take care of serious issues of race-cases. I always try to make the thread as isolated as possible with only limited way to interfere with Gtk’s loops.

There are also good support from Gio to call functions in unblocking way.

• async variants of methods.
• Gio.Task

If the function you wish to run separately will be alive as long as your application (like a server in background), you may use DBus and bridge the app and service using DBus mechanisms. Read Minimal example of GDBus in Python for more information.

Recommended read : https://pygobject.readthedocs.io/en/latest/guide/threading.html

Apart from this, if you “strict” your separate process to not mingle directly with UI, it may help in avoid some “unexpected” errors. You can show us a minimal example that demonstrates the error, so we can help in resolving it, if possible.
Hope it helps.

[jeffb]

Thanks for your reply.

Python threading does not take full advantage of multiple cores because there is only ever one Python interpreter and the GIL assures that only one thread is running at a time. There is also GLib.thread, but I do not see how it could circumvent this limitation.

As you noted in another thread, using DBus is complex. That complexity would be hard to justify for one specific application. Pyro would be easier, but it still introduces complexity that I was hoping to avoid.

I solved the problem by running a work server using Gio.Subprocess. It idles waiting for something to run. I send the function that I want to run to the work server through its input stream along with a callback for reporting the result.

[cwendling (Colomban Wendling)]

While apparently the OP has found another solution for the problem, I just wanted to mention that it definitely is possible to use multiprocessing with GTK. It has several gotchas, but if properly done it works fine.

Note that (from my experience):

• You must use the spawn multiprocessing context. fork will lead to unrecoverable wreckage similar to the one you mentioned (one way to accept this is knowing that GTK might spawn threads, and fork context will not work in that situation – but actually bare X11 won’t either, for other reasons).
• Do not call anything outside if __name__ == '__main__' block. This basically applies to any multiprocessing usage, though. If you do call anything at the root scope, who knows what will happen (hint: nothing too good).
• You cannot call any GTK methods on any objects from another process. It’s kind of obvious if you think about it, as they do not share anything, the GTK from one process doesn’t know anything about the state or objects of the one of another, just like the one from any separate apps. If you need to pass data, use pipes or multiprocessing shared objects for that (but not GTK objects).
• …and possibly others I don’t think of from the top of my head.

Anyway, here’s a basic example showing it actually just works, similar to running “normal” separate processes:

test-gtk-multiprocessing.py

#!/usr/bin/env python3
# -*- coding: utf-8; -*-

import multiprocessing

import gi
gi.require_version('Gtk', '3.0')
from gi.repository import Gtk


def on_btn_new_win_clicked(button=None):
    proc = multiprocessing.get_context('spawn').Process(target=create_window,
                                                        args=("Subprocess Window",))
    proc.start()

def create_window(title):
    w = Gtk.Window(title=title)
    w.connect('destroy', Gtk.main_quit)

    box = Gtk.Box(orientation=Gtk.Orientation.VERTICAL, spacing=6)
    w.add(box)

    sw = Gtk.ScrolledWindow(child=Gtk.TextView())
    box.pack_start(sw, True, True, 0)

    btn = Gtk.Button(label="_Open new window", use_underline=True)
    btn.connect('clicked', on_btn_new_win_clicked)
    box.pack_start(btn, False, True, 0)

    w.show_all()
    Gtk.main()

def main():
    create_window("Main Process Window")


if __name__ == '__main__':
    main()