I’ve encountered some seemingly random bugs when using gjs with gstreamer. I think the probable cause was using Gst.Bus.add_watch() instead of add_signal_watch(). The former results in the callback being called from a thread other than the default/main one. Is that something that isn’t supported by gjs?
add_watch()'s callback is also called from the thread that runs the main loop on the default main context. set_sync_handler() / the sync-message signal would be called from some random thread.
Ah, so I was wrong about what thread these callbacks are called on. But I was still getting mysterious bugs and then I realised another part of my code was calling JS callbacks off the main thread. I changed that, and it seems to have fixed the problem, although it’s difficult to tell for sure, because I’m making lots of other mistakes!
If there is an introspectable API that unconditionally calls callbacks in a non-main thread, then I believe that’s a bug that ought to be fixed. I’d be surprised if JS was the only language where that went wrong.
Yeah bindings need to be able to handle that, or otherwise they will only work with the GTK-based libraries and GIO. It’s completely normal for lots of GStreamer APIs to have callbacks and signals being called from arbitrary threads, just the example above is one of the few cases in GStreamer where that’s actually not the case.
I didn’t know that this was used so pervasively in GStreamer. From my side it’s not so easily fixable though; it’s not a question of building support for it in GJS — the entire JS language doesn’t have any concept of threads.*
Maybe there is a possibility to have an annotation in G-I telling that the callback may be called off-thread, so that bindings can either decline to expose those functions, or (in the case of a callback with no return value) schedule the user-provided callback on the main thread?
[*] Well, technically it does, through WebWorkers, but there is no sharing of data between threads and I don’t believe it would be possible to use WebWorkers in the GStreamer API
Recent versions of Javascript can share memory buffers between threads, but even if that’s supported in the versions of spidermonkey widely available in “stable” distros etc, it’s probably still too limited to allow full interoperability with posix/glib threading.
The gstreamer plugin I’m developing has a support library written in C, so it’s easy enough to add extra functions to work around problems like this.
The way how this work in Python (which also doesn’t really support threads) is that each callback goes through a global mutex (in this case provided by Python itself, the Global Interpreter Lock). Maybe something similar would be doable in GJS?
I’d like to have that and such a thing was proposed many years ago for providing more useful information to the Rust bindings, but discussions went nowhere. Right now this information is externally provided in the Rust bindings.
Thread-safety annotation needed (#119) · Issues · GNOME / gobject-introspection · GitLab is the old issue for this (and various information in there is very outdated). It’s a bit more generic but also covers callbacks: in Rust terms, a callback closure would have to be Send (at least) to allow it from being called from a different thread.
1 Like
What does Python do when the callback is supposed to give a return value? Does it block the callback thread to wait for it?
What do you mean with that? Return values of callbacks are returned to the caller of the callback, return values of the function taking the callback parameter are directly returned.
Do you have an example function for the case you’re thinking of?
I don’t have an example, but here in JS-like code:
const MY_NUMBER = 42;
obj.get_number_on_some_random_thread(() => {
return MY_NUMBER;
});
JS can only execute the callback on its single thread, so it would have to block the thread that calls the callback, until the callback is able to be executed on the main thread and return the value. With Python, does the GIL cause a similar amount of blockage? It seems to me that it might, and in that case this solution wouldn’t be so bad, or at least not worse than Python.
It would continue execution as is, and at some point at the callback would be called. At that point the callback would try to get the lock, and at the next “yield point” whatever currently executes in the interpreter would potentially be suspended so that the callback can run. (Note: I don’t know the internals of Python well but that’s my understanding and how it behaves in practice at least)
It does not block completely until the callback is actually called, but only blocks other code in the interpreter while the callback is called.
The value that is captured by the closure is kept alive via the normal mechanisms.
Knowing fairly little about gst and gjs it kinda sounds like we want behaviour like a Promise (or rather a microtask) for these callbacks
My unsolicited tuppence anyway
In addition to the global interpreter lock allowing python to be called from multiple native threads, python has a full-blown threading API which is emulated by time slicing on a single native thread, so this doesn’t require a programmer to yield explicitly. I guess spidermonkey must also have some sort of threading support under the hood to enable Web Workers etc, but I don’t know how easy/practical/possible it would be to apply it to this use case in gjs.
That’s different to the example given above though: that simply has a callback that returns a value to the caller of the callback, not to the caller of the function taking the callback as argument.
If you somehow want to return a value to the original thread then something like a promise/future would be a potential solution. That’s what most bindings are converting things like GIO async operations to (which in C are also callback based). It’s something that is already handled in all the bindings 
… a callback that is scheduled in a very specific way which may be useful for running the callback on the main thread
Of course your native dummy/trampoline callback sitting on the other thread would have to wait for the result somehow
1 Like
Ah that’s what you mean. That sounds like a recipe for deadlocks though
1 Like
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.