GStreamer and gobject-introspection

I was recently asked to add gstreamer to the Nim bindings, see

I had seen the name gstreamer a few times already, but had no idea what it is before. OK, found the wikipedia page and gstreamer homepage. Seems to be related to video.

But is there a reason that someone may need language bindings for that? I am a bit skeptical – about one year ago someone asked to add libvte, I did it, and that user never came back.

Gstreamer seems to be different to most other gtk core libs, also documentation differs. So it works not out of the box with my gintro gobject-introspection language binding generator. One issue is that there are a few classes that have no parents. I saw that before only for data types GObject and GParamSpec. But Gst has for example

/usr/share/gir-1.0/Gst-1.0.gir

    <class name="Int64Range"
           c:symbol-prefix="int64_range"
           glib:type-name="GstInt64Range"
           glib:get-type="gst_int64_range_get_type"
           glib:fundamental="1">
    </class>

Do we have an idea what is special about these classes without parent? My feeling was, that the Gobject Type system is based on GObjects as base types, well maybe on GInitiallyUnowned.

At least I have to write some special code for these classes without parent. For GObject itself that was easy, because it is just the Root class.

A google search for “GstInt64Range” gave me no great results unfortunately.

GStreamer is the media framework used by GNOME (and other projects); it doesn’t just deal with video, but with audio playback; audio/video capture; audio/video editing and processing.

Probably not for a niche and fairly unknown language like Nim; there are plenty of users of GStreamer in Python, for instance, and recently a lot of work has gone into integrating Rust with GStreamer through introspection.

That’s not really correct.

GType is the type system, and GObject is one of the fundamental types in it; GObject, though, inherits from GTypeInstance, which is the base type for other derivable, instantiatable types, like GParamSpec. Additionally, GType allows creating new fundamental types—like other GTypeInstance derived types, for instance ClutterPaintNode; or new integral types like the int64 range type in GStreamer.

There is really no GstInt64Range structure. GstInt64Range is what g_type_name(GST_TYPE_INT64_RANGE) would return. GST_TYPE_INT64_RANGE would be used to initialise a GValue to pass to g_object_set() and g_object_get().

You need to handle fundamental GTypes; they are a concept that is used in various libraries. GLib itself has fundamental types out of the box, like all the scalar types.

1 Like

Thanks for your fast and detailed reply.

there are plenty of users of GStreamer in Python, for instance, and recently a lot of work has gone into integrating Rust with GStreamer

Well, when it is used by Python and Rust people, then wish of Nim users seems to be not really that strange, so I think I really should try to support it…

Do you have an idea about how to guess the right unref/free method?

With the core gtk related libs that was not that hard – for widgets and all what is based on gobject it is g_object_unref(). For other data types my plain strategy was to search for object methods called “free” or “unref”. This strategy is stupid – I implemented that in early days, and as it was good enough for maybe 97% of all processed data types I did not change it. When there was found no method to freing/unrefing, during install of the bindings I gave a message so that someone can try to manually fix it, and having no free/unref method just means that the data types are not freed, not a too big issue.

For gstreamer this seems not to work properly, I am getting many messages like

Caution: No free/unref found for  Context (gst_element_get_context_unlocked)

Before I start investigation this further, I ask myself if I may have missed something.

In the API docs there are somtimes comments like “use functionXYZ to unref this data type” but there seems to be no easy way to access such information by gobject-introspection.

One idea which I just had is following the PARENT chain until one arrived at the well known parent class, for which free/unref is known from a data table?

Of course I have to do all these changes very carefully, as the bindings seems to work well for core gtk, and I should not damage that.

All fundamental types should have additional annotations for memory and GValue handling. The values for the annotations can be retrieved programmatically from the GIObjectInfo data structure using its API.

For GBoxed types, you should always go through g_boxed_copy() and g_boxed_free(), and never call (or try to guess) the copy/free functions yourself.

Data types that do not have a GType are meant to be placed on the stack prior to a function call, or allocated on the heap by your binding, and thus allocated and freed by your own memory allocator (you already have the size of the data type).

1 Like
g_object_info_get_unref_function

I have never seen that function before! Have to investigate, thanks for that link.

And now I really wonder why I missed it, it is there

https://developer.gnome.org/gi/stable/gi-GIObjectInfo.html

Indeed I was some kind of unhappy that I had missed g_object_info_get_unref_function() for so long, but I think that is indeed not such a big problem, as that function can be used for the GObject types only, and for that types I seems to have chosed the right unref functions already. But I will use that function in future.

Also I had never problems with all the plain structs which are generally allocated on the stack, like GtkTextIter, GtkRectangle or the types from your graphene library.

For the Boxed types, I seems to have to learn more. For example for GStreamer one type which makes some problems seems to be GstMessage. Is that a boxed type?

https://gstreamer.freedesktop.org/documentation/gstreamer/gstmessage.html?gi-language=c

I think the issue with my current strategy of guessing the unref() function is, that there exists gst_message_unref() in

only as static funtion, and is called macro in

https://gstreamer.freedesktop.org/documentation/gstreamer/gstmessage.html?gi-language=c#gst_message_unref

And it seems to be not reported in

/usr/share/gir-1.0/Gst-1.0.gir

So maybe I have indeed to learn how to use the boxing.

GstMessage is a boxed type, as far as the GType type system is concerned; you always have to go through g_boxed_copy() and g_boxed_free() for copying and freeing instances, respectively. You can simply check using something like:

if (G_TYPE_FUNDAMENTAL (some_gtype) == G_TYPE_BOXED) {
  // it's a boxed type
}

Behind the scenes, GstMessage is a GstMiniObject, which is really a mixin pattern. GstMessage does not inherit from GstMiniObject—as GBoxed types cannot have child types—but it provides its interface; this means that copying a GstMessage via g_boxed_copy() calls gst_mini_object_ref(), and freeing an instance via g_boxed_free() calls gst_mini_object_unref().

Thanks again for the explanation.

For Boxed types I found

https://developer.gnome.org/gobject/stable/gobject-Boxed-Types.html

but no concrete example yet. Will continue searching. I have already learned that GstMessage is a GstMiniObject. Indeed I had the impression and hope that GtkMessage is not a Boxed type, so thanks for clarification. I think and hope that manually providing gst_mini_object_unref() for GstMessage and GstMiniObject, which is then called from the Nim GarbageCollector, will work for now. The executed code is then similar to ordinary C code. Of course copying these data types is forbidden, but that is the case for GtkWidgets too. Nobody generally creates a GtkButton with gtk_button_new() and copy the object – at least I had never the desire to do that. I will try to support Boxed types later.

Unfortunately you seems to be the only one still knowing all these details, all others seems to have retired yaers ago.

The boxed types seems to be really hard.

I did some initial tests to find out which types are really boxed types, hoping that gobject-introspection would tell me that for example GstMessage is a boxed types. There is

https://developer.gnome.org/gi/stable/GIBaseInfo.html#g-base-info-get-type

which returns

https://developer.gnome.org/gi/stable/GIBaseInfo.html#GIInfoType

but that function gives me GI_INFO_TYPE_STRUCT., not GI_INFO_TYPE_BOXED. And I found no other functions that would tell me that the struct is really a boxed type.

Then I tried a C program like

//gcc -o t t.c `pkg-config --libs --cflags gtk+-3.0 gstreamer-1.0`
#include <gtk/gtk.h>
#include <stdio.h>
#include <gst/gst.h>
int main(int argc, char *argv[]) {

  printf("%s\n", g_type_name(G_TYPE_STRING));
  printf("%s\n", g_type_name(G_TYPE_SOURCE));

  if (G_TYPE_FUNDAMENTAL (GST_TYPE_MESSAGE) == G_TYPE_BOXED) {
    printf("GST_TYPE_MESSAGE is boxed\n");
  }

  if (G_TYPE_FUNDAMENTAL (GST_TYPE_MINI_OBJECT) == G_TYPE_BOXED) {
    printf("GST_TYPE_MINI_OBJECT is boxed\n");
  }

  if (G_TYPE_FUNDAMENTAL (G_TYPE_DATE) == G_TYPE_BOXED) {
    printf("G_TYPE_DATE is boxed\n");
  }

  printf("%s\n", g_type_name(G_TYPE_DATE));
  //printf("%s\n", g_type_name(GST_TYPE_MESSAGE));

  return 0;
}

Only for G_TYPE_DATE it tells me that it is a boxed type.

Using g-boxed-free() without testing prior that it is a boxed types seems to be strange.

I found

https://developer.gnome.org/gi/stable/gi-GIRegisteredTypeInfo.html#g-registered-type-info-get-g-type

which may give me the gtype, which again is needed for g-boxed-free()

[Last minute edit]

g_type_fundamental(g_registered_type_info_get_g_type(info)) == G_TYPE_BOXED

may be the check which I really need? At least it gives true for GstMessage, so maybe that is the right test. Have to investigate more carefully.

[EDIT 2]

Looks promising indeed :slight_smile: . I have to rewrite some code before I can test it and confirm that it really works still.

I get the feeling that g_object_info_get_unref_function() always returns an empty string. Maybe that was the reason why I did not use it when I started the bindings some years ago. I have not found its source code yet, and also found no issues in the bug tracker. But on the other hand I do not think that I am using it wrong.

But that should be no real problem, just using g_object_unref () seems to be fine for all gobjects.

GIObjectInfo does not imply it’s a GObject. Yes: it’s going to be empty for all object types that inherit from GObject, but as I said above, GLib supports fundamental, derivable types that do not inherit from GObject. For those, the ref/unref functions, as well as the GValue accessors, must be defined.

You cannot use GIRepository API without dealing with GType; they both exist for a reason, and they both need to be used to check the functionality of a type. GLib cannot use GIRepository to introspect its own types because it doesn’t depend on GIRepository; and GIRepository cannot reimplement the type system out of tree.

I’d strongly recommend you looked at how introspection-based language bindings are implemented—like pygobject or gjs. Reading the introspection API reference alone will not be enough to write a language binding, unless you’re willing to go through all the mistakes and discoveries done for the past 15 years of GObject binding development.

Maybe that should be the head line of the API page :slight_smile:

Indeed I felt a bit stupid when I started the Nim bindings three years ago.

For PyGobject the problem is, that Python is an interpreted language with dynamic typing. Nim is compiled, using a C or LLVM backend, with static typing.

Rust, D, Go, Crystal should be closer to Nim. I had a short look at these bindings – a lot of code without much comment. Rust seems to be the most active, indeed I already considering Mr Droege, one of the Rust people.

But finally the Nim bindings seems to work not bad already. I made some progress with the boxed types, and got Nim version of Basic tutorial 1: Hello world! working already. But I have to do more testing still.

Feel free to open an issue.

It doesn’t really matter: the code to handle different types depending on GType, as well as converting arguments and return values between native types and introspected types is not really based on anything dynamic.

From /usr/share/gir-1.0/GLib-2.0.gir we have

    <record name="String"
            c:type="GString"
            glib:type-name="GString"
            glib:get-type="g_gstring_get_type"
            c:symbol-prefix="gstring">

May there exist a way to get the glib:get-type enty by gobject-introspection?

I tried to find a function which may provide that name, but had no succsess. For now I guess (construct) the name myself, but it is not fully trivial. For gtk prefix is gtk, but for glib and gobject prefix is just g. And for GString it is really special g_gstring_get_type. For some other modules with long names it is also not easy to guess, as name may be shortened. Guessing is good enough for now, another option would be parsing the XML gir files and storing names in a table. But best solution would be APi support of course.

[EDIT]

Well, at least I can use g_registered_type_info_get_type_name() so I get “GString” name for the glib string, then it is not hard to construct g_gstring_get_name().

Maybe

https://developer.gnome.org/gi/stable/GIBaseInfo.html#g-base-info-iterate-attributes

is the solution – I will investigate later…

You want g_registered_type_info_get_g_type(), which will call the get_type() function for you and return the GType for the type.

No.

g_registered_type_info_get_g_type() is well known, but it returns the value during binding generation. (I am sure that you know that GType values of non fundamental type have no constant value.) Remember that we use a compiled language, so we generate bindings during install, not on the fly as scripting languages may do.

What we would need is for example G_TYPE_STRING as used in http://zetcode.com/gui/gtk2/gtktreeview/

But that C macro is not available by gobject-introspection of course. The function g_gstring_get_type() is available during runtime, which the gir files tell us, and it does the same as G_TYPE_STRING. But we can not get the exact name from gobject-introspection, guessing is possible, but some sort of ugly.

I will investigate g-base-info-iterate-attributes() soon.

It seems to me you’d have better luck generating code from the GIR data, instead of using libgirepository, if you’re generating static bindings instead of dynamic ones.

That is an interesting idea – if I had known from the beginning how difficult using the gobject-introspection API is to use, I may had considered that. Do you know a binding for a compiled language that did it?

I started about 3 years ago, at the same time someone other started Nim bindings using gobject introspection also, but stopped soon, and someone other used gobject introspection for Crystal language. I think Rust and Go bindings did not even exist at that time.

For sure I will continue using gobject introspection now, have spent more than 1k hours on it, and most works fine.

But for other new languages like Kid, Zig, VLang, Jai without GTK bindings that may be a better solution maybe?

[Edit]

Note, that I started with Nim bindings generated from the C header files 4 years ago, these are now called oldgtk3 in Nim package directory, but I think no one is using that now. Was really boring work, I wasted 600 hours on it.