Proper way to display video frames buffer (or any kind of dynamic pixels buffer) in Gtk4

Platform
,
1

Hello all!

My application need to display a video (RGB pixels buffer) from camera on Gtk4 window, after few days of research I have a list of approaches:

  1. Drawing video pixels buffer with Cairo.
  2. Use GLArea and create Texture to display (either glTexSubImage or pixel buffer object).
  3. Since Gtk4 have multiple backends (GL, Vulkan, software, …), we can utilize snapshot API from Gtk4 to display a GdkTexture for portablility.

I’ve tried all 3 approches above and this is my opinions about it:

  1. Approach 1 work fine, but in the end the cairo surface will need to be uploaded to GPU for displaying (in case we’re using GL, Vulkan, … backends) so I consisder this is not an optimized way to display a video (too much data transfer, and most drawing operations are in CPU side), especially for HD video or multiple videos.
  2. Approach 2 work fine in OpenGL backend, but Gtk4 supports other backends (software, vulkan, …), so I think writing OpenGL specific code is not a portable (I meant, it might not work on Vulkan or Software backends) and not necessary way, am I right about this?
  3. This is my preferred way (in my opinion) to do this but unfortunately I haven’t gotten it to work yet :smiling_face_with_tear:. The problems are:
    a. Gdk.Texture seems to be a static texture in design than a dynamic one (I don’t see any method for updating Gdk.Texture)
    b. There is a Gdk.GLTexture object seems a wrapper of OpenGL texture, I’ve try to append Gdk.GLTexture in snapshot method and call to glTexSubImage to update data but the widget.queue_draw() doesn’t redraw the updated texture at all (until resize the window), but using Gdk.GLTexture also losing portability as it depends on OpenGL.

So my questions are:

  1. In Gtk4 how can we update a Gdk.Texture dynamically?
  2. What is proper way to display a dynamic pixels buffer in Gtk4 in portable (and efficient) way?

Thanks for your reading!

2

1 - You should make a Gdk.Paintable instead of a looking into making a texture, the paintable will snapshot the texture whenever it changes, in your case it would be receiving a frame that you would make a MemoryTexture from.

You can look at how video/gtk4 · main · GStreamer / gst-plugins-rs · GitLab does it, it is written in Rust but should give you some ideas :slight_smile:

2 Likes
3

The most portable way to do it is probably to use gstreamer. But the gst plugin is not a good example of how to do it efficiently, until this MR is done: wip: video/gtk4: Use GL Memory Textures wherever possible. (!588) · Merge requests · GStreamer / gst-plugins-rs · GitLab

Using GdkMemoryTexture is not ideal as it does a copy on the CPU. If possible, it is best to avoid using it altogether, and only use GdkGLTexture.

4

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.