Display OpenCV Webcam in GTK 4 using PyGobject

Platform
,
1

Hi Folks!

I’m attempting to display a webcam feed from a USB camera using OpenCV in a Picture widget using GTK 4 / PyGobject. I have a thread populating a queue of OpenCV images (Numpy NDArrays) that have been converted from BGR to RGB. A timeout triggers a callback that retrieves the image from the queue and is supposed to display it but the Picture widget just shows up as a black square.

I’ve verified that the camera capture thread populates the queue correctly and the parent thread can save them as PNGs as expected. I’ve tried this on both MacOS and Ubuntu without success.

The relevant callback function is:

    def _on_camera_update(self) -> bool:
        try:
            self._current_frame = self._camera_handler.output_queue.get_nowait()
            self._current_pixbuff = GdkPixbuf.Pixbuf.new_from_data(
                data=numpy.array(self._current_frame).ravel(),  # type: ignore[union-attr]
                colorspace=GdkPixbuf.Colorspace.RGB,
                has_alpha=False,
                bits_per_sample=8,
                width=self._current_frame.shape[1],  # type: ignore[union-attr]
                height=self._current_frame.shape[0],  # type: ignore[union-attr]
                rowstride=self._current_frame.shape[1] * self._current_frame.shape[2],  # type: ignore[union-attr]
            )

            self._current_texture = Gdk.Texture.new_for_pixbuf(self._current_pixbuff)
            self._image_container.set_paintable(self._current_texture)
        except queue.Empty:
            pass

        return True
2

Hello @econeale! Do you mean that you saved self._current_pixbuff to a PNG file and it showed up correctly?

3

Not exactly. I saved self._current_frame as a png using cv2.imwrite. Then I viewed it from the system preview and it looked correct.

4

Ok, next step would be to save a few GdkPixbufs to ensure they contain the right data. See

5

Question:

GdkPixbuf.Pixbuf.new_from_data() doesn’t copy (or even ref) the data passed as argument. It could be that numpy.array(self._current_frame).ravel() must be kept alive in some way (not sure if it produces new data or just a view into an existing buffer). What happens if you write:

  self._current_frame = self._camera_handler.output_queue.get_nowait()
self.array_data=numpy.array(self._current_frame).ravel()
  self._current_pixbuff = GdkPixbuf.Pixbuf.new_from_data(
                self.array_data,
                colorspace=GdkPixbuf.Colorspace.RGB,
                has_alpha=False,
                bits_per_sample=8,
                width=self._current_frame.shape[1],  # type: ignore[union-attr]
                height=self._current_frame.shape[0],  # type: ignore[union-attr]
                rowstride=self._current_frame.shape[1] * self._current_frame.shape[2],  # type: ignore[union-attr]
            )

EDIT: not so sure. Thinking about it, new_from_data() should ref the bytes object

6

Thanks, that’s a good idea. I’ll try that.

7

So, it looks like the problem is related to framerate which I was not expecting. While trying to save the Pixbuff object as a PNG to investigate, the application started hanging. I figured the disk write was slow so I lowered the display rate of the camera. When I lower the framerate to 1 FPS it seems to display the image as expected.

I don’t really understand why that would be the case. The only thing I can think of is some kind of race condition between the capture process and the display process.

1 Like
8

I suggest using Gtk.Widget.add_tick_callback() to add callback that is called by GTK once per frame (from the GTK rendering engine). Use such callback to pull the latest frame, convert it to a GdkPixbuf and set the paintable.

9

Okay, using the tick callback seems to have worked the best. The final code becomes the following.

Run function for the background image capturing thread (needed keep the camera’s frame buffer from filling up):

 def run(self) -> None:
        """Start capturing frames from the camera and updating the frame queue"""
        tmp_frame: np.ndarray[np.uint8, Any] = np.zeros(
            (self._camera_frame_height, self._camera_frame_width, 3), dtype=np.uint8
        )
        while not self._stop_event.is_set():
            try:
                if self._capture_device.isOpened():
                    ret, tmp_frame = self._capture_device.read()

                    if time.time() - self._previous_time > 0.25:
                        if ret:
                            self._previous_time = time.time()

                            tmp_frame = cv2.cvtColor(tmp_frame, cv2.COLOR_BGR2RGB)
                            self.output_queue.put_nowait(tmp_frame)
                        else:
                            self._logger.debug("Camera read failed")
                else:
                    self._capture_device.open(self._camera_path, cv2.CAP_V4L2)

            except (queue.Full, queue.Empty):
                self._logger.debug("Output queue full")
            except Exception as err:
                self._logger.error("Camera thread received: %s", str(err))

            time.sleep(0.05)

Main activation function of the parent GTK application:

def do_activate(self) -> None:
        self._camera_handler.start()
        self._image_container = self._builder.get_object("image_container")
        self._image_container.add_tick_callback(self._on_camera_update, None)

Function toupdate the image preview widget:

def _on_camera_update(self, widget:Gtk.Widget, frame_clock:typing.Any, user_data:typing.Any) -> bool:
        """Function called on camera update timer"""
        try:
            tmp_frame = cv2.resize(self._camera_handler.output_queue.get_nowait(), (1280, 720), cv2.INTER_AREA)
            self._current_frame = tmp_frame
  
            self._current_pixbuff = GdkPixbuf.Pixbuf.new_from_data(
                data=np.array(self._current_frame).ravel(),  # type: ignore[union-attr]
                colorspace=GdkPixbuf.Colorspace.RGB,
                has_alpha=False,
                bits_per_sample=8,
                width=self._current_frame.shape[1],  # type: ignore[union-attr]
                height=self._current_frame.shape[0],  # type: ignore[union-attr]
                rowstride=self._current_frame.shape[1] * self._current_frame.shape[2],  # type: ignore[union-attr]
            )
  
            self._current_texture = Gdk.Texture.new_for_pixbuf(self._current_pixbuff)
            self._image_container.set_paintable(self._current_texture)  # type: ignore[union-attr]
        except queue.Empty:
            pass
  
        return True

Thanks for the help!

1 Like
10

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.