GStreamer - How to save matroska file

I have already asked this question here, but since I didn’t get a response I’ll try here. I will here copy paste the original question. To get my code click on the link to the original question and you’ll find it there.


Hello.

I want to save what camera “sees” in a file while also displaying it
on the screen. I have figured out that I can do that with this command

 gst-launch-1.0 -e v4l2src device=/dev/video0 ! videoconvert !queue ! tee name=t t. ! queue ! matroskamux ! queue ! filesinklocation=/tmp/test.mkv t. ! queue ! autovideosink sync=false

But since I’m working on a C program that uses GStreamer and GTK I
need to translate this command to C and I don’t know how to properly
do it. I also want my program to allow users to pause the recording
but not the video output but I successfully figured that out.

In the end my pipeline looks like this

                                       /-> queue -> autovideosink
v4l2src -> videoconvert -> queue -> tee
                                       \-> queue -> matroskamux -> queue -> filesink || fakesink

The reason for “filesink || fakesink” is because these two are swapped
when the recording state changes: when we’re recording filesink is
present and when a user pauses the recording filesink is replaced with
fakesink (and when a user unpauses the recording fakesink is replaced
by filesink, etc.).

I wrote a small program (available as an attachment) that can
replicate all relevant problems that I have encountered in my project.
Use “Pause” / “Unpause” button to pause / unpause the recording and
“Stop” button to stop the recording and quit the application (normally
quitting the application can be achieved by also closing the control
window (the one with the buttons) but not by closing preview window (I
didn’t bother changing the preview window to custom GTK window so this
is the default window provided by GStreamer)).

I hope it’s OK that this example program uses a little bit of GTK,
it’s only there for the control window and you don’t have to know
anything about GTK to help me with my problem.

I used video player MPV to test the resulting MKV files. These are the
errors that I have found when running my program (all these errors
also exist in attached program if you find more please tell me):
1. If I click on “Pause” then on “Unpause” the resulting file is
too corrupted to be recognized by MPV as an MKV file.
2. If I do not click “Pause” button or click it only once before
stopping the program MPV displays errors like “[mkv] Expected element
0x1043a770 not found” when opening the file (usually three these). If
I use gst-launch-1.0 command mentioned at the beginning of this letter
opening the resulting MKV file does not produce such errors.
3. When the resulting file produced as in error 2 and is played
using MPV, MPV seems unable to determine the duration of the file so
it constantly changes what it thinks the duration is. Again, if I use
command mentioned at the beginning of this letter this problem doesn’t
occur.

I think that’s it.

I think you may be switching to fakesink before the data has been flushed into the filesink (and the other way around)? My GStreamer knowledge is very spotty but can you simply disconnect (and later reconnect) the second branch of the tee?

The second issue sounds similar, in that the pipeline gets taken down before the data has been fully written. You may have to wait for a signal after setting the pipeline state to null before exiting.

Thanks for your response.

I’m not sure, I tried only disconnecting the file sink but it didn’t work for some reason I can’t remember.

Hmmmm doesn’t this kinda mean that first two problems (probably all three) can be solved by just waiting for the end of the “segment”? I mean, that’s what it seems to me since at least first two problems are caused because the program doesn’t wait for the end of the data. So I should just wait for the message GST_MESSAGE_SEGMENT_DONE, correct?

I tried wait for GST_MESSAGE_SEGMENT_DONE but that message never arrived so I guess this is not the solution.
Also, the code: https://pastebin.com/MaRdNSZd.

You might want to have a look at the camerabin element, it can take care of all the low-level details for you.

Some example code here: https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/tree/master/tests/examples/camerabin2 (might be a bit bitrotten, but shouldn’t be too hard to fix if so).

I have found a fix for problems 1 and 3. All I had to do is send EOS event and wait for it before I set pipeline to GST_STATE_NULL. Fix can be found here.

Now I only have to figure out how to solve second problem.

Element 0x1043a770 in Matroska is Chapters, which isn’t mandatory. The error message comes from this line in mpv but I still can’t figure out when it happens.

You can try checking the file with mkvalidator, or comparing the good and bad files with mkvtoolnix (the “Info tool” tab on the left).

That seems like an overkill for my purposes. And I have only one problem left, I’m not going to rewrite 1000 LOC now.

Ok, now I have only this problem. Please see related code. You can try setting a brakpoint on line 199.
The problem is that now when I try to unpause pipeline.sink_file fails to be recognized as GObject for some reason.

This is the only problem left. Solved by @tpm, here.

When you call gst_bin_remove(), the element will be removed from the bin. Since the pipeline took ownership of the element (as it was a newly-created element, see “floating references”) when you added it, and the pipeline drops its reference when you remove it, that means the element will get freed, unless someone else still holds a ref to it. So you can prevent this by adding a gst_object_ref() before removing it.

You can track down these kind of issues (invalid memory access, use-after-free) with valgrind by the way. It will tell you where it got freed.

Having said that, I don’t understand this whole replace-filesink-with-fakesink thing while being paused. I don’t think it makes much sense.

What is the effect you want to have when pausing? If you playback the Matroska file later and you recorded for 10 seconds, paused for 10 seconds, and then recorded again for 10 seconds, how long should the playback be in your opinion? 30 secs or 20 secs? Should it show the paused part from 10-20 secs as ‘freeze frame’ or should it skip it entirely as if it never happened?

Perhaps you want a valve element in front of the encoder/muxer instead to drop data when you’re not recording? But if you want to skip the gaps you would also have to tweak the timestamps with gst_pad_set_offset() on the recording branch tee pad perhaps (and set an offset of -$time_paused whenever you resume recording) or somesuch.

1 Like

Thanks, it works.

It should be 20 seconds, as if it never happened.

Thanks, I’ll check out valve element.

@tpm Even though your solution worked now only the last part of the recording is saved. For example,

  1. Start recording.
  2. Press “Pause” after 10 seconds. (Part A of the recording)
  3. Press “Unpause” after 5 seconds.
  4. Press “Stop” after 40 seconds. (Part B of the recording)

The produced MKV file will contains only part B (40s) of the recording, instead of A + B (50s) as it should.

I tried setting append property of filesink to TRUE before setting pipeline to GST_STATE_PLAYING when user clicks “Unpause”, but in that case only part A (10s) is recorded.

That’s probably because you set the state of the pipeline or branch or elements to NULL and then back to PLAYING. Which doesn’t seem right to me.

Don’t you want to keep seeing the camera feed even when you’re not recording?

Once you send EOS to the recording branch and/or stop writing the file, that file is finished. You can’t start again and add data to it. That’s just not how the file format works. You can’t use append mode here either.

You need to keep the pipeline running all the time, and just not feed data to the recording branch when you don’t want to save it.

So I guess using valve is the best way. I already tried to get it to work with valve, but it didn’t work so I didn’t bother because I thought what I already have will work fine.
Thanks again.

I managed to almost do it with valve.

If I just record and press “Stop” it works fine.

Pausing is now done by setting valve property drop to TRUE and unpausing by setting it to FALSE. If I click “Stop” while the recording is paused (i.e. drop = TRUE) the entire application freezes, but if I continue the recording and then stop the application works normally. So should I always set drop to TRUE when a user clicks “Stop” or is there a better way?

Also, pausing and unpausing produce a corrupted file. MPV gives me these errors.

Playing: Untitled.mkv
 (+) Video --vid=1 (*) 'Video' (rawvideo 640x480 30.000fps)
[autoconvert] Converting yuyv422 -> yuv422p
VO: [gpu] 640x480 yuv422p
V: 00:00:00 / 00:00:11 (4%)
[mkv] Corrupt file detected. Trying to resync starting from position 7373405...
V: 00:00:00 / 00:00:11 (8%)
[mkv] Cluster found at 14746363.
V: 00:00:06 / 00:00:11 (58%)
[mkv] Corrupt file detected. Trying to resync starting from position 31335548...
V: 00:00:06 / 00:00:11 (59%)
[mkv] Cluster found at 43009425.
V: 00:00:00 / 00:00:12 (4%)
[mkv] Corrupt file detected. Trying to resync starting from position 7373405...
[mkv] Cluster found at 14746363.
V: 00:00:01 / 00:00:12 (15%)


Exiting... (Quit)

When the file is played it freezes there where I paused the recording.
Again, if I just stop the application the produced file is fine. I guess I should just use gst_pad_set_offset(), as you have pointed out, but how can I get the duration that I need to supply? And can I apply it to the source pad of queue that forwards data to the file?

As for the freezing on stop, try g_object_set(filesink, "async", FALSE, NULL) on the filesink. Without this, it might wait for another preroll buffer when going from PLAYING to PAUSED state when shutting down, and that buffer never comes. Just a guess though.

I don’t know what the cause of the data corruption is, I don’t think it is related to not setting the pad offset. The reason for setting the pad offset is to shift the timestamps forward in front of the muxer after resuming the stream, so that it looks to the muxer like the pausing never happened. Without it, you should just get a long timestamp gap in the file where it keeps showing the last frame, but no data corruption.

I think you need to post your latest code again.

No, that didn’t help.

Ok. Still, how can I get the duration that I need to supply as the second argument to gst_pad_set_offset()?

Here you go.

The valve element should be before the encoder/muxer, not before the file sink.

It still doesn’t work.

http://ix.io/1MPP

Perhaps you could be more specific than “it doesn’t work”? What did you change and what was the outcome? Do you get an error, etc?