GStreamer (Rust Bindings): Tee at the end of a pipeline link chain makes it impossible to add something to the beginning

Hey,

I am currently working on a gstreamer project that takes a stream via RTSP and outputs it as HLS. This works just fine. Now I want to add sort of a snapshot function, to take a PNG picture out of the stream occasionally. To do this, I wanted to use a tee to duplicate the stream. However, when I try to dynamically link some parts, I receive an error. A reduced version of my code looks like this:

use gst::prelude::*;
use gst_video::VideoFormat;
use std::env;
use std::net::UdpSocket;
use std::thread;
use std::str;

fn main() {
    // Initialize GStreamer
    gst::init().unwrap();

	let args: Vec<String> = env::args().collect();
	let location = &args[1];
	println!("Using {} as stream location.", location);

    // Create the elements
    let source = gst::ElementFactory::make("rtspsrc", Some("source"))
        .expect("Could not create rtspsrc element.");

		source.set_property("location", &location);
		source.set_property("latency", &(500 as u32))
		.expect("Couldn't set RTSP Source latency");

	let depayloader = gst::ElementFactory::make("rtph264depay", Some("depayloader"))
		.expect("Could not create depayloader element.");
		
		let decodebin = gst::ElementFactory::make("decodebin", Some("decodebin"))
		.expect("Could not create decodebin element.");

		let convert = gst::ElementFactory::make("videoconvert", Some("video_convert"))
        .expect("Could not create convert element.");

        let convert2 = gst::ElementFactory::make("x264enc", Some("video_convert2"))
        .expect("Could not create convert element.");
		convert2.set_property("key_int_max", &(5 as u32));

        let convert3 = gst::ElementFactory::make("h264parse", Some("video_convert3"))
        .expect("Could not create convert element.");

        let sink = gst::ElementFactory::make("hlssink2", Some("udp_sink"))
                .expect("Could not create sink element.");
        sink.set_property("playlist-root", &"http://10.10.193.95:443");
        sink.set_property("playlist-location", &"/stream/playlist.m3u8");
        sink.set_property("location", &"/stream/segment%05d.ts");
        sink.set_property("target-duration", &(1 as u32))
		.expect("Not working.");
        sink.set_property("max-files", &(6 as u32))
		.expect("Not working.");
		sink.set_property("playlist-length", &(3 as u32))
		.expect("Not working.");
		sink.set_property("send-keyframe-requests", &(false))
		.expect("Not working.");

	let filter = gst::ElementFactory::make("capsfilter", Some("filter"))
	.expect("Could not create filter element.");
	let video_caps =
        gst::Caps::new_simple("video/x-h264", &[("profile", &"constrained-baseline")]);
	filter.set_property("caps", &video_caps);

	let tee = gst::ElementFactory::make("tee", Some("tee"))
                .expect("Could not create tee element.");

	let tee_queue = gst::ElementFactory::make("queue", Some("tee_queue"))
                .expect("Could not create tee queue element.");
	let tee_queue2 = gst::ElementFactory::make("queue", Some("tee_queue2"))
                .expect("Could not create tee queue 2 element.");
	tee_queue.set_property_from_str("Queue-leaky", &"downstream");
	let png_enc = gst::ElementFactory::make("pngenc", Some("pngenc"))
                .expect("Could not create PNG encoder element.");
	png_enc.set_property("snapshot", &(true))
	.expect("Could not set png_enc property.");
	let filesink = gst::ElementFactory::make("filesink", Some("filesink"))
                .expect("Could not create filesink element.");
	filesink.set_property("location", &"/stream/capture.png")
	.expect("Could not set filesink property.");

    // Create the empty pipeline
    let pipeline = gst::Pipeline::new(Some("test-pipeline"));

    // Build the pipeline Note that we are NOT linking the source at this
    // point. We will do it later.
    pipeline
        .add_many(&[&source, &decodebin, &depayloader, &convert, &convert2, &filter, &convert3, &sink, &tee, &tee_queue, &tee_queue2, &png_enc, &filesink])
		//.add_many(&[&source, &parse, &queue, &convert, &convert2, &filter, &convert3, &sink])
        .unwrap();
	
	let tee_hls_pad = tee.get_request_pad("src_%u").unwrap();
    println!(
        "Obtained request pad {} for HLS branch",
        tee_hls_pad.get_name()
    );
    let tee_queue2_pad = tee_queue2.get_static_pad("sink").unwrap();
    tee_hls_pad.link(&tee_queue2_pad).unwrap();

    let tee_img_pad = tee.get_request_pad("src_%u").unwrap();
    println!(
        "Obtained request pad {} for image capture branch",
        tee_img_pad.get_name()
    );
    let tee_queue_pad = tee_queue.get_static_pad("sink").unwrap();
    tee_img_pad.link(&tee_queue_pad).unwrap();
	
	// This links all elements in a row (something could be added in the middle)
    //gst::Element::link_many(&[&source, &parse, &queue, &convert, &convert2, &filter, &convert3, &sink]).expect("Elements could not be linked.");
	gst::Element::link_many(&[&depayloader, &decodebin]).expect("Depayloader/Decodebin elements could not be linked.");
	gst::Element::link_many(&[&convert, &tee]).expect("Source elements could not be linked.");
	gst::Element::link_many(&[&tee_queue2, &convert2, &filter, &convert3, &sink]).expect("HLS branch elements could not be linked.");
	gst::Element::link_many(&[&tee_queue, &png_enc, &filesink]).expect("Image capture elements could not be linked.");
	
/*
	let text_sink = textoverlay.get_static_pad("text_sink")
	.expect("Failed to get static sink pad from text");
	let udp_src_pad = udpsrc.get_static_pad("src")
	.expect("Failed to get static src pad from UDP");
	let res = udp_src_pad.link(&text_sink);
	if res.is_err() {
		println!("Text Link failed.");
	} else {
		println!("Text Link succeeded.");
	}
*/
	
    // Set the URI to play
    //let uri =
    //    "https://www.freedesktop.org/software/gstreamer-sdk/data/media/sintel_trailer-480p.webm";
        //      //"file:///home/leo/zal.mp4";
    //source
    //    .set_property("uri", &uri)
    //    .expect("Can't set uri property on uridecodebin");

    //Connect the pad-added signal
    //This is a function that is called when a new pad is added to the source element (which is audio and video each)
	let sink_pad = depayloader
            .get_static_pad("sink")
			.expect("Failed to get static sink pad from depayloader");
    source.connect_pad_added(move |src, src_pad| {
        println!(
            "Received new pad {} from {}",
            src_pad.get_name(),
            src.get_name()
        );

        if sink_pad.is_linked() {
            println!("We are already linked. Ignoring.");
            return;
        }

        let new_pad_caps = src_pad
            .get_current_caps()
            .expect("Failed to get caps of new pad.");
        let new_pad_struct = new_pad_caps
            .get_structure(0)
            .expect("Failed to get first structure of caps.");
        let new_pad_type = new_pad_struct.get_name();

        /*let is_audio = new_pad_type.starts_with("video/x-raw");
        if !is_audio {
            println!(
                "It has type {} which is not raw video. Ignoring.",
                new_pad_type
            );
            return;
        }*/
		
		let is_video = new_pad_type.starts_with("application/x-rtp");
        if !is_video {
            println!(
                "It has type {} which is not x-rtp. Ignoring.",
                new_pad_type
            );
            return;
        }

        let res = src_pad.link(&sink_pad);
        if res.is_err() {
            println!("Type is {} but link failed.", new_pad_type);
        } else {
            println!("Link succeeded (type {}).", new_pad_type);
        }
    });
	
	decodebin.connect_pad_added(move |src, src_pad| {
        println!(
            "Received new pad {} from {}",
            src_pad.get_name(),
            src.get_name()
        );

        let sink_pad = convert
            .get_static_pad("sink")
            .expect("Failed to get static sink pad from convert");
        if sink_pad.is_linked() {
            println!("We are already linked. Ignoring.");
            return;
        }

        let new_pad_caps = src_pad
            .get_current_caps()
            .expect("Failed to get caps of new pad.");
        let new_pad_struct = new_pad_caps
            .get_structure(0)
            .expect("Failed to get first structure of caps.");
        let new_pad_type = new_pad_struct.get_name();

        /*let is_audio = new_pad_type.starts_with("video/x-raw");
        if !is_audio {
            println!(
                "It has type {} which is not raw video. Ignoring.",
                new_pad_type
            );
            return;
        }*/
		
		let is_video = new_pad_type.starts_with("video/x-raw");
        if !is_video {
            println!(
                "It has type {} which is not video. Ignoring.",
                new_pad_type
            );
            return;
        }

        let res = src_pad.link(&sink_pad);
        if res.is_err() {
            println!("Type is {} but link failed.", new_pad_type);
        } else {
            println!("Link succeeded (type {}).", new_pad_type);
        }
    });


        //let x = source
        //.set_state(gst::State::Paused);
        //println!("{}", x.to_string());
        //.expect("Couldn't set the source to the ready state");
    // Start playing
	//textoverlay.set_property("text", &"Hello there!!!!!!");
    pipeline
        .set_state(gst::State::Playing)
        .expect("Unable to set the pipeline to the `Playing` state");
	println!("Pipeline set to playing state!!!");
	
    // Wait until error or EOS
    let bus = pipeline.get_bus().unwrap();
    for msg in bus.iter_timed(gst::CLOCK_TIME_NONE) {
        use gst::MessageView;

        match msg.view() {
            MessageView::Error(err) => {
                eprintln!(
                    "Error received from element {:?} {}",
                    err.get_src().map(|s| s.get_path_string()),
                    err.get_error()
                );
                eprintln!("Debugging information: {:?}", err.get_debug());
                break;
            }
            MessageView::StateChanged(state_changed) => {
                if state_changed
                    .get_src()
                    .map(|s| s == pipeline)
                    .unwrap_or(false)
                {
                    println!(
                        "Pipeline state changed from {:?} to {:?}",
                        state_changed.get_old(),
                        state_changed.get_current()
                    );
                }
            }
            MessageView::Eos(..) => break,
            _ => (),
        }
    }

    pipeline
        .set_state(gst::State::Null)
        .expect("Unable to set the pipeline to the `Null` state");
}

I am running this program in a docker container with Ubuntu 20, and I’m using the most recently available gstreamer version via apt (1.16). My gstreamer rust repository was pulled up to date just an hour ago. The output my program gives me is this:

Using rtsp://10.10.193.96:8554/unicast as stream location.
Obtained request pad src_0 for HLS branch
Obtained request pad src_1 for image capture branch
Pipeline set to playing state!!!
Pipeline state changed from Null to Ready
Pipeline state changed from Ready to Paused
Received new pad recv_rtp_src_0_807327211_96 from source
Link succeeded (type application/x-rtp).
Received new pad src_0 from decodebin
Type is video/x-raw but link failed.

This last line is the problem. The link fails. The problem does not exist when I only link one of the two source pads from the tee or remove it entirely from the pipeline; but obviously I need both, because otherwise it would be pointless.

Can you update the code formatting so that it reads properly? :slight_smile:
Putting a ```rust at the beginning and 3 backticks at the end should fix that.

Also please try getting a GStreamer debug log with GST_DEBUG=6 and attach it here. That should make it quite clear where the problem is.

Done. Also, I copied the last 100 lines printed before the error message here. If more are necessary, let me know :slight_smile:

docker logs thirsty_morse | grep -B 100 "Type is video"
0:00:18.054059393     6 0x7fc528027300 DEBUG               GST_PADS gstpad.c:4076:gst_pad_query:<filesink:sink> sent query 0x7fc500056140 (caps), result 1
0:00:18.054065503     6 0x7fc528027060 DEBUG        rtpjitterbuffer gstrtpjitterbuffer.c:2244:add_timer: signal timer, 1 waiters
0:00:18.054072797     6 0x7fc528027300 DEBUG               GST_CAPS gstutils.c:3110:gst_pad_peer_query_caps:<pngenc:src> peer query returned ANY
0:00:18.054076243     6 0x7fc528027060 DEBUG        rtpjitterbuffer gstrtpjitterbuffer.c:2221:add_timer:<rtpjitterbuffer0> add timer 1 for seqnum 9224 to 0:00:03.921774074, delay 0:00:00.000000000
0:00:18.054083630     6 0x7fc528027060 DEBUG        rtpjitterbuffer gstrtpjitterbuffer.c:2244:add_timer: signal timer, 1 waiters
0:00:18.054086687     6 0x7fc528027300 LOG                  default gstvideoutilsprivate.c:128:__gst_video_element_proxy_getcaps:<pngenc> template caps video/x-raw, format=(string){ RGBA, RGB, GRAY8, GRAY16_BE }, width=(int)[ 1, 2147483647 ], height=(int)[ 1, 2147483647 ], framerate=(fraction)[ 0/1, 2147483647/1 ]
0:00:18.054089032     6 0x7fc528027060 DEBUG        rtpjitterbuffer gstrtpjitterbuffer.c:2221:add_timer:<rtpjitterbuffer0> add timer 1 for seqnum 9225 to 0:00:03.921774074, delay 0:00:00.000000000
0:00:18.054097781     6 0x7fc528027060 DEBUG        rtpjitterbuffer gstrtpjitterbuffer.c:2244:add_timer: signal timer, 1 waiters
0:00:18.054098048     6 0x7fc528027300 LOG                  default gstvideoutilsprivate.c:129:__gst_video_element_proxy_getcaps:<pngenc> allowed caps image/png, width=(int)[ 16, 1000000 ], height=(int)[ 16, 1000000 ], framerate=(fraction)[ 0/1, 2147483647/1 ]
0:00:18.054107466     6 0x7fc528027060 DEBUG        rtpjitterbuffer gstrtpjitterbuffer.c:2221:add_timer:<rtpjitterbuffer0> add timer 1 for seqnum 9226 to 0:00:03.921774074, delay 0:00:00.000000000
0:00:18.054112323     6 0x7fc528027060 DEBUG        rtpjitterbuffer gstrtpjitterbuffer.c:2244:add_timer: signal timer, 1 waiters
0:00:18.054118878     6 0x7fc528027060 DEBUG        rtpjitterbuffer gstrtpjitterbuffer.c:2221:add_timer:<rtpjitterbuffer0> add timer 1 for seqnum 9227 to 0:00:03.921774074, delay 0:00:00.000000000
0:00:18.054120236     6 0x7fc528027300 LOG                  default gstvideoutilsprivate.c:147:__gst_video_element_proxy_getcaps:<pngenc> proxy caps video/x-raw, width=(int)[ 16, 1000000 ], height=(int)[ 16, 1000000 ], framerate=(fraction)[ 0/1, 2147483647/1 ], format=(string){ RGBA, RGB, GRAY8, GRAY16_BE }
0:00:18.054126863     6 0x7fc528027060 DEBUG        rtpjitterbuffer gstrtpjitterbuffer.c:2244:add_timer: signal timer, 1 waiters
0:00:18.054137786     6 0x7fc528027300 LOG             videoencoder gstvideoencoder.c:739:gst_video_encoder_sink_getcaps:<pngenc> Returning caps video/x-raw, width=(int)[ 16, 1000000 ], height=(int)[ 16, 1000000 ], framerate=(fraction)[ 0/1, 2147483647/1 ], format=(string){ RGBA, RGB, GRAY8, GRAY16_BE }
0:00:18.054140035     6 0x7fc528027060 DEBUG        rtpjitterbuffer gstrtpjitterbuffer.c:2221:add_timer:<rtpjitterbuffer0> add timer 1 for seqnum 9228 to 0:00:03.921774074, delay 0:00:00.000000000
0:00:18.054148195     6 0x7fc528027300 DEBUG               GST_PADS gstpad.c:4076:gst_pad_query:<pngenc:sink> sent query 0x7fc500056230 (caps), result 1
0:00:18.054150824     6 0x7fc528027060 DEBUG        rtpjitterbuffer gstrtpjitterbuffer.c:2244:add_timer: signal timer, 1 waiters
0:00:18.054160637     6 0x7fc528027300 DEBUG                default gstutils.c:2769:query_caps_func:<tee_queue:src> intersect with result video/x-raw, width=(int)[ 16, 1000000 ], height=(int)[ 16, 1000000 ], framerate=(fraction)[ 0/1, 2147483647/1 ], format=(string){ RGBA, RGB, GRAY8, GRAY16_BE }
0:00:18.054164937     6 0x7fc528027060 DEBUG        rtpjitterbuffer gstrtpjitterbuffer.c:2221:add_timer:<rtpjitterbuffer0> add timer 1 for seqnum 9229 to 0:00:03.921774074, delay 0:00:00.000000000
0:00:18.054169967     6 0x7fc528027060 DEBUG        rtpjitterbuffer gstrtpjitterbuffer.c:2244:add_timer: signal timer, 1 waiters
0:00:18.054170489     6 0x7fc528027300 DEBUG                default gstutils.c:2771:query_caps_func:<tee_queue:src> intersected video/x-raw, width=(int)[ 16, 1000000 ], height=(int)[ 16, 1000000 ], framerate=(fraction)[ 0/1, 2147483647/1 ], format=(string){ RGBA, RGB, GRAY8, GRAY16_BE }
0:00:18.054180731     6 0x7fc528027060 DEBUG        rtpjitterbuffer gstrtpjitterbuffer.c:2221:add_timer:<rtpjitterbuffer0> add timer 1 for seqnum 9230 to 0:00:03.921774074, delay 0:00:00.000000000
0:00:18.054184910     6 0x7fc528027300 DEBUG               GST_PADS gstpad.c:3437:gst_pad_query_default:<tee_queue:sink> not forwarding 0x7fc500056230 (caps) query
0:00:18.054186852     6 0x7fc528027060 DEBUG        rtpjitterbuffer gstrtpjitterbuffer.c:2244:add_timer: signal timer, 1 waiters
0:00:18.054194185     6 0x7fc528027300 DEBUG               GST_PADS gstpad.c:4076:gst_pad_query:<tee_queue:sink> sent query 0x7fc500056230 (caps), result 1
0:00:18.054200998     6 0x7fc528027060 DEBUG        rtpjitterbuffer gstrtpjitterbuffer.c:2221:add_timer:<rtpjitterbuffer0> add timer 1 for seqnum 9231 to 0:00:03.921774074, delay 0:00:00.000000000
0:00:18.054204030     6 0x7fc528027300 DEBUG                default gstutils.c:2769:query_caps_func:<tee:src_1> intersect with result video/x-raw, width=(int)[ 16, 1000000 ], height=(int)[ 16, 1000000 ], framerate=(fraction)[ 0/1, 2147483647/1 ], format=(string){ RGBA, RGB, GRAY8, GRAY16_BE }
0:00:18.054207307     6 0x7fc528027060 DEBUG        rtpjitterbuffer gstrtpjitterbuffer.c:2244:add_timer: signal timer, 1 waiters
0:00:18.054217120     6 0x7fc528027300 DEBUG                default gstutils.c:2771:query_caps_func:<tee:src_1> intersected EMPTY
0:00:18.054221779     6 0x7fc528027060 DEBUG        rtpjitterbuffer gstrtpjitterbuffer.c:2221:add_timer:<rtpjitterbuffer0> add timer 1 for seqnum 9232 to 0:00:03.921774074, delay 0:00:00.000000000
0:00:18.054226781     6 0x7fc528027300 DEBUG               GST_PADS gstpad.c:3437:gst_pad_query_default:<tee:sink> not forwarding 0x7fc500056230 (caps) query
0:00:18.054228150     6 0x7fc528027060 DEBUG        rtpjitterbuffer gstrtpjitterbuffer.c:2244:add_timer: signal timer, 1 waiters
0:00:18.054235704     6 0x7fc528027300 DEBUG               GST_PADS gstpad.c:4076:gst_pad_query:<tee:sink> sent query 0x7fc500056230 (caps), result 1
0:00:18.054242445     6 0x7fc528027060 DEBUG        rtpjitterbuffer gstrtpjitterbuffer.c:2221:add_timer:<rtpjitterbuffer0> add timer 1 for seqnum 9233 to 0:00:03.921774074, delay 0:00:00.000000000
0:00:18.054242883     6 0x7fc528027300 DEBUG               GST_CAPS gstutils.c:3110:gst_pad_peer_query_caps:<video_convert:src> peer query returned EMPTY
0:00:18.054250557     6 0x7fc528027060 DEBUG        rtpjitterbuffer gstrtpjitterbuffer.c:2244:add_timer: signal timer, 1 waiters
0:00:18.054257702     6 0x7fc528027300 DEBUG          basetransform gstbasetransform.c:684:gst_base_transform_query_caps:<video_convert:sink> peer caps  EMPTY
0:00:18.054271263     6 0x7fc528027060 DEBUG        rtpjitterbuffer gstrtpjitterbuffer.c:2221:add_timer:<rtpjitterbuffer0> add timer 1 for seqnum 9234 to 0:00:03.921774074, delay 0:00:00.000000000
0:00:18.054276175     6 0x7fc528027060 DEBUG        rtpjitterbuffer gstrtpjitterbuffer.c:2244:add_timer: signal timer, 1 waiters
0:00:18.054281697     6 0x7fc528027060 DEBUG        rtpjitterbuffer gstrtpjitterbuffer.c:2221:add_timer:<rtpjitterbuffer0> add timer 1 for seqnum 9235 to 0:00:03.921774074, delay 0:00:00.000000000
0:00:18.054285177     6 0x7fc528027060 DEBUG        rtpjitterbuffer gstrtpjitterbuffer.c:2244:add_timer: signal timer, 1 waiters
0:00:18.054290454     6 0x7fc528027060 DEBUG        rtpjitterbuffer gstrtpjitterbuffer.c:2221:add_timer:<rtpjitterbuffer0> add timer 1 for seqnum 9236 to 0:00:03.921774074, delay 0:00:00.000000000
0:00:18.054293889     6 0x7fc528027060 DEBUG        rtpjitterbuffer gstrtpjitterbuffer.c:2244:add_timer: signal timer, 1 waiters
0:00:18.054295149     6 0x7fc528027300 DEBUG          basetransform gstbasetransform.c:687:gst_base_transform_query_caps:<video_convert:sink> our template  video/x-raw, format=(string){ I420, YV12, YUY2, UYVY, AYUV, VUYA, RGBx, BGRx, xRGB, xBGR, RGBA, BGRA, ARGB, ABGR, RGB, BGR, Y41B, Y42B, YVYU, Y444, v210, v216, Y210, Y410, NV12, NV21, GRAY8, GRAY16_BE, GRAY16_LE, v308, RGB16, BGR16, RGB15, BGR15, UYVP, A420, RGB8P, YUV9, YVU9, IYU1, ARGB64, AYUV64, r210, I420_10BE, I420_10LE, I422_10BE, I422_10LE, Y444_10BE, Y444_10LE, GBR, GBR_10BE, GBR_10LE, NV16, NV24, NV12_64Z32, A420_10BE, A420_10LE, A422_10BE, A422_10LE, A444_10BE, A444_10LE, NV61, P010_10BE, P010_10LE, IYU2, VYUY, GBRA, GBRA_10BE, GBRA_10LE, BGR10A2_LE, GBR_12BE, GBR_12LE, GBRA_12BE, GBRA_12LE, I420_12BE, I420_12LE, I422_12BE, I422_12LE, Y444_12BE, Y444_12LE, GRAY10_LE32, NV12_10LE32, NV16_10LE32, NV12_10LE40 }, width=(int)[ 1, 2147483647 ], height=(int)[ 1, 2147483647 ], framerate=(fraction)[ 0/1, 2147483647/1 ]; video/x-raw(ANY), format=(string){ I420, YV12, YUY2, UYVY, AYUV, VUYA, RGBx, BGRx, xRGB, xBGR, RGBA, BGRA, ARGB, ABGR, RGB, BGR, Y41B, Y42B, YVYU, Y444, v210, v216, Y210, Y410, NV12, NV21, GRAY8, GRAY16_BE, GRAY16_LE, v308, RGB16, BGR16, RGB15, BGR15, UYVP, A420, RGB8P, YUV9, YVU9, IYU1, ARGB64, AYUV64, r210, I420_10BE, I420_10LE, I422_10BE, I422_10LE, Y444_10BE, Y444_10LE, GBR, GBR_10BE, GBR_10LE, NV16, NV24, NV12_64Z32, A420_10BE, A420_10LE, A422_10BE, A422_10LE, A444_10BE, A444_10LE, NV61, P010_10BE, P010_10LE, IYU2, VYUY, GBRA, GBRA_10BE, GBRA_10LE, BGR10A2_LE, GBR_12BE, GBR_12LE, GBRA_12BE, GBRA_12LE, I420_12BE, I420_12LE, I422_12BE, I422_12LE, Y444_12BE, Y444_12LE, GRAY10_LE32, NV12_10LE32, NV16_10LE32, NV12_10LE40 }, width=(int)[ 1, 2147483647 ], height=(int)[ 1, 2147483647 ], framerate=(fraction)[ 0/1, 2147483647/1 ]
0:00:18.054299183     6 0x7fc528027060 DEBUG        rtpjitterbuffer gstrtpjitterbuffer.c:2221:add_timer:<rtpjitterbuffer0> add timer 1 for seqnum 9237 to 0:00:03.921774074, delay 0:00:00.000000000
0:00:18.054309026     6 0x7fc528027300 DEBUG          basetransform gstbasetransform.c:689:gst_base_transform_query_caps:<video_convert:sink> intersected EMPTY
0:00:18.054310826     6 0x7fc528027060 DEBUG        rtpjitterbuffer gstrtpjitterbuffer.c:2244:add_timer: signal timer, 1 waiters
0:00:18.054317120     6 0x7fc528027300 DEBUG          basetransform gstbasetransform.c:471:gst_base_transform_transform_caps:<video_convert> transform caps (direction = 1)
0:00:18.054320836     6 0x7fc528027060 DEBUG        rtpjitterbuffer gstrtpjitterbuffer.c:2221:add_timer:<rtpjitterbuffer0> add timer 1 for seqnum 9238 to 0:00:03.921774074, delay 0:00:00.000000000
0:00:18.054323399     6 0x7fc528027300 LOG            basetransform gstbasetransform.c:473:gst_base_transform_transform_caps:<video_convert> from: EMPTY
0:00:18.054324695     6 0x7fc528027060 DEBUG        rtpjitterbuffer gstrtpjitterbuffer.c:2244:add_timer: signal timer, 1 waiters
0:00:18.054332634     6 0x7fc528027300 DEBUG           videoconvert gstvideoconvert.c:405:gst_video_convert_transform_caps:<video_convert> transformed EMPTY into EMPTY
0:00:18.054335246     6 0x7fc528027060 DEBUG        rtpjitterbuffer gstrtpjitterbuffer.c:2221:add_timer:<rtpjitterbuffer0> add timer 1 for seqnum 9239 to 0:00:03.921774074, delay 0:00:00.000000000
0:00:18.054340022     6 0x7fc528027300 LOG            basetransform gstbasetransform.c:475:gst_base_transform_transform_caps:<video_convert>   to: EMPTY
0:00:18.054346128     6 0x7fc528027060 DEBUG        rtpjitterbuffer gstrtpjitterbuffer.c:2244:add_timer: signal timer, 1 waiters
0:00:18.054356680     6 0x7fc528027300 DEBUG          basetransform gstbasetransform.c:498:gst_base_transform_transform_caps:<video_convert> to: EMPTY
0:00:18.054358945     6 0x7fc528027060 DEBUG        rtpjitterbuffer gstrtpjitterbuffer.c:2221:add_timer:<rtpjitterbuffer0> add timer 1 for seqnum 9240 to 0:00:03.921774074, delay 0:00:00.000000000
0:00:18.054361237     6 0x7fc528027300 DEBUG          basetransform gstbasetransform.c:697:gst_base_transform_query_caps:<video_convert:sink> transformed  EMPTY
0:00:18.054366472     6 0x7fc528027060 DEBUG        rtpjitterbuffer gstrtpjitterbuffer.c:2244:add_timer: signal timer, 1 waiters
0:00:18.054372830     6 0x7fc528027300 DEBUG          basetransform gstbasetransform.c:731:gst_base_transform_query_caps:<video_convert> returning  EMPTY
0:00:18.054382419     6 0x7fc528027300 DEBUG               GST_PADS gstpad.c:4076:gst_pad_query:<video_convert:sink> sent query 0x7fc5000560a0 (caps), result 1
0:00:18.054385981     6 0x7fc528027060 DEBUG        rtpjitterbuffer gstrtpjitterbuffer.c:2221:add_timer:<rtpjitterbuffer0> add timer 1 for seqnum 9241 to 0:00:03.921774074, delay 0:00:00.000000000
0:00:18.054388286     6 0x7fc528027300 DEBUG               GST_CAPS gstutils.c:3064:gst_pad_query_caps:<video_convert:sink> query returned EMPTY
0:00:18.054390704     6 0x7fc528027060 DEBUG        rtpjitterbuffer gstrtpjitterbuffer.c:2244:add_timer: signal timer, 1 waiters
0:00:18.054402901     6 0x7fc528027060 DEBUG        rtpjitterbuffer gstrtpjitterbuffer.c:2221:add_timer:<rtpjitterbuffer0> add timer 1 for seqnum 9242 to 0:00:03.921774074, delay 0:00:00.000000000
0:00:18.054404901     6 0x7fc528027300 DEBUG               GST_CAPS gstpad.c:2277:gst_pad_link_check_compatible_unlocked:<decodebin:src_0> src caps video/x-raw, format=(string){ I420, YUY2, RGB, BGR, Y42B, Y444, YUV9, Y41B, GRAY8, RGB8P, I420, Y42B, Y444, UYVY, NV12, NV21, ARGB, RGBA, ABGR, BGRA, GRAY16_BE, GRAY16_LE, A420, RGB16, RGB15, I420_10BE, I420_10LE, I422_10BE, I422_10LE, Y444_10BE, Y444_10LE, GBR, GBR_10BE, GBR_10LE, A420_10BE, A420_10LE, A422_10BE, A422_10LE, A444_10BE, A444_10LE, GBRA, xRGB, RGBx, xBGR, BGRx, I420_12BE, I420_12LE, I422_12BE, I422_12LE, Y444_12BE, Y444_12LE, GBR_12BE, GBR_12LE, GBRA_12BE, GBRA_12LE }
0:00:18.054412902     6 0x7fc528027060 DEBUG        rtpjitterbuffer gstrtpjitterbuffer.c:2244:add_timer: signal timer, 1 waiters
0:00:18.054419307     6 0x7fc528027300 DEBUG               GST_CAPS gstpad.c:2279:gst_pad_link_check_compatible_unlocked:<video_convert:sink> sink caps EMPTY
0:00:18.054427402     6 0x7fc528027300 DEBUG               GST_CAPS gstpad.c:2297:gst_pad_link_check_compatible_unlocked: caps are not compatible
0:00:18.054427640     6 0x7fc528027060 DEBUG        rtpjitterbuffer gstrtpjitterbuffer.c:2221:add_timer:<rtpjitterbuffer0> add timer 1 for seqnum 9243 to 0:00:03.921774074, delay 0:00:00.000000000
0:00:18.054434034     6 0x7fc528027300 INFO                GST_PADS gstpad.c:2434:gst_pad_link_prepare: caps are incompatible
0:00:18.054441032     6 0x7fc528027060 DEBUG        rtpjitterbuffer gstrtpjitterbuffer.c:2244:add_timer: signal timer, 1 waiters
0:00:18.054448980     6 0x7fc528027300 INFO                GST_PADS gstpad.c:2527:gst_pad_link_full: link between decodebin:src_0 and video_convert:sink failed: no common format
0:00:18.054455550     6 0x7fc528027060 DEBUG        rtpjitterbuffer gstrtpjitterbuffer.c:2221:add_timer:<rtpjitterbuffer0> add timer 1 for seqnum 9244 to 0:00:03.921774074, delay 0:00:00.000000000
0:00:18.054459346     6 0x7fc528027300 LOG              GST_MESSAGE gstmessage.c:303:gst_message_new_custom: source sink: creating new message 0x7fc514034300 structure-change
0:00:18.054461915     6 0x7fc528027060 DEBUG        rtpjitterbuffer gstrtpjitterbuffer.c:2244:add_timer: signal timer, 1 waiters
0:00:18.054475712     6 0x7fc528027300 DEBUG                GST_BUS gstbus.c:315:gst_bus_post:<bus4> [msg 0x7fc514034300] posting on bus structure-change message: 0x7fc514034300, time 99:99:99.999999999, seq-num 287, element 'sink', GstMessageStructureChange, type=(GstStructureChangeType)GST_STRUCTURE_CHANGE_TYPE_PAD_LINK, owner=(GstElement)"\(GstDecodeBin\)\ decodebin", busy=(boolean)false;
0:00:18.054475858     6 0x7fc528027060 DEBUG        rtpjitterbuffer gstrtpjitterbuffer.c:2221:add_timer:<rtpjitterbuffer0> add timer 1 for seqnum 9245 to 0:00:03.921774074, delay 0:00:00.000000000
0:00:18.054485127     6 0x7fc528027300 DEBUG                    bin gstbin.c:3718:gst_bin_handle_message_func:<test-pipeline> [msg 0x7fc514034300] handling child sink message of type structure-change
0:00:18.054487573     6 0x7fc528027060 DEBUG        rtpjitterbuffer gstrtpjitterbuffer.c:2244:add_timer: signal timer, 1 waiters
0:00:18.054493824     6 0x7fc528027300 LOG                      bin gstbin.c:860:message_check: looking at message 0x7fc5280592c0: 1
0:00:18.054500477     6 0x7fc528027060 DEBUG        rtpjitterbuffer gstrtpjitterbuffer.c:2221:add_timer:<rtpjitterbuffer0> add timer 1 for seqnum 9246 to 0:00:03.921774074, delay 0:00:00.000000000
0:00:18.054501355     6 0x7fc528027300 DEBUG                    bin gstbin.c:956:bin_remove_messages:<video_convert:sink> deleting message 0x7fc5280592c0 of type structure-change (types 0x00001000)
0:00:18.054508236     6 0x7fc528027060 DEBUG        rtpjitterbuffer gstrtpjitterbuffer.c:2244:add_timer: signal timer, 1 waiters
0:00:18.054516306     6 0x7fc528027300 LOG              GST_MESSAGE gstmessage.c:208:_gst_message_free: finalize message 0x7fc5280592c0, structure-change from sink
0:00:18.054522839     6 0x7fc528027060 DEBUG        rtpjitterbuffer gstrtpjitterbuffer.c:2221:add_timer:<rtpjitterbuffer0> add timer 1 for seqnum 9247 to 0:00:03.921774074, delay 0:00:00.000000000
0:00:18.054523699     6 0x7fc528027300 LOG                      bin gstbin.c:860:message_check: looking at message 0x562feee1b1b0: 0
0:00:18.054530895     6 0x7fc528027060 DEBUG        rtpjitterbuffer gstrtpjitterbuffer.c:2244:add_timer: signal timer, 1 waiters
0:00:18.054542367     6 0x7fc528027300 DEBUG                    bin gstbin.c:962:bin_remove_messages:<decodebin> not deleting message 0x562feee1b1b0 of type 0x00100000
0:00:18.054543219     6 0x7fc528027060 DEBUG        rtpjitterbuffer gstrtpjitterbuffer.c:2221:add_timer:<rtpjitterbuffer0> add timer 1 for seqnum 9248 to 0:00:03.921774074, delay 0:00:00.000000000
0:00:18.054549427     6 0x7fc528027300 LOG                      bin gstbin.c:860:message_check: looking at message 0x562feede0420: 0
0:00:18.054556037     6 0x7fc528027060 DEBUG        rtpjitterbuffer gstrtpjitterbuffer.c:2244:add_timer: signal timer, 1 waiters
0:00:18.054566937     6 0x7fc528027300 DEBUG                    bin gstbin.c:962:bin_remove_messages:<udp_sink> not deleting message 0x562feede0420 of type 0x00100000
0:00:18.054567386     6 0x7fc528027060 DEBUG        rtpjitterbuffer gstrtpjitterbuffer.c:2221:add_timer:<rtpjitterbuffer0> add timer 1 for seqnum 9249 to 0:00:03.921774074, delay 0:00:00.000000000
0:00:18.054573955     6 0x7fc528027300 LOG                      bin gstbin.c:860:message_check: looking at message 0x562feede0120: 0
0:00:18.054583095     6 0x7fc528027300 DEBUG                    bin gstbin.c:962:bin_remove_messages:<filesink> not deleting message 0x562feede0120 of type 0x00100000
0:00:18.054584062     6 0x7fc528027060 DEBUG        rtpjitterbuffer gstrtpjitterbuffer.c:2244:add_timer: signal timer, 1 waiters
0:00:18.054591132     6 0x7fc528027300 DEBUG                GST_BUS gstbus.c:351:gst_bus_post:<bus4> [msg 0x7fc514034300] dropped
0:00:18.054599743     6 0x7fc528027300 LOG              GST_MESSAGE gstmessage.c:208:_gst_message_free: finalize message 0x7fc514034300, structure-change from sink

I assume the relevant part is this:

0:00:18.054404901     6 0x7fc528027300 DEBUG               GST_CAPS gstpad.c:2277:gst_pad_link_check_compatible_unlocked:<decodebin:src_0> src caps video/x-raw, format=(string){ I420, YUY2, RGB, BGR, Y42B, Y444, YUV9, Y41B, GRAY8, RGB8P, I420, Y42B, Y444, UYVY, NV12, NV21, ARGB, RGBA, ABGR, BGRA, GRAY16_BE, GRAY16_LE, A420, RGB16, RGB15, I420_10BE, I420_10LE, I422_10BE, I422_10LE, Y444_10BE, Y444_10LE, GBR, GBR_10BE, GBR_10LE, A420_10BE, A420_10LE, A422_10BE, A422_10LE, A444_10BE, A444_10LE, GBRA, xRGB, RGBx, xBGR, BGRx, I420_12BE, I420_12LE, I422_12BE, I422_12LE, Y444_12BE, Y444_12LE, GBR_12BE, GBR_12LE, GBRA_12BE, GBRA_12LE }
0:00:18.054412902     6 0x7fc528027060 DEBUG        rtpjitterbuffer gstrtpjitterbuffer.c:2244:add_timer: signal timer, 1 waiters
0:00:18.054419307     6 0x7fc528027300 DEBUG               GST_CAPS gstpad.c:2279:gst_pad_link_check_compatible_unlocked:<video_convert:sink> sink caps EMPTY
0:00:18.054427402     6 0x7fc528027300 DEBUG               GST_CAPS gstpad.c:2297:gst_pad_link_check_compatible_unlocked: caps are not compatible
0:00:18.054427640     6 0x7fc528027060 DEBUG        rtpjitterbuffer gstrtpjitterbuffer.c:2221:add_timer:<rtpjitterbuffer0> add timer 1 for seqnum 9243 to 0:00:03.921774074, delay 0:00:00.000000000
0:00:18.054434034     6 0x7fc528027300 INFO                GST_PADS gstpad.c:2434:gst_pad_link_prepare: caps are incompatible
0:00:18.054441032     6 0x7fc528027060 DEBUG        rtpjitterbuffer gstrtpjitterbuffer.c:2244:add_timer: signal timer, 1 waiters
0:00:18.054448980     6 0x7fc528027300 INFO                GST_PADS gstpad.c:2527:gst_pad_link_full: link between decodebin:src_0 and video_convert:sink failed: no common format

I didn’t look in detail, but the problem is most likely that you only have a videoconvert before the tee, and then two tee branches: one to x264enc, another to pngenc. pngenc only supports RGB/grayscale, x264enc only supports YUV formats.

So the videoconvert there could convert to either of the two, but there’s not a single format that is supported by both encoders so it can’t work like this. Try putting two videoconverts after the tee right in front of each encoder.

Then this should work fine.

Thank you, this did do the trick!

However, now the pipeline does not go to the PLAYING state anymore (unless I remove the branch with the pngenc entirely from the pipeline). Do you, by any chance, have an idea what might be the cause in this case?

I saw in a different post (http://gstreamer-devel.966125.n4.nabble.com/tee-won-t-go-in-playing-state-td4680128.html) that it seemed to have something to do with the x264enc requiring 3 seconds of material and possibly getting stuck there because of the queue on the other branch not being able to hold that. However, I have added a tee_queue.set_property_from_str("Queue-leaky", &"downstream"); to that queue to ensure it will just drop frames if it’s full. I can’t tell if that does work, however, as the set_property_from_str function does not provide a result.

The answer will be in the logs again :slight_smile: I assume the PNG queue runs full and blocks on the sink, while the other queue is always empty and a whole GOP is tried to be queued up in hlssink2 before it would pass the first buffer to the sink. So the whole pipeline locks up because it never reaches a point where both sinks receive a buffer.

async=false on the sinks can help (can’t do that directly with hlssink2, you’d have to get the filesink inside it and set it on that), making the queues big enough would also help (the PNG need to be able to hold at least one GOP).

There’s also the problem that x264enc only outputs after 3s so your queues need to be big enough on the PNG side for that, but that’s going to be a smaller problem here compared to the GOP size.

If you don’t get a critical warning printed on the terminal, setting the property worked. Making the PNG queue leaky would also potentially work but it would mean that you’re dropping some data.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.