Don't know if this should be tagged 'maj'. It does not work in this particular case; it might be isolated to this setup.

You don't have the sprop-parameter-sets on the receiver udpsrc so it does not know about SPS and PPS needed to decode h264. Why exactly do you think this should work?

I might be missing something, but 

it does work if the decoder is started before the sender is started; so the data seems to be sent at that point.

Only the receiver/decoder is not able to pick in on the stream once the sender is started.

so:

1. start receiver then sender: OK
2. start sender then receiver: NOK

I'm trying to understand what is the difference between the two cases. The payloader only sends this packet once (gets it only once too):

0:00:00.584424652 27649      0x106d240 DEBUG             rtph264pay gstrtph264pay.c:748:gst_rtp_h264_pay_handle_buffer:<rtph264pay0> got 30 bytes
0:00:00.584436546 27649      0x106d240 DEBUG             rtph264pay gstrtph264pay.c:806:gst_rtp_h264_pay_handle_buffer:<rtph264pay0> found first start at 0, bytes left 30
0:00:00.584446946 27649      0x106d240 DEBUG             rtph264pay gstrtph264pay.c:833:gst_rtp_h264_pay_handle_buffer:<rtph264pay0> found next start at 18 of size 18
0:00:00.584456513 27649      0x106d240 DEBUG             rtph264pay gstrtph264pay.c:456:gst_rtp_h264_pay_decode_nal: NAL payload len=18
0:00:00.584465079 27649      0x106d240 DEBUG             rtph264pay gstrtph264pay.c:467:gst_rtp_h264_pay_decode_nal: Found SPS 0 3 7 Len=18
0:00:00.584473810 27649      0x106d240 DEBUG             rtph264pay gstrtph264pay.c:493:gst_rtp_h264_pay_decode_nal: Profile level IDC = 420029

at startup; type 7 is never encountered again. This seems to match the depayloader:

0:00:01.439597751 27702      0x153d400 DEBUG           rtph264depay gstrtph264depay.c:255:gst_rtp_h264_depay_setcaps:<rtph264depay0> decoding param 0 (Z0IAKeKQFoe2BqwYBBuHiRFQ)
0:00:01.439625204 27702      0x153d400 DEBUG           rtph264depay gstrtph264depay.c:261:gst_rtp_h264_depay_setcaps:<rtph264depay0> decoded 18 bytes
0:00:01.439655622 27702      0x153d400 DEBUG           rtph264depay gstrtph264depay.c:255:gst_rtp_h264_depay_setcaps:<rtph264depay0> decoding param 1 (aM48gA==)
0:00:01.439666585 27702      0x153d400 DEBUG           rtph264depay gstrtph264depay.c:261:gst_rtp_h264_depay_setcaps:<rtph264depay0> decoded 4 bytes


So probably this packet is inserted in the steam at startup of the depayloader; but is never repeated. If it would be repeated; I guess the decoder could pick in on the sender.

Yes, the PPS and SPS are probably inserted at the start of the stream (although that is purely coincidental). There is currently no code to parse and insert the config params at regular intervals into the stream.

A patch to the payloader that inserts the config params into the payloaded packets (with a configurable delay) would be accepted.

Created attachment 150167 [details] [review]
patch rtph264pay to send sps/pps in data stream

Waits delay seconds to re-insert SPS/PPS into the stream right in front of an IDR frame to avoid decoding errors.

The patch resets the counter once it finds a Type 7/8 packet in the stream. So if the stream sends 7/8 type packets in a regular fashion, this option will not have any effect. It will only affect streams that do not send out SPS/PPS params regularly.

This option could probably be enabled by default to e.g. 3 or 5 seconds.

It is currently set to 0 (disabled) to keep the same default data stream behaviour as previously.

Ideally, it would also re-send the sps/pps on a downstream GstForceKeyUnit event (which should come just before a new forced keyframe).

Created attachment 150173 [details] [review]
param tweak spspps_send

Changed the parameter to an int instead of uint.

Setting it to 0 will send the sps/pps parameters in front of every IDR frame instead of waiting for a timeout; should minimise the time for the decoder to pick in on the stream in network applications.

default (disabled) is -1.

 /* resend SPS */
 sps = gst_buffer_new_and_alloc (rtph264pay->sps_len);
 gst_buffer_set_data (sps, rtph264pay->sps, rtph264pay->sps_len);

This leaks the allocated data and frees the sps multiple times. Maybe you meant to memcpy into the allocated buffer data.

Also your code does nothing for the packetized case. I'm rewriting it a bit now to reset the counters in the _decode_nal() function and to prepend the SPS in the _payload_nal() function (when needed).

Rewrote most of the patch, please test this. It should also work with AVC streams.

commit 46871993488a30c0544f5c07c2b4a580ece59cc1
Author: Wim Taymans <wim.taymans@collabora.co.uk>
Date:   Mon Dec 21 20:39:53 2009 +0100

    rtph264pay: add option to insert PPS/SPS in streams
    
    Add a new spspps-interval property to instruct the payloader to insert
    SPS and PPS at periodic intervals in the stream.
    Rework the SPS/PPS handling so that bytestream and AVC sample code both use the
    same code paths to handle sprop-parameter-sets. This also allows to have the AVC
    code to insert SPS/PPS like the bytestream code.
    
    Fixes #604913

Tested the rebuilt code with the test setup as described; and the decoder is able to pick in as expected.