GNOME Bugzilla – Bug 760980
splitmuxsink: corrupt output when receiving H264 in RTP over UDP and using mpegtsmux
Last modified: 2018-11-03 15:07:18 UTC
We observe that splitmuxsink+mpegtsmux generates corrupt segments when receiving H264 in RTP over UDP. (Note: mp4mux works just fine) Sender pipeline: gst-launch-1.0 -eevv videotestsrc ! capsfilter caps="video/x-raw,format=I420,width=1280,height=720,framerate=(fraction)25/1" ! queue ! x264enc key-int-max=25 bitrate=3096 option-string="no-scenecut" bframes=0 ! h264parse config-interval=1 ! rtph264pay ! udpsink port=5004 Receiver pipeline to make sure data is received correctly: gst-launch-1.0 -eeevv udpsrc port=5004 caps="application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264, packetization-mode=(string)1, profile-level-id=(string)64001f,payload=(int)96, ssrc=(uint)2359631590, timestamp-offset=(uint)1144720463, seqnum-offset=(uint)20301" ! rtpjitterbuffer ! rtph264depay ! h264parse ! avdec_h264 ! queue ! xvimagesink Receiver pipeline that generates corrupt MPEG2TS files: gst-launch-1.0 -eeevv udpsrc port=5004 caps="application/x-rtp\,\ media\=\(string\)video\,\ clock-rate\=\(int\)90000\,\ encoding-name\=\(string\)H264\,\ packetization-mode\=\(string\)1\,\ profile-level-id\=\(string\)64001f\,\ payload\=\(int\)96\,\ ssrc\=\(uint\)3675893665\,\ timestamp-offset\=\(uint\)1120179275\,\ seqnum-offset\=\(uint\)19\,\ a-framerate\=\(string\)25" ! rtpjitterbuffer ! rtph264depay ! h264parse ! identity sync=true ! queue ! splitmuxsink muxer=mpegtsmux location=segment_%05d.ts max-size-time=3000001000 We can "play" the first segment, but ffmpeg reports corrupt timestamps. Every subsequent segment is unplayable. On a side note (but perhaps that should go in a separate bug report?), we cannot pass the sprop-parameter-sets on udpsrc. When we add the following sprop-parameter-sets: sprop-parameter-sets\=\(string\)\"Z2QAH6yyAKALdgIgAAADACAAAAZR4wZJ\\\,aOvMsiw\\\=\"\,\ in the caps above, gst-launch throws an error about incorrect caps. e.g gst-launch-1.0 -eeevv udpsrc port=5004 caps="application/x-rtp\,\ media\=\(string\)video\,\ clock-rate\=\(int\)90000\,\ encoding-name\=\(string\)H264\,\ packetization-mode\=\(string\)1\,\ profile-level-id\=\(string\)64001f\,\ sprop-parameter-sets\=\(string\)\"Z2QAH6yyAKALdgIgAAADACAAAAZR4wZJ\\\,aOvMsiw\\\=\"\,\ payload\=\(int\)96\,\ ssrc\=\(uint\)3675893665\,\ timestamp-offset\=\(uint\)1120179275\,\ seqnum-offset\=\(uint\)19\,\ a-framerate\=\(string\)25" ! rtpjitterbuffer ! rtph264depay ! h264parse ! identity sync=true ! queue ! splitmuxsink muxer=mpegtsmux location=segment_%05d.ts max-size-time=3000001000 Throws this error message: WARNING: erroneous pipeline: could not set property "caps" in element "udpsrc0" to "application/x-rtp,\ media=(string)video,\ clock-rate=(int)90000,\ encoding-name=(string)H264,\ packetization-mode=(string)1,\ profile-level-id=(string)64001f,\ sprop-parameter-sets=(string)"Z2QAH6yyAKALdgIgAAADACAAAAZR4wZJ\\,aOvMsiw\\=",\ payload=(int)96,\ ssrc=(uint)3675893665,\ timestamp-offset=(uint)1120179275,\ seqnum-offset=(uint)19,\ a-framerate=(string)25" Not sure whether this problem is related to https://bugzilla.gnome.org/show_bug.cgi?id=752999 (already marked as RESOLVED). I am on 1.7.0.1, but we have observed the same behavior on other versions (1.6.x) as well. I am also willing to test against git HEAD if that helps.
I can confirm that this issue also applies to GIT master. An example output of ffprobe of the first and second segment are now attached to this bug report.
Created attachment 319965 [details] ffprobe of segment 0 (can be played with ffplay but timestamping errors are reported)
Created attachment 319966 [details] ffprobe of segment 1 (cannot be played)
Anyone care to look into this problem? Given the output of ffprobe it looks to me that SPS/PPS data is missing at the beginning of all but the first segment. This problem is still prevalent in git master. ffprobe: [h264 @ 0x15307e0] decode_slice_header error [h264 @ 0x15307e0] no frame! [mpegts @ 0x152bc40] decoding for stream 0 failed [mpegts @ 0x152bc40] Could not find codec parameters for stream 0 (Video: h264 (HDMV / 0x564D4448), none): unspecified size Consider increasing the value for the 'analyzeduration' and 'probesize' options Input #0, mpegts, from 'segment_00001.ts': Duration: 00:00:04.00, start: 3606.432956, bitrate: 3190 kb/s Program 1 Stream #0:0[0x41]: Video: h264 (HDMV / 0x564D4448), none, 25 fps, 25 tbr, 90k tbn, 180k tbc [mpegts @ 0x152bc40] Invalid timestamps stream=0, pts=324578966, dts=324578992, size=21954 [mpegts @ 0x152bc40] Invalid timestamps stream=0, pts=324582566, dts=324582582, size=18157
I'd expect h264parse config-interval=-1 to fix this (requires git master), but it would be better if it worked as expected out of the box.
I upgraded to git master (of today) and indeed, config-interval=-1 (in both the sending receiving pipeline) fixes the problem (although ffmpeg keeps throwing warnings about invalid timestamps for all segments (including the first one). I guess config-interval=-1 insert an sps/pps nal unit after each frame?
No, before each keyframe
Ah, if I read the docs I would have known that myself ;) But given the fact that my framerate is 25/1 and my keyframe interval is 25 as well (and config-interval=1), I would've expected that both align. Even if there would have been a slight misalignment (causing a segment to start with an I-frame but no SPS/PPS), I would've expected that a decoder recovers after the next keyframe. This is clearly not the case. Any thoughts on that?
In that case, config-interval=1 on the receiver's h264parse should also have worked. In your pipelines above, you're not using config-interval=1 on the receiver, only on the sender. The SPS/PPS are carried in the RTP prop-parameter-sets, not the stream itself - the receiver will only get them from your caps and h264parse will only insert them into the converted byte-stream once unless told otherwise.
Well, you are right sir. Adding config-interval=1 to the receiver pipeline did fix the issue indeed. But in that case, shouldn't splitmuxsink hold a copy of the SPS/PPS and insert it at segment start if none is provided (I guess mp4mux is doing just that)? Because right now, it generates corrupt mpe2ts segments. I could imagine that splitmuxsink at least should throw a warning about missing SPS/PPS at the beginning of a segment.
You guys mean config-interval=-1 on the receiver side, right? With mp4mux SPS/PPS will be in the input caps for mp4mux (codec_data), so they will always be available. With mpegtsmux SPS/PPS are not in the caps but are sent inline in the stream. splitmux doesn't know about h264 presumably, so can't really handle that nicely. This is another use case where it would be good to signal SPS/PPS in the caps as well for byte-stream format too. We can do that by adding them as 'streamheader' buffers in the caps (in byte-stream format). mpegtsmux could/should pick up streamheaders from the input caps then and mux them first thing before any other payloads.
The problem is that splitmuxsink doesn't know anything about formats or SPS/PPS. In the mp4mux case, it works because the SPS/PPS are in the caps in H.264 stream-format=avc mode, so it can restart cleanly. I'm not sure what the right way to fix it is. Teaching splitmuxsink about specific formats isn't it.
(In reply to Tim-Philipp Müller from comment #11) > You guys mean config-interval=-1 on the receiver side, right? We meant that in the case where key-interval = 1 second, they should be the same thing. > This is another use case where it would be good to signal SPS/PPS in the > caps as well for byte-stream format too. We can do that by adding them as > 'streamheader' buffers in the caps (in byte-stream format). mpegtsmux > could/should pick up streamheaders from the input caps then and mux them > first thing before any other payloads. I was thinking something like that, but it's not great to have the muxer futzing with the stream like that either. If you have a constant-bitrate stream, for example.
-- GitLab Migration Automatic Message -- This bug has been migrated to freedesktop.org's GitLab instance and has been closed from further activity. You can subscribe and participate further through the new bug through this link to our GitLab instance: https://gitlab.freedesktop.org/gstreamer/gst-plugins-good/issues/252.