GNOME Bugzilla – Bug 758023
unable to play rtsp stream with framerate range excluding zero on caps downstream
Last modified: 2015-11-16 12:38:22 UTC
Debian Linux 8.2, 64bit, kernel 3.16.7-ckt11-1+deb8u5 (2015-10-09) x86_64 I've encountered very strange behaviour on Sony and FLIR cameras rtsp streams when I was porting our software from gstreamer-0.10 to gstreamer-1.0: If some element downstream specifies framerate range that does not include zero, e.g. "framerate=[1/100, 100/1]", then the whole pipeline: gst-launch-1.0 rtspsrc location='rtsp://<addr>/media/video1' \ ! decodebin ! video/x-raw, framerate=[1/100, 100/1] ! xvimagesink fails to start with: WARN rtpjitterbuffer rtpjitterbuffer.c:185:rtp_jitter_buffer_set_clock_rate: Clock rate changed from 0 to 90000 WARN rtpsource rtpsource.c:1056:update_receiver_stats: duplicate or reordered packet (seqnr 10182) WARN rtpsource rtpsource.c:1056:update_receiver_stats: duplicate or reordered packet (seqnr 10183) WARN rtpsource rtpsource.c:1056:update_receiver_stats: duplicate or reordered packet (seqnr 10184) WARN rtpsource rtpsource.c:1056:update_receiver_stats: duplicate or reordered packet (seqnr 10185) WARN libav gstavcodecmap.c:2408:gst_ffmpeg_caps_to_pixfmt: ignoring insane framerate 1/0 WARN basesrc gstbasesrc.c:2933:gst_base_src_loop:<udpsrc1> error: Internal data flow error. WARN basesrc gstbasesrc.c:2933:gst_base_src_loop:<udpsrc1> error: streaming task paused, reason not-linked (-1) When there are no framerate restriction, e.g.: gst-launch-1.0 rtspsrc location='rtsp://<addr>/media/video1' \ ! decodebin ! xvimagesink the pipeline plays just fine, but all elements downstream of rtspsrc have "framerate=(fraction)0/1" in caps when playing (inspected using generated DOT files). It's probably not a very serious problem, but it took me 2 days to discover that reason for not-linking was caps with "framerate=[1/100, 1000/1]" I had on sinkpad of my custom element. Then I was able to duplicate the bug using pipeline above, so I just removed that framerate restriction and everything started working. But all caps have "framerate=(fraction)0/1" even when the pipeline runs. This seems quite wrong to me.. especially when rtspsrc reports: DEBUG rtspsrc gstrtspsrc.c:1892:gst_rtspsrc_sdp_attributes_to_caps: adding caps: a-framerate=12.0 even before playing (the camera is configured to 12 fps).
framerate 0/1 basically means "variable/unknown framerate" here. That's not really wrong :) Arguably the depayloader could convert a-framerate=12.0 to framerate=12/1. Which depayloader is used here, which codec?
Ok, maybe it's not "wrong", but I've never seen it in documentation anywhere (I'm not saying it's not there.. I just wasn't able to find it) and since it worked (framerate=[1/100, 100/1]) in gstreamer-0.10 I just assumed it's wrong now.. Anyway, it would be great if someone documented this in "porting from 0.10 to 1.0" guide, since it may save a lot of debugging for someone.. It was a h264 stream and depayloader was rtph264depay. I'll attach relevant .dot file
Created attachment 315395 [details] dot file
a-framerate is not a valid SDP attribute for h264: https://tools.ietf.org/html/rfc6184#section-8.2 And there seems to be no standard way of providing the framerate for the RTP h264 mapping, so not much we can do here when just following the standards. Not sure why it worked in 0.10 :)
Shouldn't then rtspsrc provide attribute which is valid?
How should it do that? The a-framerate attribute that your server puts in the SDP is not according to any standard (but rtspsrc should put that into the caps, just that rtph264depay does not handle it). There seems to be no standard way to provide the framerate for H264 unless I'm missing something.
I understand that it's not according to standard. But from user's (my) point of view, new Sony and FLIR cameras are not some "obscure" HW so it's not unreasonable to expect GStreamer to handle them "correctly" and report valid framerate. It's not "my server" that puts "a-framerate" there, but quite good, modern and expensive cameras (Sony SNC-VB600 and FLIR FC-690-PAL). Also I just tried it on some random publicly accessible cameras with rtsp streams and it works on those which have mpeg4 stream. So I will check this with one of our cameras - I suspect it will work with mpeg4. I'm willing to test this more thoroughly, because I believe that it's better to make things work the way anyone would expect them to work (e.g. having correct framerate if it's reported somewhere) even if it means "bending the rules a little", than adhering to standards and rules and saying it can't be done because it's not according to standard. As far as I know, most hw manufactures don't really care much about standards as long as their gadgets can be made to work. It may not be ideal, but we don't live in ideal world. I don't think that my original "problem" could be solved by talking to people at Sony or FLIR (actually, flir techsupport is excelent, so maybe there is a small chance.., with Sony - no chance) and demanding fixing their cameras. On the other hand - creating some exception and allowing some nonstandard attribute from well known and widespread HW is perhaps not so bad a solution..
That's also why I didn't close this bug yet :) rtph264depay could in theory convert the a-framerate=12.0 to framerate=12/1, risk of breaking anything with that seem non-existing. But it's also not clear to me what exactly this is going to solve? The framerate is mostly informational, what matters is the timestamps on the actual frames. If for whatever reason the framerate is relevant for you and you require a fixed framerate, you'll have to look at the timestamps (which is also what videorate does).
With that I mean, I would also like to understand why exactly you want to forbid a framerate of 0/1. Maybe there's something better you can do, independent of handling a-framerate or not (which might make sense to support in the depayloader, not in the payloader!, anyway).
You spotted the problem - I was reporting the framerate into our system on GST_EVENT_CAPS in our element as "default-fps" so our system knows immediately what to expect.. this could be a bit problematic with computing framerate, because if I want some precision, I would have to wait at least a few seconds to compute the framerate which would slightly break our system which expects (and needs) to know the framerate before first frame (or more precisely, before second frame) arrives. I made a quickfix and just replaced this with fixed "12.0", but you probably see that this is not a long term solution.. But that's my problem - I shouldn't have designed it that way - I'm just writting that so you'd understand why I won't let it go :) Anyway, my problem is mine, and has nothing to do with GStreamer. But I still feel it's not right to have 0/1 (=unknown) framerate if it actually IS known..
Oh, I wrote to fast - but the problem is not exactly that I want to forbid a "0/1" framerate - as you've explained, it means "unknown" and that is quite ok with me. I reduced the original caps to "video/x-raw, format=(string)I420" and I have no linking errors. The problem which stays is the one I described above - I was expecting to know the framerate at GST_EVENT_CAPS. I confess, I don't really understand how elements are supposed to work and I just put together anything I found in plugin sources and it seemed to work so I didn't touch it for more than 4 years. But I would only very humbly ask if "framerate=0/1 == unknown" could be documented...
> But I would only very humbly ask if "framerate=0/1 == unknown" could be > documented... http://gstreamer.freedesktop.org/data/doc/gstreamer/head/pwg/html/section-types-definitions.html#table-video-types http://cgit.freedesktop.org/gstreamer/gst-plugins-base/tree/docs/design/part-mediatype-video-raw.txt#n11 (for raw video)
..stupid me.. sorry
well.. maybe not completely stupid.. more like confused: framerate, GST_TYPE_FRACTION, default 0/1 The framerate of the video 0/1 for variable framerate I never expected variable framerate, so I suppose (it was 4 years ago) I just thought that because we don't use variable framerate (it's actually a requirement for our system to have fixed framerate) I could ignore this..
Also not stupid because the documentation is not that easy to find between all the other docs :) Anyway, independent of all that it might make sense to handle a-framerate in rtph264depay. It's information that might be useful for something and we just drop it currently. Should get a new bug without all the other discussions though, if someone thinks this should be implemented
Yeah, finding documentation which is relevant to one specific GStreamer version is not always an easy task.. when I was writing 0.10 plugin, I was often led away by 0.8 documentation or examples and the same happened when I tried to port it to 1.0 - a lot of documentation still retains old code examples, or is simply wrong, obsolete, misleading or missing crucial information - so I generally found it easier just to read GStreamer and plugin sources. :) As for a new bug - I may be a bit slow - are you saying that I should file a new bug requesting "a-framerate" to "framerate" conversion in rtph264depay?
Yes, should also be trivial to implement in the depayloader.
Let's close this one then and continue in bug #758169.