GNOME Bugzilla – Bug 704214
h264parse: splits frame with two fields
Last modified: 2018-11-03 13:16:46 UTC
The video contains interlaced frames, however when parsing h264parse doesn't keep the fields together, which breaks decoding, timestamps, etc. One reason might be that there is AU delimiter, SEI and PPS between the fields, but when I tried make parser ignore those it still didn't split the frames right. Maybe someone with better H264 knowledge can fix this? File can be found here: https://s3.amazonaws.com/MatejK/Samples/00046.MTS
Also related to bug https://bugzilla.gnome.org/show_bug.cgi?id=696770
h264parse outputs AU as defined by spec (or otherwise individual NALUs). As such, we can't have 2 AU delimiter in 1 AU. And afaik the spec defines things in a way that even if there is no AU delimiter (and those other ignored ones), it would still lead to separate AU (due to slice numbering in this case). In essence, each AU is regarded as a separate coded picture (= field) in this case, even though at decode-screen time, it ends up in one frame (at least if not rendered at once, in such a setting there would never really be a single "frame" they both end up in). As such, IMHO there is no "fixing" here at parser level, but the decoder should handle this (having to do with there-is-more-than-meaning-to-a-frame), to which I have been giving some thoughts (still to sort out).
Well, this is going to be a bit problematic then. All timestamp handling assumes that we have one frame (two fields) in one buffer. That's why it is broken currently (DTS interpolation doesn't work, etc). Also if this is indeed the case then the decoders will need to be modified to merge PAFF fields together, at least for libav.
Tricky/problematic in any case, yes. Either producing output claiming it is an AU, though it is not really an AU (unless that is redefined), and what problems that might introduce in other cases (?) Either all you mention. Still have to check things here, but the referenced bug seems to indicate that the (at least libav) decoder already decodes the 2 fields into 1 output frame (?) (hence the obviously-to-be-fixed-as-said-above frame leaking). And in case of E-AC3, there are possibly also several output frames/buffers, which really count as 1 audio frame wrt time-stamping, and special care is taken for that as well (to treat the others as "auxiliary frames" or so). The point here is; if we can come up with code that would know when to put them together in one frame [*], then the same code can keep them apart and properly handle e.g. timestamping. So, either of the above, depending on taste. Obviously biased, but if (mind *if*) the second one can be made to work ... [*] which always makes me feel on the top a slippery slope that leads to parsers getting delusions of decoder-hood
The libav issue is that when frame-based multi threaded decoding is enabled, it needs both fields in one buffer, otherwise it fails to decode. The access unit is a H.264 concept. Most of gstreamer assumes that one buffer = one frame. To me it seems that at this point having the parser output one frame (two access units) breaks less things than having it output two access units. That said, I think the proper thing to do is to output two access units and fix the rest of the gstreamer. From the top of my head, what needs to be fixed is a) DTS interpolation in baseparse b) VideoCodecFrame leaking in decoder (two frames outputting one picture) c) libav decoder complaining about not receiving both fields in buffer
a) and b) can be a "bit" ugly (ehm), but not entirely surprising. c) is more of a problem; afaik the libav parser would have the same parsing result as h264parse, and if so that would then seem to end up with something the corresponding decoder does not like (in some or the other mode). That could be specifically patched up for h264 in avdec, but feels even uglier. Also/alternatively, nothing prevents (other than complexity ;) ) coming up with another alignment, e.g. alignment=frame, which avdec could then add to sink caps, and we could then cater for it (and possibly others) that way in h264parse. And indeed, AU is a H.264 concept. It also happens e.g. to be what makes up a AVC sample as spec'ed for the MP4 format (14496-15). One would also rather expect a whole *frame* in there (in view of timeing info a.o.), as most of GStreamer expects, but that's how it is.
Small note on the complexity, sometimes there is only the first field, and that forms a complete frame. This is bottom field first, but few frame only have 1 field. http://livehopper.com/N12_mpeg4.ts
Yeah, then there is the thing with mpeg ts muxing. According to my experience, H.264 in mpeg ts must be muxed one AU per packet, otherwise it fails to play on Apple devices. (this is more complicated, i found out that A7 devices won't play PAFF stream at all, which is kinda sad, but it used to work pre A7).
Yes, and if you transmux from qtmux to mpegts, you need to insert Access Unit delimters between AUs, and how h264parse currently doesn't do that either.
-- GitLab Migration Automatic Message -- This bug has been migrated to freedesktop.org's GitLab instance and has been closed from further activity. You can subscribe and participate further through the new bug through this link to our GitLab instance: https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/issues/101.