GNOME Bugzilla – Bug 776712
vaapih264enc doesn't output AU delimiters
Last modified: 2017-07-05 08:16:11 UTC
Follow-up bug report of the mailing list discussion here: https://lists.freedesktop.org/archives/gstreamer-devel/2016-December/062109.html Tim Müller wrote: > I suspect the problem is that vaapih264enc doesn't output AU delimiters > by default, and it also seems to have no property for it ("aud" > property on x264enc which is set to true by default). When using vaapih262enc to create a ts muxed (and presumably other as well) stream many player have problems playing it. Especially Browsers that try to play a hls stream don't like what they get from vaapih262enc, but x264enc works. Reproducing: gst-launch-1.0 videotestsrc ! vaapih264enc ! h264parse ! mpegtsmux ! hlssink The resulting "playlist.m3u8" is sometimes playable in players like ffplay with repeating messages like: [mpegts @ 0x7f06d81f7940] Invalid timestamps stream=0, pts=324107999, dts=324108000, size=3555 And sometimes it exits with this message: [h264 @ 0x7fcad807be80] sps_id 0 out of range Trying to play the playlist.m3u8 in browsers like Safari (native hls support) or Chrome (with hlsjs) (always)brings black picture / "mediaError" events. To reproduce, upload the created .ts and .m3u8 files to a webserver and use the hlsjs demo: http://dailymotion.github.io/hls.js/demo/ A workaround I found is to apply the proposed patch of bug #736213 (needs some minor changes to fix patch errors) and then use something like this: gst-launch-1.0 videotestsrc ! vaapih264enc ! \ h264parse config-interval=-1 inline-au-delimiter=true ! \ mpegtsmux ! hlssink The "inline-au-delimiter" from the patch alone doesn't do the trick, but "config-interval=-1" (any value other than 0) is required to get it to work. "config-interval" is described as "Send SPS and PPS Insertion Interval". I don't really know what that means, but it may be possible that the problem is more than just AUD delimiters and I wan't to share what I discovered. I open this new bug even though there is another more or less related one because my understanding from the last days reading is that a encoder should include the AU delimiters by itself in byte-stream mode and not rely on something else to inline it.
Seems there are 2 options. 1. Finish bug #736213. It looks that most people agree with insertion of au delimeter in h264parse unless negotiated as avc stream-format. 2. Create aud property in vaapih264enc just like x264enc. or both :)
My impression is that the aud property was a mistake, you should always have the AUD in byte-stream mode.. and never in avc1 mode (especially if you have alignment=au).
I don't know why a dependency on bug #736213 was added for this? I agree with Olivier here. When vaapih264enc outputs byte-stream, it should just output/add AU delimiters by default. I don't mind if we have a property that defaults to TRUE or just not have a property until someone asks for one (can't think of a really good reason why or scenario where one would want to disable it. Maybe when the container takes byte-stream but keeps AU together already. Then you can save a few bytes for each frame.)
(In reply to Tim-Philipp Müller from comment #3) > I don't know why a dependency on bug #736213 was added for this? True. I just wanted to explicit that you could workaround the issue with the h264parser. > > I agree with Olivier here. When vaapih264enc outputs byte-stream, it should > just output/add AU delimiters by default. > > I don't mind if we have a property that defaults to TRUE or just not have a > property until someone asks for one (can't think of a really good reason why > or scenario where one would want to disable it. Maybe when the container > takes byte-stream but keeps AU together already. Then you can save a few > bytes for each frame.)
Honestly I thought that if #bug 736213 is finished, this could be closed. But I'm aligned with that it's necessary to do something here in gst-vaapi. By the way, patches of bug #736213 landed, @Uwe Mueller, can you confirm with that?
Created attachment 349334 [details] [review] encoder: h264: Insert AU Delimiter Note: Don't push this patch since it will break the backward compatibility :) Insert an AUD as the first NAL of each encoded frame. Some of the application require Access Unit Delimiter for decoding the stream. https://bugzilla.gnome.org/show_bug.cgi?id=776712 Driver fix needed for this. https://github.com/01org/intel-vaapi-driver/pull/122/commits/762929b6b6312822fc1c04f52fdef136574508d2 Issue: Unfortunately this is going to break the backward compatibility. Because the older driver code (with out the above driver fix) has mechanism to insert anything we push as RawDataHeader. So even if the driver fix is not in place, the AUD will get inserted , but only after SPS and PPS, irrespective of how/when we insert AUD from middleware. Which is against the h264 specification: As per specification "When an access unit delimiter NAL unit is present, it shall be the first NAL unit. There shall be at most one access unit delimiter NAL unit in any access unit." So in nutshell: 1: If we apply the gstreamer-vaapi patch and use latest driver(with the fix mentioned above): Everything will work as expected and AUD will get inserted as the first NAL. 2: If we apply the gstreamer-vaapi patch and use the old driver: AUD will get inserted only after SPS and PPS
I would add this, but under a property to disable (or enable).
Created attachment 351981 [details] [review] encoder: h264: Insert AU Delimiter Insert an AUD as the first NAL of each encoded frame. Some applications require Access Unit Delimiter for decoding the stream.
I have updated the patch, now it is only enabled when property aud is set as true. By default the property is false. Why a property? Because it is only available for an Intel Gen9 and the va intel driver should be 1.8 or superior. Otherwise, the output is corrupted, as Sree said in comment #6 With this patch (and the other two dependencies), this pipeline gst-launch-1.0 videotestsrc num-buffers=1000 ! vaapih264enc aud=true keyframe-period=1 ! mpegtsmux ! hlssink -ev creates an output capable to be played with no errors using: ffplay playlist.m3u8
(In reply to Víctor Manuel Jáquez Leal from comment #9) > > gst-launch-1.0 videotestsrc num-buffers=1000 ! vaapih264enc aud=true > keyframe-period=1 ! mpegtsmux ! hlssink -ev d'oh keyframe-period=1 ... of course it will work. though, removing that parameter will work too.. with some "Invalid timestamps" messages :)
Hello, I have fetched this patched and tried to use it but I can't make it working. The streams works (in Safari at Mac OS Sierra) only if I start streaming from the first HLS chunk. If I start streaming from subsequent one, it does not work. I am using Ubuntu 16.04 and I have fetched GStreamer 1.12.0 and compiled it from sources. I've fetched libdrm-2.4.80 and libva 1.8.2. I use Intel driver 5f479acd6d856c13be18321642eac8e9635b25c9 (master, contains patch mentioned above). I've applied patch from attachment #351981 [details] to gst-vaapi before building it. This is my test pipeline: GST_DEBUG=vaapi:5 /opt/xxx/gstreamer-dist-1.12.0/bin/gst-launch-1.0 videotestsrc pattern=ball is-live=true ! "video/x-raw,width=1280,height=720,framerate=15/1" ! vaapih264enc aud=true ! mpegtsmux ! hlssink If I use ffplay to display stream when first HLS chunk is still available, I see the video and that output: $ ffplay http://164.132.206.204:8000/playlist.m3u8 ffplay version 3.3.1 Copyright (c) 2003-2017 the FFmpeg developers built with Apple LLVM version 8.1.0 (clang-802.0.42) configuration: --prefix=/usr/local/Cellar/ffmpeg/3.3.1 --enable-shared --enable-pthreads --enable-gpl --enable-version3 --enable-hardcoded-tables --enable-avresample --cc=clang --host-cflags= --host-ldflags= --enable-ffplay --enable-libmp3lame --enable-libx264 --enable-libxvid --enable-opencl --disable-lzma --enable-vda libavutil 55. 58.100 / 55. 58.100 libavcodec 57. 89.100 / 57. 89.100 libavformat 57. 71.100 / 57. 71.100 libavdevice 57. 6.100 / 57. 6.100 libavfilter 6. 82.100 / 6. 82.100 libavresample 3. 5. 0 / 3. 5. 0 libswscale 4. 6.100 / 4. 6.100 libswresample 2. 7.100 / 2. 7.100 libpostproc 54. 5.100 / 54. 5.100 Input #0, hls,applehttp, from 'http://164.132.206.204:8000/playlist.m3u8': Duration: N/A, start: 3600.072422, bitrate: N/A Program 0 Metadata: variant_bitrate : 0 Stream #0:0: Video: h264 (High) (HDMV / 0x564D4448), yuv420p, 1280x720 [SAR 1:1 DAR 16:9], 15 fps, 15 tbr, 90k tbn, 30 tbc Metadata: variant_bitrate : 0 3610.45 M-V: -0.067 fd= 1 aq= 0KB vq= 6KB sq= 0B f=0/0 When I try to display the stream via ffplay after first chunk is not available any more I get the following errors: $ ffplay http://164.132.206.204:8000/playlist.m3u8 ffplay version 3.3.1 Copyright (c) 2003-2017 the FFmpeg developers built with Apple LLVM version 8.1.0 (clang-802.0.42) configuration: --prefix=/usr/local/Cellar/ffmpeg/3.3.1 --enable-shared --enable-pthreads --enable-gpl --enable-version3 --enable-hardcoded-tables --enable-avresample --cc=clang --host-cflags= --host-ldflags= --enable-ffplay --enable-libmp3lame --enable-libx264 --enable-libxvid --enable-opencl --disable-lzma --enable-vda libavutil 55. 58.100 / 55. 58.100 libavcodec 57. 89.100 / 57. 89.100 libavformat 57. 71.100 / 57. 71.100 libavdevice 57. 6.100 / 57. 6.100 libavfilter 6. 82.100 / 6. 82.100 libavresample 3. 5. 0 / 3. 5. 0 libswscale 4. 6.100 / 4. 6.100 libswresample 2. 7.100 / 2. 7.100 libpostproc 54. 5.100 / 54. 5.100 [NULL @ 0x7fd694026e00] sps_id 0 out of range 0KB sq= 0B f=0/0 [NULL @ 0x7fd694026e00] non-existing PPS 0 referenced [h264 @ 0x7fd695008a00] sps_id 0 out of range [h264 @ 0x7fd695008a00] non-existing PPS 0 referenced [h264 @ 0x7fd695008a00] decode_slice_header error [h264 @ 0x7fd695008a00] no frame! [NULL @ 0x7fd694026e00] non-existing PPS 0 referenced [h264 @ 0x7fd695008a00] non-existing PPS 0 referenced [h264 @ 0x7fd695008a00] decode_slice_header error [h264 @ 0x7fd695008a00] no frame! [NULL @ 0x7fd694026e00] non-existing PPS 0 referenced [h264 @ 0x7fd695008a00] non-existing PPS 0 referenced [h264 @ 0x7fd695008a00] decode_slice_header error [h264 @ 0x7fd695008a00] no frame! I've added GST_DEBUG to the patch to check if the new add_packed_au_delimiter is being called, and it is: root@fa2f1e24085b:~/tmp# GST_DEBUG=vaapi:5 /opt/xxx/gstreamer-dist-1.12.0/bin/gst-launch-1.0 videotestsrc pattern=ball is-live=true ! "video/x-raw,width=1280,height=720,framerate=15/1" ! vaapih264enc aud=true ! mpegtsmux ! hlssink error: XDG_RUNTIME_DIR not set in the environment. 0:00:00.013185506 9602 0x1830c70 INFO vaapi gstvaapiutils.c:58:gst_vaapi_log: VA-API version 0.40.0 0:00:00.013201366 9602 0x1830c70 INFO vaapi gstvaapiutils.c:58:gst_vaapi_log: va_getDriverName() returns 0 0:00:00.013207187 9602 0x1830c70 INFO vaapi gstvaapiutils.c:58:gst_vaapi_log: Trying to open /opt/xxx/gstreamer-dist-1.12.0/lib/dri/i965_drv_video.so 0:00:00.013368958 9602 0x1830c70 INFO vaapi gstvaapiutils.c:58:gst_vaapi_log: Found init function __vaDriverInit_0_40 0:00:00.013599744 9602 0x1830c70 INFO vaapi gstvaapiutils.c:58:gst_vaapi_log: va_openDriver() returns 0 0:00:00.013605417 9602 0x1830c70 INFO vaapi gstvaapiutils.c:77:vaapi_initialize: VA-API version 0.40 0:00:00.013738823 9602 0x1830c70 INFO vaapi gstvaapiutils.c:58:gst_vaapi_log: VA-API version 0.40.0 0:00:00.013747038 9602 0x1830c70 INFO vaapi gstvaapiutils.c:58:gst_vaapi_log: va_getDriverName() returns 0 0:00:00.013751403 9602 0x1830c70 INFO vaapi gstvaapiutils.c:58:gst_vaapi_log: Trying to open /opt/xxx/gstreamer-dist-1.12.0/lib/dri/i965_drv_video.so 0:00:00.013759811 9602 0x1830c70 INFO vaapi gstvaapiutils.c:58:gst_vaapi_log: Found init function __vaDriverInit_0_40 0:00:00.013949536 9602 0x1830c70 INFO vaapi gstvaapiutils.c:58:gst_vaapi_log: va_openDriver() returns 0 0:00:00.013954962 9602 0x1830c70 INFO vaapi gstvaapiutils.c:77:vaapi_initialize: VA-API version 0.40 Setting pipeline to PAUSED ... error: XDG_RUNTIME_DIR not set in the environment. 0:00:00.016064385 9602 0x1830c70 INFO vaapi gstvaapiutils.c:58:gst_vaapi_log: VA-API version 0.40.0 0:00:00.016074071 9602 0x1830c70 INFO vaapi gstvaapiutils.c:58:gst_vaapi_log: va_getDriverName() returns 0 0:00:00.016079233 9602 0x1830c70 INFO vaapi gstvaapiutils.c:58:gst_vaapi_log: Trying to open /opt/xxx/gstreamer-dist-1.12.0/lib/dri/i965_drv_video.so 0:00:00.016088202 9602 0x1830c70 INFO vaapi gstvaapiutils.c:58:gst_vaapi_log: Found init function __vaDriverInit_0_40 0:00:00.016290031 9602 0x1830c70 INFO vaapi gstvaapiutils.c:58:gst_vaapi_log: va_openDriver() returns 0 0:00:00.016295524 9602 0x1830c70 INFO vaapi gstvaapiutils.c:77:vaapi_initialize: VA-API version 0.40 0:00:00.016389214 9602 0x1830c70 INFO vaapi gstvaapiutils.c:58:gst_vaapi_log: VA-API version 0.40.0 0:00:00.016397125 9602 0x1830c70 INFO vaapi gstvaapiutils.c:58:gst_vaapi_log: va_getDriverName() returns 0 0:00:00.016401484 9602 0x1830c70 INFO vaapi gstvaapiutils.c:58:gst_vaapi_log: Trying to open /opt/xxx/gstreamer-dist-1.12.0/lib/dri/i965_drv_video.so 0:00:00.016410109 9602 0x1830c70 INFO vaapi gstvaapiutils.c:58:gst_vaapi_log: Found init function __vaDriverInit_0_40 0:00:00.016598574 9602 0x1830c70 INFO vaapi gstvaapiutils.c:58:gst_vaapi_log: va_openDriver() returns 0 0:00:00.016603649 9602 0x1830c70 INFO vaapi gstvaapiutils.c:77:vaapi_initialize: VA-API version 0.40 0:00:00.016676186 9602 0x1830c70 DEBUG vaapi gstvaapiutils.c:86:vaapi_check_status: vaGetConfigAttributes(): the requested VAEntryPoint is not supported 0:00:00.016684904 9602 0x1830c70 DEBUG vaapi gstvaapiutils.c:86:vaapi_check_status: vaGetConfigAttributes(): the requested VAEntryPoint is not supported Pipeline is live and does not need PREROLL ... Got context from element 'vaapiencodeh264-0': gst.vaapi.Display=context, gst.vaapi.Display=(GstVaapiDisplay)"\(GstVaapiDisplayDRM\)\ vaapidisplaydrm1"; 0:00:00.016836122 9602 0x184db20 DEBUG vaapi gstvaapicontext.c:100:context_destroy: context 0xffffffff 0:00:00.016849538 9602 0x184db20 DEBUG vaapi gstvaapicontext.c:100:context_destroy: context 0xffffffff 0:00:00.016857637 9602 0x184db20 DEBUG vaapi gstvaapicontext.c:100:context_destroy: context 0xffffffff 0:00:00.016864698 9602 0x184db20 DEBUG vaapi gstvaapicontext.c:100:context_destroy: context 0xffffffff 0:00:00.016871521 9602 0x184db20 DEBUG vaapi gstvaapicontext.c:100:context_destroy: context 0xffffffff 0:00:00.016878002 9602 0x184db20 DEBUG vaapi gstvaapicontext.c:100:context_destroy: context 0xffffffff 0:00:00.016885214 9602 0x184db20 DEBUG vaapi gstvaapicontext.c:100:context_destroy: context 0xffffffff Setting pipeline to PLAYING ... 0:00:00.016891813 9602 0x184db20 DEBUG vaapi gstvaapicontext.c:100:context_destroy: context 0xffffffff New clock: GstSystemClock 0:00:00.017057273 9602 0x184db20 DEBUG vaapi gstvaapiencoder_h264.c:2843:gst_vaapi_encoder_h264_reconfigure: resolution: 1280x720 0:00:00.017067476 9602 0x184db20 DEBUG vaapi gstvaapiencoder_h264.c:2430:ensure_profile_and_level: selected high profile at level 3.1 0:00:00.017074635 9602 0x184db20 INFO vaapi gstvaapiencoder.c:542:get_packed_headers: supported packed headers: 0x0000001f 0:00:00.017087202 9602 0x184db20 DEBUG vaapi gstvaapisurface.c:337:gst_vaapi_surface_new: size 1280x720, chroma type 0x1 0:00:00.017094024 9602 0x184db20 DEBUG vaapi gstvaapisurface.c:121:gst_vaapi_surface_create: surface 0x4000000 0:00:00.017099014 9602 0x184db20 DEBUG vaapi gstvaapisurface.c:337:gst_vaapi_surface_new: size 1280x720, chroma type 0x1 0:00:00.017104255 9602 0x184db20 DEBUG vaapi gstvaapisurface.c:121:gst_vaapi_surface_create: surface 0x4000001 0:00:00.017108654 9602 0x184db20 DEBUG vaapi gstvaapisurface.c:337:gst_vaapi_surface_new: size 1280x720, chroma type 0x1 0:00:00.017114349 9602 0x184db20 DEBUG vaapi gstvaapisurface.c:121:gst_vaapi_surface_create: surface 0x4000002 0:00:00.017118008 9602 0x184db20 DEBUG vaapi gstvaapisurface.c:337:gst_vaapi_surface_new: size 1280x720, chroma type 0x1 0:00:00.017123661 9602 0x184db20 DEBUG vaapi gstvaapisurface.c:121:gst_vaapi_surface_create: surface 0x4000003 0:00:00.017127835 9602 0x184db20 DEBUG vaapi gstvaapisurface.c:337:gst_vaapi_surface_new: size 1280x720, chroma type 0x1 0:00:00.017133302 9602 0x184db20 DEBUG vaapi gstvaapisurface.c:121:gst_vaapi_surface_create: surface 0x4000004 0:00:00.017137906 9602 0x184db20 DEBUG vaapi gstvaapisurface.c:337:gst_vaapi_surface_new: size 1280x720, chroma type 0x1 0:00:00.017142924 9602 0x184db20 DEBUG vaapi gstvaapisurface.c:121:gst_vaapi_surface_create: surface 0x4000005 0:00:00.017147684 9602 0x184db20 DEBUG vaapi gstvaapisurface.c:337:gst_vaapi_surface_new: size 1280x720, chroma type 0x1 0:00:00.017152803 9602 0x184db20 DEBUG vaapi gstvaapisurface.c:121:gst_vaapi_surface_create: surface 0x4000006 0:00:00.017156951 9602 0x184db20 DEBUG vaapi gstvaapisurface.c:337:gst_vaapi_surface_new: size 1280x720, chroma type 0x1 0:00:00.017162594 9602 0x184db20 DEBUG vaapi gstvaapisurface.c:121:gst_vaapi_surface_create: surface 0x4000007 0:00:00.017167388 9602 0x184db20 DEBUG vaapi gstvaapisurface.c:337:gst_vaapi_surface_new: size 1280x720, chroma type 0x1 0:00:00.017172867 9602 0x184db20 DEBUG vaapi gstvaapisurface.c:121:gst_vaapi_surface_create: surface 0x4000008 0:00:00.017387740 9602 0x184db20 DEBUG vaapi gstvaapicontext.c:217:context_create: context 0x02000000 0:00:00.017426091 9602 0x184db20 DEBUG vaapi gstvaapisurface.c:376:gst_vaapi_surface_new_full: size 1280x720, format I420, flags 0x00000000 0:00:00.017436822 9602 0x184db20 DEBUG vaapi gstvaapisurface.c:213:gst_vaapi_surface_create_full: surface 0x4000009 0:00:00.017441867 9602 0x184db20 DEBUG vaapi gstvaapiimage.c:293:gst_vaapi_image_new_with_image: VA image 0x0a000000, format I420, size 1280x720 0:00:00.017453345 9602 0x184db20 DEBUG vaapi gstvaapiimage.c:118:gst_vaapi_image_destroy: image 0xa000000 0:00:00.017457956 9602 0x184db20 DEBUG vaapi gstvaapisurface.c:78:gst_vaapi_surface_destroy: surface 0x4000009 0:00:00.017595610 9602 0x184db20 DEBUG vaapi gstvaapisurface.c:376:gst_vaapi_surface_new_full: size 1280x720, format I420, flags 0x00000000 0:00:00.017607938 9602 0x184db20 DEBUG vaapi gstvaapisurface.c:213:gst_vaapi_surface_create_full: surface 0x4000009 0:00:00.017613584 9602 0x184db20 DEBUG vaapi gstvaapiimage.c:293:gst_vaapi_image_new_with_image: VA image 0x0a000000, format I420, size 1280x720 0:00:00.022926177 9602 0x184db20 DEBUG vaapi gstvaapiimage.c:118:gst_vaapi_image_destroy: image 0xa000000 0:00:00.022958087 9602 0x184db20 DEBUG vaapi gstvaapicodedbuffer.c:49:coded_buffer_create: coded buffer 0x8000001 0:00:00.022967146 9602 0x184db20 DEBUG vaapi gstvaapiencoder_h264.c:1296:add_packed_au_delimiter: PACKED_AU 0:00:00.022981830 9602 0x184db20 DEBUG vaapi gstvaapiencoder_objects.c:517:gst_vaapi_enc_picture_encode: encode picture 0x04000009 0:00:00.025881849 9602 0x184db20 DEBUG vaapi gstvaapisurface.c:376:gst_vaapi_surface_new_full: size 1280x720, format I420, flags 0x00000000 0:00:00.025906477 9602 0x184db20 DEBUG vaapi gstvaapisurface.c:213:gst_vaapi_surface_create_full: surface 0x400000e 0:00:00.025913507 9602 0x184db20 DEBUG vaapi gstvaapiimage.c:293:gst_vaapi_image_new_with_image: VA image 0x0a000000, format I420, size 1280x720 0:00:00.030556475 9602 0x184db20 DEBUG vaapi gstvaapiimage.c:118:gst_vaapi_image_destroy: image 0xa000000 0:00:00.089833948 9602 0x184db20 DEBUG vaapi gstvaapiimage.c:293:gst_vaapi_image_new_with_image: VA image 0x0a000000, format I420, size 1280x720 0:00:00.108076458 9602 0x184db20 DEBUG vaapi gstvaapiimage.c:118:gst_vaapi_image_destroy: image 0xa000000 0:00:00.156504396 9602 0x184db20 DEBUG vaapi gstvaapiencoder_h264.c:1296:add_packed_au_delimiter: PACKED_AU 0:00:00.156571453 9602 0x184db20 DEBUG vaapi gstvaapiencoder_objects.c:517:gst_vaapi_enc_picture_encode: encode picture 0x04000009 0:00:00.163281331 9602 0x184db20 DEBUG vaapi gstvaapicodedbuffer.c:49:coded_buffer_create: coded buffer 0x8000009 0:00:00.163325494 9602 0x184db20 DEBUG vaapi gstvaapiencoder_h264.c:1296:add_packed_au_delimiter: PACKED_AU 0:00:00.163357322 9602 0x184db20 DEBUG vaapi gstvaapiencoder_objects.c:517:gst_vaapi_enc_picture_encode: encode picture 0x0400000e 0:00:00.170309279 9602 0x184db20 DEBUG vaapi gstvaapiimage.c:293:gst_vaapi_image_new_with_image: VA image 0x0a000000, format I420, size 1280x720 0:00:00.180434838 9602 0x184db20 DEBUG vaapi gstvaapiimage.c:118:gst_vaapi_image_destroy: image 0xa000000 0:00:00.223154784 9602 0x184db20 DEBUG vaapi gstvaapiimage.c:293:gst_vaapi_image_new_with_image: VA image 0x0a000000, format I420, size 1280x720 0:00:00.241637612 9602 0x184db20 DEBUG vaapi gstvaapiimage.c:118:gst_vaapi_image_destroy: image 0xa000000 0:00:00.289812813 9602 0x184db20 DEBUG vaapi gstvaapiencoder_h264.c:1296:add_packed_au_delimiter: PACKED_AU 0:00:00.289898767 9602 0x184db20 DEBUG vaapi gstvaapiencoder_objects.c:517:gst_vaapi_enc_picture_encode: encode picture 0x0400000e 0:00:00.296093498 9602 0x184db20 DEBUG vaapi gstvaapiencoder_h264.c:1296:add_packed_au_delimiter: PACKED_AU 0:00:00.296173650 9602 0x184db20 DEBUG vaapi gstvaapiencoder_objects.c:517:gst_vaapi_enc_picture_encode: encode picture 0x04000009 0:00:00.300306042 9602 0x184db20 DEBUG vaapi gstvaapiimage.c:293:gst_vaapi_image_new_with_image: VA image 0x0a000000, format I420, size 1280x720 0:00:00.319103314 9602 0x184db20 DEBUG vaapi gstvaapiimage.c:118:gst_vaapi_image_destroy: image 0xa000000 0:00:00.356499085 9602 0x184db20 DEBUG vaapi gstvaapiimage.c:293:gst_vaapi_image_new_with_image: VA image 0x0a000000, format I420, size 1280x720 0:00:00.374830563 9602 0x184db20 DEBUG vaapi gstvaapiimage.c:118:gst_vaapi_image_destroy: image 0xa000000 0:00:00.423137887 9602 0x184db20 DEBUG vaapi gstvaapiencoder_h264.c:1296:add_packed_au_delimiter: PACKED_AU 0:00:00.423220350 9602 0x184db20 DEBUG vaapi gstvaapiencoder_objects.c:517:gst_vaapi_enc_picture_encode: encode picture 0x04000009 0:00:00.427222655 9602 0x184db20 DEBUG vaapi gstvaapiencoder_h264.c:1296:add_packed_au_delimiter: PACKED_AU 0:00:00.427306978 9602 0x184db20 DEBUG vaapi gstvaapiencoder_objects.c:517:gst_vaapi_enc_picture_encode: encode picture 0x0400000e 0:00:00.432674674 9602 0x184db20 DEBUG vaapi gstvaapiimage.c:293:gst_vaapi_image_new_with_image: VA image 0x0a000000, format I420, size 1280x720 0:00:00.450814858 9602 0x184db20 DEBUG vaapi gstvaapiimage.c:118:gst_vaapi_image_destroy: image 0xa000000 0:00:00.489830630 9602 0x184db20 DEBUG vaapi gstvaapiimage.c:293:gst_vaapi_image_new_with_image: VA image 0x0a000000, format I420, size 1280x720 0:00:00.508233018 9602 0x184db20 DEBUG vaapi gstvaapiimage.c:118:gst_vaapi_image_destroy: image 0xa000000 Here is the output of vainfo: # /opt/xxx/gstreamer-dist-1.12.0/bin/vainfo error: XDG_RUNTIME_DIR not set in the environment. error: can't connect to X server! libva info: VA-API version 0.40.0 libva info: va_getDriverName() returns 0 libva info: Trying to open /opt/xxx/gstreamer-dist-1.12.0/lib/dri/i965_drv_video.so libva info: Found init function __vaDriverInit_0_40 libva info: va_openDriver() returns 0 vainfo: VA-API version: 0.40 (libva ) vainfo: Driver version: Intel i965 driver for Intel(R) Kabylake - 1.8.3.pre1 (glk-alpha-51-g5f479ac) vainfo: Supported profile and entrypoints VAProfileMPEG2Simple : VAEntrypointVLD VAProfileMPEG2Simple : VAEntrypointEncSlice VAProfileMPEG2Main : VAEntrypointVLD VAProfileMPEG2Main : VAEntrypointEncSlice VAProfileH264ConstrainedBaseline: VAEntrypointVLD VAProfileH264ConstrainedBaseline: VAEntrypointEncSlice VAProfileH264ConstrainedBaseline: VAEntrypointEncSliceLP VAProfileH264Main : VAEntrypointVLD VAProfileH264Main : VAEntrypointEncSlice VAProfileH264Main : VAEntrypointEncSliceLP VAProfileH264High : VAEntrypointVLD VAProfileH264High : VAEntrypointEncSlice VAProfileH264High : VAEntrypointEncSliceLP VAProfileH264MultiviewHigh : VAEntrypointVLD VAProfileH264MultiviewHigh : VAEntrypointEncSlice VAProfileH264StereoHigh : VAEntrypointVLD VAProfileH264StereoHigh : VAEntrypointEncSlice VAProfileVC1Simple : VAEntrypointVLD VAProfileVC1Main : VAEntrypointVLD VAProfileVC1Advanced : VAEntrypointVLD VAProfileNone : VAEntrypointVideoProc VAProfileJPEGBaseline : VAEntrypointVLD VAProfileJPEGBaseline : VAEntrypointEncPicture VAProfileVP8Version0_3 : VAEntrypointVLD VAProfileVP8Version0_3 : VAEntrypointEncSlice VAProfileHEVCMain : VAEntrypointVLD VAProfileHEVCMain : VAEntrypointEncSlice VAProfileHEVCMain10 : VAEntrypointVLD VAProfileHEVCMain10 : VAEntrypointEncSlice VAProfileVP9Profile0 : VAEntrypointVLD VAProfileVP9Profile0 : VAEntrypointEncSlice VAProfileVP9Profile2 : VAEntrypointVLD CPU is i7 i7700K (KabyLake). Any ideas what can be the cause?
Created attachment 352650 [details] Log of mpegts skipping keyframe
I have added attachment #352650 [details] that is a log where numerous "skipping downstream force key unit" from mpegtsmux are present, I am not sure if that is related or not. I can confirm that workaround with h264parse allows to open stream on Safari, but ffplay still return errors: $ ffplay http://164.132.206.204:8000/playlist.m3u8 ffplay version 3.3.1 Copyright (c) 2003-2017 the FFmpeg developers built with Apple LLVM version 8.1.0 (clang-802.0.42) configuration: --prefix=/usr/local/Cellar/ffmpeg/3.3.1 --enable-shared --enable-pthreads --enable-gpl --enable-version3 --enable-hardcoded-tables --enable-avresample --cc=clang --host-cflags= --host-ldflags= --enable-ffplay --enable-libmp3lame --enable-libx264 --enable-libxvid --enable-opencl --disable-lzma --enable-vda libavutil 55. 58.100 / 55. 58.100 libavcodec 57. 89.100 / 57. 89.100 libavformat 57. 71.100 / 57. 71.100 libavdevice 57. 6.100 / 57. 6.100 libavfilter 6. 82.100 / 6. 82.100 libavresample 3. 5. 0 / 3. 5. 0 libswscale 4. 6.100 / 4. 6.100 libswresample 2. 7.100 / 2. 7.100 libpostproc 54. 5.100 / 54. 5.100 [NULL @ 0x7fa4c0047000] sps_id 0 out of range 0KB sq= 0B f=0/0 [h264 @ 0x7fa4c002ae00] sps_id 0 out of range Input #0, hls,applehttp, from 'http://164.132.206.204:8000/playlist.m3u8': Duration: N/A, start: 3648.074078, bitrate: N/A Program 0 Metadata: variant_bitrate : 0 Stream #0:0: Video: h264 (High) (HDMV / 0x564D4448), yuv420p, 1280x720 [SAR 1:1 DAR 16:9], 15 fps, 15 tbr, 90k tbn, 30 tbc Metadata: variant_bitrate : 0 [h264 @ 0x7fa4bf8c1c00] sps_id 0 out of range Failed to open file 'http://164.132.206.204:8000/playlist.m3u8' or configure filtergraph nan : 0.000 fd= 0 aq= 0KB vq= 0KB sq= 0B f=0/0 This is the pipeline I used: /opt/xxx/bin/gst-launch-1.0 videotestsrc pattern=ball is-live=true ! "video/x-raw,width=1280,height=720,framerate=15/1" ! vaapih264enc aud=true keyframe-period=1 ! h264parse ! mpegtsmux ! hlssink target-duration=1 &
Created attachment 353436 [details] [review] libs: encoder: h264: set the frame as IDR if forced key unit Currently the event GstForceKeyUnit is usually being used to manage frame-lost in case of streaming such as RTP. We need to start new gop to handle this if we get this event rathe than just producing i-frame.
Created attachment 353437 [details] [review] libs: encoder: h264: submit sps in case of IDR picture If the picture is IDR, submit SPS header. This means when frame number reaches to keyframe-period or force key unit event arrives, we insert SPS/PPS again.
I propose patches fixing hlssink issues. This is based on Victor's au delimiter patch. Need to verify if this patch breaks other cases. @Marcin, could you test your cases with this patch?
(In reply to Hyunjun Ko from comment #15) > Created attachment 353437 [details] [review] [review] > libs: encoder: h264: submit sps in case of IDR picture > > If the picture is IDR, submit SPS header. > This means when frame number reaches to keyframe-period or force key unit > event arrives, we insert SPS/PPS again. Are these patches part of the spec??
(In reply to Víctor Manuel Jáquez Leal from comment #17) > > Are these patches part of the spec?? No. It's just a kind of policy of encoder. Basically, SPS and PPS are placed in the first Access unit mandatorily and doesn't need to be placed any more under assumption that decoder got sequence/picture parameters successfully. But normally, they are inserted to the stream periodically for 2 reasons. 1\ A decoder sometimes needs to start decoding mid-stream. 2\ If the stream has been changed for some parameters, an encoder inserts SPS/PPS again. Currently, vaapi h264 encoder does this with the flag config_changed. But in the case of this issue, hlssink produces seperated segment file (since target-duration property is set to 1). Meanwhile, hlssink sends GstForceKeyUnit event if the duration is expired and about to start new segment file, which means when an encoder get this event, it needs to refresh status of encoder and produce IDR(Instantaneous Decoding Refresh) picture. This is what these patches are doing. In addition, these patches also fix a kind of bug. Currently, if it gets GstForceKeyUnit, sets to produce I-frame. But there's no changed config, so it doesn't produce SPS but produces PPS, which is absolutely wrong. This is exactly the failure of this issue.
So far it works fine. Awesome! This is my exact setup, for future reference: I use gst 1.12.0 with patches 351981, 353436, 353437, without h264parse workaround, intel driver 5f479acd6d856c13be18321642eac8e9635b25c9, libdrm-2.4.80, libva 1.8.2, i7-7700K CPU.
While it does seem to work, it does make actual browsers/players mad (but mplayer et al seem happy): https://video-dev.github.io/hls.js/demo/?src=https%3A%2F%2Fwww.ubicast.eu%2Fmedia%2Fdownloads%2Fhls%2Fplaylist_360p.m3u8&enableStreaming=true&autoRecoverError=true&enableWorker=true&dumpfMP4=false&levelCapping=-1&defaultAudioCodec=undefined https://bitmovin.com/hls-mpeg-dash-test-player/ The stream has been generated with vaapih264enc bitrate=1000 keyframe-period=30 tune=low-power ! "video/x-h264,profile=baseline" ! queue ! mpegtsmux name=mux ! hlssink location=/tmp/360p/video%02d.ts playlist-location=/tmp/playlist_360p.m3u8 target-duration=1 playlist-root=360p playlist-length=12 max-files=50 ... fdkaacenc ! bitrate=128000 ! queue ! mux. I created an issue on the hls.js player too: https://github.com/video-dev/hls.js/issues/1239
The developer of hls.js said: "there is no AUD NAL unit at the beginning of each PES packet. I rechecked the spec (summarized here) and it is true that AUD are optional. I made a fix for that" It now works as expected with the hls.js playr. However, aren't these patches supposed to do exactly that (i.e. add AUD NAL unit at the beginning of each PES packet) ? https://github.com/video-dev/hls.js/issues/1239#issuecomment-311859900
Hi Florent (In reply to Florent Thiéry from comment #20) > The stream has been generated with vaapih264enc bitrate=1000 > keyframe-period=30 tune=low-power it looks you forgot to add aud=TRUE because with my patch, it is not enabled by default > ! "video/x-h264,profile=baseline" ! queue > ! mpegtsmux name=mux ! hlssink location=/tmp/360p/video%02d.ts > playlist-location=/tmp/playlist_360p.m3u8 target-duration=1 > playlist-root=360p playlist-length=12 max-files=50 ... fdkaacenc ! > bitrate=128000 ! queue ! mux.
Oops right, sorry my bad, it's better now. Why is mplayer happy even without this then (comparing master: happy without aud=true and 1.12.1 unhappy) ?
(In reply to Florent Thiéry from comment #23) > Oops right, sorry my bad, it's better now. Why is mplayer happy even without > this then (comparing master: happy without aud=true and 1.12.1 unhappy) ? Probably, it's because the last 2 patches make it working, which are not related to insertion of aud.
Attachment 351981 [details] pushed as 02a6e9a - libs: encoder: h264: insert AU delimiter Attachment 353436 [details] pushed as 465d586 - libs: encoder: h264: set the frame as IDR if forced key unit Attachment 353437 [details] pushed as 6ebf7b1 - libs: encoder: h264: submit sps in case of IDR picture
I rebuilt master, seems to work pretty well too (Marcin was using 1.12).