After an evaluation, GNOME has moved from Bugzilla to GitLab. Learn more about GitLab.
No new issues can be reported in GNOME Bugzilla anymore.
To report an issue in a GNOME project, go to GNOME GitLab.
Do not go to GNOME Gitlab for: Bluefish, Doxygen, GnuCash, GStreamer, java-gnome, LDTP, NetworkManager, Tomboy.
Bug 797075 - hls segfaults
hls segfaults
Status: RESOLVED OBSOLETE
Product: GStreamer
Classification: Platform
Component: gst-plugins-bad
git master
Other Linux
: Normal normal
: git master
Assigned To: GStreamer Maintainers
GStreamer Maintainers
Depends on:
Blocks:
 
 
Reported: 2018-09-04 20:50 UTC by Nicola
Modified: 2018-11-03 14:31 UTC
See Also:
GNOME target: ---
GNOME version: ---



Description Nicola 2018-09-04 20:50:33 UTC
I see some crashs using playbin for an hls stream

- crash 1:

  • #0 malloc
  • #1 __vasprintf_chk
  • #2 g_vasprintf
  • #3 g_strdup_vprintf
  • #4 g_strdup_printf
  • #5 uri_join
    at ../subprojects/gst-plugins-bad/ext/hls/m3u8.c line 1111
  • #6 gst_m3u8_update
    at ../subprojects/gst-plugins-bad/ext/hls/m3u8.c line 522
  • #7 gst_hls_demux_update_playlist
    at ../subprojects/gst-plugins-bad/ext/hls/gsthlsdemux.c line 1459
  • #8 gst_hls_demux_change_playlist
    at ../subprojects/gst-plugins-bad/ext/hls/gsthlsdemux.c line 1599
  • #9 gst_hls_demux_select_bitrate
    at ../subprojects/gst-plugins-bad/ext/hls/gsthlsdemux.c line 1149
  • #10 gst_adaptive_demux_stream_select_bitrate
    at ../subprojects/gst-plugins-bad/gst-libs/gst/adaptivedemux/gstadaptivedemux.c line 4280
  • #11 gst_adaptive_demux_stream_advance_fragment_unlocked
    at ../subprojects/gst-plugins-bad/gst-libs/gst/adaptivedemux/gstadaptivedemux.c line 4232
  • #12 gst_adaptive_demux_stream_advance_fragment
    at ../subprojects/gst-plugins-bad/gst-libs/gst/adaptivedemux/gstadaptivedemux.c line 4153
  • #13 gst_hls_demux_finish_fragment
    at ../subprojects/gst-plugins-bad/ext/hls/gsthlsdemux.c line 953
  • #14 gst_adaptive_demux_eos_handling
    at ../subprojects/gst-plugins-bad/gst-libs/gst/adaptivedemux/gstadaptivedemux.c line 2730
  • #15 _src_event
    at ../subprojects/gst-plugins-bad/gst-libs/gst/adaptivedemux/gstadaptivedemux.c line 2748
  • #16 gst_pad_send_event_unchecked
    at ../subprojects/gstreamer/gst/gstpad.c line 5757
  • #17 gst_pad_push_event_unchecked
    at ../subprojects/gstreamer/gst/gstpad.c line 5402
  • #18 push_sticky
    at ../subprojects/gstreamer/gst/gstevent.h line 438
  • #19 events_foreach
    at ../subprojects/gstreamer/gst/gstpad.c line 608
  • #20 check_sticky
    at ../subprojects/gstreamer/gst/gstpad.c line 3977
  • #21 gst_pad_push_event
    at ../subprojects/gstreamer/gst/gstpad.c line 5533
  • #22 event_forward_func
    at ../subprojects/gstreamer/gst/gstevent.h line 438
  • #23 gst_pad_forward
    at ../subprojects/gstreamer/gst/gstpad.c line 3004
  • #24 gst_pad_event_default
    at ../subprojects/gstreamer/gst/gstpad.c line 3101
  • #25 gst_pad_send_event_unchecked
    at ../subprojects/gstreamer/gst/gstpad.c line 5757
  • #26 gst_pad_push_event_unchecked
    at ../subprojects/gstreamer/gst/gstpad.c line 5402
  • #27 push_sticky
    at ../subprojects/gstreamer/gst/gstevent.h line 438
  • #28 events_foreach
    at ../subprojects/gstreamer/gst/gstpad.c line 608
  • #29 check_sticky
    at ../subprojects/gstreamer/gst/gstpad.c line 3977
  • #30 gst_pad_push_event
    at ../subprojects/gstreamer/gst/gstpad.c line 5533
  • #31 gst_queue_push_one
    at ../subprojects/gstreamer/plugins/elements/gstqueue.c line 1455
  • #32 gst_queue_loop
    at ../subprojects/gstreamer/plugins/elements/gstqueue.c line 1537
  • #33 gst_task_func
    at ../subprojects/gstreamer/gst/gsttask.c line 328
  • #34 0x00007ffff7c37bf6 in
  • #35 0x00007ffff7c371ea in
  • #36 start_thread
  • #0 __strlen_avx2
  • #1 g_strdup
  • #2 gst_hls_demux_update_fragment_info
    at ../subprojects/gst-plugins-bad/ext/hls/gsthlsdemux.c line 1102
  • #3 gst_adaptive_demux_stream_update_fragment_info
    at ../subprojects/gst-plugins-bad/gst-libs/gst/adaptivedemux/gstadaptivedemux.c line 4303
  • #4 gst_adaptive_demux_stream_download_loop
    at ../subprojects/gst-plugins-bad/gst-libs/gst/adaptivedemux/gstadaptivedemux.c line 3716
  • #5 gst_task_func
    at ../subprojects/gstreamer/gst/gsttask.c line 328
  • #6 0x00007ffff7c37bf6 in
  • #7 0x00007ffff7c371ea in
  • #8 start_thread
  • #9 clone

the same stream works with no crash in ffplay
Comment 1 Nicolas Dufresne (ndufresne) 2018-09-04 21:48:28 UTC
Do you have an uri to share so we can test ? Or the m3u8 file ?
Comment 2 Sebastian Dröge (slomo) 2018-09-05 07:29:23 UTC
Or alternatively backtraces without optimizations, or you'll have to debug yourself first why those string functions are crashing (which of the pointers is wrong, and why).
Comment 3 Nicola 2018-09-07 07:51:59 UTC
the problem does not happen anymore so I cannot reproduce now, I'll have to wait for few days ... it happens periodically

I have no control on the server, anyway when this edge case happen then GStreamer crashs every few minutes so basically is unable to play the hls stream while ffmpeg and flowplayer work. 

Probably the m3u8 file is corrupted and other players redownload it, I remember some 404 and consecutive m3u8 download in the browser developer console.

For your info GStreamer has more delay than flowplayer and ffmpeg too: the server store 2 segments, I suppose GStreamer starts to play from the first one, in ffmpeg you can set live_start_index to start from the last one, this help a lot if hls is a "live" streaming
Comment 4 Nicola 2018-09-08 13:58:57 UTC
it probably happens because sometime the m3u8 is a binary file,

for example

curl "<url to m3u8>"

returns
 
Warning: Binary output can mess up your terminal. Use "--output -" to tell 
Warning: curl to output it to your terminal anyway, or consider "--output 
Warning: <FILE>" to save to a file.

this is obvious a server error, but if you try to download the m3u8 again after few milliseconds it works as expected.

Another strage behaviour of this broken server:

it returns something like this as m3u8

#EXTM3U
#EXT-X-VERSION:3
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=420000
live2_lq/index.m3u8
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=670000
live2_mq/index.m3u8
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=1100000
live2_hq/index.m3u8

sometime the links to live2_lq/index.m3u8 and live2_mq/index.m3u8 return 404 http error while the hq link is ok
Comment 5 Nicolas Dufresne (ndufresne) 2018-09-08 15:28:16 UTC
It's not calling into the binary case. gst_hls_src_buf_to_utf8_playlist() runs g_utf8_validate() which would detect this case.

After that, we clearly have gone passed g_str_has_prefix (data, "#EXTM3U")) and g_strrstr (data, "\n#EXT-X-STREAM-INF:") validation, so the file is probably not that corrupted.

So I can only blame the hand written parser that follow this code. If you manage to reproduce, got to frame 6, and and print "data". Attach the output here. We can then plug that data into our unit test and easily reproduce (and fix) that bug.

For the refefence:
 gdb) f 6
 gdb) printf "%s",data
Comment 6 Nicolas Dufresne (ndufresne) 2018-09-08 15:36:56 UTC
Another note, at the line of the crash, gst_uri_is_valid() returns success, so the data does looks like a valid URI. The code also thinks we have a relative URI. As we crash in malloc, the size passed to malloc should be traced. It also possible that it's some other memory corruption that lead to that ...
Comment 7 Nicola 2018-09-10 11:47:16 UTC
here is another crash with hopefully the needed info

  • #0 malloc
  • #1 __vasprintf_chk
  • #2 g_vasprintf
  • #3 g_string_append_vprintf
  • #4 g_string_append_printf
  • #5 g_log_writer_format_fields
  • #6 g_log_writer_standard_streams
  • #7 g_log_writer_default
  • #8 g_log_structured_array
  • #9 g_log_default_handler
  • #10 g_logv
  • #11 g_log
  • #12 gst_structure_set_field
    at gststructure.c line 801
  • #13 gst_structure_set_valist_internal
    at gststructure.c line 612
  • #14 gst_structure_new_valist
    at gststructure.c line 286
  • #15 gst_structure_new
    at gststructure.c line 255
  • #16 gst_hls_demux_change_playlist
    at gsthlsdemux.c line 1605
  • #17 gst_hls_demux_select_bitrate
    at gsthlsdemux.c line 1149
  • #18 gst_adaptive_demux_stream_select_bitrate
    at gstadaptivedemux.c line 4280
  • #19 gst_adaptive_demux_stream_advance_fragment_unlocked
    at gstadaptivedemux.c line 4232
  • #20 gst_adaptive_demux_stream_advance_fragment
    at gstadaptivedemux.c line 4153
  • #21 gst_hls_demux_finish_fragment
    at gsthlsdemux.c line 953
  • #22 gst_adaptive_demux_eos_handling
    at gstadaptivedemux.c line 2730
  • #23 _src_event
    at gstadaptivedemux.c line 2748
  • #24 gst_pad_send_event_unchecked
    at gstpad.c line 5757
  • #25 gst_pad_push_event_unchecked
    at gstpad.c line 5402
  • #26 push_sticky
    at ../gst/gstevent.h line 438
  • #27 events_foreach
    at gstpad.c line 608
  • #28 check_sticky
    at gstpad.c line 3977
  • #29 gst_pad_push_event
    at gstpad.c line 5533
  • #30 event_forward_func
    at ../gst/gstevent.h line 438
  • #31 gst_pad_forward
    at gstpad.c line 3004
  • #32 gst_pad_event_default
    at gstpad.c line 3101
  • #33 gst_pad_send_event_unchecked
    at gstpad.c line 5757
  • #34 gst_pad_push_event_unchecked
    at gstpad.c line 5402
  • #35 push_sticky
    at ../gst/gstevent.h line 438
  • #36 events_foreach
    at gstpad.c line 608
  • #37 check_sticky
    at gstpad.c line 3977
  • #38 gst_pad_push_event
    at gstpad.c line 5533
  • #39 gst_queue_push_one
    at gstqueue.c line 1455
  • #40 gst_queue_loop
    at gstqueue.c line 1537
  • #41 gst_task_func
    at gsttask.c line 328
  • #42 0x00007ffff7501bd6 in
  • #43 0x00007ffff74fa3eb in
  • #44 start_thread
  • #45 clone
  • #16 gst_hls_demux_change_playlist
    at gsthlsdemux.c line 1605
  • #12 gst_structure_set_field
    at gststructure.c line 801
g(gdb) printf "%x",field
e0ba0450(gdb)
Comment 8 Nicola 2018-09-10 11:51:23 UTC
f 16
  • #16 gst_hls_demux_change_playlist
    at gsthlsdemux.c line 1605
  • #12 gst_structure_set_field
    at gststructure.c line 801

Comment 9 Nicolas Dufresne (ndufresne) 2018-09-10 13:23:02 UTC
It's starting to look bad, the only thing these two have in common is malloc. Looks like memory corruption. We need to produce in valgrind (or similar) to figure out.
Comment 10 Nicola 2018-09-10 13:39:06 UTC
Sorry I gave up, I switched to ffmpeg this weekend and now I use GStreamer only for audio and video sinks,

I'll try to find some spare time to provide the needed info but it is quite difficult since the stream is available only during working hours and most of the time hlsdemux hangs (see #797101) so it is a very time consuming task.

As soon as I have some spare time I would like to summarize the steps and the workarounds needed to correctly handle this hls "live" stream in my ffmpeg + gst app, maybe this will help to improve the hls support in GStreamer
Comment 11 GStreamer system administrator 2018-11-03 14:31:03 UTC
-- GitLab Migration Automatic Message --

This bug has been migrated to freedesktop.org's GitLab instance and has been closed from further activity.

You can subscribe and participate further through the new bug through this link to our GitLab instance: https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/issues/780.