After an evaluation, GNOME has moved from Bugzilla to GitLab. Learn more about GitLab.
No new issues can be reported in GNOME Bugzilla anymore.
To report an issue in a GNOME project, go to GNOME GitLab.
Do not go to GNOME Gitlab for: Bluefish, Doxygen, GnuCash, GStreamer, java-gnome, LDTP, NetworkManager, Tomboy.
Bug 774428 - qtdemux: Outputting unaligned raw audio/video buffers
qtdemux: Outputting unaligned raw audio/video buffers
Status: RESOLVED FIXED
Product: GStreamer
Classification: Platform
Component: gst-plugins-good
git master
Other Linux
: Normal normal
: 1.10.2
Assigned To: GStreamer Maintainers
GStreamer Maintainers
Depends on:
Blocks:
 
 
Reported: 2016-11-14 20:23 UTC by Thibault Saunier
Modified: 2016-11-24 11:41 UTC
See Also:
GNOME target: ---
GNOME version: ---


Attachments
Pipeline dump (45.15 KB, image/svg+xml)
2016-11-17 19:21 UTC, Thibault Saunier
Details

Description Thibault Saunier 2016-11-14 20:23:45 UTC
You can find an example test here: https://ci.gstreamer.net/job/GStreamer-master-validate/lastCompletedBuild/testReport/validate.http.transcode/to_vorbis_and_theora_in_ogg/raw_video_mov/

This bug only happens when orc is enabled and only when running the GstValidate HTTP server ported to python3 (if you run the old version in python2 it seems to never happen). This issue seems to only happen when transcoding raw video(UYVY) inside MOV files.

It is very simple to reproduce on master doing:

   gst-validate-launcher -t validate.http.transcode.to_vorbis_and_vp8_in_webm.raw_video_mov -f

(it is somehow racy but it fails almost all the time still).

Another way of reproducing with a sensibly smaller pipeline:

    gst-validate-launcher --http-only --http-server-port=8000 && gst-validate-1.0 uridecodebin uri=http://127.0.0.1:8000/defaults/mp4/raw_video.mov  ! videoconvert ! video/x-raw,format=I420 ! autovideosink

---

The stack trace:

           PID: 937 (gst-validate-tr)
           UID: 1001 (jenkins)
           GID: 1001 (jenkins)
        Signal: 11 (SEGV)
     Timestamp: Mon 2016-11-14 18:18:02 UTC (15s ago)
  Command Line: /home/jenkins/workspace/GStreamer-master-validate/gst-devtools/validate/tools/gst-validate-transcoding-1.0-debug -o application/ogg:video/x-theora http://127.0.0.1:8039/defaults/mp4/raw_video.mov file:///home/jenkins/workspace/GStreamer-master-validate/validate-output/rendered/validate/http/to_vorbis_and_theora_in_ogg/raw_video_mov
    Executable: /home/jenkins/workspace/GStreamer-master-validate/gst-devtools/validate/tools/gst-validate-transcoding-1.0-debug
 Control Group: /
         Slice: -.slice
       Boot ID: b8064adcba4347fbb05f72f8a3318336
    Machine ID: 4d2c5bb8d53a4138bff9fd3a6cfe9d8f
      Hostname: london.bilboed.com
      Coredump: /var/lib/systemd/coredump/core.gst-validate-tr.1001.b8064adcba4347fbb05f72f8a3318336.937.1479147482000000.xz
       Message: Process 937 (gst-validate-tr) of user 1001 dumped core.
                
                Stack trace of thread 1058:
                #0  0x00007f7dea104180 n/a (n/a)

Thread apply all bt:

GNU gdb (GDB) Fedora 7.10.1-31.fc23
Copyright (C) 2015 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /home/jenkins/workspace/GStreamer-master-validate/gst-devtools/validate/tools/gst-validate-transcoding-1.0-debug...done.

warning: core file may not match specified executable file.
[New LWP 1058]
[New LWP 937]
[New LWP 1029]
[New LWP 1041]
[New LWP 1046]
[New LWP 980]
[New LWP 981]
[New LWP 1059]
[New LWP 1132]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/home/jenkins/workspace/GStreamer-master-validate/gst-devtools/validate/tools/g'.
Program terminated with signal SIGSEGV, Segmentation fault.

Thread 9 (Thread 0x7f7dc0ece700 (LWP 1132))

  • #0 syscall
  • #1 g_cond_wait_until
  • #2 g_async_queue_pop_intern_unlocked
  • #3 g_thread_pool_thread_proxy
  • #4 g_thread_proxy
  • #5 start_thread
  • #6 clone

Thread 8 (Thread 0x7f7dc16cf700 (LWP 1059))

  • #0 syscall
  • #1 g_cond_wait
  • #2 gst_queue_loop
    at gstqueue.c line 1494
  • #3 gst_task_func
    at gsttask.c line 334
  • #4 default_func
    at gsttaskpool.c line 68
  • #5 g_thread_pool_thread_proxy
  • #6 g_thread_proxy
  • #7 start_thread
  • #8 clone

Thread 7 (Thread 0x7f7dde518700 (LWP 981))

  • #0 syscall
  • #1 g_cond_wait_until
  • #2 g_async_queue_pop_intern_unlocked
  • #3 g_async_queue_timeout_pop
  • #4 g_thread_pool_thread_proxy
  • #5 g_thread_proxy
  • #6 start_thread
  • #7 clone

Thread 5 (Thread 0x7f7dc3147700 (LWP 1046))

  • #0 syscall
  • #1 g_cond_wait
  • #2 gst_queue_chain_buffer_or_list
    at gstqueue.c line 1222
  • #3 gst_queue_chain
    at gstqueue.c line 1320
  • #4 gst_validate_pad_monitor_chain_func
    at gst-validate-pad-monitor.c line 2121
  • #5 gst_pad_chain_data_unchecked
    at gstpad.c line 4206
  • #6 gst_pad_push_data
    at gstpad.c line 4458
  • #7 gst_pad_push
    at gstpad.c line 4577
  • #8 gst_proxy_pad_chain_default
    at gstghostpad.c line 126
  • #9 gst_pad_chain_data_unchecked
    at gstpad.c line 4206
  • #10 gst_pad_push_data
    at gstpad.c line 4458
  • #11 gst_pad_push
    at gstpad.c line 4577
  • #12 gst_proxy_pad_chain_default
    at gstghostpad.c line 126
  • #13 gst_pad_chain_data_unchecked
    at gstpad.c line 4206
  • #14 gst_pad_push_data
    at gstpad.c line 4458
  • #15 gst_pad_push
    at gstpad.c line 4577
  • #16 gst_proxy_pad_chain_default
    at gstghostpad.c line 126
  • #17 gst_pad_chain_data_unchecked
    at gstpad.c line 4206
  • #18 gst_pad_push_data
    at gstpad.c line 4458
  • #19 gst_pad_push
    at gstpad.c line 4577
  • #20 gst_single_queue_push_one
    at gstmultiqueue.c line 1611
  • #21 gst_multi_queue_loop
    at gstmultiqueue.c line 1915
  • #22 gst_task_func
    at gsttask.c line 334
  • #23 default_func
    at gsttaskpool.c line 68
  • #24 g_thread_pool_thread_proxy
  • #25 g_thread_proxy
  • #26 start_thread
  • #27 clone

Thread 4 (Thread 0x7f7dc3fff700 (LWP 1041))

  • #0 syscall
  • #1 g_cond_wait
  • #2 gst_data_queue_push
    at gstdataqueue.c line 520
  • #3 gst_multi_queue_chain
    at gstmultiqueue.c line 2104
  • #4 gst_validate_pad_monitor_chain_func
    at gst-validate-pad-monitor.c line 2121
  • #5 gst_pad_chain_data_unchecked
    at gstpad.c line 4206
  • #6 gst_pad_push_data
    at gstpad.c line 4458
  • #7 gst_pad_push
    at gstpad.c line 4577
  • #8 gst_qtdemux_decorate_and_push_buffer
    at qtdemux.c line 5350
  • #9 gst_qtdemux_process_adapter
    at qtdemux.c line 6671
  • #10 gst_qtdemux_chain
    at qtdemux.c line 6140
  • #11 gst_validate_pad_monitor_chain_func
    at gst-validate-pad-monitor.c line 2121
  • #12 gst_pad_chain_data_unchecked
    at gstpad.c line 4206
  • #13 gst_pad_push_data
    at gstpad.c line 4458
  • #14 gst_pad_push
    at gstpad.c line 4577
  • #15 gst_type_find_element_chain
    at gsttypefindelement.c line 894
  • #16 gst_validate_pad_monitor_chain_func
    at gst-validate-pad-monitor.c line 2121
  • #17 gst_pad_chain_data_unchecked
    at gstpad.c line 4206
  • #18 gst_pad_push_data
    at gstpad.c line 4458
  • #19 gst_pad_push
    at gstpad.c line 4577
  • #20 gst_proxy_pad_chain_default
    at gstghostpad.c line 126
  • #21 gst_pad_chain_data_unchecked
    at gstpad.c line 4206
  • #22 gst_pad_push_data
    at gstpad.c line 4458
  • #23 gst_pad_push
    at gstpad.c line 4577
  • #24 gst_queue2_push_one
    at gstqueue2.c line 2908
  • #25 gst_queue2_loop
    at gstqueue2.c line 3030
  • #26 gst_task_func
    at gsttask.c line 334
  • #27 default_func
    at gsttaskpool.c line 68
  • #28 g_thread_pool_thread_proxy
  • #29 g_thread_proxy
  • #30 start_thread
  • #31 clone

Thread 3 (Thread 0x7f7dc8c93700 (LWP 1029))

  • #0 syscall
  • #1 g_cond_wait
  • #2 gst_task_func
    at gsttask.c line 319
  • #3 default_func
    at gsttaskpool.c line 68
  • #4 g_thread_pool_thread_proxy
  • #5 g_thread_proxy
  • #6 start_thread
  • #7 clone

Thread 1 (Thread 0x7f7dc1ed0700 (LWP 1058))

  • #0 0x00007f7dea104180 in
  • #1 video_orc_convert_UYVY_Y42B
    at tmp-orc.c line 15891
  • #2 convert_UYVY_Y42B
    at video-converter.c line 3147
  • #3 gst_video_converter_frame
    at video-converter.c line 2374
  • #4 gst_video_convert_transform_frame
    at gstvideoconvert.c line 692
  • #5 gst_video_filter_transform
    at gstvideofilter.c line 271
  • #6 default_generate_output
    at gstbasetransform.c line 2183
  • #7 gst_base_transform_chain
    at gstbasetransform.c line 2336
  • #8 gst_validate_pad_monitor_chain_func
    at gst-validate-pad-monitor.c line 2121
  • #9 gst_pad_chain_data_unchecked
    at gstpad.c line 4206
  • #10 gst_pad_push_data
    at gstpad.c line 4458
  • #11 gst_pad_push
    at gstpad.c line 4577
  • #12 gst_base_transform_chain
    at gstbasetransform.c line 2372
  • #13 gst_validate_pad_monitor_chain_func
    at gst-validate-pad-monitor.c line 2121
  • #14 gst_pad_chain_data_unchecked
    at gstpad.c line 4206
  • #15 gst_pad_push_data
    at gstpad.c line 4458
  • #16 gst_pad_push
    at gstpad.c line 4577
  • #17 gst_base_transform_chain
    at gstbasetransform.c line 2372
  • #18 gst_validate_pad_monitor_chain_func
    at gst-validate-pad-monitor.c line 2121
  • #19 gst_pad_chain_data_unchecked
    at gstpad.c line 4206
  • #20 gst_pad_push_data
    at gstpad.c line 4458
  • #21 gst_pad_push
    at gstpad.c line 4577
  • #22 gst_stream_splitter_chain
    at gststreamsplitter.c line 140
  • #23 gst_validate_pad_monitor_chain_func
    at gst-validate-pad-monitor.c line 2121
  • #24 gst_pad_chain_data_unchecked
    at gstpad.c line 4206
  • #25 gst_pad_push_data
    at gstpad.c line 4458
  • #26 gst_pad_push
    at gstpad.c line 4577
  • #27 gst_queue_push_one
    at gstqueue.c line 1359
  • #28 gst_queue_loop
    at gstqueue.c line 1506
  • #29 gst_task_func
    at gsttask.c line 334
  • #30 default_func
    at gsttaskpool.c line 68
  • #31 g_thread_pool_thread_proxy
  • #32 g_thread_proxy
  • #33 start_thread
  • #34 clone

Comment 1 Sebastian Dröge (slomo) 2016-11-15 06:01:17 UTC
Can you also get a valgrind log (with --track-origins=yes) of it? What are the exact caps on both sides of the videoconvert here, the following?

> /GstPipeline:pipeline0/GstVideoConvert:videoconvert0.GstPad:src: caps = video/x-raw, width=(int)320, height=(int)240, interlace-mode=(string)progressive, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)30/1, format=(string)I420
> /GstPipeline:pipeline0/GstVideoConvert:videoconvert0.GstPad:sink: caps = video/x-raw, format=(string)UYVY, width=(int)320, height=(int)240, interlace-mode=(string)progressive, pixel-aspect-ratio=(fraction)1/1, chroma-site=(string)jpeg, colorimetry=(string)bt601, framerate=(fraction)30/1
Comment 2 Edward Hervey 2016-11-16 16:31:08 UTC
in caps:  video/x-raw, format=(string)UYVY, width=(int)320, height=(int)240, interlace-mode=(string)progressive, pixel-aspect-ratio=(fraction)1/1, chroma-site=(string)jpeg, colorimetry=(string)bt601, framerate=(fraction)30/1
out caps: video/x-raw, width=(int)320, height=(int)240, interlace-mode=(string)progressive, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)30/1, format=(string)I420

And I confirm it's been failing on ci.gst since the switch to python3 (and not before).

I can also reproduce it locally (so not specific to ci.gst machine).
Comment 3 Thibault Saunier 2016-11-17 19:21:02 UTC
Created attachment 340182 [details]
Pipeline dump

Output of valgrind --track-origins=yes (does not sound useful to me, am I missing something?):

==7862== Process terminating with default action of signal 11 (SIGSEGV): dumping core
==7862==  General Protection Fault
==7862==    at 0x9AAD1C0: ??? (in /run/user/1000/orcexec.BpSdB0 (deleted))
==7862==    by 0x7BC80F2: convert_UYVY_I420 (video-converter.c:3074)
==7862==    by 0x7BC51BC: gst_video_converter_frame (video-converter.c:2374)
==7862==    by 0xB5E8A9B: gst_video_convert_transform_frame (gstvideoconvert.c:692)
==7862==    by 0x7BAF45A: gst_video_filter_transform (gstvideofilter.c:271)
==7862==    by 0x7E5F866: default_generate_output (gstbasetransform.c:2183)
==7862==    by 0x7E5FF1E: gst_base_transform_chain (gstbasetransform.c:2336)
==7862==    by 0x4E56AF6: gst_validate_pad_monitor_chain_func (gst-validate-pad-monitor.c:2121)
==7862==    by 0x512CABD: gst_pad_chain_data_unchecked (gstpad.c:4206)
==7862==    by 0x512D68C: gst_pad_push_data (gstpad.c:4458)
==7862==    by 0x512DDB4: gst_pad_push (gstpad.c:4577)
==7862==    by 0x510D698: gst_proxy_pad_chain_default (gstghostpad.c:126)
==7862== 
==7862== HEAP SUMMARY:
==7862==     in use at exit: 34,386,589 bytes in 62,430 blocks
==7862==   total heap usage: 265,164 allocs, 202,734 frees, 135,261,758 bytes allocated
==7862== 
==7862== LEAK SUMMARY:
==7862==    definitely lost: 16,472 bytes in 4 blocks
==7862==    indirectly lost: 1,573 bytes in 57 blocks
==7862==      possibly lost: 19,635 bytes in 258 blocks
==7862==    still reachable: 34,176,229 bytes in 61,295 blocks
==7862==                       of which reachable via heuristic:
==7862==                         length64           : 4,448 bytes in 86 blocks
==7862==                         newarray           : 2,000 bytes in 45 blocks
==7862==         suppressed: 0 bytes in 0 blocks
==7862== Rerun with --leak-check=full to see details of leaked memory
==7862== 
==7862== For counts of detected and suppressed errors, rerun with: -v
==7862== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
[1]    7862 segmentation fault (core dumped)
Comment 4 Sebastian Dröge (slomo) 2016-11-17 19:31:26 UTC
That looks like valgrind did not work well for you. Usually it a) prints the address that was tried to be accessed, and b) with --track-origins=yes tells you things like "which was freed here and allocated here", or "which is 1000 bytes into an area of size 1234 allocated here".
Comment 5 Sebastian Dröge (slomo) 2016-11-17 19:31:48 UTC
Does it also crash when using ORC_CODE=backup?
Comment 6 Thibault Saunier 2016-11-17 19:36:49 UTC
(In reply to Sebastian Dröge (slomo) from comment #4)
> That looks like valgrind did not work well for you. Usually it a) prints the
> address that was tried to be accessed, and b) with --track-origins=yes tells
> you things like "which was freed here and allocated here", or "which is 1000
> bytes into an area of size 1234 allocated here".

Yes, let me check why (coming back to you :))

(In reply to Sebastian Dröge (slomo) from comment #5)
> Does it also crash when using ORC_CODE=backup?

No, it only fails when actually using orc.
Comment 7 Thibault Saunier 2016-11-17 20:32:07 UTC
Looks like I am not able to get information from valgrind, any idea why?
Comment 8 Sebastian Dröge (slomo) 2016-11-20 10:45:15 UTC
Doesn't crash here for me on Debian.

Python 3.5.2
python-gi 3.22.0
Intel Core i7-4790K
Comment 9 Sebastian Dröge (slomo) 2016-11-20 10:46:08 UTC
I spoke to fast. It crashes when running in valgrind, but not otherwise.
Comment 10 Sebastian Dröge (slomo) 2016-11-20 10:56:26 UTC
This is an alignment problem probably. The src pointers passed in convert_UYVY_I420() to video_orc_convert_UYVY_I420() are required to be 32 bit aligned (we give them as 32 bit words to ORC!) but at the time it crashes they are even odd here.

Similar the first two dest pointers must be 2-byte aligned (the two Y destinations), but that's the case unless odd width (or someone decides to do an odd stride... we don't prevent that, do we?).
Comment 11 Sebastian Dröge (slomo) 2016-11-20 10:58:45 UTC
Problem here is that qtdemux does not ensure any alignment of the output buffers. For raw audio and video we require that though
Comment 12 Tim-Philipp Müller 2016-11-20 10:58:58 UTC
I can reproduce this quite easily on Debian too by serving the file with twistd -n web --path .... and then doing:

gst-launch-1.0 uridecodebin uri=http://127.0.0.1:8080/raw_video.mov  ! videoconvert ! video/x-raw,format=I420 ! autovideosink

I'm quite certain it's an alignment issue.
Comment 13 Sebastian Dröge (slomo) 2016-11-20 11:09:42 UTC
commit bb35f15d44e6a881f2c64fe731345c6d840fe789
Author: Sebastian Dröge <sebastian@centricular.com>
Date:   Sun Nov 20 13:08:27 2016 +0200

    qtdemux: Ensure that raw audio and video have properly aligned buffers
    
    That is, aligned to the basic type for audio and to 32 bytes for video.
    Fixes crashes if the raw buffers are passed to SIMD processing functions.
    
    https://bugzilla.gnome.org/show_bug.cgi?id=774428
Comment 14 Sebastian Dröge (slomo) 2016-11-20 11:10:29 UTC
It's correct in matroskademux, we should check how it is in other demuxers.
Comment 15 Sebastian Dröge (slomo) 2016-11-20 11:14:57 UTC
commit b8265e95a7c75f7a932a421419d6eeda67645d8e
Author: Sebastian Dröge <sebastian@centricular.com>
Date:   Sun Nov 20 13:14:08 2016 +0200

    avidemux: Ensure that raw video have properly aligned buffers
    
    That is, aligned to to 32 bytes for video. Fixes crashes if the raw
    buffers are passed to SIMD processing functions.
    
    https://bugzilla.gnome.org/show_bug.cgi?id=774428
Comment 16 Nicolas Dufresne (ndufresne) 2016-11-20 16:41:52 UTC
I think we should on top run an allocation query and re-use the allocation params there. Even better would be using a downstream pool when available.
Comment 17 Tim-Philipp Müller 2016-11-20 16:48:28 UTC
There are/were problems with having demuxers do allocation queries because of the blocking nature of the allocation queries.
Comment 18 Sebastian Dröge (slomo) 2016-11-21 07:18:55 UTC
Yes, see

commit b001da292626c16c3cfa995585673380f65a9f4f
Author: Sebastian Dröge <slomo@circular-chaos.org>
Date:   Wed Jun 19 11:06:37 2013 +0200

    qtdemux: Disable usage of allocation queries
    
    This can only reliably work if demuxers have a
    separate streaming thread per srcpad. This should be
    done in a demuxer base class, which integrates parts
    of multiqueue
    
    https://bugzilla.gnome.org/show_bug.cgi?id=701856