GNOME Bugzilla – Bug 576997
floating point stack corruption : emms_c macro not working in ffdeinterlace
Last modified: 2009-05-07 10:53:56 UTC
Hi, I spent lots of time tracking a voodoo issue on one of our encoder on a 64 bit machine : There was a big floating point error once when the encoder was compiled with icc. Doing the calculation once (to get the error) and again will give correct result afterward... By compiling with icc using the -fp-stack-check, I got a crash in my encoder at the place of the floating point calculation error. This means that the floating point context was not clean at this point. After lots of trial, I found the cause of the problem : ffdeinterlace. There is some mmx instruction in the deinterlace code. Normally, the emms_c() macro will take care of calling emms() function to clear the state of the floating point unit. However, the emms_c() macro expect the global mm_flags to be set properly (mmx support) or else, the macro does nothing. I also see that this problem occur in the current ffmpeg (2009-03-24) when using a deinterlace in a raw to raw scenario like this : ./ffmpeg_g -i test.avi -vcodec rawvideo -an -deinterlace out2.avi using a compress input file or choosing an encoder will call the dsputil_init() function and properly set the mm_flags. I made a patch to ffdeinterlace that fix the issue for me. If there is a fix in ffmpeg, my patch might not be need after that point. I am using gst-ffmpeg 0.10.5 but since the problem is cause by ffmpeg code, this patch still apply for latest gst-ffmpeg
Created attachment 131508 [details] [review] patch for this problem
Here is a quick test to confirm this issue : gdb --args gst-launch-0.10 videotestsrc num-buffers=2 ! ffdeinterlace ! fakesink and put a break point in avpicture_deinterlace b avpicture_deinterlace run it. Once it break, check the value of the mm_flags. Without the patch, it is 0. After applying the patch and redoing this test, it should be different than 0 if running on a pc with mmx. if the mm_flags is not properly set, the emms_c macro defined in gst-libs/ext/ffmpeg/libavcodec/dsputil.h will not call emms(); #define emms_c() \ {\ if (mm_flags & MM_MMX)\ emms();\ } This will leave the floating point stack potentially corrupted after the deinterlace mmx function.
Hi, this is not the correct fix as it uses ffmpeg private API (the dsputil.h stuff is private API). The correct fix would be to make sure that the code that uses this calls dsputil_init(). Please file this bug at https://roundup.mplayerhq.hu/roundup/ffmpeg/ avpicture_deinterlace() should call this or somewhere there should be a public initialization function for this.