Created attachment 158924 [details]
patch fixing problem

The GOOM plugin crashes on Solaris when you build it with MMX mode enabled.
I filed a bug against the Sun Studio compiler about this and they informed me that
the problem is a bug in the assembly code.

Here is the comment from the engineer on the Sun Studio team who looked into this:

Actually the problem is not in our compiler. 
goom.c contains wrong code (in aten this problem is masked by differences in the registers allocation):
...
170     __asm__ __volatile__ ("#2 \n\t psrld $4, %%mm0" "#2 \n\t psrld $4, %%mm1"   /* PERTEDEC = $4 */
171         "#4 \n\t movd %%mm1,%%eax"
172         "#3 \n\t movq %%mm3,%%mm5"
173         "#4 \n\t mull %[prevX]"
174         "#4 \n\t movd %%mm0,%%esi"
175         "#3 \n\t punpcklbw %%mm5, %%mm3"
176         "#4 \n\t addl %%esi, %%eax"
177         "#3 \n\t movq %%mm3, %%mm4"
178         "#3 \n\t movq %%mm3, %%mm5"
179         "#4 \n\t movl %[expix1], %%esi"
180         "#3 \n\t punpcklbw %%mm5, %%mm3"
181         "#4 \n\t movq (%%esi,%%eax,4),%%mm0"
182         "#3 \n\t punpckhbw %%mm5, %%mm4"
183         "#4 \n\t addl %[prevX],%%eax"
184         "#4 \n\t movq (%%esi,%%eax,4),%%mm2"::[expix1] "g" (expix1)
185         ,[prevX] "g" (prevX)
186         :"eax", "esi");
...

See at line 173: instruction 'mull' change %edx so '%edx' should be mentioned in the clobber list. 

Updated goom.c is compilled/executed successfully:
[as158432@int4-sol3 tests]$ cat -n 1.c|sed -n '170,186 p'
   170	    __asm__ __volatile__ ("#2 \n\t psrld $4, %%mm0" "#2 \n\t psrld $4, %%mm1"   /* PERTEDEC = $4 */
   171	        "#4 \n\t movd %%mm1,%%eax"
   172	        "#3 \n\t movq %%mm3,%%mm5"
   173	        "#4 \n\t mull %[prevX]"
   174	        "#4 \n\t movd %%mm0,%%esi"
   175	        "#3 \n\t punpcklbw %%mm5, %%mm3"
   176	        "#4 \n\t addl %%esi, %%eax"
   177	        "#3 \n\t movq %%mm3, %%mm4"
   178	        "#3 \n\t movq %%mm3, %%mm5"
   179	        "#4 \n\t movl %[expix1], %%esi"
   180	        "#3 \n\t punpcklbw %%mm5, %%mm3"
   181	        "#4 \n\t movq (%%esi,%%eax,4),%%mm0"
   182	        "#3 \n\t punpckhbw %%mm5, %%mm4"
   183	        "#4 \n\t addl %[prevX],%%eax"
   184	        "#4 \n\t movq (%%esi,%%eax,4),%%mm2"::[expix1] "g" (expix1)
   185	        ,[prevX] "g" (prevX)
   186	        :"eax", "esi", "edx");

Sure enough, fixing the code with the attached patch fixes the problem