Created attachment 336440 [details]
Demo program provoking the crash on 64 bit machines.

Under some circumstances the range check in gegl/buffer/gegl-sampler.h fails to recognize, that it needs to fetch data.

This happens when x and y coordinate are close to 2^31 (at least on my 64 bit machine).

Attached is a test program. For me this crashes on the 3rd gegl_sampler_get().

I have added some debug output to gegl that prints out the ROIs around the range checks. This is the output from a run with this debug output enabled:


This is the output from a gdb run with a LINEAR sampler:

(gdb) run
Starting program: /home/simon/src/unstable/gegl/tests/simple/.libs/test-sampler 
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
need (2147483647, 2147483647; 3, 3), have (0, 0; 0, 0)
0.000000,0.000000,0.000000,0.000000
need (2147483647, 85707687; 3, 3), have (0, 0; 0, 0)
fetching (2147483645, 85707685; 9, 9)
0.000000,0.000000,0.000000,0.000000
need (2147483647, 2147483647; 3, 3), have (2147483645, 85707685; 9, 9)

Program received signal SIGSEGV, Segmentation fault.
gegl_sampler_linear_get (self=0x6c20a0 [GeglSamplerLinear], 
    absolute_x=<optimized out>, absolute_y=<optimized out>, 
    scale=<optimized out>, output=0x7fffffffdf60, repeat_mode=GEGL_ABYSS_NONE)
    at gegl-sampler-linear.c:208
208         const gfloat bot_rite_3 = *in_bptr;
(gdb) bt

+ Trace 236707

Note that for the 1st coordinate it does not recognize, that it doesn't have the necessary area available. It does recognize this for the 2nd coordinate (where y is in a managable range), but on the 3rd attept it again doesn't recognize the need to fetch data. This time however, it crashes, since it probably tries to read way outside of the area cached.