GNOME Bugzilla – Bug 509988
"unsigned integer value" functions fail for large numbers
Last modified: 2009-01-26 11:33:08 UTC
This report has been filled here: https://bugs.launchpad.net/ubuntu/+source/gcalctool/+bug/181763 "The "&16" and "&32" functions both fail for large inputs: u16(11111111111111111111) u32(11111111111111111111) Both evaluate to 0 (and their least significant bit is 0) But doing the "near" equivalent (which stuffs up for negatives and behaves differently for non-integer inputs) works: 11111111111111111111 Mod 65536 = 29127 11111111111111111111 Mod 4294967291 = 3098111297"
Yup, it indeed appears to be a bug. I noticed that u16(1111111111111111111) and u32(1111111111111111111) work though. I.e. 19 1's rather than 20.
The functions both work by converting the number to a double, and then to an integer. With the larger numbers, the double runs out of precision. I plan on making a patch when I have the time (hopefully within a month)
This would be calc_u32() and calc_u16() in mpmath.c Yeah, anywhere in the gcalctool code that does a conversion to a double first for something like this, is broken. Thanks.
So I've gone a bit sour on the idea of patching it, mainly because I think that #505608 is a better way of fixing it (switching to GMP for maths). One of the test cases which I was using was u32(2^32), and it exposed another bug (enter "1/(2^32-4294967296)" (or even "1/(2^1-2)") and note the lack of divide by zero - 2^32 returns a number slightly less than 2^32). So I'm not sure if the correct way to fix it is to round the numbers to integers first (currently they are truncated), do a dodgy hack, or fix the other bug. I'm less enthused about writing a power function in the same auto-converted fortran style of C, given that there are probably heaps of other bugs like this around which could all be fixed with a more to GMP. If you really want the patch, then I can finish it off (though I'd like some direction about how to handle the rounding issue), but otherwise I'm going to see how much fun it would be to move to GMP.
Fixed for 2.24.2 and 2.25.2: http://svn.gnome.org/viewvc/gcalctool?view=revision&revision=2310 http://svn.gnome.org/viewvc/gcalctool?view=revision&revision=2311 The calc_u32 and boolean operations are now done by converting the number to a hexadecimal string and performing the operations on each 4 bit nibble. The should have much larger precision now (depends on the string buffer size). This is a stop-gap until they access the MP data directly (if anyone ever notices a performance problem).
*** Bug 569034 has been marked as a duplicate of this bug. ***