After an evaluation, GNOME has moved from Bugzilla to GitLab. Learn more about GitLab.
No new issues can be reported in GNOME Bugzilla anymore.
To report an issue in a GNOME project, go to GNOME GitLab.
Do not go to GNOME Gitlab for: Bluefish, Doxygen, GnuCash, GStreamer, java-gnome, LDTP, NetworkManager, Tomboy.
Bug 509988 - "unsigned integer value" functions fail for large numbers
"unsigned integer value" functions fail for large numbers
Status: RESOLVED FIXED
Product: gnome-calculator
Classification: Core
Component: general
5.20.x
Other Linux
: High normal
: 2.24.0
Assigned To: Robert Ancell
gcalctool maintainers
: 569034 (view as bug list)
Depends on:
Blocks:
 
 
Reported: 2008-01-16 20:02 UTC by Pedro Villavicencio
Modified: 2009-01-26 11:33 UTC
See Also:
GNOME target: ---
GNOME version: 2.19/2.20



Description Pedro Villavicencio 2008-01-16 20:02:58 UTC
This report has been filled here:

https://bugs.launchpad.net/ubuntu/+source/gcalctool/+bug/181763

"The "&16" and "&32" functions both fail for large inputs:

u16(11111111111111111111)
u32(11111111111111111111)

Both evaluate to 0 (and their least significant bit is 0)

But doing the "near" equivalent (which stuffs up for negatives and behaves differently for non-integer inputs) works:

11111111111111111111 Mod 65536 = 29127
11111111111111111111 Mod 4294967291 = 3098111297"
Comment 1 Rich Burridge 2008-01-16 20:40:48 UTC
Yup, it indeed appears to be a bug.
I noticed that u16(1111111111111111111)
and u32(1111111111111111111) work though. 
I.e. 19 1's rather than 20.
Comment 2 andy-bugreport 2008-01-17 13:23:21 UTC
The functions both work by converting the number to a double, and then to an integer. With the larger numbers, the double runs out of precision.

I plan on making a patch when I have the time (hopefully within a month)
Comment 3 Rich Burridge 2008-01-17 15:42:57 UTC
This would be calc_u32() and calc_u16() in mpmath.c
Yeah, anywhere in the gcalctool code that does a 
conversion to a double first for something like this, 
is broken.

Thanks.


Comment 4 Andy Owen 2008-03-07 04:01:57 UTC
So I've gone a bit sour on the idea of patching it, mainly because I think that #505608 is a better way of fixing it (switching to GMP for maths). 

One of the test cases which I was using was u32(2^32), and it exposed another bug (enter "1/(2^32-4294967296)" (or even "1/(2^1-2)") and note the lack of divide by zero - 2^32 returns a number slightly less than 2^32). So I'm not sure if the correct way to fix it is to round the numbers to integers first (currently they are truncated), do a dodgy hack, or fix the other bug. 

I'm less enthused about writing a power function in the same auto-converted fortran style of C, given that there are probably heaps of other bugs like this around which could all be fixed with a more to GMP.

If you really want the patch, then I can finish it off (though I'd like some direction about how to handle the rounding issue), but otherwise I'm going to see how much fun it would be to move to GMP.
Comment 5 Robert Ancell 2008-11-23 09:58:14 UTC
Fixed for 2.24.2 and 2.25.2:
http://svn.gnome.org/viewvc/gcalctool?view=revision&revision=2310
http://svn.gnome.org/viewvc/gcalctool?view=revision&revision=2311

The calc_u32 and boolean operations are now done by converting the number to a hexadecimal string and performing the operations on each 4 bit nibble. The should have much larger precision now (depends on the string buffer size). This is a stop-gap until they access the MP data directly (if anyone ever notices a performance problem).
Comment 6 Robert Ancell 2009-01-26 11:33:08 UTC
*** Bug 569034 has been marked as a duplicate of this bug. ***