After an evaluation, GNOME has moved from Bugzilla to GitLab. Learn more about GitLab.
No new issues can be reported in GNOME Bugzilla anymore.
To report an issue in a GNOME project, go to GNOME GitLab.
Do not go to GNOME Gitlab for: Bluefish, Doxygen, GnuCash, GStreamer, java-gnome, LDTP, NetworkManager, Tomboy.
Bug 526406 - rounding errors
rounding errors
Status: RESOLVED DUPLICATE of bug 515297
Product: Gnumeric
Classification: Applications
Component: Main System
1.7.x
Other All
: Normal normal
: ---
Assigned To: Jody Goldberg
Jody Goldberg
Depends on:
Blocks:
 
 
Reported: 2008-04-05 21:37 UTC by Graham Cranston
Modified: 2008-04-06 00:16 UTC
See Also:
GNOME target: ---
GNOME version: ---


Description Graham Cranston 2008-04-05 21:37:36 UTC 
Gnumeric is exhibiting rounding errors on long fills and repeated formulae.  For example, entering 0, then 0.01 in the first two rows of a column, then filling down a large number of rows eventually displays 5.3499999999999.  About every 20 rows, another digit is lost (i.e. 5.5499999999998).  This also occurs using a starting value in say A1, then filling down the formula =A1+0.01 to try to create the same x axis of values.
Comment 1 Morten Welinder 2008-04-06 00:16:56 UTC 
This is a fundamental difference between real numbers, see
http://en.wikipedia.org/wiki/Real_number, and finite precision numbers
such as http://en.wikipedia.org/wiki/IEEE_754

Short version: 0.01 is not representable.


*** This bug has been marked as a duplicate of 515297 ***