After an evaluation, GNOME has moved from Bugzilla to GitLab. Learn more about GitLab.
No new issues can be reported in GNOME Bugzilla anymore.
To report an issue in a GNOME project, go to GNOME GitLab.
Do not go to GNOME Gitlab for: Bluefish, Doxygen, GnuCash, GStreamer, java-gnome, LDTP, NetworkManager, Tomboy.
Bug 660564 - non-zero stdev with equal data
non-zero stdev with equal data
Status: RESOLVED FIXED
Product: libgoffice
Classification: Other
Component: General
0.8.x
Other All
: Normal trivial
: ---
Assigned To: Jody Goldberg
Jody Goldberg
Depends on:
Blocks:
 
 
Reported: 2011-09-30 13:18 UTC by Kevin Woerner
Modified: 2011-09-30 19:18 UTC
See Also:
GNOME target: ---
GNOME version: ---

Attachments
Tentative patch (2.40 KB, patch)
2011-09-30 15:34 UTC, Morten Welinder 		accepted-commit_now Details | Review

Description Kevin Woerner 2011-09-30 13:18:28 UTC 
The computed standard deviation of 3415 cells each of which contain 0.3 is non-zero (2.71090237130979E-020).  This does not happen if all the cells are 0.1, 0.2, 0.4, 0.5, 0.7, or 0.8.  If the 3415 cells are all 0.6 then the standard deviation is computed to be 5.42180474261958E-020.  When 2277 (==ceil(3415*2/3)) cells are 0.9, the computed standard deviation is 5.42220163912776E-020.  I'm using Gnumeric v1.10.16 on a Windows XP OS.
Comment 1 Andreas J. Guelzow 2011-09-30 14:28:56 UTC 
I don't really see a problem with that. We know that the standard deviation should be 0 but it shoudl be clear that there will be some precision error when calculating the the standard deviation. Compared with the size of the data values,  2.71090237130979E-020 is neglegible.

For comparison, with 3415 data values of 0.3, LibreOffice calculates 1.87E-14.
Comment 2 Morten Welinder 2011-09-30 14:43:27 UTC 
We can do better with little effort.  I will fix (in goffice).
Comment 3 Morten Welinder 2011-09-30 15:34:43 UTC 
Created attachment 197888 [details] [review]
Tentative patch

This ought to do it.
Comment 4 Andreas J. Guelzow 2011-09-30 18:00:10 UTC 
Review of attachment 197888 [details] [review]:

looks good
Comment 5 Morten Welinder 2011-09-30 19:18:08 UTC 
This problem has been fixed in the development version. The fix will be available in the next major software release. Thank you for your bug report.