After an evaluation, GNOME has moved from Bugzilla to GitLab. Learn more about GitLab.
No new issues can be reported in GNOME Bugzilla anymore.
To report an issue in a GNOME project, go to GNOME GitLab.
Do not go to GNOME Gitlab for: Bluefish, Doxygen, GnuCash, GStreamer, java-gnome, LDTP, NetworkManager, Tomboy.
Bug 671863 - Cannot open a big file
Cannot open a big file
Status: RESOLVED FIXED
Product: Gnumeric
Classification: Applications
Component: Charting
1.10.x
Other Windows
: Normal normal
: ---
Assigned To: Jean Bréfort
Jody Goldberg
Depends on:
Blocks:
 
 
Reported: 2012-03-12 02:42 UTC by Lee Li
Modified: 2012-03-14 07:18 UTC
See Also:
GNOME target: ---
GNOME version: ---



Description Lee Li 2012-03-12 02:42:23 UTC
when I open(writing) a big file, stop the opening by Gnumeric. Why?

Can(would) you remote connect to my computer to see what happen when Gnumeric write(open) the big file?
Comment 1 Andreas J. Guelzow 2012-03-12 03:16:56 UTC
Of which file type is your big file?

You wrote "open(writing)", so do you see  a problem both when you read and when you save the file?
Comment 2 Lee Li 2012-03-12 12:14:52 UTC
Hi, Andreas.

Thank you for quick replying.

The type of this big file is ods.

Follow are the errors of opening(reading) this big .ods file by Gnumeric :

errors:
 error even name:      APPCRASH
 App. :        gnumeric.exe
 App. verison: 1.10.16.0
 Date/time:    4dfa9942
 error model's name:   libglib-2.0-0.dll
 錯誤模組版本:       2.26.0.0
 錯誤模組時間戳記:     4ddc4d13
 例外狀況代碼:       40000015
 例外狀況位移:       00044fc3
 作業系統版本:       Win 7 :6.1.7601.2.1.0.768.3
 地區設定識別碼:      1028
 其他資訊 1:       bd69
 其他資訊 2:       bd69c66909ea6759726eb4cc95e17554
 其他資訊 3:       303d
 其他資訊 4:       303db9a9cd9995d360de3c8903cd1126


This big .ods file come from a small .ods file,  Gnumeric can opens(reads)the small file is okay, but open the big file is always crash.


The big .ods file is too big to attach.. so I'd have emailed you a email with the big .ods file. pls. view and check it.


Thanks.
Comment 3 Andreas J. Guelzow 2012-03-12 14:03:58 UTC
Hi,

I can open the large file just fine. It takes a while but the progress is appropriately shown. The file, when loaded in memory, requires nearly 2GB of memory. During loading not significantly more memory seems to be used.

Your error information indicates that the problem occurs in glib, this lends credence to the guess that you may simply run out of memory.

Note that I was using 1.11.2 and 1.10.17 while you are using 1.10.16. (There is no Windows version for 1.11.2).
Comment 4 Lee Li 2012-03-12 14:25:30 UTC
Andreas,

Thank you for your helping testing the trouble file and explaining the errors in detail.

I'd have the point as same as you, maybe this big file eat up all my computer resource..... RAM or CPU, it's 16.5 M big file.

By the way, what time will I have the latest version that can run in Windows( win7), and any big file make the same errors can be saved by using version 1.11.2 Gnumeric in Windows? 

Thank you again!
Comment 5 Andreas J. Guelzow 2012-03-12 15:04:31 UTC
Gnumeric 1.11.2 depends on gtk3. As far as I know so far there is no viable port of that to MSWindows. There is no specific Gnumeric development on MSWindows, so a Windows version should become available whenever teh underlying libraries are ported.

I don't think there is any significant change between these versions fo rthis kind of file. You are just running out of memory...
Comment 6 Morten Welinder 2012-03-12 18:39:53 UTC
How many cells are there in the file?

There used to be an inefficiency in glib that caused hashes to use more
memory than necessary.  See bugs 644197 and 644437.
Comment 7 Andreas J. Guelzow 2012-03-12 22:10:16 UTC
Morten, there don't seem to be that many cells. I tried to send you the file but got:

A message that you sent could not be delivered to one or more of its
recipients. This is a permanent error. The following address(es) failed:

  mortenw@gnome.org
    SMTP error from remote mail server after MAIL FROM:<aguelzow@pyrshep.ca> SIZE=23232320:
    host barmail1.idig.net [64.34.111.254]: 552 Message size exceeds fixed limit

------ This is a copy of the message, including all the headers. ------
------ The body of the message is 22932635 characters long; only the first
------ 106496 or so are included here.
Comment 8 Lee Li 2012-03-13 00:51:41 UTC
Thank Andreas, Morten for checking and understanding this big and trouble file(size: 16.57M).


Just for more your information:

My computer system :CPU(inteli3-2310M.10GHz),RAM(4GB),Win7(32).


Would you like to tell me next step how to fix the error of opening(reading) this big and trouble file by Gnumeric?
Comment 9 Lee Li 2012-03-13 02:52:10 UTC
(In reply to comment #6)
> How many cells are there in the file?
> 
> There used to be an inefficiency in glib that caused hashes to use more
> memory than necessary.  See bugs 644197 and 644437.

Morten,

I've sent Andreas an email with the file.

This big and trouble .ods file(均線統計Gn.ods) can be downloaded from:

https://docs.google.com/open?id=0B1r2GzCRhZEhUU94Wk41RjhRNGFtNjlYdnZ3Um1Jdw


Looking forward to hearing your advice for fix the errors of opening(reading) this big and trouble .ods file.
Comment 10 Andreas J. Guelzow 2012-03-13 06:31:55 UTC
Looking closely at the loaded file I see about 1.65 Million cells. So its obviously bigger than I thought.
Comment 11 Lee Li 2012-03-13 07:24:33 UTC
(In reply to comment #10)
> Looking closely at the loaded file I see about 1.65 Million cells. So its
> obviously bigger than I thought.

Yes, the trouble  file is so big that over Gnumeric's ability to open(read) it okay.

Andreas, I can't open the big file now. for resizing(reduce size) all sheets, can you help delting all blank cells in the bottom of all sheets in the big file.

And then sendback me the big file with all resized(reduce size) sheets.
Comment 12 Andreas J. Guelzow 2012-03-13 15:43:11 UTC
Lee, deleting blank cells will not change the memory used: we do not allocate memory for blank cell location. I'll see what I can do to make the sheet a little less memory intensive.
Comment 13 Morten Welinder 2012-03-13 17:29:47 UTC
What we have here is a serious flaw in the ODF format.  It was rushed out
by the text processing people and never vetted by spreadsheet people.
Specifically the file format does not allow for shared expressions, so
if we look at this file we see that it's almost 700M underneath:

# unzip -v ~/Download/b671863.ods
Archive:  /home/welinder/Download/b671863.ods
 Length   Method    Size  Cmpr    Date    Time   CRC-32   Name
--------  ------  ------- ---- ---------- ----- --------  ----
      46  Stored       46   0% 03-12-2012 07:08 8a396c85  mimetype
695574597  Defl:N 16971416  98% 03-12-2012 07:08 c0e1853b  content.xml
    5742  Defl:N     1194  79% 03-12-2012 07:13 7a710a42  styles.xml
    1126  Defl:N      409  64% 03-12-2012 07:13 d8c3b944  meta.xml
    3374  Defl:N      698  79% 03-12-2012 07:13 6922acd0  settings.xml
     642  Defl:N      211  67% 03-12-2012 07:13 ff4c2973  META-INF/manifest.xml
--------          -------  ---                            -------
695585527         16973974  98%                            6 files

The way Gnumeric currently works is to load the spreadsheet and then go
hunting for cells that can be shared.  That too late.  I have a patch that
does the sharing also during ODF load and with that patch we store just
43 different expressions instead of 1022130.
Comment 14 Morten Welinder 2012-03-13 17:33:19 UTC
This problem has been fixed in the development version. The fix will be available in the next major software release. Thank you for your bug report.
Comment 15 Morten Welinder 2012-03-13 17:45:02 UTC
Just for reference.  The corresponding .gnumeric file is 8M based on
~150M of xml.

# ./ssconvert ~/Download/b671863.ods /tmp/b671863.gnumeric
# ls -l /tmp/b671863.gnumeric
-rw-r--r-- 1 welinder research 8713126 Mar 13 13:38 /tmp/b671863.gnumeric
# gzip --list /tmp/b671863.gnumeric
         compressed        uncompressed  ratio uncompressed_name
            8713126           155657785  94.4% /tmp/b671863.gnumeric
Comment 16 Lee Li 2012-03-13 20:28:36 UTC
(In reply to comment #12)
> Lee, deleting blank cells will not change the memory used: we do not allocate
> memory for blank cell location. I'll see what I can do to make the sheet a
> little less memory intensive.

Morten,

I'm sorry, it's NOT to delete blank cells, but to Resize(adjust) every sheet(operation: mouse right click every sheet to select Resize, and then adjust the number of column and row. ) and then save the resized file.

Does the resizing can reduce the memory of the big file?
Comment 17 Andreas J. Guelzow 2012-03-13 20:37:45 UTC
Lee, resizing the sheet. deleting cells etc. will have little effect. Tonight (my time) I will be able to create a .gnumeric file for you that you should be able to load to access the file content.
Comment 18 Lee Li 2012-03-13 20:44:43 UTC
(In reply to comment #17)
> Lee, resizing the sheet. deleting cells etc. will have little effect. Tonight
> (my time) I will be able to create a .gnumeric file for you that you should be
> able to load to access the file content.

Andreas, thank you very much, and good work.
Comment 19 Lee Li 2012-03-13 20:47:44 UTC
(In reply to comment #14)
> This problem has been fixed in the development version. The fix will be
> available in the next major software release. Thank you for your bug report.

Morten,

Really? It's a good news.

I'd can't wait to see and use your newest version Gnumeric.

May the Gnumeric be a world-class spreadsheet software in the world in short time.
Comment 20 Andreas J. Guelzow 2012-03-14 05:16:55 UTC
Morten,

do we need this console output:?

Expressions in: 1022130
Expressions stored: 43
Expressions killed: 1022087
Comment 21 Lee Li 2012-03-14 07:18:00 UTC
(In reply to comment #17)
> Lee, resizing the sheet. deleting cells etc. will have little effect. Tonight
> (my time) I will be able to create a .gnumeric file for you that you should be
> able to load to access the file content.

Andreas,

Yes, your hope does come true, I've got your email, it'so great that I can download it by firefox in 3 seconds,and then can open(read)it by Gnumeric in 5 seconds.

How can you do reducing 17 MB to 8.3MB! and let me open it in seconds , it's a great job.


I'd believe that Gnumeric will become a world-class spreadsheet software in a short time, because your nice and good helping.

Thank you for your nice and good helping.

Now, I want to use Gnumeric's functions to programming my spreadsheet.

You are good job! 

Thank you again.