After an evaluation, GNOME has moved from Bugzilla to GitLab. Learn more about GitLab.
No new issues can be reported in GNOME Bugzilla anymore.
To report an issue in a GNOME project, go to GNOME GitLab.
Do not go to GNOME Gitlab for: Bluefish, Doxygen, GnuCash, GStreamer, java-gnome, LDTP, NetworkManager, Tomboy.
Bug 686858 - gnumeric-1.11.6 takes a long time to open and manipulate a million row TSV file
gnumeric-1.11.6 takes a long time to open and manipulate a million row TSV file
Status: RESOLVED FIXED
Product: Gnumeric
Classification: Applications
Component: Main System
1.11.x
Other Linux
: Normal normal
: ---
Assigned To: Jody Goldberg
Jody Goldberg
Depends on:
Blocks:
 
 
Reported: 2012-10-25 10:44 UTC by Shlomi Fish
Modified: 2012-10-27 20:15 UTC
See Also:
GNOME target: ---
GNOME version: ---


Attachments
Tentative patch (547 bytes, patch)
2012-10-26 23:29 UTC, Morten Welinder
none Details | Review

Description Shlomi Fish 2012-10-25 10:44:53 UTC
Hi all,

I'm on x86-64 Mageia Linux 3/Cauldron with gnumeric-1.11.6-1.mga3 on a Core i3 machine with 8 GB of RAM. On that machine (which is very new and fast), gnumeric takes a long time to load and process this file (unpack with "unxz"):

http://www.shlomifish.org/Files/files/code/gnumeric/all_in_a_row.tsv.xz

Opening it takes a while, and then selecting the columns is problematic.

Regards,

-- Shlomi Fish
Comment 1 Andreas J. Guelzow 2012-10-25 18:33:43 UTC
This sounds like you are running out of memory. On my 16GB 32 bit machine it opens fine (it takes less than 1.5 minutes to open).

Since you are running on a 64 bit machine you will likely use twice the memory I am using and depending what else is happening on your machine that could be too much.

Please observe your memory usage while you are opening the file!
Comment 2 Shlomi Fish 2012-10-26 13:08:02 UTC
Hi Andreas,

(In reply to comment #1)
> This sounds like you are running out of memory. On my 16GB 32 bit machine it
> opens fine (it takes less than 1.5 minutes to open).

Well, here it takes something like 2 minutes. Furthermore, afterwards, selecting columns is time-consuming. It seems like too much.

> 
> Since you are running on a 64 bit machine you will likely use twice the memory
> I am using and depending what else is happening on your machine that could be
> too much.

Well, not everything in 64-bit land consumes twice the RAM. gnumeric is currently using 1,034M VIRT 730M RES and 17,672 SHR.

> 
> Please observe your memory usage while you are opening the file!

It consumes 9.2% of my RAM (then grows beyond 10% after I run the histogram tool on column C).

Regards,

-- Shlomi Fish
Comment 3 Morten Welinder 2012-10-26 13:15:28 UTC
It takes about 2 minutes here to open.  Most of that is actually column
resizing.

The problems you observe with selecting columns sound like something
is requesting the clipboard.  With 10^6 cells selected generating the
clipboard contents takes a while.

To check if something is playing games with you, do


    GNM_DEBUG=clipboard gnumeric all_in_a_row.tsv

I get no output when I select a row, but I get something when I press
Ctrl-C.
Comment 4 Morten Welinder 2012-10-26 23:29:06 UTC
Created attachment 227389 [details] [review]
Tentative patch

This brings down load time to ~26s on my old laptop.

This is done by only looking 1000 rows when resizing the columns.
Andreas: how do you like that?
Comment 5 Andreas J. Guelzow 2012-10-26 23:58:22 UTC
We'll probably see a bug report some day complaining that the columns aren't wide enough for some entry in row #10000, but as far as I am concerned looking at teh first 100 rows only should be fine.
Comment 6 Shlomi Fish 2012-10-27 00:10:48 UTC
(In reply to comment #3)
> It takes about 2 minutes here to open.  Most of that is actually column
> resizing.
> 
> The problems you observe with selecting columns sound like something
> is requesting the clipboard.  With 10^6 cells selected generating the
> clipboard contents takes a while.
> 
> To check if something is playing games with you, do
> 
> 
>     GNM_DEBUG=clipboard gnumeric all_in_a_row.tsv
> 
> I get no output when I select a row, but I get something when I press
> Ctrl-C.

With these flags, when I select a column (not a row!) I'm not getting anything, but when I press Ctrl+C, I'm getting an event. (Like you, I guess).

Regards,

-- Shlomi Fish
Comment 7 Andreas J. Guelzow 2012-10-27 02:32:54 UTC
Shlomi, which version of gtk do you have installed and which version of pango? It occurred to me that Mageia Linux 3/Cauldron is a beta of a new distribution and may have some really new libraries...
Comment 8 Shlomi Fish 2012-10-27 10:40:22 UTC
Andreas: here are the installed versions of Pango and Gtk+3:

shlomif@telaviv1:~$ ldd /usr/bin/gnumeric
        linux-vdso.so.1 (0x00007fff01798000)
        libspreadsheet-1.11.6.so => /lib64/libspreadsheet-1.11.6.so (0x00007fdecb0e8000)
        libgoffice-0.10.so.9 => /lib64/libgoffice-0.10.so.9 (0x00007fdecad70000)
        libgtk-3.so.0 => /lib64/libgtk-3.so.0 (0x00007fdeca6bd000)
        libgdk-3.so.0 => /lib64/libgdk-3.so.0 (0x00007fdeca413000)
        libgdk_pixbuf-2.0.so.0 => /lib64/libgdk_pixbuf-2.0.so.0 (0x00007fdeca1f4000)
        libgobject-2.0.so.0 => /lib64/libgobject-2.0.so.0 (0x00007fdec9fa5000)
        libglib-2.0.so.0 => /lib64/libglib-2.0.so.0 (0x00007fdec9caf000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fdec9a93000)
        libc.so.6 => /lib64/libc.so.6 (0x00007fdec96e7000)
        libgsf-1.so.114 => /lib64/libgsf-1.so.114 (0x00007fdec94a5000)
        libxml2.so.2 => /lib64/libxml2.so.2 (0x00007fdec9141000)
        libgmodule-2.0.so.0 => /lib64/libgmodule-2.0.so.0 (0x00007fdec8f3d000)
        libgthread-2.0.so.0 => /lib64/libgthread-2.0.so.0 (0x00007fdec8d3b000)
        libatk-1.0.so.0 => /lib64/libatk-1.0.so.0 (0x00007fdec8b18000)
        libpangocairo-1.0.so.0 => /lib64/libpangocairo-1.0.so.0 (0x00007fdec890b000)
        libpango-1.0.so.0 => /lib64/libpango-1.0.so.0 (0x00007fdec86bf000)
        libcairo.so.2 => /lib64/libcairo.so.2 (0x00007fdec83ad000)
        libm.so.6 => /lib64/libm.so.6 (0x00007fdec80b2000)
        librsvg-2.so.2 => /lib64/librsvg-2.so.2 (0x00007fdec7e7d000)
        libgio-2.0.so.0 => /lib64/libgio-2.0.so.0 (0x00007fdec7b2a000)
        libX11.so.6 => /lib64/libX11.so.6 (0x00007fdec77f0000)
        libXi.so.6 => /lib64/libXi.so.6 (0x00007fdec75e1000)
        libXfixes.so.3 => /lib64/libXfixes.so.3 (0x00007fdec73da000)
        libcairo-gobject.so.2 => /lib64/libcairo-gobject.so.2 (0x00007fdec71d1000)
        libatk-bridge-2.0.so.0 => /lib64/libatk-bridge-2.0.so.0 (0x00007fdec6fa7000)
        libpangoft2-1.0.so.0 => /lib64/libpangoft2-1.0.so.0 (0x00007fdec6d93000)
        libfontconfig.so.1 => /lib64/libfontconfig.so.1 (0x00007fdec6b59000)
        libXinerama.so.1 => /lib64/libXinerama.so.1 (0x00007fdec6956000)
        libXrandr.so.2 => /lib64/libXrandr.so.2 (0x00007fdec674c000)
        libXcursor.so.1 => /lib64/libXcursor.so.1 (0x00007fdec6542000)
        libXcomposite.so.1 => /lib64/libXcomposite.so.1 (0x00007fdec633f000)
        libXdamage.so.1 => /lib64/libXdamage.so.1 (0x00007fdec613c000)
        libXext.so.6 => /lib64/libXext.so.6 (0x00007fdec5f2a000)
        libffi.so.6 => /lib64/libffi.so.6 (0x00007fdec5d22000)
        libpcre.so.0 => /lib64/libpcre.so.0 (0x00007fdec5ae4000)
        librt.so.1 => /lib64/librt.so.1 (0x00007fdec58dc000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fdecb617000)
        libz.so.1 => /lib64/libz.so.1 (0x00007fdec56c4000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007fdec54c0000)
        liblzma.so.5 => /lib64/liblzma.so.5 (0x00007fdec528e000)
        libfreetype.so.6 => /lib64/libfreetype.so.6 (0x00007fdec5001000)
        libpixman-1.so.0 => /lib64/libpixman-1.so.0 (0x00007fdec4d6f000)
        libEGL.so.1 => /lib64/libEGL.so.1 (0x00007fdec4b4d000)
        libpng15.so.15 => /lib64/libpng15.so.15 (0x00007fdec48ff000)
        libxcb-shm.so.0 => /lib64/libxcb-shm.so.0 (0x00007fdec46fc000)
        libxcb-render.so.0 => /lib64/libxcb-render.so.0 (0x00007fdec44f2000)
        libxcb.so.1 => /lib64/libxcb.so.1 (0x00007fdec42d3000)
        libXrender.so.1 => /lib64/libXrender.so.1 (0x00007fdec40c9000)
        libGL.so.1 => /lib64/libGL.so.1 (0x00007fdec3e44000)
        libcroco-0.6.so.3 => /lib64/libcroco-0.6.so.3 (0x00007fdec3c09000)
        libresolv.so.2 => /lib64/libresolv.so.2 (0x00007fdec39f0000)
        libatspi.so.0 => /lib64/libatspi.so.0 (0x00007fdec37cb000)
        libdbus-1.so.3 => /lib64/libdbus-1.so.3 (0x00007fdec3585000)
        libharfbuzz.so.0 => /lib64/libharfbuzz.so.0 (0x00007fdec3321000)
        libX11-xcb.so.1 => /lib64/libX11-xcb.so.1 (0x00007fdec311f000)
        libxcb-dri2.so.0 => /lib64/libxcb-dri2.so.0 (0x00007fdec2f1a000)
        libxcb-xfixes.so.0 => /lib64/libxcb-xfixes.so.0 (0x00007fdec2d13000)
        libwayland-client.so.0 => /lib64/libwayland-client.so.0 (0x00007fdec2b07000)
        libwayland-server.so.0 => /lib64/libwayland-server.so.0 (0x00007fdec28f6000)
        libgbm.so.1 => /lib64/libgbm.so.1 (0x00007fdec26f0000)
        libudev.so.1 => /lib64/libudev.so.1 (0x00007fdec24e0000)
        libdrm.so.2 => /lib64/libdrm.so.2 (0x00007fdec22d4000)
        libXau.so.6 => /lib64/libXau.so.6 (0x00007fdec20d0000)
        libXdmcp.so.6 => /lib64/libXdmcp.so.6 (0x00007fdec1eca000)
        libglapi.so.0 => /lib64/libglapi.so.0 (0x00007fdec1c7a000)
        libxcb-glx.so.0 => /lib64/libxcb-glx.so.0 (0x00007fdec1a62000)
        libXxf86vm.so.1 => /lib64/libXxf86vm.so.1 (0x00007fdec185c000)
        libicule.so.49 => /lib64/libicule.so.49 (0x00007fdec161a000)
        libicuuc.so.49 => /lib64/libicuuc.so.49 (0x00007fdec12a3000)
        libicudata.so.49 => /lib64/libicudata.so.49 (0x00007fdebff83000)
        libkms.so.1 => /lib64/libkms.so.1 (0x00007fdebfd7f000)
        libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007fdebfa7b000)
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fdebf865000)

shlomif@telaviv1:~$ rpm -qf /lib64/libgtk-3.so.0
lib64gtk+3_0-3.6.1-2.mga3
shlomif@telaviv1:~$ 
shlomif@telaviv1:~$ rpm -qf /lib64/libpango-1.0.so.0
lib64pango1.0_0-1.32.1-1.mga3
shlomif@telaviv1:~$
Comment 9 Morten Welinder 2012-10-27 20:15:33 UTC
This problem has been fixed in the development version. The fix will be available in the next major software release. Thank you for your bug report.