GNOME Bugzilla – Bug 686858
gnumeric-1.11.6 takes a long time to open and manipulate a million row TSV file
Last modified: 2012-10-27 20:15:33 UTC
Hi all, I'm on x86-64 Mageia Linux 3/Cauldron with gnumeric-1.11.6-1.mga3 on a Core i3 machine with 8 GB of RAM. On that machine (which is very new and fast), gnumeric takes a long time to load and process this file (unpack with "unxz"): http://www.shlomifish.org/Files/files/code/gnumeric/all_in_a_row.tsv.xz Opening it takes a while, and then selecting the columns is problematic. Regards, -- Shlomi Fish
This sounds like you are running out of memory. On my 16GB 32 bit machine it opens fine (it takes less than 1.5 minutes to open). Since you are running on a 64 bit machine you will likely use twice the memory I am using and depending what else is happening on your machine that could be too much. Please observe your memory usage while you are opening the file!
Hi Andreas, (In reply to comment #1) > This sounds like you are running out of memory. On my 16GB 32 bit machine it > opens fine (it takes less than 1.5 minutes to open). Well, here it takes something like 2 minutes. Furthermore, afterwards, selecting columns is time-consuming. It seems like too much. > > Since you are running on a 64 bit machine you will likely use twice the memory > I am using and depending what else is happening on your machine that could be > too much. Well, not everything in 64-bit land consumes twice the RAM. gnumeric is currently using 1,034M VIRT 730M RES and 17,672 SHR. > > Please observe your memory usage while you are opening the file! It consumes 9.2% of my RAM (then grows beyond 10% after I run the histogram tool on column C). Regards, -- Shlomi Fish
It takes about 2 minutes here to open. Most of that is actually column resizing. The problems you observe with selecting columns sound like something is requesting the clipboard. With 10^6 cells selected generating the clipboard contents takes a while. To check if something is playing games with you, do GNM_DEBUG=clipboard gnumeric all_in_a_row.tsv I get no output when I select a row, but I get something when I press Ctrl-C.
Created attachment 227389 [details] [review] Tentative patch This brings down load time to ~26s on my old laptop. This is done by only looking 1000 rows when resizing the columns. Andreas: how do you like that?
We'll probably see a bug report some day complaining that the columns aren't wide enough for some entry in row #10000, but as far as I am concerned looking at teh first 100 rows only should be fine.
(In reply to comment #3) > It takes about 2 minutes here to open. Most of that is actually column > resizing. > > The problems you observe with selecting columns sound like something > is requesting the clipboard. With 10^6 cells selected generating the > clipboard contents takes a while. > > To check if something is playing games with you, do > > > GNM_DEBUG=clipboard gnumeric all_in_a_row.tsv > > I get no output when I select a row, but I get something when I press > Ctrl-C. With these flags, when I select a column (not a row!) I'm not getting anything, but when I press Ctrl+C, I'm getting an event. (Like you, I guess). Regards, -- Shlomi Fish
Shlomi, which version of gtk do you have installed and which version of pango? It occurred to me that Mageia Linux 3/Cauldron is a beta of a new distribution and may have some really new libraries...
Andreas: here are the installed versions of Pango and Gtk+3: shlomif@telaviv1:~$ ldd /usr/bin/gnumeric linux-vdso.so.1 (0x00007fff01798000) libspreadsheet-1.11.6.so => /lib64/libspreadsheet-1.11.6.so (0x00007fdecb0e8000) libgoffice-0.10.so.9 => /lib64/libgoffice-0.10.so.9 (0x00007fdecad70000) libgtk-3.so.0 => /lib64/libgtk-3.so.0 (0x00007fdeca6bd000) libgdk-3.so.0 => /lib64/libgdk-3.so.0 (0x00007fdeca413000) libgdk_pixbuf-2.0.so.0 => /lib64/libgdk_pixbuf-2.0.so.0 (0x00007fdeca1f4000) libgobject-2.0.so.0 => /lib64/libgobject-2.0.so.0 (0x00007fdec9fa5000) libglib-2.0.so.0 => /lib64/libglib-2.0.so.0 (0x00007fdec9caf000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fdec9a93000) libc.so.6 => /lib64/libc.so.6 (0x00007fdec96e7000) libgsf-1.so.114 => /lib64/libgsf-1.so.114 (0x00007fdec94a5000) libxml2.so.2 => /lib64/libxml2.so.2 (0x00007fdec9141000) libgmodule-2.0.so.0 => /lib64/libgmodule-2.0.so.0 (0x00007fdec8f3d000) libgthread-2.0.so.0 => /lib64/libgthread-2.0.so.0 (0x00007fdec8d3b000) libatk-1.0.so.0 => /lib64/libatk-1.0.so.0 (0x00007fdec8b18000) libpangocairo-1.0.so.0 => /lib64/libpangocairo-1.0.so.0 (0x00007fdec890b000) libpango-1.0.so.0 => /lib64/libpango-1.0.so.0 (0x00007fdec86bf000) libcairo.so.2 => /lib64/libcairo.so.2 (0x00007fdec83ad000) libm.so.6 => /lib64/libm.so.6 (0x00007fdec80b2000) librsvg-2.so.2 => /lib64/librsvg-2.so.2 (0x00007fdec7e7d000) libgio-2.0.so.0 => /lib64/libgio-2.0.so.0 (0x00007fdec7b2a000) libX11.so.6 => /lib64/libX11.so.6 (0x00007fdec77f0000) libXi.so.6 => /lib64/libXi.so.6 (0x00007fdec75e1000) libXfixes.so.3 => /lib64/libXfixes.so.3 (0x00007fdec73da000) libcairo-gobject.so.2 => /lib64/libcairo-gobject.so.2 (0x00007fdec71d1000) libatk-bridge-2.0.so.0 => /lib64/libatk-bridge-2.0.so.0 (0x00007fdec6fa7000) libpangoft2-1.0.so.0 => /lib64/libpangoft2-1.0.so.0 (0x00007fdec6d93000) libfontconfig.so.1 => /lib64/libfontconfig.so.1 (0x00007fdec6b59000) libXinerama.so.1 => /lib64/libXinerama.so.1 (0x00007fdec6956000) libXrandr.so.2 => /lib64/libXrandr.so.2 (0x00007fdec674c000) libXcursor.so.1 => /lib64/libXcursor.so.1 (0x00007fdec6542000) libXcomposite.so.1 => /lib64/libXcomposite.so.1 (0x00007fdec633f000) libXdamage.so.1 => /lib64/libXdamage.so.1 (0x00007fdec613c000) libXext.so.6 => /lib64/libXext.so.6 (0x00007fdec5f2a000) libffi.so.6 => /lib64/libffi.so.6 (0x00007fdec5d22000) libpcre.so.0 => /lib64/libpcre.so.0 (0x00007fdec5ae4000) librt.so.1 => /lib64/librt.so.1 (0x00007fdec58dc000) /lib64/ld-linux-x86-64.so.2 (0x00007fdecb617000) libz.so.1 => /lib64/libz.so.1 (0x00007fdec56c4000) libdl.so.2 => /lib64/libdl.so.2 (0x00007fdec54c0000) liblzma.so.5 => /lib64/liblzma.so.5 (0x00007fdec528e000) libfreetype.so.6 => /lib64/libfreetype.so.6 (0x00007fdec5001000) libpixman-1.so.0 => /lib64/libpixman-1.so.0 (0x00007fdec4d6f000) libEGL.so.1 => /lib64/libEGL.so.1 (0x00007fdec4b4d000) libpng15.so.15 => /lib64/libpng15.so.15 (0x00007fdec48ff000) libxcb-shm.so.0 => /lib64/libxcb-shm.so.0 (0x00007fdec46fc000) libxcb-render.so.0 => /lib64/libxcb-render.so.0 (0x00007fdec44f2000) libxcb.so.1 => /lib64/libxcb.so.1 (0x00007fdec42d3000) libXrender.so.1 => /lib64/libXrender.so.1 (0x00007fdec40c9000) libGL.so.1 => /lib64/libGL.so.1 (0x00007fdec3e44000) libcroco-0.6.so.3 => /lib64/libcroco-0.6.so.3 (0x00007fdec3c09000) libresolv.so.2 => /lib64/libresolv.so.2 (0x00007fdec39f0000) libatspi.so.0 => /lib64/libatspi.so.0 (0x00007fdec37cb000) libdbus-1.so.3 => /lib64/libdbus-1.so.3 (0x00007fdec3585000) libharfbuzz.so.0 => /lib64/libharfbuzz.so.0 (0x00007fdec3321000) libX11-xcb.so.1 => /lib64/libX11-xcb.so.1 (0x00007fdec311f000) libxcb-dri2.so.0 => /lib64/libxcb-dri2.so.0 (0x00007fdec2f1a000) libxcb-xfixes.so.0 => /lib64/libxcb-xfixes.so.0 (0x00007fdec2d13000) libwayland-client.so.0 => /lib64/libwayland-client.so.0 (0x00007fdec2b07000) libwayland-server.so.0 => /lib64/libwayland-server.so.0 (0x00007fdec28f6000) libgbm.so.1 => /lib64/libgbm.so.1 (0x00007fdec26f0000) libudev.so.1 => /lib64/libudev.so.1 (0x00007fdec24e0000) libdrm.so.2 => /lib64/libdrm.so.2 (0x00007fdec22d4000) libXau.so.6 => /lib64/libXau.so.6 (0x00007fdec20d0000) libXdmcp.so.6 => /lib64/libXdmcp.so.6 (0x00007fdec1eca000) libglapi.so.0 => /lib64/libglapi.so.0 (0x00007fdec1c7a000) libxcb-glx.so.0 => /lib64/libxcb-glx.so.0 (0x00007fdec1a62000) libXxf86vm.so.1 => /lib64/libXxf86vm.so.1 (0x00007fdec185c000) libicule.so.49 => /lib64/libicule.so.49 (0x00007fdec161a000) libicuuc.so.49 => /lib64/libicuuc.so.49 (0x00007fdec12a3000) libicudata.so.49 => /lib64/libicudata.so.49 (0x00007fdebff83000) libkms.so.1 => /lib64/libkms.so.1 (0x00007fdebfd7f000) libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007fdebfa7b000) libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fdebf865000) shlomif@telaviv1:~$ rpm -qf /lib64/libgtk-3.so.0 lib64gtk+3_0-3.6.1-2.mga3 shlomif@telaviv1:~$ shlomif@telaviv1:~$ rpm -qf /lib64/libpango-1.0.so.0 lib64pango1.0_0-1.32.1-1.mga3 shlomif@telaviv1:~$
This problem has been fixed in the development version. The fix will be available in the next major software release. Thank you for your bug report.