After an evaluation, GNOME has moved from Bugzilla to GitLab. Learn more about GitLab.
No new issues can be reported in GNOME Bugzilla anymore.
To report an issue in a GNOME project, go to GNOME GitLab.
Do not go to GNOME Gitlab for: Bluefish, Doxygen, GnuCash, GStreamer, java-gnome, LDTP, NetworkManager, Tomboy.
Bug 526936 - gda_connection_execute_select_command() slow even when using ITER_MODEL_ONLY
gda_connection_execute_select_command() slow even when using ITER_MODEL_ONLY
Status: RESOLVED FIXED
Product: libgda
Classification: Other
Component: Client library
3.0.x
Other Linux
: Normal normal
: ---
Assigned To: malerba
gnome-db Maintainers
Depends on:
Blocks:
 
 
Reported: 2008-04-08 12:53 UTC by Murray Cumming
Modified: 2008-11-17 10:16 UTC
See Also:
GNOME target: ---
GNOME version: ---


Attachments
libgda_test_bigtable.c (1.81 KB, text/plain)
2008-04-08 12:54 UTC, Murray Cumming
Details
postgres_test_bigtable.c (3.38 KB, text/plain)
2008-04-08 12:55 UTC, Murray Cumming
Details
libgda4_test_importtest.c (1.84 KB, text/plain)
2008-04-08 20:36 UTC, Murray Cumming
Details
libgda4_test_bigtable_gettingvalues.c (2.02 KB, text/plain)
2008-04-10 04:14 UTC, Murray Cumming
Details

Description Murray Cumming 2008-04-08 12:53:56 UTC
I'm using a copy of the musicbrainz database on my PostgreSQL server on localhost, just to test this. I'm using the "artist" table, which has almost 600,000 records.

As the attached libgda_test_bigtable.c test case shows, it takes about 1 minute (on my 3Mhz PC with 1.5Gb RAM) to return from gda_connection_execute_select_command() and uses about 300Mb of memory (as seen in System Monitor).

postgres_test_bigtable.c shows that the equivalent (I think) with libpq takes no more than 10 seconds and uses about 35Mb of memory. That's not great but it's much better than libgda.
Comment 1 Murray Cumming 2008-04-08 12:54:41 UTC
Created attachment 108855 [details]
libgda_test_bigtable.c
Comment 2 Murray Cumming 2008-04-08 12:55:15 UTC
Created attachment 108856 [details]
postgres_test_bigtable.c
Comment 3 Murray Cumming 2008-04-08 20:36:41 UTC
Created attachment 108887 [details]
libgda4_test_importtest.c

The same thing with libgda-4.0. I don't know what you did, but both speed and memory seem not much more than when using libpq directly. Well don.
Comment 4 malerba 2008-04-09 07:24:19 UTC
This is strange... In the last attachment, the returned data model is a random access one: the ITER_MODEL_ONLY option does not exist anymore in V4 (this should be in the doc about migration, I'll mention it), you need to use gda_connection_statement_execute_select_full() and pass GDA_STATEMENT_MODEL_CURSOR as the model_usage argument to have a cusron based data model. The params argument of gda_connection_statement_execute_select() is not needed in your case as the statement does not have any variable.

So in that last attachment, libpq transfers all the data from the postgres server (it is all stored in memory), but compared to V3, the data is not converted to GValue values until actually requested. This is why it is so close to what you'd get with libq. Reading the contents of the data model will make memory usage increase.
Comment 5 Murray Cumming 2008-04-10 04:14:32 UTC
Created attachment 108969 [details]
libgda4_test_bigtable_gettingvalues.c

It's wonderfully fast when I get values too, and when I correctly request a cursor/iter-based model.
Comment 6 Murray Cumming 2008-11-17 10:16:29 UTC
Closing because people should now just use libgda-4.0 instead.