After an evaluation, GNOME has moved from Bugzilla to GitLab. Learn more about GitLab.
No new issues can be reported in GNOME Bugzilla anymore.
To report an issue in a GNOME project, go to GNOME GitLab.
Do not go to GNOME Gitlab for: Bluefish, Doxygen, GnuCash, GStreamer, java-gnome, LDTP, NetworkManager, Tomboy.
Bug 694857 - Gtk.TreeModel._convert_value() has a huge performance cost
Gtk.TreeModel._convert_value() has a huge performance cost
Status: RESOLVED OBSOLETE
Product: pygobject
Classification: Bindings
Component: general
Git master
Other Linux
: Normal minor
: ---
Assigned To: Nobody's working on this now (help wanted and appreciated)
Python bindings maintainers
Depends on: 688081
Blocks:
 
 
Reported: 2013-02-28 11:36 UTC by Martin Pitt
Modified: 2018-01-10 20:23 UTC
See Also:
GNOME target: ---
GNOME version: ---


Attachments
Set GValue.g_type to a local in get_value/set_value dispatch (6.70 KB, patch)
2013-02-28 12:34 UTC, Simon Feltman
committed Details | Review
Gtk.TreeModel insertion speedup (broken) (1.48 KB, patch)
2013-02-28 14:51 UTC, Martin Pitt
rejected Details | Review
Replace GValue get/set dispatching overrides with C dispatching (5.17 KB, patch)
2013-07-25 03:13 UTC, Simon Feltman
needs-work Details | Review

Description Martin Pitt 2013-02-28 11:36:26 UTC
While debugging bug 693402 I noticed that our _convert_value() magic for List/TreeStore has a rather large performance penalty.

Doing this:

    st = Gtk.ListStore(*[str]*5)
    for r in range(10000):
        st.append(['Aaaaaaa', 'Bbbbbbb', 'Ccccccc', 'Ddddddd', 'Eeeeeee'])

takes about 5 seconds on my machine. When short-circuiting _convert_value() to just return the input value, it takes about half a second only.

As pygobject marshals native types into GValues itself, and much more efficiently, we should only construct GObject.Values from Python in the corner cases where it matters, e. g. for derived int types.
Comment 1 Simon Feltman 2013-02-28 12:25:15 UTC
The GValue.set/get_value method (used during construction) may be part of this because they have a large dispatch of if/elifs. At one point I started working on a patch to remove this whole dispatch and reuse gi/_gobject/pygtype.c:pyg_value_from_pyobject because does the same thing and is more complete (I recall setting TYPE_PYOBJECT has problems in the current setup). I will work on it again at some point because it will be needed in order to fully deprecate the current ability to use PyObjects as gpointer arguments (https://bugzilla.gnome.org/show_bug.cgi?id=688081)

In the mean time we can get almost a 2x performance improvement for items further in the dispatch like TYPE_STRING by moving the self.g_type into a local. e.g.

gtype = self.g_type
if gtype == TYPE_BOOLEAN:
    ...

In [1]: from gi.repository import GObject, Gtk
In [2]: st = Gtk.ListStore(*[str]*5)
In [3]: %timeit -n 10000 st.append(['Aaaaaaa', 'Bbbbbbb', 'Ccccccc', 'Ddddddd', 'Eeeeeee'])
10000 loops, best of 3: 344 us per loop

Moving self.g_type into a local:

In [3]: %timeit -n 10000 st.append(['Aaaaaaa', 'Bbbbbbb', 'Ccccccc', 'Ddddddd', 'Eeeeeee'])
10000 loops, best of 3: 188 us per loop
Comment 2 Simon Feltman 2013-02-28 12:34:58 UTC
Created attachment 237594 [details] [review]
Set GValue.g_type to a local in get_value/set_value dispatch

This increases performance by a factor of 2x for types later
in the dispatch.
Comment 3 Martin Pitt 2013-02-28 13:50:07 UTC
Comment on attachment 237594 [details] [review]
Set GValue.g_type to a local in get_value/set_value dispatch

This is a good first improvement, please push.

For the record, I was going to try and avoid the Value building in python for all gtypes <= 255 (FUNDAMENTAL_MAX), to leave the GValue building to the C part for the common case, and only use the GObject.Value override for derived classes.

Also, this code would be much more elegant with a static gtype -> method dictionary, and perhaps faster too?
Comment 4 Martin Pitt 2013-02-28 14:11:25 UTC
I pushed a new performance test case: http://git.gnome.org/browse/pygobject/commit/?id=91f76dd9

Before Simon's patch my machine took 255 µs per append operation, with Simon's patch it's 185 µs/append.
Comment 5 Martin Pitt 2013-02-28 14:23:51 UTC
> Also, this code would be much more elegant with a static gtype -> method
dictionary, and perhaps faster too?

I tried this, and it's not faster.
Comment 6 Martin Pitt 2013-02-28 14:51:11 UTC
Created attachment 237601 [details] [review]
Gtk.TreeModel insertion speedup (broken)

This is what I meant with not building Python GObject.Value objects for the common cases. However, it causes a regression as with this we don't throw proper ValueErrors on invalid values any more, so I don't push that for now.
Comment 7 Simon Feltman 2013-07-25 03:13:20 UTC
Created attachment 250090 [details] [review]
Replace GValue get/set dispatching overrides with C dispatching

Based on the refactoring and fixing of boxed types in bug #688081,
this patch simplifies the overrides by using existing C code.
This gives a performance improvement of about %15 for appending
GValues to ListStores.

Before:
In [1]: from gi.repository import GObject, Gtk
In [2]: st = Gtk.ListStore(*[str]*5)
In [3]: %timeit -n 10000 st.append(['Aaaaaaa', 'Bbbbbbb', 'Ccccccc', 'Ddddddd', 'Eeeeeee'])
10000 loops, best of 3: 178 us per loop

After:
In [3]: %timeit -n 10000 st.append(['Aaaaaaa', 'Bbbbbbb', 'Ccccccc', 'Ddddddd', 'Eeeeeee'])
10000 loops, best of 3: 133 us per loop
Comment 8 Simon Feltman 2013-07-25 03:15:09 UTC
Adding bug #688081 as a dependency. This does not require the deprecation but the patch exposing of _gobject._gvalue_get/set to Python.
Comment 9 Martin Pitt 2013-07-25 13:11:13 UTC
Comment on attachment 250090 [details] [review]
Replace GValue get/set dispatching overrides with C dispatching

I'm glad that this ugly piece of python overrides will be gone, thanks!
Comment 10 Simon Feltman 2013-07-29 20:06:46 UTC
Review of attachment 250090 [details] [review]:

This actually causes a few exceptions because the C dispatcher is a bit more strict.
Comment 11 wonghow 2015-08-07 04:45:53 UTC
python gi 3.12 

I realized the gi liststore in my software was much slower than my gtk2 app that I was porting to gi, then I found this bug report here.

this bug still exists.

I compared with gtk2. 
If I loop for 30000, gtk2 only 0.5s and gi takes 8s in the example above.

I also find the length of the row affects performance much more in gi than gtk2

Is there a fix?
Comment 12 Christoph Reiter (lazka) 2015-08-07 06:05:27 UTC
Try using Gtk.ListStore.insert_with_valuesv directly:

    st = Gtk.ListStore(*[str]*5)
    cols = [0, 1, 2, 3, 4]
    for r in range(10000):
        st.insert_with_valuesv(-1, cols, ['Aaaaaaa', 'Bbbbbbb', 'Ccccccc', 'Ddddddd', 'Eeeeeee'])
Comment 13 wonghow 2015-08-07 06:20:01 UTC
yes this is fast, is this a workaround? as liststore append cannot be used this way
Comment 14 wonghow 2015-08-07 06:34:11 UTC
some of these rows I get error using insert_with_valuesv

Unable to convert from void to gchararray

what does this mean? rows that contains str
Comment 15 wonghow 2015-08-07 06:34:44 UTC
Unable to convert from PyObject to gchararray
Comment 16 Christoph Reiter (lazka) 2015-08-07 06:56:44 UTC
Yes, it's a workaround. It assumes that the default gtype assumed by PyGObject for the passed in values matches the list store column type.

In case that doesn't work (you don't pass in str values, but values which provide a __str__ or are None) you have to create the GValue yourself (this works since PyGObject 3.12):

    st = Gtk.ListStore(*[GObject.TYPE_STRING]*5)
    values = []
    for _ in xrange(5):
        value = GObject.Value()
        value.init(GObject.TYPE_STRING)
        values.append(value)
    cols = [0, 1, 2, 3, 4]
    for r in xrange(10000):
        for v, n in zip(values, ['Aaaaaaa', 'Bbbbbbb', 'Ccccccc', 'Ddddddd', 'Eeeeeee']):
            v.set_boxed(n)
        st.insert_with_valuesv(-1, cols, values)

If you need more speed I recommend using only one column with the type "object", create your own Python row type and use cellrenderer functions to extract the values when needed.

    Entry = namedtuple("Entry", ["a", "b", "c"])
    st = Gtk.ListStore(object)
    cols = [0]
    for r in xrange(10000):
        value = Entry("foo", "bar", "quux")
        st.insert_with_valuesv(-1, cols, [value])
Comment 17 wonghow 2015-08-07 10:53:20 UTC
I found out why there was a type conversion problem

when data rows returned from python sqlite that contains value None, insert_valuesv does convert to int 0 or empty str.
the liststore append does this automatically. 

more code need to process all the rows manually convert all None to appropriate types.

without using liststore append become so much mode code.
Comment 18 wonghow 2015-08-11 00:20:20 UTC
extra problem 
integers from sqlite become negative when using insert_with_valuesv
Comment 19 GNOME Infrastructure Team 2018-01-10 20:23:20 UTC
-- GitLab Migration Automatic Message --

This bug has been migrated to GNOME's GitLab instance and has been closed from further activity.

You can subscribe and participate further through the new bug through this link to our GitLab instance: https://gitlab.gnome.org/GNOME/pygobject/issues/46.