The GValue.set/get_value method (used during construction) may be part of this because they have a large dispatch of if/elifs. At one point I started working on a patch to remove this whole dispatch and reuse gi/_gobject/pygtype.c:pyg_value_from_pyobject because does the same thing and is more complete (I recall setting TYPE_PYOBJECT has problems in the current setup). I will work on it again at some point because it will be needed in order to fully deprecate the current ability to use PyObjects as gpointer arguments (https://bugzilla.gnome.org/show_bug.cgi?id=688081)

In the mean time we can get almost a 2x performance improvement for items further in the dispatch like TYPE_STRING by moving the self.g_type into a local. e.g.

gtype = self.g_type
if gtype == TYPE_BOOLEAN:
    ...

In [1]: from gi.repository import GObject, Gtk
In [2]: st = Gtk.ListStore(*[str]*5)
In [3]: %timeit -n 10000 st.append(['Aaaaaaa', 'Bbbbbbb', 'Ccccccc', 'Ddddddd', 'Eeeeeee'])
10000 loops, best of 3: 344 us per loop

Moving self.g_type into a local:

In [3]: %timeit -n 10000 st.append(['Aaaaaaa', 'Bbbbbbb', 'Ccccccc', 'Ddddddd', 'Eeeeeee'])
10000 loops, best of 3: 188 us per loop

Created attachment 237594 [details] [review]
Set GValue.g_type to a local in get_value/set_value dispatch

This increases performance by a factor of 2x for types later
in the dispatch.

Comment on attachment 237594 [details] [review]
Set GValue.g_type to a local in get_value/set_value dispatch

This is a good first improvement, please push.

For the record, I was going to try and avoid the Value building in python for all gtypes <= 255 (FUNDAMENTAL_MAX), to leave the GValue building to the C part for the common case, and only use the GObject.Value override for derived classes.

Also, this code would be much more elegant with a static gtype -> method dictionary, and perhaps faster too?

I pushed a new performance test case: http://git.gnome.org/browse/pygobject/commit/?id=91f76dd9

Before Simon's patch my machine took 255 µs per append operation, with Simon's patch it's 185 µs/append.

> Also, this code would be much more elegant with a static gtype -> method
dictionary, and perhaps faster too?

I tried this, and it's not faster.

Created attachment 237601 [details] [review]
Gtk.TreeModel insertion speedup (broken)

This is what I meant with not building Python GObject.Value objects for the common cases. However, it causes a regression as with this we don't throw proper ValueErrors on invalid values any more, so I don't push that for now.

Created attachment 250090 [details] [review]
Replace GValue get/set dispatching overrides with C dispatching

Based on the refactoring and fixing of boxed types in bug #688081,
this patch simplifies the overrides by using existing C code.
This gives a performance improvement of about %15 for appending
GValues to ListStores.

Before:
In [1]: from gi.repository import GObject, Gtk
In [2]: st = Gtk.ListStore(*[str]*5)
In [3]: %timeit -n 10000 st.append(['Aaaaaaa', 'Bbbbbbb', 'Ccccccc', 'Ddddddd', 'Eeeeeee'])
10000 loops, best of 3: 178 us per loop

After:
In [3]: %timeit -n 10000 st.append(['Aaaaaaa', 'Bbbbbbb', 'Ccccccc', 'Ddddddd', 'Eeeeeee'])
10000 loops, best of 3: 133 us per loop

Adding bug #688081 as a dependency. This does not require the deprecation but the patch exposing of _gobject._gvalue_get/set to Python.

Comment on attachment 250090 [details] [review]
Replace GValue get/set dispatching overrides with C dispatching

I'm glad that this ugly piece of python overrides will be gone, thanks!

Review of attachment 250090 [details] [review]:

This actually causes a few exceptions because the C dispatcher is a bit more strict.

python gi 3.12 

I realized the gi liststore in my software was much slower than my gtk2 app that I was porting to gi, then I found this bug report here.

this bug still exists.

I compared with gtk2. 
If I loop for 30000, gtk2 only 0.5s and gi takes 8s in the example above.

I also find the length of the row affects performance much more in gi than gtk2

Is there a fix?

Try using Gtk.ListStore.insert_with_valuesv directly:

    st = Gtk.ListStore(*[str]*5)
    cols = [0, 1, 2, 3, 4]
    for r in range(10000):
        st.insert_with_valuesv(-1, cols, ['Aaaaaaa', 'Bbbbbbb', 'Ccccccc', 'Ddddddd', 'Eeeeeee'])

yes this is fast, is this a workaround? as liststore append cannot be used this way

some of these rows I get error using insert_with_valuesv

Unable to convert from void to gchararray

what does this mean? rows that contains str

Unable to convert from PyObject to gchararray

Yes, it's a workaround. It assumes that the default gtype assumed by PyGObject for the passed in values matches the list store column type.

In case that doesn't work (you don't pass in str values, but values which provide a __str__ or are None) you have to create the GValue yourself (this works since PyGObject 3.12):

    st = Gtk.ListStore(*[GObject.TYPE_STRING]*5)
    values = []
    for _ in xrange(5):
        value = GObject.Value()
        value.init(GObject.TYPE_STRING)
        values.append(value)
    cols = [0, 1, 2, 3, 4]
    for r in xrange(10000):
        for v, n in zip(values, ['Aaaaaaa', 'Bbbbbbb', 'Ccccccc', 'Ddddddd', 'Eeeeeee']):
            v.set_boxed(n)
        st.insert_with_valuesv(-1, cols, values)

If you need more speed I recommend using only one column with the type "object", create your own Python row type and use cellrenderer functions to extract the values when needed.

    Entry = namedtuple("Entry", ["a", "b", "c"])
    st = Gtk.ListStore(object)
    cols = [0]
    for r in xrange(10000):
        value = Entry("foo", "bar", "quux")
        st.insert_with_valuesv(-1, cols, [value])

I found out why there was a type conversion problem

when data rows returned from python sqlite that contains value None, insert_valuesv does convert to int 0 or empty str.
the liststore append does this automatically. 

more code need to process all the rows manually convert all None to appropriate types.

without using liststore append become so much mode code.

extra problem 
integers from sqlite become negative when using insert_with_valuesv

-- GitLab Migration Automatic Message --

This bug has been migrated to GNOME's GitLab instance and has been closed from further activity.

You can subscribe and participate further through the new bug through this link to our GitLab instance: https://gitlab.gnome.org/GNOME/pygobject/issues/46.