To be exact, since the methods are annotated with (transfer full), the get_item should make a copy:

gpointer
mylib_container_get_item (MylibContainer *container, int index)
{
   return mylib_container_copy_item (container, container->priv->data[index]);
}

My initial comments on this are:

* It'd be technically more accurate if the C methods returned a "GTypeInstance*", since it is always one.  I know this means you lose the warningless conversion in C to a different type, but honestly there is large swaths of GObject usage that require casts anyways
* All introspection bindings would need special code for this

The further problem is that it doesn't fit into the current introspection GI_TYPE_TAG_BOXED and GI_TYPE_TAG_OBJECT.

Anything we do here is actually going to require a set of changes across the stack; so basically consuming public Gee APIs from introspectable bindings is not going to happen soon.

My recommendation for libraries targeting introspection is to stick with GList * return values.

> * It'd be technically more accurate if the C methods returned a
> "GTypeInstance*", since it is always one.

What about strings, ints and other fundamental types?

(In reply to comment #3)
> > * It'd be technically more accurate if the C methods returned a
> > "GTypeInstance*", since it is always one.
> 
> What about strings, ints and other fundamental types?

I didn't realize one could put those in these generic containers, so then you're right, GTypeInstance wouldn't work.

So my main concern is that trying to implement this would be a lot of typelib and binding work; a new type tag would be required, which is tricky and there's not much space left for new type tags.

Currently in Vala the parameters are passed as triple:

 - GType
 - duplicate function
 - free function

For classes they are passed to constructor and for methods they are just after instance method.

*** Bug 677960 has been marked as a duplicate of this bug. ***

For Comment 5:

Any GType must have a init and finalize function, even a GValue handler to copy/free and is a prerequisite of any GBoxed derived types.

Is really necessary to pass duplicate and free function?

(In reply to comment #7)
> For Comment 5:
> 
> Any GType must have a init and finalize function, even a GValue handler to
> copy/free and is a prerequisite of any GBoxed derived types.
> 
> Is really necessary to pass duplicate and free function?

Not everything people want to use with generics are registered GTypes.

(In reply to comment #8)
> (In reply to comment #7)
> > For Comment 5:
> > 
> > Any GType must have a init and finalize function, even a GValue handler to
> > copy/free and is a prerequisite of any GBoxed derived types.
> > 
> > Is really necessary to pass duplicate and free function?
> 
> Not everything people want to use with generics are registered GTypes.

If we define a GGenericFunc as:

void GGenericFunc (gpointer returnval, GType returntype, GHash *params)


where params use string keys for named parameters and values as its values and paramtypes use string keys for named parameters and values a GType for each type.

Then a Vala func:

string random_name (G data)

can be handle for PyGObject to get all generic params and return values.

*** Bug 669686 has been marked as a duplicate of this bug. ***

This limitation prohibits using libfolks (among others) from JavaScript. As it become official GNOME language it become more severe - are there any plans fixing this bug?

(In reply to comment #9)
> (In reply to comment #8)
> > (In reply to comment #7)
> > > For Comment 5:
> > > 
> > > Any GType must have a init and finalize function, even a GValue handler to
> > > copy/free and is a prerequisite of any GBoxed derived types.
> > > 
> > > Is really necessary to pass duplicate and free function?
> > 
> > Not everything people want to use with generics are registered GTypes.
> 
> If we define a GGenericFunc as:
> 
> void GGenericFunc (gpointer returnval, GType returntype, GHash *params)
> 
> 
> where params use string keys for named parameters and values as its values and
> paramtypes use string keys for named parameters and values a GType for each
> type.
> 
> Then a Vala func:
> 
> string random_name (G data)
> 
> can be handle for PyGObject to get all generic params and return values.

What about generics on class level?

GValue is already supported as a generic container, can handle any GType defined value.

If any generic function use GValue as parameter they could be easy to box and unbox the value.

string random_name (G data);

Could be handle as:

gchar* random_name (GValue data);

This is already done by PyGObject, it automatically pack and unpacks values to GValue structure.

GObject Introspection can handle Generics as GValue (even a GValue can hold a GValue).

(In reply to comment #12)
> GValue is already supported as a generic container, can handle any GType
> defined value.
> 
> If any generic function use GValue as parameter they could be easy to box and
> unbox the value.
> 
> string random_name (G data);
> 
> Could be handle as:
> 
> gchar* random_name (GValue data);
> 
> This is already done by PyGObject, it automatically pack and unpacks values to
> GValue structure.
> 
> GObject Introspection can handle Generics as GValue (even a GValue can hold a
> GValue).

It's a bit more complicated.

 - If the change was handled on user side than it would impose additional code to manually pack/unpack values (in addition to 'normal' code), dealing with casts[1] and it would impose strange interface:

map = ...
it = map.iterator()
while it.next():
    it.set_value_value(1)
it.set_value(2)                 # This lines will crash program without warning
it.set_value(0xdeadbeet) # Or write in random pieces of memory - yay

 - Other option would be to include it in Vala. Since the change would impose (backward compatible) change of ABI it would require the additional attribute to ensure that there is no ABI change (otherwise the ABI would silently depend on version of compiler). Oh - and interface to GIR would still be as complicated as previously.

My feeling is that the better option would be to push it to GIR as:
 - There would be no pre-existing ABI so it wouldn't be broken by accident
 - The automagic casting in a language would work as intended instead of shifting the policy to language (say PHP bindings can cast integer to string freely while python would throw an error or vice versa)

[1] Which might be strange if some languages support automatic conversion from integer to string - such policy would need to be done on per-library basis

I just found out the hard way that libfolks is unusable from JavaScript due to this bug. We use libgee structures as a critical part of our API, so the fact that their elements can't be accessed from JS totally breaks us in that regard.

It sounds like it could be a fair amount of work from the g-i side, but I don't see any other route for us. Is there anything we can do to help out?

For GDA vala extensions and for libfolks, there are a workaround:

Add accessors with defined type.

If you have a class with ArrayList<int> member, when you call GEE method get(),
from iterator or from the class, will thrown errors on any languages accessing
through GObject Introspection (GI), but if you declare your own class as:

class MyArrayList<int> : ArrayList<int>
{
   int get_data() { return get(); }
}

You can use get_data() to get data stored in Gee.ArrayList, but GI will always
know the way to manage returned values from your methods.

Unfortunately, you must define your own methods to replace the Gee ones to
access the data in order to avoid Generics.

(In reply to comment #15)
> For GDA vala extensions and for libfolks, there are a workaround:
> 
> Add accessors with defined type.
> 
> If you have a class with ArrayList<int> member, when you call GEE method get(),
> from iterator or from the class, will thrown errors on any languages accessing
> through GObject Introspection (GI), but if you declare your own class as:
> 
> class MyArrayList<int> : ArrayList<int>
> {
>    int get_data() { return get(); }
> }
> 

Possibly it should be better documented but I'm not sure we consider deriving from classes as stable ABI. Even in case we do it creates long-term (API-long - not even ABI-long) commitment to use MyArrayList - even if there will be proper implementation in place. It also is against the policy of using interfaces instead of classes - you are basically forcing to always use ArrayList even if you find out that LinkedList gives you more performance (or ArrayQueue, or whatever). 

> You can use get_data() to get data stored in Gee.ArrayList, but GI will always
> know the way to manage returned values from your methods.
> 

Still this leaves an issue of "I called collection.add and my application segfault. The stacktrace shows it is in libgee." type of bugs.

I'm tempted to say "don't use libgee as part of your API". It's a mess to use from C already with it's gpointer API. If you write a sane API that a C programmer would be happy to use, introspection will be happy to use it as well.

libgee and Vala's generics is really pushing the limits of C and introspection.

(In reply to comment #17)
> I'm tempted to say "don't use libgee as part of your API". It's a mess to use
> from C already with it's gpointer API. If you write a sane API that a C
> programmer would be happy to use, introspection will be happy to use it as
> well.
> 
> libgee and Vala's generics is really pushing the limits of C and introspection.

Setting aside the unimplemented part I fail to see what is hard or wrong about Vala approach. I fail to see how to implement it differently other then using GValue[1] (and I'm not sure that it would be much easier to use for C programmer). My guess is that it was modelled after existing GLib API to reuse GList, GArray etc.

If you have any proposal for generics you might want to share it (please have in mind however the limitations of GValue I have pointed). [As of C API - my guess is that it is hard to use more HL constructs in C anyway so I don't think it is reasonable target for library targetting more HL languages].

Anyway whatever the ideal solution would be the problem is that there are important libraries which are using libgee. Furthermore I don't think there exists a collection library which is currently introspectable or accessable from Vala so people would use it and there might be more. 

[1] Which would either require Vala to understend the concept of extenal API or deal with packing/unpacking at each funtion. As the Vala is compiled language and there is no JIT it would be an overhead

(In reply to comment #17)
> I'm tempted to say "don't use libgee as part of your API".

OK, but that's not an option for Folks. We aren't planning to break API any time soon.

> It's a mess to use
> from C already with it's gpointer API. If you write a sane API that a C
> programmer would be happy to use, introspection will be happy to use it as
> well.

It's not much harder to use from C than GHashTable if you're mostly reading values out of it (like most Folks clients).

We went out of our way to not expose the generic types too often (which is where most of the pain in C is - when you have to add the GType, copy, and delete parameters to each function).

If we did use generics more extensively, the API would have been simpler in Vala. Assuming this could also be guaranteed in other key introspected language(s) (mostly JavaScript) by the time we get around to Folks 2 (not in the immediate future), I would definitely take advantage of them.

> libgee and Vala's generics is really pushing the limits of C and introspection.

It seems like it's certainly /doable/ in C, even if it means some extra function parameters. The concept doesn't seem too complicated (but I admit I've only dealt with introspection from a few languages, and not in great detail outside of C and Vala).

Possibly it is good to take into account while planning  - it would be nice to add covariance/variance support. While many languages can simply erase this information (transform <? extends T> -> T) others might use.

(In reply to comment #18)
> If you have any proposal for generics you might want to share it (please have
> in mind however the limitations of GValue I have pointed). [As of C API - my
> guess is that it is hard to use more HL constructs in C anyway so I don't think
> it is reasonable target for library targetting more HL languages].
> 

Also - GValues as base of generics would not allow for implementation of lock-free data structures in libgee.

I spoke with Colin during this GUADEC and we agreed that it would be nice to have higher integration between Vala and g-i (in particular regarding this bug).

I believe that the goals would look more or less as:
 - Make the generics just work behind the scene. In other words they should feel native to language and follow its casting rules. So adding int to Gee.List<string> in PHP should work with implicit atoi while it
should throw an error in Python etc.
 - Make the code in Vala as nice as possible without need to add much code. And without implementing the policy for the language (see previous point)
 - Keep the C API of Vala library stable across this change
 - Keep the ABI of Vala library stable across this change

Description of generics in Vala:
 - Currently each Vala generic type can be casted internally to another. So Gee.List<Gtk.Widget> l = new Gee.ArrayList<Gtk.Button>() is legal. However I would like to introduce proper covariance and contravariance (bug #694895 and bug #700142) support so it is no longer supported - however something like Gee.List<? : Gtk.Widget> l = new Gee.ArrayList<Gtk.Button>() would be legal.
 - Currently the generic is implemented as a 3 arguments passed either to function or constructor which describes it. It contains GType, GCopyFunc and GDestroy. As it distiguish between owned and unowned values it is not a 1-1 relationship. There are SimpleGenerics as well which don't pass the arguments.

Proposition  (after doing 5 minute research - probably needs fine-tuning or even complete rewrite)
  To the .gir format:
    - Add the <generic name=name> element to the classes, methods etc. 
    - The generic element can be used as a type in .gir
    - Either 
      - If the interface have prerequisite it should have the generics matching to parent (i.e. specifying which generics argument are assign
to each type)
      - Either move the parent inside class or add the <parent-generics element to the class (former have benefit of being consistent with interfaces while latter might be partially compatible with current .gir format)
      - Add the generic elements to the <type
      Or
      - Extend the types naming by arguments
  To the C API:
    - Extend the g_callable_info_invoke by generic arguments or require binding to push them
    - Allow querying for generics for types so have g_object_info_get_n_generic_arguments etc.
    Or (possibly more work in gir but much less work for bindings - I think no work on bindings side would be required other then calling generic functions and creating generic objects)
    - Add new GIGenericObjectInfo and GIGenericFunctionInfo which allows to query for generics
    - Have something like:
        GIObjectInfo *
        g_generic_object_bind (GIGenericObjectInfo *info,
                               int n_generic_arguments,
                               GITypeInfo *generic_arguments,
                               bool *generic_arguments_owned);
        GIFunctionInfo *
        g_generic_function_bind (GIGenericFunctionInfo *info,
                                 int n_generic_arguments,
                                 GITypeInfo *generic_arguments,
                                 bool *generic_arguments_owned);
    - If an API returns a generic type just substitute correct GIObjectInfo. So for folks Gee.Collection<Folks.Backend> list_backends(); the API for return type returns the GIObjectInfo for Gee.Collection<Folks.Backend>. In other words GIObjectInfo corresponds to concrete types/functions which can be passed around in code while GIGenericObjectInfo refers to 'metaclass'.

Just as an example of .gir:

interface Test<G, H> : Object {
    H get(Gee.Set<G> g, H h);
}
interface Test2<G, H> : Test<G, Test<unowned H>> {}
class Test3<G, H> : Object, Test<G, Test<unowned Test<H>>, Test2<G,
Test<H>> {
    ...
}

Pseudo-GIR:
<interface name="Test" c:type=Test" ...>
    <generic-parameter name="G" />
    <generic-parameter name="H" />
    <prerequisite name="GObject.Object"/>
    <method name="get" c:identifier="test_get">
         <return-value transfer-ownership="full">
             <type name="H" c:type="void*"/>
         </return-value>
         <parameters>
             <parameter name="g" transfer-ownership="none">
                 <type name="Gee.Set" c:type="GeeSet *">
                     <type-argument name="G"
transfer-ownership="full" />
                 </type>
             </parameter>
             <parameter name="h" transfer-ownership="none">
                 <type name="H" c:type="void *"/>
             </parameter>
         </parameters>
    </method>
    ...
</interface>
<interface name="Test2" c:type="Test2" ...>
    <generic-parameter name="G" />
    <generic-parameter name="H" />
    <prerequisite name="Test">
        <type-argument name="G" transfer-ownership="full" />
        <type-argument name="Test" transfer-ownership="full">
            <type-argument name="H" transfer-ownership="none" />
        </type-argument>
    </prerequisite>
    ...
</interface>
<class name="Test3" c:type="Test3" ...>
    <generic-parameter name="G" />
    <generic-parameter name="H" />
    <implements name="Test">
        <type-argument name="G" transfer-ownership="full" />
        <type-argument name="Test" transfer-ownership="full">
            <type-argument name="Test" transfer-ownership="none">
                <type-argument name="H" transfer-ownership="full" />
            </type-argument>
        </type-argument>
    </implements>
    ...
</class>

PS. Adding Jürg to the discussion.

(In reply to comment #19)
> We went out of our way to not expose the generic types too often (which is
> where most of the pain in C is - when you have to add the GType, copy, and
> delete parameters to each function).

I just spotted it - you don't. Only to generic functions and during creation of generic classes. So if you call generic member you are free to omit it.

[Mass-moving gobject-introspection tickets to its own Bugzilla product - see bug 708029. Mass-filter your bugmail for this message: introspection20150207 ]

-- GitLab Migration Automatic Message --

This bug has been migrated to GNOME's GitLab instance and has been closed from further activity.

You can subscribe and participate further through the new bug through this link to our GitLab instance: https://gitlab.gnome.org/GNOME/gobject-introspection/issues/42.