Created attachment 131144 [details] [review]
Legitimate dots in names

Another way would be to create a NamespaceBlob, and additionnal API in girepository. Indeed, since the resolution of Bug 555960 (Nested structs and unions), it seems obvious we could nest NamespaceBlobs.

I don't have much involvement in the introspection work anymore, but to me this appears quite backwards. Just because consumers of introspection have a feature, gir doesn't have to support it. The purpose of gir is to faithfully represent enough of the C abi to allow good auto-generated bindings, not to somehow supplant higher-level language constructs onto the C api.

(In reply to comment #3)
Matthias,

In fact, Vala is not just a consumer of introspection. It is also a producer of introspection data, as it produces C code and thus C libraries.

When programming an API in C, it is natural to prefix function, interface and object names to group them in correlated set. On big APIs, one may want to have two or more levels of prefixing. This stays one big package, but the API is split in many sub-namespaces in order to clarify the grouping of concepts and ease the understand of the API by the users.

Vala introduced the concept of namespace to match the fact that people tend to group their API concepts. It uses dots (".") to nest namespaces, as it uses dots to name the members of a class. Those dots are translated to _ for lower-case names and removed for the upper-case names, just any other C library.

Gtk+ is an example of library set that could take benefit of nested namespaces. Instead of having GtkSource, GtkImage, GtkHtml, and GtkMath, one would have Gtk.Source, Gtk.Image, etc... It seems important to me to make the logical grouping of those library apparent in the namespacing.

Not having the "." allowed in the typelib seems a non-sense to me. Removing that limitation would permit to transport the logical grouping of namespaces.

What would help evaluate this bug is:

* A concrete example here of a "." namespaced Vala file
* The current generated .gir for that file (without this patch)
* The proposed changes to the .gir

Now, when doing any change like this we need to consider the impact on the bindings.  How does a Vala-namespaced library map to say JavaScript?

Another tack on this bug would be - there's no reason Vala couldn't also add a vala:type="Foo.Bar" attribute on various things if it needs to know the Vala-namespacing for consuming its own .gir files, but no one other than Vala needs to know.

You can find an example here of a multi-level namespace Vala library:
http://git.freesmartphone.org/?p=libfso-glib.git;a=tree;f=src;hb=HEAD

As this is interface-only definitions, there is no point to deliver these in separate libraries. However, valac generates DBus wrapping code along to the interfaces GTypes definitions, and one of the purpose of this library is to share that common code.

Note that this library is also meant to be used by C clients.

I will attach the gir currently produced by valac for that library.
I will also attach the gir produced by valac when the explicit barrier for nested namespaces is set off.

You will notice that in the first version, there is only interfaces and DBus code belonging to the top-level namespace.

I could propose two different changes to the GIR format:

- either, we support for nested namespace tags (and thus the notion of module that uses the typelib compile is fully representative with arbitrary namespace levels inside a module)

- either, we support for names with dots

In both case, either a relative name resolution must be introduced either c:type attributes must use fully-namespaced names, which I would recommend.

Created attachment 131722 [details]
The libfso-glib GIR file generated by valac with nested namespace barier on

Created attachment 131728 [details]
The libfso-glib GIR file generated by valac with nested namespace barier off

First class nested namespaces just aren't going to happen for the typelib.  It's far too much nontrivial work with a huge impact on the bindings.  I'm not saying "never" but not this pass.

Allowing names with dots is possible, but again we need to have a plan for how this works for bindings.  Taking your example vala file, the GI namespace would be "FreeSmartphone.Device".  In Gjs, the import mechanism uses magic on an "imports.gi" prototype, like:

    let Clutter = imports.gi.Clutter;

How would the import work if we allowed "." in GI namespaces?  We can't just do:

    let FSMDevice = imports.gi.FreeSmartphone.Device;

Because the "imports.gi.FreeSmartphone" lookup is going to fail, and the import doesn't know that we're then trying to look up a "Device" property after that.   We could take the replacement route, and say "." is replaced with "_":

    let GSM = imports.gi.FreeSmartphone_Device;

Would work, but it's kind of arbitrary and would need docs.

Concretely what I'm suggesting here is that there is one .gir file per namespace.  This may actually imply one .vala file could generate multiple .gir files if you did:

namespace Foo {
  namespace Bar {
     class SomeClass { ... }
  }
  
  namespace Baz {
     class AnotherClass { ... }
  }
}

I don't know if Vala supports that, but if it did, you'd have "Foo.Bar-1.0.gir" with a SomeClass element and "Foo.Baz-1.0.gir" with AnotherClass element.

I'm with Matthias and Colin here, I think that we should not support nested namespaces neither in GIR nor the typelib.

The solution for this is to make Vala export multiple girs, one per namespace.

(In reply to comment #9)
>     let FSMDevice = imports.gi.FreeSmartphone.Device;
> 
> Because the "imports.gi.FreeSmartphone" lookup is going to fail, and the import
> doesn't know that we're then trying to look up a "Device" property after that. 

It seems to me that the 'FreeSmartphone' namespace would also exist here. So could you explain why importing FreeSmartphone would fail?

> Concretely what I'm suggesting here is that there is one .gir file per
> namespace. [...]

Why not... However this leads to two important things:
 - This may bloat the gir and typelib directories with a large number of files that may have lots of interdepencies. I don't think it is really is a problem, but it seems to me that one goal of the typelib was to have a self-contained compact representation of introspection data. Splitting corelated introspection data in multiple typelib files will lead in the need to open a large number of files to get that data and thus reducing its performance in type discovery scenarios.
 - This would mean that for one (pkgconfig;library) pair there would exist multiple (gir;typelib) pairs of file. Again, this may not be a problem, but then we have to modify the pkgconfig variable that tells the gi-module name to describe multiple gi-modules. For instance:

gi-modules: FreeSmartphone-1.0 FreeSmartphone.Device-1.0 FreeSmartphone-GSM-1.0 FreeSmartphone-Time.1.0

I'm not against that "solution", but it seems to me this is a bit like going backward.

Just another real world example, that uses nested namespaces and is not related to Vala or some other high-level language at all: GStreamer uses Gst as it's main namespace, additional libraries use Gst.Base, Gst.Rtp, Gst.Audio, .....
In C this looks like gst_base_foo_bar() for example.

With restricting GIR to plain namespaces this would instead look like GstBase, etc. which is really ugly, was not the intention of the C API and is not used by any official GStreamer bindings so far.


If there are languages that don't support nested namespaces, like what you said above, they could as well strip the additional dots from the namespace and use GstBase as namespace. The other way around, adding additional dots here and there, can't be automated (should GstFooBar be Gst.Foo.Bar, GstFoo.Bar, Gst.FooBar?)

*** Bug 594223 has been marked as a duplicate of this bug. ***

This affects us too, lack of nested namespace support makes GIR unusable for us.

(In reply to comment #9)

The *magic* used by Gjs to do imports is very limited (supposedly not magic at all) and it seems it can't evolve nor be changed to another way, right ? I guess we should limit the use of g-i to Gjs. Or...

You should realize that every other bindable languages do support nested namespaces and make the necessary changes to Gjs!

(In reply to comment #14)
> This affects us too, lack of nested namespace support makes GIR unusable for
> us.

Hi Arc, who are "us" ? What project do you work on ?

(In reply to comment #11)
>
> > Concretely what I'm suggesting here is that there is one .gir file per
> > namespace. [...]
> 
> Why not... However this leads to two important things:
>  - This may bloat the gir and typelib directories with a large number of files
> that may have lots of interdepencies. I don't think it is really is a problem,
> but it seems to me that one goal of the typelib was to have a self-contained
> compact representation of introspection data. Splitting corelated introspection
> data in multiple typelib files will lead in the need to open a large number of
> files to get that data and thus reducing its performance in type discovery
> scenarios.

Maybe; we'd need to look at a given app.  But generally on linux+ext[34], small files are fairly cheap.

>  - This would mean that for one (pkgconfig;library) pair there would exist
> multiple (gir;typelib) pairs of file. Again, this may not be a problem, but
> then we have to modify the pkgconfig variable that tells the gi-module name to
> describe multiple gi-modules. For instance:

True, but that's fairly easy.  Having multiple namespaces in a .gir file would be a huge rewrite of the parser, and for the typelib...well, even smaller things in the typelib can be nontrivial, and this would be huge.

(In reply to comment #15)
> (In reply to comment #9)
> You should realize that every other bindable languages do support nested
> namespaces and make the necessary changes to Gjs!

+1.  I don't know of a modern OO language that doesn't support nested namespaces.  Every language has some method of supporting this.
 

> (In reply to comment #14)
> > This affects us too, lack of nested namespace support makes GIR unusable for
> > us.
> 
> Hi Arc, who are "us" ? What project do you work on ?

http://www.pysoy.org/

We've been working on a migration from Pyrex (C/Python hybrid language) to a GObject-based "libsoy" plus Python+Javascript bindings for more than a year (we already made heavy use of GLib libraries)

Like GStreamer, our API is arranged in a 2-level namespace as package.module.class.  For example, "soy.joints.Ball".  Multiple .gir/.typelib files would be fine, Python needs one file per module as well (and groups modules in a package in a directory, ie, /usr/lib/python3.1/site-packages/soy/joints.so).  

I am not willing to flatten our namespace just to work with GIR.  Vala/Genie has had nested namespaces for some time, so the lack of this basic feature in GIR leads me to work on a .vapi -> Python/Javascript bindings instead.

(In reply to comment #16)
> (In reply to comment #11)
> >
> > > Concretely what I'm suggesting here is that there is one .gir file per
> > > namespace. [...]
> > 
> > Why not... However this leads to two important things:
> > [...]
> 
> Maybe; we'd need to look at a given app.  But generally on linux+ext[34], small
> files are fairly cheap.
> 
> >  - This would mean that for one (pkgconfig;library) pair there would exist
> > multiple (gir;typelib) pairs of file. Again, this may not be a problem, but
> > then we have to modify the pkgconfig variable that tells the gi-module name to
> > describe multiple gi-modules. For instance:
> 
> True, but that's fairly easy.  Having multiple namespaces in a .gir file would
> be a huge rewrite of the parser, and for the typelib...well, even smaller
> things in the typelib can be nontrivial, and this would be huge.

Colin,

Please consider how both your above propositions lead to more responsibility to the users of g-i, instead of being integrated in g-i itself :
 - having multiple files for nested stuff : the algorithm for apps to know whether a specific name *is a* nested namespace or *is in a* nested namespace is let to the applications using g-i,
 - having to parse some, even if *fairly easy*, properties in the pkgconfig file : this is something to be in the scope of g-i's code, and that should be manage transparently by g-i.

Both those things to be managed by application are two much error-prone. 
What is the point to make an introspection library, if part of the introspection logic is let at the application discretion ?

Also, you postpone a decision and changes you'll have to do anyway for Gjs. How will you use Gst, or some others that use nested namespaces, from Gjs ? Will you make a hack to add _ in your nested names when you encounter girs that are possibly nested, will you implement specific hacks for specific libraries like Gst (which would completely be ridiculous and a way to prove g-i inadequacy), or will you implement a correct import statement to handle correctly nested namespace ? I guess you will be forced to do the later. Hence, the only thing we will have gain : incompatible versions of gir/typelib formats in their way to handle nested namespaces, complexification of applications, and thus more possible failures...

Please consider this is very important to do the stuff correctly to avoid future incompatibilities. What is the point to make things that will introduce new breakage possibilities in the future ?

Doing it well the first time is particularly important for g-i. What's the problem to take some months more to finish that correctly ?

I can already envision how this will be difficult to make that change in a future version, not only because the technical difficulties to keep backward compatibility, but mostly because of gnome's ancient no-goers.

First, why are you marking this OBSOLETE?  I'd like to keep a discussion going, and I'd definitely like to support this in some form, but we need to figure out how it could work.  I'm going to reopen.

(In reply to comment #18)
> 
> Please consider how both your above propositions lead to more responsibility to
> the users of g-i, instead of being integrated in g-i itself :
>  - having multiple files for nested stuff : the algorithm for apps to know
> whether a specific name *is a* nested namespace or *is in a* nested namespace
> is let to the applications using g-i,

Not necessarily?  I could certainly imagine having a GIRepository API for this.

> Both those things to be managed by application are two much error-prone. 
> What is the point to make an introspection library, if part of the
> introspection logic is let at the application discretion ?

Well, introspection in general I think should mostly be a data source; we do have some convenience utilities for bindings using ffi right now, and I wouldn't be opposed to more use-case-specific things like that, but we also don't want to just be a huge grab bag of stuff in one library.
 
> Also, you postpone a decision and changes you'll have to do anyway for Gjs. How
> will you use Gst, or some others that use nested namespaces, from Gjs ?

Right now we're just calling them "Gst*".  I don't think it's a serious problem, so far, but yes, nested namespaces for this 

> Will
> you make a hack to add _ in your nested names when you encounter girs that are
> possibly nested, will you implement specific hacks for specific libraries like
> Gst (which would completely be ridiculous and a way to prove g-i inadequacy),
> or will you implement a correct import statement to handle correctly nested
> namespace ? I guess you will be forced to do the later. Hence, the only thing
> we will have gain : incompatible versions of gir/typelib formats in their way
> to handle nested namespaces, complexification of applications, and thus more
> possible failures...

Yes, I agree, we should ideally update GScript to handle this.  At the same time, I don't have a lot of time to work on introspection and large-scale changes are difficult.

I think it would be possible to support "const GstBase = imports.Gst.Base" with some small scope work in introspection; all we need to do is have a function to scan for all typelib files which start with "Gst.", and that resolves to a Namespace object of some sort.

Multiple files are ok.  I would prefer, personally, if they were grouped in a subdirectory as Python does.  For example, "/usr/lib/girepository-1.0/foo-1.0/bar.typelib" would make it much easier to manage.

Namespace flattening is not ok.  In Python, modules have explicit file and name members which are often referred to in documentation and tracebacks.  While we could certainly setup a script to import modules other than what they're actually called, looking at a traceback for soy.windows.Window and seeing references to SoyWindows.Window is not acceptable to me.

What kind of timescale are we looking at here?  Can we decide on a way to implement this in the next week?  I am planning to implement the Python 3.1 bindings for this, the question is whether to use GIR or Vala as the source.

With a month of no progress, its clear this bug holds no priority for resolution.

Given the weak state of GIR, we've abandoned it in favor of Python bindings generation using .vapi (which .gir can be converted to easily).

Well, I personally have a ton of projects to work on, not having this feature implemented doesn't mean no priority, it means it just hasn't happened yet.

Are you planning then to pre-generate static bindings from the .vapi?  That loses several of the advantages of the introspection approach, e.g.:

* That the bindings are guaranteed to be up to date as they're built at the same time as the library
* The on-demand efficient typelib lookup (i.e. you only pay in memory cost for functions you use)

Do you not understand that this is a blocker?  Promote any cool features you want, if the end result is not a clean, Pythonic API then it is not usable by us.

I'm getting a sprint organized for this at PyCon. The current strategy is dynamic using libvala and ctypes, generating a Python API using either a .vapi or .gir. More details here; http://arcriley.blogspot.com/2010/02/working-on-gtypes.html

I would start by passing --ns-name=Gst.Base into g-ir-scanner and see what breaks.

Just so its documented, the way we're working around this is the following format:

<?xml version="1.0"?>
<repository version="1.0"
            xmlns="http://www.gtk.org/introspection/core/1.0"
            xmlns:c="http://www.gtk.org/introspection/c/1.0"
            xmlns:glib="http://www.gtk.org/introspection/glib/1.0">
  <package name="soy"/>
  <namespace name="soy.windows" version="1.0" c:prefix="soywindows">
  </namespace>
</repository>

This is in contrast to actually nesting namespace which would break the XML schema, such as:
<?xml version="1.0"?>
<repository version="1.0"
            xmlns="http://www.gtk.org/introspection/core/1.0"
            xmlns:c="http://www.gtk.org/introspection/c/1.0"
            xmlns:glib="http://www.gtk.org/introspection/glib/1.0">
  <package name="soy"/>
  <namespace name="soy" version="1.0" c:prefix="soy">
    <namespace name="windows" version="1.0" c:prefix="soywindows">
    </namespace>
  </namespace>
</repository>

Colin suggested that we should use one .gir file per namespace, so we will likely use a "full package" .gir that pulls in all the sub-namespaces, such as:
<?xml version="1.0"?>
<repository version="1.0"
            xmlns="http://www.gtk.org/introspection/core/1.0">
  <include name="soy.bodies" version="1.0"/>
  <include name="soy.widgets" version="1.0"/>
  <include name="soy.scenes" version="1.0"/>
  <include name="soy.windows" version="1.0"/>
  <include name="soy.joints" version="1.0"/>
  <include name="soy.datatypes" version="1.0"/>
  <include name="soy.shapes" version="1.0"/>
</repository>

Unless someone has a strong argument against this I'll begin working on a patch for valagirwriter to support this format, as most code that uses nested namespace is likely to be written in either Vala or Genie.  Note that at present Vala's generated GIR files contain only an empty root namespace when compiling a library with nested namespaces.

In case anyone is waiting on this, we (PySoy) stopped working on it roughly a year ago after spending a month trying to document GIR's existing xml schema.

We've abandoned GIR due to this bug in favor of hand-coded Python bindings written in C since fixing this would take us far more time than maintaining bindings by hand.  Since GIR lacks many other features we believe are critical for proper Python support (operator overloading, iterator objects, etc) we have no plans to move to GIR even if this bug is fixed.

The benefits of GIR simply do not justify limiting our Python API to use it.

We could take inspiration from Python:

/usr/lib64/girepository-1.0/FreeSmartphone/__init__-1.0.gir
/usr/lib64/girepository-1.0/FreeSmartphone/Device-1.0.gir

That would be workable for both gjs and pygobject. There's still an issue of collisions ("what happens if __init__-1.0.gir has a thing called a 'Device'"), but I think that's solvable with a simple answer of "don't do that".

[Mass-moving gobject-introspection tickets to its own Bugzilla product - see bug 708029. Mass-filter your bugmail for this message: introspection20150207 ]

Considering nothing happened in 5+ years, I think this can be closed as WONTFIX.

If somebody wants to come up with a solution and implement it, please: send an email to the mailing list for discussion, before reopening this bug.