a couple of thoughts i had since:

The argument about ~/.local/ vs some-other-dir comes down to an argument about convenience for the end-user vs. niceness when viewed from dconf-editor.  Obviously some standard path in ~/.local/ would be required for the editor to work properly.  This route would be quite easy to take by just checking the user datadir in addition to the system data dirs.

At the same time, plugins tend to be self-contained things that don't want to install files other places in the homedir.  So this gives a desire for a add_gsettings_path() API.

The problem with that is a couple of things.

First, if we add schema paths, we probably want to have some way to remove them (when the plugin is unloaded?).  This would be difficult with the way the code currently works.  Second is that we don't want the added path to leak into other uses of GSettings from the same process.

This pushes me toward some way to create a GSettings with a custom schema path.  Something like

  g_settings_new_with_schemadir("org.gnome.GEdit.Plugin.foo",
                                "/home/you/etc/etc");

Then we have to get into the idea of providing all of the variants of with_schemadir, with_path, with_backend, etc.  I'd like to dodge that.

One way might be

  g_settings_push_schema_dir ("/home/you/etc/etc");
  g_settings_new ("org.gnome.GEdit.Plugin.foo");
  g_settings_pop_schema_dir ();

that keeps a stack of custom schema directories in a TLS variable.  On creating the GSettings object, it would keep a copy of this stack internally so that any child settings object of the main object get created with the same set of schema dirs (so that you can pop immediately when you return and not have to push again).

For bonus points then the GEdit plugin API could either:

  a) add a function to get a GSettings object in a 'pushed' environment

  b) push some directories before calling the initialisation code of the
     plugin and pop them after

This would make the "what directory am I installed in?" question quite transparent to the plugin itself...

(In reply to comment #1)
> a couple of thoughts i had since:
> 
> The argument about ~/.local/ vs some-other-dir comes down to an argument about
> convenience for the end-user vs. niceness when viewed from dconf-editor. 
> Obviously some standard path in ~/.local/ would be required for the editor to
> work properly.  This route would be quite easy to take by just checking the
> user datadir in addition to the system data dirs.
> 
> At the same time, plugins tend to be self-contained things that don't want to
> install files other places in the homedir.  So this gives a desire for a
> add_gsettings_path() API.
> 
> The problem with that is a couple of things.
> 
> First, if we add schema paths, we probably want to have some way to remove them
> (when the plugin is unloaded?).  This would be difficult with the way the code
> currently works.  Second is that we don't want the added path to leak into
> other uses of GSettings from the same process.
> 
> This pushes me toward some way to create a GSettings with a custom schema path.
>  Something like
> 
>   g_settings_new_with_schemadir("org.gnome.GEdit.Plugin.foo",
>                                 "/home/you/etc/etc");
> 
> Then we have to get into the idea of providing all of the variants of
> with_schemadir, with_path, with_backend, etc.  I'd like to dodge that.
> 
> One way might be
> 
>   g_settings_push_schema_dir ("/home/you/etc/etc");
>   g_settings_new ("org.gnome.GEdit.Plugin.foo");
>   g_settings_pop_schema_dir ();
I like this approach, it means to just add 2 new methods and we would keep managing everything as always. Also as you say below we could add some api in gedit to even make this transparent, so I am all for this approach.

> 
> that keeps a stack of custom schema directories in a TLS variable.  On creating
> the GSettings object, it would keep a copy of this stack internally so that any
> child settings object of the main object get created with the same set of
> schema dirs (so that you can pop immediately when you return and not have to
> push again).
> 
> For bonus points then the GEdit plugin API could either:
> 
>   a) add a function to get a GSettings object in a 'pushed' environment
> 
>   b) push some directories before calling the initialisation code of the
>      plugin and pop them after
> 
> This would make the "what directory am I installed in?" question quite
> transparent to the plugin itself...

I like the proposed idea of having a way for gedit to load the custom schema and expose it to only the plugin. If needed we could even have a settings schema line in the .plugin file.


Another possible idea that comes to my mind is having a way in the gedit schema file itself to "refer" or "map" to another schema tree: for instance gedit's schema would say that schemas for org.gedit.plugins should be fetched from ~/.gedit/plugins/ or something like that.

(In reply to comment #1)
> Second is that we don't want the added path to leak into
> other uses of GSettings from the same process.

Would this be a problem in practice ?
Obviously, if your different locations contain conflicting schemas, you are in big trouble anyway. 
I would expect all of this to be namespaced uniquely.

From the point of view of an application that needs to introspect the schemas/settings (e.g. the editor) I think the schemas must be in a fixed location (e.g. ~/.local).

A possible solution might be to symlink schemas from ~/.local to the plugin location.  This could be done transparently to the user, e.g.

g_settings_new_from_dir ("org.gnome.GEdit.Plugin.foo", "/home/you/.local/share/gedit/plugins/foo");

In this case, the call to g_settings_new first looks in ~/.local/share/glib-2.0/schemas/ for the schema.  If it doesn't exist, it looks in ~/.local/share/gedit/plugins/foo, and makes a symlink to point to the schema.  If there is no schema, GSettings fails as normal (i.e. this schema is never looked for in the system settings).

So, first run creates the symlink, second run uses the symlink already there.  DConf editor shows this schema after the first time you use a plugin.

A broken symlink should be treated as the plugin is uninstalled.  Perhaps the GSettings cleanup code could check for these and delete them.  We should probably also provide an explicit g_settings_remove_schema () for well behaved applications.

If symlinks aren't appropriate (if dconf is used on systems that don't support them), we could have a file that contains the list of schemas.

Created attachment 192362 [details] [review]
GSettings: support for dynamic schema directories

Add g_settings_push_schema_dir() and g_settings_pop_schema_dir() to
support the temporary addition of a directory in which to search for
schemas.

This is intended to support self-contained plugins that wish to include
their schemas in the same directory as the rest of their files.

Add a --symlink-files option to glib-compile-schemas which links schema
xml and override files to ~/.local/glib-2.0/linked-schemas/.

g_spawn() the compiler with this new option from
g_settings_push_schema_dir() so that dconf-editor has a chance to find
the schemas for the plugin.

two things are still on my list to consider:

first: can we prevent the plugin from having to distribute the gschemas.compiled file?  we're already spawning the schema compiler to take care of the xml file linking, so why not also have it compile the schemas and output the result to stdout to be captured by the process and used directly?

second: can we integrate this with libpeas?

(In reply to comment #7)
> two things are still on my list to consider:
> 
> first: can we prevent the plugin from having to distribute the
> gschemas.compiled file?  we're already spawning the schema compiler to take
> care of the xml file linking, so why not also have it compile the schemas and
> output the result to stdout to be captured by the process and used directly?

It is not high prio but it would definitely would be a good idea. Python/javascript devs are quite lazy in relation to compile anything.

> 
> second: can we integrate this with libpeas?

I will ask steve

** Disclaimer ;) **

This may be a very bad idea, I'm talking as a user of the gsettings api and I'm not aware at all of the implementations detail of gsettings / dconf.

** The requirement as my understanding of the problem **

The specific problem that we want to solve is:

to permit the installation of plugins in the home dir of the user

These are the basic requirements:

1) try to make dconf-editor happy so it can find the plugin options easly
2) try to make the user happy so he can install all the plugin file in the same directory (the .so file and the schema)

These are things that would be nice to have (I will not cover this):

a) The user should just install the file and the system will take care of all the rest (spawn the schema compiler etc...)

** My proposal **

Extend the schema dtd so an "user" directory (or event an arbitrary directory) can be included in the scanning process of the schema compiler.

For example for gEdit the org.gnome.gedit.gschema.xml could include a line like:

<include-schema-dir>~/.local/share/gedit/plugins</include-schema-dir>

And than the schema compiler when scanning the .schema files will include all the file found in that directory too.

So the user when installing a gEdit plugin should just copy the required files in the .local/share/gedit/plugin directory and execute the schema compiler.

This should solve the point 1 (may be that info should be included also in the gschemas.compile file) and 2 (locality of the plugin installation).

** Some considerations about the schema compiler **

I don't really know if we can have multiple gschema.compled file at the moment, but if this is feasible we can have two distinct modes:

when issued a glib-compile-schemas from root we can compile the system schemas in /usr/share/glib-2.0/schemas

when issued from a normal user we can read the system schemas and compile only the included schema dir if found

** Advantage **

No API change is required and all the complexity is moved to the schema compiler and not in glib

> second: can we integrate this with libpeas?

Sure, but it needs some thinking wrt the API (the plugin info struct only contains static data as of today).

Also it won't solve the issue from the editor's standpoint.

I do not think that the patch in comment #6 considers a very basic use case:

I want to be able to install an arbitrary gsettings-based *application* (not just a plugin!) in my home directory.

Without root privileges. Without needing to ask my system administrator to install a schema file somewhere under /usr.

Without needing to link to random libraries like libpeas that the application does not need.

And without having to add any additional gsettings API calls to the application. After all, the application could well be partially or fully closed-source!

I should be able to just type "./configure --prefix=/home/me; make; make install" and then run ~/bin/application.

This use case is addressed by the patches attached to bug #645254 (recently closed as WONTFIX), but is not addressed by the patch in comment #6 here.

Well for user installed applications you already have .local where gsettings would work perfectly and it is the recommended place for it.

(In reply to comment #11)
> I do not think that the patch in comment #6 considers a very basic use case:
> 
> I want to be able to install an arbitrary gsettings-based *application* (not
> just a plugin!) in my home directory.
> 
> Without root privileges. Without needing to ask my system administrator to
> install a schema file somewhere under /usr.
> 
> Without needing to link to random libraries like libpeas that the application
> does not need.
> 
> And without having to add any additional gsettings API calls to the
> application. After all, the application could well be partially or fully
> closed-source!
> 
> I should be able to just type "./configure --prefix=/home/me; make; make
> install" and then run ~/bin/application.
> 
> This use case is addressed by the patches attached to bug #645254 (recently
> closed as WONTFIX), but is not addressed by the patch in comment #6 here.

(In reply to comment #12)
> Well for user installed applications you already have .local where gsettings
> would work perfectly and it is the recommended place for it.

It *ought to* work perfectly, but currently fails.

Gio currently only loads gsettings schemas from XDG_DATA_DIRS and GSETTINGS_SCHEMA_DIR. It does not load gsettings schemas that are installed in XDG_DATA_HOME (i.e. ~/.local) or anywhere else in ~. See initialise_schema_sources() in gio/gsettingsschema.c and the discussion in bug #645254.

How is it that this bug has been lingering for so long with no resolution??? I mean, gnome shell extensions are entirely broken because settings are loaded from user config dies but schemas are not loaded from user data dies. This is entirely irrational.

If settings are read from systemwwide dies but schemas to interpret them are NOT, that is a bug. The any proposal that does not address this mismatch is fundamentally broken.

(In reply to comment #14)
> How is it that this bug has been lingering for so long with no resolution??? I
> mean, gnome shell extensions are entirely broken because settings are loaded
> from user config dies but schemas are not loaded from user data dies. This is
> entirely irrational.

Nobody has written a patch, that is why.

Extensions are broken if their authors don't test them, naturally...they simply cannot use gsettings at this point.

(In reply to comment #16)
> Nobody has written a patch, that is why.

People have written patches (https://bugzilla.gnome.org/show_bug.cgi?id=645254), but the patches have been marked as WONTFIX.

Created attachment 200425 [details] [review]
gsettings: Add g_settings_push_runtime_dir() and pop_dir APIs

Certain clients of gsettings want to provide a way for plugin systems
to use their own gsettings schemas. Because plugins are often bundled
somewhere in the home directory, we need a way to specify directories
other than the system data dirs. We introduce two new APIs to provide
for these clients. Use them like:

  GSettings *settings;

  g_settings_push_runtime_dir ("/usr/share/gedit/plugins/foo-1.0/");
  settings = g_settings_new ("org.foomatic.FooPlugin");
  g_settings_pop_runtime_dir ();

Obviously clients are more likely to use path names generated at
runtime rather than hardcoded ones.



--

Alexandre, that's because the patches didn't do the thing that desrt wanted. Fortunately, we don't have to wait for patches anymore, just for desrt to review this one that I'm attaching now.

I'm a moron and just realized that desrt already produced a patch. Sorry about the noise.

(In reply to comment #18)
> Alexandre, that's because the patches didn't do the thing that desrt wanted.
> Fortunately, we don't have to wait for patches anymore, just for desrt to
> review this one that I'm attaching now.

No, the issue is that there are two different problems at hand, which are, unfortunately, being conflated.

The problem that shell extension writers, average users, and yours truly want to solve is the ability to install *new applications* in a home directory. The natural solution to such a problem is to allow schemas to be installed under a single location standard for all applications - namely, XDG_DATA_HOME.

The problem that desrt wants to solve is the ability to install *new plugins for a previously installed application* in a home directory. The natural solution to such a problem is to add new API that plugin systems could use to load additional schemas from arbitrary, plugin-specific locations.

Apparently, desrt believes that these two problems are identical. I must disagree. I believe that both problems should be solved, that the solutions to them are necessarily different, that these two solutions are not in conflict, and that implementing *both* solutions in glib would be a good thing.

Alexandre:

Notice that reading schemas from XDG_DATA_HOME in addition to reading them from XDG_DATA_DIRS, actually solves both problems* quite effectively.

* Problem "installing new applications to homedir" and "installing new plugins for existing applications to homedir", both fundamentally need the problem "transparently reading and incorporating schemas from homedir / XDG_DATA_HOME" solved, one way or another.

Patch looks great.

Unfortunately the patch doesn't ACTUALLY solve the gnome-shell extensions problems (or, really, ANY application's problems if it's installed in $HOME) unless the extension or the application goes OUT OF ITS WAY to do what OUGHT to be automatic (incorporating schemas from XDG_DATA_HOME).

Which will be an API call that people will forget to use, because people (developers and users) already expect that schemas will be read first from DATA_DIRS, then from DATA_HOME, because it's already established behavior, because it makes SENSE.

And the patch doesn't fix that.

Why the opposition to automatically reading schemas from XDG_DATA_HOME?

http://standards.freedesktop.org/basedir-spec/basedir-spec-latest.html

I ask: WTH do we have an XDG_DATA_HOME in the spec if core components of the desktop will ignore it?

(In reply to comment #22)
> Patch looks great.
> 
> Unfortunately the patch doesn't ACTUALLY solve the gnome-shell extensions
> problems (or, really, ANY application's problems if it's installed in $HOME)
> unless the extension or the application goes OUT OF ITS WAY to do what OUGHT to
> be automatic (incorporating schemas from XDG_DATA_HOME).
> 
> Which will be an API call that people will forget to use, because people
> (developers and users) already expect that schemas will be read first from
> DATA_DIRS, then from DATA_HOME, because it's already established behavior,
> because it makes SENSE.
> 
> And the patch doesn't fix that.

We can fix GNOME Shell. That's the entire reason I made the patch.

> Why the opposition to automatically reading schemas from XDG_DATA_HOME?
> 
> http://standards.freedesktop.org/basedir-spec/basedir-spec-latest.html
> 
> I ask: WTH do we have an XDG_DATA_HOME in the spec if core components of the
> desktop will ignore it?

desrt explained to me at one time the reason that we don't read from XDG_DATA_HOME. I forget the explanation, so you'll have to ask him.

(In reply to comment #23)
> We can fix GNOME Shell. That's the entire reason I made the patch.

Jasper, you can fix GNOME Shell, but you can't fix the rest of the world. To allow all gsettings-based applications to be installable in a user's home directory, the choice comes down to
a. modifying EVERY application that installs a gsettings schema to use desrt's proposed new API calls, or
b. fixing gsettings to do by default what is logical and expected

and it seems clear to me that choice (b) is better.

> desrt explained to me at one time the reason that we don't read from
> XDG_DATA_HOME. I forget the explanation, so you'll have to ask him.

I eagerly await to hear his reasoning.

CHOICE B, for the love of gawd.

Created attachment 200441 [details] [review]
GSettings: support for dynamic schema directories

Add g_settings_push_schema_dir() and g_settings_pop_schema_dir() to
support the temporary addition of a directory in which to search for
schemas.

This is intended to support self-contained plugins that wish to include
their schemas in the same directory as the rest of their files.

Add a --symlink-files option to glib-compile-schemas which links schema
xml and override files to ~/.local/glib-2.0/linked-schemas/.

g_spawn() the compiler with this new option from
g_settings_push_schema_dir() so that dconf-editor has a chance to find
the schemas for the plugin.



Merged the two patches. Included my tests, but used desrt's original code. Other things. We're also compiling on-the-fly now in the event that we have a gschema.xml, but not a gschemas.compiled.

This should be usable enough as it is to work. If you're fine with this, desrt, I can start working on a GNOME Shell patch.

As for other people, just reading the schemas from the homedir won't be enough in the case of GNOME Shell Extensions, since we never install into ~/.local/share/glib-2.0/schemas.

(In reply to comment #26)
> As for other people, just reading the schemas from the homedir won't be enough
> in the case of GNOME Shell Extensions, since we never install into
> ~/.local/share/glib-2.0/schemas.

If I am building gnome-shell-extensions for installation in my home directory, I would run

./configure --prefix=$HOME/.local

to configure for installation in ~/.local/share/gnome-shell/extensions, where gnome-shell will look for them. And that configure call results in schemas being installed in ~/.local/share/glib-2.0/schemas.

(In reply to comment #24)
> (In reply to comment #23)
> > desrt explained to me at one time the reason that we don't read from
> > XDG_DATA_HOME. I forget the explanation, so you'll have to ask him.
> 
> I eagerly await to hear his reasoning.
One explanation I gave on the other report, not sure it's what Ryan was thinking about, but that would be a problem anyway AFAICT:

I think one of the problems with reading schemas from XDG_DATA_HOME is that it
can create a real mess if two schemas with the same name are installed, one on
the system, and one in the user's home. What for example if you run two
versions of Nautilus that need different versions of the schemas? The system
version would load the one stored in XDG_DATA_HOME, and probably crash.

> (In reply to comment #28)
> I think one of the problems with reading schemas from XDG_DATA_HOME is that it
> can create a real mess if two schemas with the same name are installed, one on
> the system, and one in the user's home. hat for example if you run two
> versions of Nautilus that need different versions of the schemas? The system
> version would load the one stored in XDG_DATA_HOME, and probably crash.

That's not really a significant problem. The same thing already happens if you install two versions of Nautilus, one prefixed in /usr and the other in /usr/local. Both will try to load the schema from /usr/local/share/glib-2.0/schemas first, and the Nautilus from /usr might crash as a result. And yet we don't hear too many complaints about this behavior :)

Correct.  That crap is not a significant problem, because the people who encounter that situation (DEVELOPERS, ALWAYS) know how to strace and gdb.

I fully agree with Alexandre.  So, can you fix your programs to work for USERS?  If someone was, uh, uninformed enough to install schemas to the wrong place and then forget to set environment variables / reconfigure your systems, I don't think that's the case that the code should be serving.

If you run two different versions of Nautilus, FIX YOUR ENVIRONMENT (you can set your XDG_DATA_HOME, can't you?).  Let the rest of us, who want to install applications and plugins, you know, your USERS, use our programs in peace.

Created attachment 200543 [details] [review]
GSettings: support for dynamic schema directories

Add g_settings_push_schema_dir() and g_settings_pop_schema_dir() to
support the temporary addition of a directory in which to search for
schemas.

This is intended to support self-contained plugins that wish to include
their schemas in the same directory as the rest of their files.

Add a --symlink-files option to glib-compile-schemas which links schema
xml and override files to ~/.local/glib-2.0/linked-schemas/.

g_spawn() the compiler with this new option from
g_settings_push_schema_dir() so that dconf-editor has a chance to find
the schemas for the plugin.



--

Fix a silly compile error (forgot to commit --amend), also,
shove output from the compiler to /dev/null

Created attachment 201272 [details] [review]
GSettings: support for dynamic schema directories

Add g_settings_push_schema_dir() and g_settings_pop_schema_dir() to
support the temporary addition of a directory in which to search for
schemas.

This is intended to support self-contained plugins that wish to include
their schemas in the same directory as the rest of their files.

Add a --symlink-files option to glib-compile-schemas which links schema
xml and override files to ~/.local/glib-2.0/linked-schemas/.

g_spawn() the compiler with this new option from
g_settings_push_schema_dir() so that dconf-editor has a chance to find
the schemas for the plugin.



Fix a whitespace issue

This has been addressed by exposing GSchemaSource

For whoever (as me) comes here and gets lost: Matthias refers to
https://developer.gnome.org/gio/stable/gio-GSettingsSchema-GSettingsSchemaSource.html#GSettingsSchemaSource

see function g_settings_schema_source_new_from_directory ().