I have this same problem I can reproduce with using my Accerelated-X server
running on Gentoo linux. I can also reproduce the problem with the Xorg server
running on cygwin. 

Well, not exactly like that. libxklavier does not access any configuration files
(it is configured by the level above, using setXX functions). Yes, in the build
process it uses include files from include/X11/extensions. Yes, it relies on
XKB_RULES_NAMES property - this is true. This property is described in 
include/X11/extensions/XKBrules.h. This file is copyrighted by SGI, so I assumed
it to be a part of any XKB implementation (sorry, it seems I was wrong) -
because SGI implemented XKB first AFAIK. Unfortunately, I do not know portable
way to determine the current keyboard configuration (in terms of
model/layouts/options - this is important). If you could point me to the
alternative better solution - I'd really appreciate it.

I always realized that libxklavier read/write the property on X server and use
xml repository from the X client machine. But again - this is beyond my control.
XKB does not provide any means to detect actually available configuration items
on the X server side. If you can tell me how to fix/workaround this - again, I
am open and would be grateful.

Sergey,  the best source of information for what functions to use is the XKB specification released with X11R6.
4, included in any recent X11 source trees.  There are many query functions available, they should provide all 
the information needed for XKB configuration (e.g. XkbGetKeyboard).

While we are no experts in libxklavier, our devel team did not see that the XKB functions designated in the 
spec are being used to query/set the keyboard layout.

Portable XKB clients should not have runtime dependencies on the /usr/X11R6/lib/X11/xkb or /etc/X11 
configuration directories.  These directories are XFree86 specific and will not work with other X 
implementations that do not use those directories for configuration.  Fortunately, most of X11 is a 
standardized so it should not be necessary to use these directories unless the client wants to perform 
proprietary X-server configuration.

I read these specs. I could not find any function which would return, say, the
list of available models (I'd prefer, with localized descriptions:). Existing
functions deal with the CURRENT configuration, not the list of AVAILABLE ones.
Also, they deal with low-level configuration components (like XkbGetKeyboard) -
I cannot obtain model/layouts/options from them (they are already decomposited).

I repeat - I use local xml file just because I have no other means to get the
list of available XKB models/layouts/options. It is not really good, but it is
the best I can do. Or there is some other way to get this info - but I cannot
find it.

Could you help me with these two questions?

Sergey:  Please look at Chapter 20 of the XKB extension specification 'Server Database of Keyboard 
Components'.

I read it already. I found no mentioning of the models/layouts/options - either
low-level components (like symbols, geometrie) or keymaps.

Sergey:  The 'XKM design' currently being used in libxklavier is different from the XKB specification, so you 
cannot compare between the two.  It is unclear how libxklavier will choose to map the XKB information into a 
format libxklavier will use -- this will likely require redisign to model XKB instead of XKM.  Note that individual 
components of an XKB keyboard layout can be changed at any time during server runtime, so trying to find an 
'explicit' XKM based mapping is inconsistent with XKB.

I realize that using high-level "model-layouts-options" (MLO) approach instead
of  low-level "keycodes-types-symbols-geometry-compat" (KTSGC) makes us
vulnerable, to some extent, to the people using "setxkbmap -symbols ...."
(actually, GNOME still feels ok if people use setxkbmap in MLO mode). I know
there is no "backward" mapping from the low level config options to the high
level ones. But I intentionally use 3-part configuration instead of 5-part. IIRC
MLO paradigm was introduced long ago for a good reason - to simplify the
configuration task, to make it less confusing. That is why I use it in
libxklavier. Do I understand you right - do you advise to redesign libxklavier
so it would entirely work on KTSGC basis? I'm afraid, on the user level, this
will cause even more confusion than what we have now. Also, this would still
leave us without i18n which we have now in xorg.xml. This would create
semantical mess of national layouts, and options (they are nothing more than
symbols) etc etc.

Sergey: It's actually worse than that.  Many people still use xmodmap for a variety of keyboard remappings.  
The goal is for control-center to be consistent with the underlying window system (not the other way around)?

The issue is unfortunately the design :-(  libxklavier makes assumptions about the X server that are not true.  
This does not bode well for portability or for future support (e.g. there was a time when GNOME used a static 
global variable to abstract X display connections because that was like Windows -- that was fun to reconcile).

Here are some questions:

- Should the X-server's keyboard configuration be the reference for keyboard settings, or a configuration 
client such as control-center?  How about the other way around, where only the client reconfigures the default 
server settings?  Most users need to have a working keyboard configuration in the absence of GNOME, KDE, 
etc... so do not need GNOME for a functioning keyboard?

- One goal of libxklavier (or control-center?) seems to be to allow users to dynamically select a different 
keyboard layout at runtime.  In that case, a user could select from a list of well known XKM files and the layout 
could be loaded via the XKB extension (not XF86 specific)?  Otherwise, it could download the current X-server 
configuration (XKB) and modify it using whatever intermediary format was useful?

- Is there a need to assume/know in advance what specific keyboard layout is being used?

> The goal is for control-center to be consistent with the underlying window
system (not the other way around)?

The goal is NOT to cover ALL possible way of configuring the keybord (because
IMHO this is not possible at all in X keyboard world) - but to cover typical
scenarious, give most possible flexibility without loosing on the usability side
of things.

Well, regarding your questions:

1. Sorry, as a gnomer, I do not really care at this point about users not having
GNOME. But as a fd.o guy, I made libxklavier desktop-agnostic, so in theory any
environment can use it - libxklavier does not worry about the way user stores
his configuration. I intentionally do not use X Server configuration because it
is not portable (/etc/X11/XF86Config? /etc/x11/xorg.conf? $HOME/xorg.conf?
...????) - but in theory, I can imagine some application, using libxklavier,
which would use some kind of xinitrc-like scripting like

#!/bin/sh
# $HOME/.libxklavier is a simple name=value config file
. $HOME/.libxklavier.conf
setxkbmap -layout $LAYOUT -model $MODEL - option $option

I just do not see any real advantage for the _GNOME_ users - other than
inability to use standard gnome configuration architecture.

2. I am not exactly sure what you idea is. Do you advise to wrap our own
prebuilt XKM files and load them onto the server? What makes you think this is
portable? Do you want to cover with XKM files all possible scenarious (Nmodels *
Nlayout_combinations * Noption_combinations)?

3. Yes. Because gnome is trying to use (if possible) existing configuration if
user wants to. So how can I do it without analyzing XKB_RULES_NAMES?

Just one small note. Could we please discuss separately libxklavier and
gnome-control-center (and gnome-applets, if it comes to the Keyboard Indicator)
? It seems you are mixing them somehow.

Sergey:

The goal is to get you to think about the design problem and how it affects GNOME.  Assuming you are in 
charge of this module, you are in a position to solve this problem.

Is it right to categorize this as a user interface problem?  Please try to understand how the specifications and 
the operation conflict with the current design.

>2. I am not exactly sure what you idea is. Do you advise to wrap our own
>prebuilt XKM files and load them onto the server? What makes you think this is
>portable? Do you want to cover with XKM files all possible scenarious (Nmodels *
>Nlayout_combinations * Noption_combinations)?

This approach is being used with commercial X servers (and is portable).

You may choose to ennumerate all possible keyboard layout choices using the XF86 reference files and then 
load them via the XKB extension, but it is mistaken to then represent it as the 'current' keyboard state.

What other applications use the libxklavier library?

> Is it right to categorize this as a user interface problem? 

Well, if we change the architecture in a way which would make UI more complex
(instead of making it simpler) - yes. All the architecture should be streamlined
in a way to make user experience better (we are talking about the desktop,
aren't we?:) Right now it is portability problem, if you are asking about it. I
could consider possible solutions as long as they do not make user's experience
worse.

> This approach is being used with commercial X servers (and is portable).

Well, assuming that GNOME works with the set of X libraries which can deal with
XKM files (which are not a part of the XKB standard, are they?).

> but it is mistaken to then represent it as the 'current' keyboard state.

I know. Just to illustrate this, I am under constant internal (my very own
usability daemon at work:) pressure to consider "layouts" as "groups" - and to
some extent, I am doing it. But I still remember the difference - and do not do
's/group/layout' everywhere in the UI. So I realize - I control layouts, but I
do not control separate symbols etc.

libxklavier is currently used by the gnome-control-center (Keyboard Preferences
capplet + gnome-seetings-daemon) and gnome-applets (Keyboard Indicator). No
other cases of usage is detected. So I could afford incompatible changes - as
long as I am sure they are really necessary.

Still, I do not really understand about the XKM files. Currently, we have 100+
models, ~70 layouts (I do not count variants). So even in "single layout mode"
we should have ~7K XKM files? Or do you still want to add geometries to the XKB
on the fly? What about multiple layouts (like "us,ru,de,il")? Do you offer to
merge XKM files on the fly? Still, I really do not see how client-side XKM files
can provide viable option...

I think there is some misunderstanding at the begin of this discussion.
Libxklavier doesn't work with "raw XKB configuration files" and doesn't use the 
functions that deals with XKM files (binary XKB config files).

What it actually use is some additional layer _above_ XKB files.  Apparently 
this layer was added later than 'basic XKB' and is absent in XKB specs.  OTOH 
the corresponded files are copyrighted by SGI and I (as Sergey) always think 
that it is an 'official' part of XKB but not an XFree86's proprietary addition.

Why this addition was needed and why I would insist on using it...
The 'basic XKB' is very poor in the part of choosing _available_ keyboard 
maps.  Whereas it has quite reach API for manipulation with the current keymap 
(the keymap that is currently loaded into the server), it offers almost nothing 
for users who just want to select prepared keyboard map that satisfy their 
needs.

The basic XKB specs include a function 'load keyboard by its name' that 
actually accepts a set of filenames for all XKB config components.  Also it 
provides a function that should deliver from the server some 'list of available 
components'.  But these components are just names of files where parts of the 
complete configuration is scattered.  If I want, say, a keyboard map with 
English and Russian alphabet, that can be switched with Alt+Shift and I want 
the CapsLock operating as 'locking Shift' and my keyboard has some multimedia 
keys that I'd like to use, it's very cryptic to gather needed small files from 
lists of their names.

Also note that the division of complete keymap to 'XKB components' probably is 
adequate to the XKB internal sructure but is very unclear for ordinary users.
I can guess that 'symbols files' with names 'en' and 'ru' have some relations 
to the map I want but who advise me what 'types file' or 'compatibility file' I 
have to choose?  And what kind of components I need to add to my complete map 
if I want multimedia keys working?

Basic XKB also provides a set of 'precomposed keymaps' where I could choose a 
complete 'Russian keymap' with all neede components.  But this way is extremely 
unflexible.  I somebody who made this keymap didn't include there all possible 
combinations for switching between English an Russina layouts, I can't use one 
I  am used to or I have to learn ... where the files are placed, how XKB works 
and where and what lines I have to add into these files.

The addition I'm talking about offers instead of XKB components some more human-
friendly terms:  'keyboard model' (that hides such details as scancodes that 
are different on different vendors keybords), 'keymap layout and layout 
variant' that actually means the language (alphabet or scripting) a user needs 
and set of 'options'  that makes small (but useful) changes for the keymap such 
as 'a layout swithching combination', LEDs behaviour, Compose key position and 
so on.

Note that this addition is build completely _above_ 'basic XKB'.  It doesn't 
touch Xserver internals, it doesn't need changes in the protocol and it doesn't 
work directly with XKM files.  It is just small set of functions and own config 
file 'XKB rules' (that can be placed anywhere).  The addition takes a few 
strings that represent names of needed Model, Layout(Variant) and Options_list 
and using 'XKB rules file' composes chains of 'XKB component' names that then 
can be passed to the 'basic XKB' as if a user himself chose them.  Also note it 
works mainly on a client side.

It's a pitty that such simple but very handly tool is absent in commercial 
Xservers. Therefore I'm aganst to refuse its using in GNOME just because some 
Xservers lacks it. 

I think the real issue is not the absence of this 'proprietary addition' on the 
server (because keyborad configure applet uses it on the client side and 
doesn't need such thing in the server one) but the mismatch between the 
client 'XKB repository' and the server one.  Since the applet uses 'rules' and 
their descriptions from the client side machine it can compose component names 
chain that will be unacceptable on the server side.

(I wonder how such commercial servers load XKB keymap.  If their XKB are 
compliant to basic XKB specs they should have XkbGetKeyboardByName function.  
This function accepts a set of strings that are names of 'XKB components'.  But 
if the server doesn't have /usr/X11R6/lib/X11/xkb hierarchy where it gathers 
components to fulfill such Get*ByName request?)

If the server had somewhere such XKB repository we would just suggest to 
install a set of XKB config files there.

As for XKB_RULES_NAMES property, I think the applet should not consider its 
absence as fatal error but use some defaults for the first start and set the 
property by itself.  (I hope commercial servers allow to set such property 
anyway.)

Even if non-XFree86 servers don't support any XKB repository I can suggest a 
way.  The configuration can be composed and compiled completely on the client 
side.  (Sergey, do you remember we had already discussed it when Keith said 
that running xkbcomp by Xserver is insecure?)  Now the applet finally calls 
GetKeyboardByName (I guess).  But instead of it it could run xkbcomp passing 
component names to its input and specifying the display as its output (or save 
XKM to a file and then use the code the same as in xkbcomp to 'inject' the 
keymap to the server).

Ivan, thanks for your insightful comments. Just a couple of questions:

> As for XKB_RULES_NAMES property, I think the applet should not consider its 
absence as fatal error but use some defaults for the first start and set the 
property by itself.  (I hope commercial servers allow to set such property 
anyway.)

What would be the "default" value for the rules file name? "xree86", "xorg",
"base", "xig", ...? Actually, at the moment this is the most important part of
the property - because libxklavier keeps it intact and needs _valid_ initial value.

Invoking xkbcomp on the client side looks more or less ok to me - I just would
not like to launch it but just to call some function which would do its job.
Well, this is purely technical question...

> What would be the "default" value for the rules file name?

It is a hardest question.
Frankly speaking I have not good ideas.
Since we make efforts to make our config set some 'standard' for all distros, I
think "base" could be the default.
Of course the better solution would be to scan the corresponded directory and
try to find there al least well-known names: "base", "xorg", "xfree86", etc. 
But I don't think the applet should do it.  Maybe the 'configure' script could
perform this task.  (I realize that it would be done at binary package creation
but not at its installation.  But I hope packages are being created in the
invironments the same as one where they will work.)

> I just would not like to launch it but just to call some function which would
do its job.

If you mean XKM reading and sending the keymap to the server, it's OK.  But for
compiling a complete keymap from XKB 'sources' you need a whole xkbcomp.

> It is a hardest question.

I would not bore you with easy questions:) OK. I can assume "base", "pc101" -
and make it configure-time option. NP.

> for compiling a complete keymap from XKB 'sources' you need a whole xkbcomp.

It is a pity that xkbcomp is not a library with some thin command line
frontend:( Very pity... Why is it not??? (rhethorical question, actually).

Now, the question to Michael: what do you think about the proposed solutions?

Guys:

I would like to understand in greater detail the goals of the GNOME project w.r.t. keyboard configuration.  
Apparently it is difficult to separate the function that libxklavier is trying to perform on behalf of the GNOME 
project from the functions is is currently performing.  Is someone able to clarify what requirements GNOME has 
to configure X keyboard state at runtime?

With GNOME, a problem emerges because there the X server keyboard configuration and usage is very 
different from the Windows keyboard (which GNOME is also tasked to deal with). In the X window system, 
localization is handled on a per application (client) basis. Therefore mapping a keyboard layout to 'language' 
does not apply to the X window system as in the (possibly simplified) Windows sense.  GNOME may have 
different ideas about the current state of localization and how that applies specifically to GNOME applications 
in general; however that concept would not apply to X11.

Onto XKB: one goal of the XKB specification (as in most 'public' specifications) is not to impose an 
implementation on any X server vendor (including XF86); the spec specifies protocol and behavior.  Included 
with the spec was a sample implementation provided by SGI, but the sample implementation is not the 
specification.  Servers can only comply with the spec (but obviously they have no option to control client 
behavior).

In this case, libxklavier is using proprietary files meant for X server configuration (specifically XF86).  However, 
those files are not part of the spec and should not be used in general for X configuration at runtime.  It seems 
that there is a 'hope' that these files could/should be used in a portable and generic way, but that is not the 
case.  Another problem is that XKB extension has to operate in a distributed environment.  Configuration files 
are intentionally left out of the specification because (as is often the case) that brings up problems of local/
remote access.

However, XKB provides a very powerful programatic API for configuring the keyboard at runtime.  Dynamic 
configuration of the keyboard has always been an important characteristic of X11.   Since the state of the 
keyboard layout is not fixed by the X server, X11 clients cannot make any assumptions about the current state 
of the keyboard (whenever a server keyboard change is made, the clients react.)  But keyboard layout changes 
do not dictate the localization behavior of the client.  The XKB spec is correct in that it only describes a 
keyboard layout, as opposed to how clients are going to use that layout.

A user might find it useful to change keyboard layouts at runtime, using GNOME control center or any other 
non-GNOME application.  I think the GNOME goal would be to configure what keyboard layout the user wants 
from a list.  If the layout has a relationship to the GNOME language, then GNOME must use it's own (probably 
user configurable) abstraction to handle this.

To simplify the keyboard layout selection, the user could choose from a user specific list of XKM files on their 
system.  While XKM files are not portable between X servers, they are portable between architectures.  This 
removes the responsibility from GNOME of parsing/manipulating the proprietary configuration files and allows 
the user to generate a list of layouts based on XKMs specific to their setup.  Potentially a user could create 
these XKM files manually or using a GNOME UI.  This is just a hypothetical solution, but you may be able to 
come up with a better solution keeping the above issues in mind.

First of all, there is no direct relation between GNOME localization
architecture and keyboard management (at the moment, at least). So when we talk
about "languages" here - we only mean national layouts, nothing more. 

Regarding the requirements - they are pretty much clear. As far I see it, user
should be able to pick several layouts (usually - per-country, per-language
basis), pick his keyboard type (well, in the ideal world this should be
automatic - but not now...) and tweak functionality of the special keys
(Ctl,Alt,CapsLock) and indicators), to some extent. User should be able to do it
at any point of his desktop session - and the changes should be actualized
immediately. Does this make sense?

Yes, I see the difference between the specs and the reference implementation (I
am sure Ivan does too). Yeah, we assumed some details to be "standard" while
they were not in the specs, only in the default implementation. And again, I
agree with Ivan, that NOT having these things in specs is a very unfortunate
circumstance. The specs are weak at this point. If we stick firmly to the
features described in the spec, user should either deal with low-level component
files (which is very difficult to manage and understand for non-technical
person) - or use keymaps  (or XKM files if you like), which are very very rigid.
I still do not see how user could use XKM files if he needs multiple layouts
"us,ru,de,il" - to the best of my knowledge, they are not mergeable, are they?

There were usability reasons why we made some assumptions based on the reference
implementation (and the implementations in xfree/xorg servers). Leaving aside
the client/server keymap loading issure (which is generally covered by running
xkbcomp on the client side, as Ivan proposed) - I'm still inclined to think that
our tradeoff between usability and compatibility problems is good. The only real
fault I see here is that we do not have documentation on these extra
requirements (well, we did not think about them seriously till that point).

OK -- let's be clear.  If you compile the files distributed with XF86, they will not work with any server except 
XF86.  The fact that they work on homogenous XF86 environments is an accident.  You cannot compile the 
XF86 files and get them to work on onther X server implementations.  This is not a server problem or bug, it is 
a result of the specification which separates these files from the implementation.

The XKM file format is different; it is an encoding of the XKB information that is practical to use with the XKB 
extension.  The XKM file contains the complete state of the keyboard layout, but how it is created is purely 
proprietary (so XKM for XF86 will not work on XiG).  The file format is portable.

The XKB specification is made this way for a reason; there are a lot of keyboards, and keyboards cannot 
simply be abstracted on the basis of language.  This requires a great deal of flexibility in the backend, 
something that a client should never see (even configuration clients). Also (sorry to repeat), X11 does not 
impose localization on the clients so describing the server keyboard state with a single language is wrong 
(there may be clients with German localization at the same time there are clients running with French. using the 
same keyboard layout?)

I am curious why GNOME is trying to control the keyboard layout for X11?  The keyboard is always configured 
by the user (as part of the X server configuration) before GNOME is even started.  Adding an additional X 
server configuration through GNOME would seem to be a good idea only if GNOME already had a good idea of 
exactly what keyboard was connected.  Where did the need for keyboard layout configuration by GNOME start?

> This is not a server problem or bug, it is 
a result of the specification which separates these files from the implementation

Agree. It is not a server problem - it is a poor specification problem (or
missing additional specification). The XKB specification does not specify some
necessary things. Things, which would make XKB configuration easier. Things
which would allow to build (easily) nice GUI configuration tools.

I realize the advantages of the XKM format. But this "complete state of the
keyboard layout" makes it practically unusable. XKM repository would be
unmaintainable and monstrous - if we want to provide any concievable combination
of national layouts (just estimate the size of the "all combinations up to 4"
repository for 100 layouts - it is about O(1E8)!!!) 

Well, now I probably see why you are linking localization with layouts. OK. I am
nearly ready to skip the requitement to have localized layout descriptions
(though it is still not clear how it should be handled - but let's forget about
it). It is not localization problem which made us choose layout-based approach
(not the keymap-based). It is flexibility and maintainability.

> I am curious why GNOME is trying to control the keyboard layout for X11?

I cannot belive you don't understand it! Or - probably - I do not understand
your question. GNOME configures layouts because each user can have his own
preferences. When the system is installed, admin chooses some reasonable default
configuration - but each user can have different requirements. Currently GNOME
does not have idea about the connected keyboard (though it can pick the system
settings). I hope, eventually (with the Project Utopia) GNOME will find the
keyboard model automatically. BUT we cannot predict which layouts user would
need. User should be able to pick the layouts he wants - national layouts, their
variants. I want to have American  and Russian layouts - with any attached
keyboard. Should GNOME provide XKM for "us,ru(winkeys)" for me (well, also I
want to switch using RCtrl)? Or should I tweak all these things in .xmodmap?

I filed a bug in the X.org bugzilla (see
http://freedesktop.org/bugzilla/show_bug.cgi?id=931 ) after getting annoyed at
some of the problems I saw with libxklavier & libxkbfile when it was pulled into
GNOME on Solaris.   (It doesn't cover the biggest one - that libxklavier
completely fails with older versions of libxkbfile that don't contain the
incompatible changes put into XFree86 & Xorg 6.7 and later, but doesn't warn you
in any useful way, an error dialog just pops up on login stating that your X
server configuration is broken, when it's really a libxklavier compatibility
problem.)   I could not find in my investigation any sign that libxkbfile was
ever intended as anything but a private implmentation detail of the X server and
not a public API, but I also couldn't find a definitive statement that said that
either.

[offtopic on]

Although it's almost irrelevant to libxklavier/libxkbfile issues I'd like to 
explain why I'm saying 'language' about layouts.
A layout as a complete thing is a set of symbols enough for ... somethig.  And 
in most cases this something means "typing a text on some language".    There 
are some exceptions such as "US Internatinal" or "Canadian Multilingual" 
layouts that are proposed for typing on many languages.  But such layouts are 
explicitly named 'International', 'Multilingual', etc.  Since even particular 
language symbols can be placed on a keyboard in different ways there are 
national standards for it.  Therefore it would be more correct to refer on a 
layout by some standard or at least a country.  But I think most of end-users 
don't distinguish a country and a language except it really does matter.
I as a user configuring newly installed system always search in the list of 
layouts  "Russian" and I don't care whether it means 'Russian language' 
or 'Russian standard keyboard' or Russians country.
> there may be clients with German localization at the same time there are 
clients running with French. using the same keyboard layout?
If some national layout by accident has all symbols needed for typing on some 
another language, of course it can be used for both.  But anyway for typing 
German you use "a QWERTY layout with precomposed Ö Ä Ü, etc." or "a AZERTY 
layout with dead diaeresis, etc." or even "a QWERTY layout with a lot of dead 
keys suitable for many languages...".  How we should refer to those layouts?  I 
realize that the 'language' word here loosely connected with 'locale language' 
and it can confuse somebody.  But I'm sure a 'language' in most cases properly 
describes a layouts contents.  And I don't see a reason why we cannot refer to 
particular layout in such way except some special cases.

> I am curious why GNOME is trying to control the keyboard layout for X11?

Since it claims to be "an intuitive and attractive desktop for end-users".  Of 
course, instead of trying to solve issues you can just say to end-users: "You 
can choose here a language for messages/menus/titles/etc. but if you also want 
to configure keyboard for typing on this language use some other software you 
managed to find".  But will GNOME still be such attractive? :)

[offtopic off]

> Potentially a user could create these XKM files manually or using a GNOME UI. 

It is almost the same as I suggested.  The 'proprietary addition' we are taking 
about is such tool for preprocessing or precomposing client-side XKM files.  If 
it is absent in other vendors X's it would be better  to investigate a way to 
incorporate it into gnome 'keyboard configuration software' (I realize there 
can be licence issues).  But it allows us don't reinvent a wheel and provide a 
consistence with XFree86/Xorg keyboard layouts/options list. Is not it worth 
efforts?

> OK -- let's be clear. If you compile the files distributed with XF86, they 
will not work with any server except XF86. ... You cannot compile the XF86 
files and get them to work on onther X server implementations.
> The XKM file format is different;

OK.  It was interesting to get to know about it.  But what relations it has to 
discussed issues?
XFree86 clients never nowhere pass XKM files to the server.

The XFree86 clients use two ways.
- The xkbcomp program compiles (on the client side) a complete 'Xkb Keyboard 
Description' from *.xkb files and then loads it into the server using XKB 
standard requests like XkbSetMap, XkbSetCompatMap, XkbSetGeometry, etc.  The 
server receives standard XKB protocol requests and knows nothing how this 
keyboard description was made.
-  The setxkbmap program on the client side composes chains of XKB component 
names and sends them to the server by XKB standard request XkbGetKbdByName.

- All other clients use the same ways (or just call one of these two utilities).

Nothing more. No XKM files are used there.

Of course, the second way is worse because it doesn't use a list of available 
components the server has but uses own (client side) rules.  And it is possible 
that the filenames the client uses are absent on the server or has another 
contents.  BTW this problem can arise even in XFRee86 environment if the client 
and the server have differnt set of *.xkb files.
That is why I suggested to use the first way.

I still don't understand what real issues are behind all talking about XKM 
files. I see some:

- The applet uses the second way and faces the mismatch between the client side 
and the server side 'XKB component' sets.  Moreover, if the server side is not 
XFree/Xorg server we can't simply install our 'component set' onto the server 
side because that server can use absolutely different format of those 
components.  OK.  This issue can be solved using the first way insted 
(compiling Xkb Keyboard Description completely on the client side and sending 
it to the server by standard protocol requests).
- The libxklavier uses some 'proprietary addition', namely 'XKB rules parser'  
that is absent in other vendors libxkbfile.  It is not an issue at all (except 
legal issues).  This is a small set of functions gathered in one file.  I think 
it (or its analogue) can be incorporated directly into libxklavier.  Also it 
works completely on the client side and doesn't influence client-server 
interaction.  There can be only previous 'XKB component set mismatching' 
problem here.
- The applet relies on some root window property that XFree/Xorg servers set 
but none other does it.  OK.  It is solvable.  The applete should only try to 
get that property but use some reasonable defaults in the case of its absence.  
Then the applete can set the property by itself and operate as it does now.

- And finaly, if GNOME is supposed to work in _any_ X11 system, it can happen 
that even on the client side there are not XFree-style XKB files set, XFree's 
xkbcomp and libxkbfile.  (BTW the rest of libxkbfile on the client side is used 
by xkbcomp only.).
I can to suggest two solutions.  The better for users (but worse for 
maintainers) one is to install XFree's xkbcomp and config files set as a 
separate packages on the client side indpendly on what the environment is.  
Anyway GNOME depends on a lot of third party libraries (and probably utilities) 
and we just add two dependences.
And the second solution - the applet should somehow investigate the environment 
(just try to find some files and maybe test their versions) and disabe itself 
if it found the environment insufficient for its tasks.

Is there other problrms I don't see?  Or issues with proposed solutions?

Sergey:

Let's explain how XKB works from the server implementation and how that relates to libxklavier.  Here is a 
description of a keyboard event (the terminology is simplified so it is not technically correct):

1. The OS takes the keyboard input as a scancode. This scancode is proprietary per OS/architecture.

2. The X-server converts this scancode to a keycode.  This conversion is proprietary per server.

3. The keycode is mapped to a keysym via XKB configuration/state machine.  This mapping is proprietary per 
server (but controlled via XKB layout)

The files (/usr/X11R6/lib/X11/xkb) distributed with XF86 describe how to map the proprietary keycode to a 
keysym.  Since the keycodes only apply to XF86 servers, this mapping does not apply to any other servers.  By 
loading a XKB configuration based on the client files, on XF86 you may physically press the key '1' and get the 
keysym 'XK_1', but for any other server implementation '1' might generate 'XK_a' or any other random 
garbage.  This is why it is impossible to use the client files generically, and there is no hope for using them 
correctly in distributed environments  (other than accidentally for XF86).  In addition, XF86/Xorg could change 
their internal keycode remapping at any time (as they are allowed to do), leading to further inconsistencies 
between libxklavier and the X server.

Also, the server may not have the same MLO as the client side, leading to further inconsistencies.

xkbcomp and setxkbmap do use these proprietary files (/usr/X11R6/lib/X11/xkb), but they are distributed 
with the X implementation (X server tree) which, GNOME and libxklavier are not.  Please read the setxkbmap 
man page, the paragraph starting:

>If  you  have  an  Xserver and a client shell running on differnt computers and XKB configuration
>files sets on those machines are different you can get problems  specifying  a  keyboard  map  by
>model,  layout,  options names.

If you accept that using these files is a design problem and should not be there 'at all', then you can start to 
think how you will accomplish whatever taks GNOME/libxalkvier is trying to do without using these files.

One option is to allow the user to create/select XKM files.

Michael. Thanks for the quick introduction into the XKB internals:) I realize
the problem caused by different mapping files and formats on the server and
client sides. I just do not see how XKM files (not portable between X servers!)
could help here. Pure "choose XKM file" approach can not work because users want
to combine options - so the number of files to ship (for which X server??) would
be endless. The "create" function is problematic - at some point people crash
into portability of these files. And, worst of all, there is a stopper problem
mentioned by Ivan - XFree86 does not provide machnery to send XKM file to the
server.

PS Sorry, I did not mention the usability problems - instead of just choosing
two layouts (usually - by the country/language name) user has to work with some
obscure files. Do you mean hiding the details by maintaining the files
transparently, under the hood?

Sergey:

Why is it necessary to model the XF86 configuration files?  That gives you a lot of flexibility, for sure, but I 
would argue that this not a proper/usable abstraction for GNOME.  These files are there to allow the X-server 
to configure itself given a very specific set of parameters.  Most users do not want to see all the parameters for 
a configuration (all the time) and find it much more useful to create abstraction for the details that they use 
often.  This is the same goal of a usable UI.

With this in mind, there are several keyboard layouts that are mostly compatible with major locales.  These 
layouts are being used by the majority of users.  Then there are those who need to configure very special 
keyboards, and they are the exception.   Someone may find it useful to have a keyboard with features 
w+x+y+z, but they would rather choose a single name for that, e.g. 'my favorite keyboard layout', and even 
better, have that configuration in a single XKM file they can transfer between systems (granted, using the same 
X-server).

There is always a portability problem so you can only mitigate the problem by having some controls -- this is 
the problem with modelling something that is not really meant to be modelled.  The problem of keyboards has 
always been a 'portability' problem (a GNOME application is not going to solve this.)  That is yet another 
reason why the implementation of the XKB backend is not in the spec, to not expose applications to this mess?

Using XKM files, most details can be hidden, but the user also has the option to tinker.

What XKB functions does XF86 not support?

> Why is it necessary to model the XF86 configuration files?

Well, I would not call this "modeling". I'd rather consider it as "flexible
mechanics of combination". Even though users really do not want to see tricky
options (that is why there is always some tension around the "Layout Options"
tab) - users still want to have the freedom in combining layouts. You probably
do not know how excited they became when XFree4.3 allowed multiple layouts. Till
that point it was a real nightmare for people who want to have several groups
(or groups not supported int the "official repository"). People do want to
combine layouts. I really see users hanging me if I remove their ability to
combine layouts (I would hang myself for this).

> With this in mind, there are several keyboard layouts that are mostly
compatible with major locales. These layouts are being used by the majority of users

Absolutely not agree. What would you call "major locale"? Is Georgian (Caucasus,
not USA) a major locale? Is Irish a major locale? Would you consider me, who
wants both Russian and American layouts - as some "exception" - (especially if I
want to input euro sign as well)? Why should I bother transferring some obscure
file between systems - if I can just pick all the settings easily in the UI? I
am absolutely confident - this would be unaffordable usability regression for
GNOME. Sorry, but if we are talking in majority-minority terms - we should
remember that most of GNOME users really do use XFree/XOrg servers and have all
supported layouts and flexibility already (except for unfortunate people using
XFree4.2.-).

> Using XKM files, most details can be hidden, but the user also has the option
to tinker.

Command line tinkering is never (AFAIK) considered as a recepie for the GNOME
users. And giving them just a choice of "major layouts" is absolutely
unacceptable usability-wise.

As I said, our fault it that we conidered some things in the reference
implementation as a part of the standard. That is true. Unfortunately, these
things were not standartised - this is not our fault. I am sure they should be -
because otherwise we cannot deliver both standartization and acceptable
usability. What I offer is to think about extension (or addition) to XKB
standard which would cover the MLO machinery. I think freedesktop.org would be
interested in such a standard.

 
The fact is, most users don't care about keyboard configuration. They will configure it once on 
system installation, take the mapping that conforms to the symbols engraved on their 
keyboard, and forget about it. These are the users whose keyboard mappings happen to 
correlate to their POSIX locales. xklavier is not for these users. (The fact that keyboard 
mappings happen to correlate to POSIX locales for many people can be confusing, but this is not 
really a localization problem.) 
 
There are two places to configure a keyboard for X:  the client and server. On the server side, an 
admin normally configures it for the attached keyboard. Perhaps this could eventually be done 
via Project Utopia (especially with hot-pluggable USB keyboards). But for the moment, admins 
expect to configure the keyboard for each X server system. (Actually, they configure the 
keyboard twice:  once for the kernel and once for X. It might be nice if the Xorg server could 
take the kernel's configured keyboard and just use that as the default, but that's another story.) 
Client-side configuration has been traditionally done via xmodmap, but XKB can do this as well. 
 
There are two types of users who want to configure their keyboard:  those who want a simple 
change (e.g. swap the caps lock and control keys) and those who want to type in multiple 
scripts (e.g. Latin and Cyrillic). 
 
For the "simple change" users,  xmodmap is enough. (One bad limitation of xmodmap is that it 
doesn't support ISO_Level3_Shift, so it can't be used to, say, make EuroSign the third level on 
the E-key. I think that could be fixed on the client side.) 
 
For the "multiple scripts" users, xmodmap is unwieldy, and the Level3 limitation becomes quite 
severe. What xklavier tries to do is present the end-user with the same options that the X server 
admin had. 
 
So, xklavier tries to solve both of these problems using XKB's client-side configuration abilities. 
However, it seems that there are some design flaws in how XKB does this. Most of the problems 
seem to be that client tools expect to be able to read server files, and expect them to be the 
same on both sides. This happens to work for most users because most users are in a 
homogeneous environment. But it is not the "X way" of doing things, and isn't portable across X 
servers. Then again, the reference implementation  
 
Moving forward, I think that two things need to happen. First, the Gnome team needs to decide 
that this problem is an important problem to solve. Second, there needs to be changes made at 
the X level so that xklavier can solve this problem properly. 
 
For Gnome in the short-term, xklavier should probably check if the root window property is set, 
and if not, give a nice message that the tool can't be used with the X server. This should 
probably be released in a Gnome 2.8.1 so that users without XFree86 don't get ugly error 
messages on startup or when starting the control center. 
 
In the longer term, I think that we need to think about the different ways this problem can be 
solved properly. (And what "properly" means here.) 
 
Requirements: 
client-only configuration of X server keyboard that is consistent with the underlying window 
system. What are the UI requirements? 
 
Possible Solutions: 
XKM files 
XKB files 
xmodmap (with repaired Level3 abilities) 
XKB models/layouts/options (MLO) 
XKB keycodes/types/symbols/geometry/compat (KTSGC) 
a new XKB2 extension 
others? 
 
XKM files 
  - not portable from one X server to another 
 
XKB files 
  - problems when redefining keycodes? 
   
xmodmap (with repaired Level3 abilities) 
  - requires changes to xmodmap 
  - may not deal be able to affect LEDs (e.g. changing Caps Lock key to do Shift Lock and have 
the Caps Lock LED reflect the state of Shift Lock) 
  - how portable are xmodmap keysyms and keycodes? 
   
XKB models/layouts/options (MLO) 
  - need to get a list of available MLO options and localized descriptions 
  + perhaps xklavier could come with it's own rules/ directory and parse the rules on the client 
side. Then it could check with the X server if particular KTSGCs are available (with 
XkbListComponents()), and only present the MLO options that the X server can do. This would 
also deal with the localization problem. It wouldn't be perfect if the server has wildly different 
maps defined. 
   
XKB keycodes/types/symbols/geometry/compat (KTSGC) 
  - exposed too much complexity to the user. 
   
a new XKB2 extension 
  - would only be available for Xorg/XFree86 servers, at least to start (but that's the case now, 
too) 
 
other possible solutions? 

Frank: Generally, I am agree. The only detail is that libxklavier is NOT a part
of gnome. So even when I release libxkavier 1.1 - it will have no direct
relation with any gnome release (as long as API is stable). Also, the "MLO"
approach has one bad "-" which started this thread - it is not required by XKB
implementation to support this model at all - it is non-standard. And even
shipping rules with libxklavier would not help much - because, as you said, we
can never guarantee that consistency of the KTSGC files between servers. I do
not even mention that our rules would most-probably build based on single-group
symbols (like in XFree 4.3+) which cannot be guaranteed, can it?

XKB2 extension (I'd rather call it XKBC - XKB configuration - something in
addition to XKB) could at least guarantee us API functions (like XkbRF_* etc)
which are now just "implementation details" of XFree/XOrg server. Or - better -
allow to query the server regarding available MLO components (which we can not
do at all now - that is why we have to play with the client-side files).

So, I would propose to get together and think about extending XKB - IMHO this is
the only way to resolve our problems without losing in usability and standard
compliance.

Would it be possible in the interim, to release a version of libxkavier, that
would detect that the XServer was not XFree86 or perhaps that the root window
property was not set, and thus not provide the annoying error message, that
makes an otherwise excellent and professional system, look like it has some
serious bug.

mark: I put already the code for the "default" root window property value. I
would really appreciate if you could test it from CVS and report your result. I
am going to release new version soon. But this fix still does not improve the
situation if you want to have several layouts. In any case, XkbGetKeyboardByName
function should be able to process the parameters you chosen in the
gnome-keyboard-properties. If this does not work (you chosen some unsupported
config) - you'll still get this message.

Guys:

While I respect the goal to standardize keyboard layouts for all
systems (for the sake of easier keyboard configuration), there are
problems that make that goal a little impractical.

The XKB extension is meant to insulate applications from the details
of keyboard setup.  This is because the keyboard setup is a difficult
problem. If you are designing a portable system, the XKB authors
correctly realized that abstracting all keyboards and the systems they
operate with is not tractable.

The problem they were faced with, and the problem we still have today
is how to use individual keyboard descriptions and apply them to all
OS and all (possible) X server implementations?  Since the beginning
of X, it was realized that the keycode to keysym mapping had to be
proprietary, and that is why we still have the notion of a keycode and
an arbitrary mapping to a keysym.

There are several technical problems in trying to map a OS dependent
scancode to a fixed keycode representation that would be common among
all X servers on all OSs.  If this were possible, all servers could
share configuration files even in a distributed environment.

One example of a technical problem with this approach is that the
keyboard scancode ranges exceed the X11 keycode ranges, therefore not
all keys on a keyboard can be portably represented in a single X
server keymap.  Basically an X server implementation is forced to come
up with an 'arbitrary' mapping for the OS provided scancodes, and
those scancodes are not the same for every OS implementation.

Therefore, to 'realistically' deal with the problem of standardizing
X11 would probably require yet a lower level POSIX standard to
standardize all keyboard scancodes.  It would be tempting to treat
XF86/Linux as a reference for a standard w.r.t. keyboards (but
designing applications to a specific OS was one of the problems
plaguing fragmented UNIXes in the past.)

Now the above is an analysis of why the XKB extension avoids specific
issues when it comes to keyboard layout and keycodes.  But, a keyboard
layout could also be configured with a 'different' abstract principle
such as a symbolic representation of a keyboard that is maintained on
the server side that does not involve keycodes?

Having an abstract representation is also a portability problem.  How
can a client 'guarantee' that X server 'A' has the same keyboard
layout database as X server 'B'?  And if A & B did (accidentally)
share a layout with the same 'name', would the keyboard behavior be
allowed to be different between the 2 implementations?  This approach
might work well in controlled situations but not in a desktop
environment intended for a general non-homogenous environments?

So, XKB does provide a complex and high level approach to keyboard
layout configuration.  By using the XKB API defined in the spec a
client can configure a keyboard based on abstractions that apply to
many if not all systems that have keyboards.  However, the XKB
approach is difficult to use because of the complexity of the
abstraction (for end users).  Hence XKB is not perfect, but yes, it is
trying to forcefully avoid other problems.

If we agree that a keyboard is quite specific to a system then
keyboard configuration is specific to the system.  There are very
general layouts and locales, but that is not what libxklavier (and the
GNOME control center) is dealing with -- it trying to deal with system
level files/configuration for specific keyboards.

This is why using specific, user created XKMs is appealing. XKMs can
be created for a specific system, but they contain a state that can be
used on other systems (in the absence of other files).  The user has
the flexibility to use the XKM on server implementations that support
it (does XF86?)

XKMs can be created from the XF86/Xorg configuration files with a UI
or by other means.  I do not understand why it would be too difficult
for a user to create an XKM and how this would (substantially) be
different from the current keyboard selection mechanism.  It is more
work, though.

I am glad to see more discussion on possible solutions since the above
solution is only example.

Sergey: I do not know where you get the idea that you have to encode
every possible XF86 layout as an XKM file.  Making an XKM file
could be done on the basis of individual layouts the user selects.

On another note: I am a little anxious? that libxklavier is being
distributed as a library.  Other applications that link with
libxklavier might assume that the abstractions in the library apply to
X11 in general.

Michael: I think I realyze why scancodes are not standartized. I even agree
(absolutely!) that generally it would not make sense to standartize the list of
layouts. But that was not my idea. I offered to standartize the means for the
client to obtain the list of available layouts (and find out the active one),
the way X server processes layout rules etc - provide some protocol extension
for querying the config DB (yeah, that's what libxklavier is doing by reading
local files - which is only correct for the local servers). Sure, we cannot
expect each implementation to support the same set of MLO config items (though
with xkeyboard-config project this could be achieved, at least with the most
popular PC hardware).

Now, back to XKM files. I still see no substantial difference between
client-side xkbcomp invocation (Ivan's idea) and XKM files - other than you are
trying to persist (for some unknown reason) the result of the XKM file
compilation. From the usability POV, user should NEVER not know about these
files anyway - he just should pick up the layouts he needs, the system should
transparently do "something" to actualize his idea. Why whould we need xkbcomp
to save these files? The Ivan's idea, as I see it, would be:

1. Get locally available MLO rules

2. Convert MLO options (selected by user) to the low-level component names

3. Using the local repository (as opposite to the current approach, where the
remote repository is used - which can lead to inconsistency), do xkbcomp and
pass the result to the server.

In your approach (with building XKM files on the fly), 

1. User has to get the list of available "options" (I am not sure whether you
are talking about MLO or other, new? level of abstraction). How? Do you want to
show the list of low-level component names (from the ListComponents function)?

2. Build XKM file (user should not know about this detail of the process, it
should be under the hood) - which effectively means calling xkbcomp with output
to the file. The building procedure should provide "multiple layouts" feature
anyway.

3. Loads this XKM file to the server.

So, I do not understand what makes the second approach better.

Well, libxklavier was intended to be a library. I intentionally separated
gnome-independent XKB-related functions into it (sorry, they happened to be
specific to XFree-derived implementations in some details). Sure, I will try to
make it as XFree-free as I can - but only where it does not hit the abstraction
level.

 
The goal here is not to standardize keyboard layouts for all systems. Rather it is to enable users 
to make limited configuration changes from an X client. I don't think this is an impossible task. 
 
You say that "the XKB extension is meant to insulate applications from the details 
of keyboard setup."  That's true for all cases except for when an X client application wants to 
set up the keyboard. Then, the client needs to pierce that insulation. One way is the free-for-all 
xmodmap. xklavier is trying to present common configuration options using XKB. 
 
You talked a bit about scancodes, keycodes, and keysyms. Just to define these, scancodes are 
OS-level numbers generated when a key is pressed or release. In Linux, these can be seen using 
`showkey -s`. X keycodes are also numbers, but different from scancodes. For XFree86, the X 
keycodes can be seen in keycodes/xfree86. The keysyms are 4-character strings like TLDE or 
AE01 that are used within X. 
 
At least for XFree86, the mapping between scancode and keycode is magic and hardcoded. (In 
xf86PostKbdEvent() in 
http://cvsweb.xfree86.org/cvsweb/xc/programs/Xserver/hw/xfree86/common/xf86Events.c, I 
believe.) This is surely proprietary, and this mapping changes for each OS platform. Plus there's 
the problem you mentioned, where there can be more scancodes than X keycodes. 
 
(In terms of keyboard scancode standards, there are a couple that are widely used:  XT set 2 
and USB. XT set 2 is used under Windows, and Linux uses it, too. But I don't think we want (or 
can) change anything at this low level. We should concentrate 
on the XKB-level.) 
 
The mapping between X keycode and keysym is not necessarily proprietary, but I don't think it 
was ever standardized, so I imagine there are many, many different versions. But I do believe 
that the X keysyms were standardized, but I'm not positive. 
 
An X server which supports XKB can send a client a list of available KTSGC components, so even 
if a server uses some totally different configuration method internally, it can send this list. Even 
if the list is very restricted, xklavier can still find out what server-side options it can use (and so 
display to the user). 
 
I agree that the set of X server configuration files (in /etc/X11/xkb for XFree86) 
should be specific to that systems keyboard, OS, and X server. But an X client should be able to 
choose from what's available. The 
only way an X client should know what keyboards can be configured is via 
XkbListComponents() which provides the KTSGC list. It is not possible to know what 
the 'us' symbols file contains, but we can assume that it will be for an 
American keyboard. If that assumption fails, the keyboard will be 
misconfigured, so there needs to be a "fallback after 20 seconds" or something. 
 
XKMs: 
I don't totally understand your proposal for XKMs, so let me guess, and you tell me where I'm 
wrong. Make a GUI that has access to the X server config files, and let the user pick (via MLO or 
KSTGC) a configuration. Then run xkbcomp (hidden from the user) to generate an xkm file. Store 
this xkm file in the home directory, and send it to the X server on each login. 
 
Also, XKMs are not a very good solution because they are not portable among X server 
implementations. Plus, I think that they store the keycode to keysym maps, which we agree are 
proprietary, and so non-portable? 
 
New extension: XKBcfg 
As for thoughts on a new XKBcfg extension, I want to make sure we're talking about the same 
thing. Writing a new X extension means writing a library that would be integrated into an X 
server. In XFree86, it would be activated by adding a 'Load "XKBcfg"' line to the XF86Config-4 
file. The original XKB spec is over 200 pages long, so writing a new extension would at least 
have to cover that. Plus, it would be non-standard, so anything using it would not be supported 
on other platforms. 
 
I think we could gain the API stability just by having a separate, client-side library that would 
transform MLOs into KSTGCs. This new lib could take what is currently done in the XFree86 or 
Xorg servers, and split it into a library that could be used either by an X server (for system-wide 
configuration) or by an X client (for per-user configuration). 
 
I would see an API like getXKBcomponents(rulesfile, available_KSTGCs, model, layout, option) 
that would be used to get the KSTGCs for a particular MLO, or like getXKBcomponents(rulesfile, 
available_KSTGCs, &models, &layouts, &options) that would return a list of valid models, 
layouts, and options given the rules file and available KSTGCs. This would be to populate the 
GUI with good MOLs. 

Frank:
> The keysyms are 4-character strings like TLDE or AE01 that are used within X. 

No.  Keysyms are names of symbols like 'Agrave', 'Cyrillic_E' or 'comma'.
The things you mention are 'keycode names'.  Something like MS virtual-key codes
but used at keymap compilation time only (not in events or key mapping tables).

But these names could be standartized. Some of them are 'key positions' others
are 'key physical labels'.  If it was done it would allow to avoid the problem
with proprietary scan-codes to keycodes mapping.  It is such intermediate level
that allows to separate hardware- , OS- and Xserver-specific mapping from
keysyms (symbols) maps that in this case become fully portable.

Ideally the 'keycodes' component only should be derived from the Xserever
components database but all other parts can be get from anywhere.

But such separation can't be done in existent XKB.  Now we can send to the
server names of all components and the sever gets them from its database or we
can send to the server a complete keymap but in this case we can't get the
keycodes (keycodes to their names mapping) table from the server.

Argh. Sorry. I think I get this wrong every time I try. The terms are just too
similar, and there's one translation too many. Let me try again.

scancodes:  OS-level numbers given for each key press and release. On Linux,
they can be seen with the showkey -s command. They are usually given in hex:  e0 d1

keycodes: X server specific numeric codes for each key. Ususally given in
decimal. XFree86 translates the scancodes to keycodes internally. The conversion
is hard-coded in
http://cvsweb.xfree86.org/cvsweb/xc/programs/Xserver/hw/xfree86/common/xf86Events.c
in Xf86PostKbdEvent(). Listed on the right-hand side in keycodes/xfree86.

keynames: 4-char strings describing either position or labels (e.g. <TLDE> or
<AE00>). Used during keyboard compilation. Listed on the left-hand side in
keycodes/xfree86 and on the right-hand side in the symbols/* files.

keysyms: strings (or potentially Unicode hex values) that describe the symbol
(e.g. 'Agrave', 'Cyrillic_E' or 'comma').

I always forget the difference between scancodes and keycodes because XFree86
converts these on the fly, and there is no file listing the conversion.

But like you said, because the keynames are not standardized, we can't rely on
the keycodes being mapped to the right keynames.

It's amazing that you can send a complete keymap, but there's no way to get an
existing keymap. It really ties our hands here.

Sergey;

> XKMs: I don't totally understand your proposal for XKMs, so let me
> guess, and you tell me where I'm wrong. Make a GUI that has access
> to the X server config files, and let the user pick (via MLO or
> KSTGC) a configuration. Then run xkbcomp (hidden from the user) to
> generate an xkm file. Store this xkm file in the home directory, and
> send it to the X server on each login.

There is a big difference between having a single file that contains a
'complete' keyboard state (based on XKB abstractions) and an symbolic
abstraction of specific keyboard states (MLO/KSTGC).  Remember that
XKB is an abstract/complex state machine.

The difference is that the symbolic abstraction does not allow all the
possibilities that the XKM file allows and the XKB extension supports.
In other words, by having these symbolic names/abstractions for
keyboard (and forcing the user to use the symbolic abstractions), the
user does not have the option to completely configure the keyboard.
They would have to use the server/system provided abstraction even
when they can/could/should be able to configure whatever is possible
in the state machine.  xmodmap/XKM does not have this restriction.

While MLO/KSTGC may be detailed databases, they obviously do not
include all keyboard layouts.  Therefore modelling libxklavier only
with these abstractions prevents the user from creating their own
keyboard layouts (potentiall with XKM).

XKM files can be created in a variety of ways, and do not require you
to 'take along'/consider ancilliary system files to reproduce the
keyboard state on another system.  The file format is portable, but
the contents will not be portable across systems.  However, it is
always a given that keyboard configuration (at this level) will not be
portable so there is always a risk that the UI/user takes.

OK. I got the difference. But this difference is not substantial here at all
IMHO - and I can explain way. Indeed, MLO abstraction does not cover ALL the
configuration variants users can get with KSTGC (I do not even mention the
flexibility of XKM files). But I never heard any complains that current set of
config items in XFree/Xorg is not flexible enough (on MLO level - as gnome works
with it). There are complains that we show too many options - and I am fighting
with the oversimplification - but our current experience shows that showing all
MLO items gives _just_enough_ flexibility - users do not even ask ability to
tweak KSTGC. If we turn current g-k-p into "XKM editor + XKM transmitter" thing
(and hide the fact of XKM file existence from user - this is definitely) - it
will still work on some level of abstraction close to MLO (because we would not
expose too much details, for usability sake). So what would be the benefit with
this approach? There should not be an option "Load custom XKM", I am pretty sure
- users should not even know there are some "files" involved. If all above is
reasonable, then Ivan's approach seems simpler and cheaper - with the same result.

 
I think that the idea of a user taking along an XKM or XKB file file from system to system would 
be considered an "expert" thing. And it could be done using xkbcomp directly. So an expert user 
could add a line in .xinitrc like this: 
 
xkbcomp custom.xkb $DISPLAY 
 
This of course doesn't deal with the problem of merging the server keycodes with the new map, 
but something could be done by first getting the server's xkb, by doing `xkbcomp -xkb 
$DISPLAY server.xkb` then doing a merge, then uploading the file. 
 
But this is really an expert requirement. Joe User will just want to pick a nice customization off a 
list. 

Michael: One more question. It seems (from mark's letters) AcceleratedX does
implement XkbRF* functions - but the implementation is empty - they do not work
properly (do not return sensible results). Why do you include them? Why could
not you use the reference implementation approach - if you used it as a basis
anyway? Is there any special reason?

Michael: Sorry, I just checked - it is not AccX which delivers these functions,
it seems it was leftover from xfree.. So, would it be possible to ship
libxkbfile with your server? The X11 licence allows this, you know...

*** Bug 144159 has been marked as a duplicate of this bug. ***

svu: can we close this?

Given that the discussion is about libxklavier and not anything GNOME related, I'll close this.

I guess this is fair