We see the same problem on our Sun Blade machines; the Num Lock
key also does not work.  I also reproduced the problem on Linux
by installing a version of libgdk-x11-2.0.so without XKB support.

The problem seems to be the result of a rewrite of the relevant
code for translating a key event to a keysym, in response to
bug 93575 (dealing with the Compose key not working).  That
change replaced the call to XLookupString() with new code which
does its own key translation.

I tried enabling the XKEYBOARD extension in the Solaris X
server; that got the Caps Lock working, but Num Lock was still
broken.

*** Bug 117590 has been marked as a duplicate of this bug. ***

Created attachment 18834 [details] [review]
Address several problems with Num Lock and Caps Lock handling

I found several problems, and fixed them in the attached patch:

    * translate_keysym() is not checking whether the Num Lock
      modifier is set.

    * update_keymaps() is failing to determine the keysym for
      the Lock modifier, which is used to distinguish between
      Shift Lock and Caps Lock.  The problem is that the Lock
      modifiers are the second set in the modifier map, but
      the existing code skips past the first three sets.

    * In the Caps Lock case, translate_keysym() checks for the
      Shift modifier being on instead of Lock.

In addition, I found that the Sun X server does Num Lock handling
in a non-standard way.  Normally, the keysym to use for a keypad
key when Num Lock is on is the second element in the array for that
keycode; however, in the Sun implementation, it is the third element.
(I verified that the version of XLookupString() in the Openwindows
X library functions accordingly).  As far as I can tell, the only
feasible way to deal with this is to hack the code to determine
whether it is dealing with a Sun X server; thus the patch uses
ServerVendor() to determine which index to use for Num Lock.

Note that my patch does not address a problem with Num Lock
handling on Sun when the XKEYBOARD extension is used.  I imagine
this is related to the aforementioned non-standard implementation
in the Sun X server.  However, I found that XLookupString() has
the same problem in this case, so I saw no compelling reason to
address this.  (The extension is apparently disabled by default).

Some comments on the patch:

* I think it would read better if you simply converted
  the loop to go over all 8 modifiers and not skip
  the first three.

  Looking at the logic, it probably could be done as:

  switch (mask)
    {
    case GDK_LOCK_MASK:
      /* find lock keysym */
      break;
    case GDK_CONTROL_MASK:
    case GDK_SHIFT_MASK:
    case GDK_MOD1_MASK:
      /* Comment about extra mod1 modifier */
      break;
    default:
      /* Search for Mode_Switch/Num_Lock */
      break;
    }

* The comment:

+                  /* If any Lock keysym is Caps_Lock, we will interpret
+                   * Lock as Caps_Lock; otherwise, if any is Shift_Lock,
+                   * we will interpret Lock as Shift_Lock.
+                   */

   Isn't clear to me. If the code needs to be commented,
   the correct would be something like:

    /* If any keysym bound to this modifier is Caps_Lock,
     * we use that as the interpretation. Otherwise we
     * use Shift_Lock as the interpretation.
     */
   
   (See usage elsewhere for the fact that Shift_Lock is the 
    default. If that matches the Xlib spec, we probably
    should make it more explicit in the search for the
    lock symbol. If it doesn't we should fix it.)

* It would be nice if the code for handling non-existant
  keysyms you introduced:

+       shift_level = ((state & shift_modifiers) ||
+                      !SYM (keymap_x11, group,
keymap_x11->num_lock_index))
+         ? 0 : keymap_x11->num_lock_index;

  matched the existing code in the next block:

      /* Fall back to the the first level if no symbol for the level
       * we were passed.
       */
      shift_level = (state & shift_modifiers) ? 1 : 0;
      if (!SYM (keymap_x11, group, shift_level) && SYM (keymap_x11,
group, 0))
        shift_level = 0;

* There is something fishy about the Sun stuff - by definition
  there are two shift levels per group in a class X keyboard,
  see e.g.:

#define KEYSYM_INDEX(keymap_impl, group, level) \
  (2 * ((group) % (keymap_impl->keysyms_per_keycode / 2)) + (level))

  that doesn't work if you have more levels than that.

  I was going to ask you for xmodmap -pk output, but 
  bug 79184 already has that. (I express some puzzlement
  over this issue there, but it turned out to be an entirely
  different problem.)
 
  Cc'ing a couple of Sun people on the bug - perhaps someone
  could take a look at the Sun Xlib sources and:

  A) See what the algorithm there is for handling the 
     NumLock key is, since it clearly is not the same
     as in section 12.7 of the Xlib manual.
 
  B) See if there is any attempt to handle non-Sun X keymaps,
     and if so, how does it tell the difference.

  (I know Rich was recently dealing with a problem with  
  the numeric keypad and GCalcTool, not sure if it was
  the same as this.)

The problem I had with gcalctool was bug #117691. The cursor 
keys on my Solaris 9 Update 3 Ultra 60 weren't generating
GDK_Up, GDK_Down, GDK_Left, GDK_Right.

I note that using the 072203 GNOME build from Sun Ireland,
that this problem is now fixed.

You mean "the gcalctool problem is now fixed"?

Sorry, I should have been more specific. The RealFix(TM)
for the gcalctool problem was as described in the comments
for bug #117691. This fix unfortunately didn't work for
me on my Solaris box until I upgraded to the 0722 build 
of GNOME 2.4 from Sun Ireland (I was using a GNOME 2.2 + 
latest AT build before).

Created attachment 18970 [details] [review]
Revised patch to fix num lock and caps lock problems

I attached a new patch which addresses the issues raised:

    * I changed the modifier loop to go over all 8 modifiers,
      as suggested.

    * I revised the comment about the Lock keysym, but I don't
      see that the "usage elsewhere" is for Shift_Lock to be the
      default.  As far as I can tell, the default is NoSymbol,
      i.e. ignore Lock.  I modified the code to be a bit clearer,
      and to ensure that, if present, Caps_Lock trumps Shift_Lock.
      This seems to be the algorithm used in the standard Xlib,
      as well as the intention of the existing (albeit unused) code
      in update_keymaps().  (Section 12.7 of the Scheifler/Gettys
      manual implies that there are no strict requirements at all
      about Lock, default or otherwise).

    * I made the code for handling non-existent keysyms in the
      Num_Lock case match the existing code in the next block
      (except for the requisite difference in logic).

    * As for the Sun hack...their implementation is certainly
      non-standard, but it is not clear that it is wrong, since
      key translation seems to be loosely defined.  In any case,
      we have access to the Sun Xlib sources, which confirmed the
      behavior we observed empirically.  The only check they seem
      to do is whether the third element in the array is a keypad
      key; whether to use the second or third element is determined
      at build time.

      I revised the hack as follows:

      1) Don't apply Mode_Switch for a keypad key on Sun -- this
         matches the Sun behavior.

      2) Ensure that the keysyms array contains at least 3 elements.

      I don't see a good alternative way to make Num_Lock work on
      Sun.

One correction to the patch:

+      if (server_vendor &&
+          (strcmp (server_vendor, "Sun Microsystems, Inc.") == 0) &&
+          (keymap_x11->keysyms_per_keycode > 4))
+        keymap_x11->num_lock_index = 2;

The third line above should read:

+          (keymap_x11->keysyms_per_keycode > 2))

Since we won't apply Mode_Switch to a keypad key, there is no
need to require more than 3 keysyms for Num_Lock.  (The patch
works as written because the Sun server returns 5 keysyms per
keycode).

Applied with some cosmetic changes:

 - Used a boolean keypad_x11->sun_keypad instead of 
   keypad_x11->num_lock_index since it's not just the index that 
   changes.

 - Simplified the logic in the loop:

                  for (j = 0; j < keymap_x11->keysyms_per_keycode; j++)
                    {
                      if (syms[j] == GDK_Caps_Lock)

   by figuring that the loop was short enough it wasn't 
   worth trying to bypass it.

 - Don't check for NULL ServerVendor; it can't be NULL,
   and we've been assuming it is non-NULL in GTK+ for years,
   so there aren't Xlib/Xserver's that violate that.

Mon Aug 25 16:15:41 2003  Owen Taylor  <otaylor@redhat.com>
                                                                     
                                                      
        * gdk/x11/gdkkeys-x11.c: Fix up CapsLock and
        NumLock handling for non-XKB. Add a special case
        hack for NumLock on Sun servers. (Patch from
        Robert Basch, #115819)

Created attachment 19500 [details] [review]
Patch as applied