Our detection of buttons > 3 is not too reliable currently... could you provide the output of WAYLAND_DEBUG=1 nautilus (or ephy) while you press those buttons? preferably on weston, I think we are also figuring out the wrong evcode for the pressed button on mutter on those buttons...

Ok, I ran
WAYLAND_DEBUG=1 nautilus > logfile 2&>1
and filtered the log file to match only .button() events.

mouse buttons I used:
left-click (two double-clicks. It seems like nautilus didn't get both of them completely because weston's terminal interfered for a very short time)
scroll up (I don't think this counts as button)
scroll down
back button
forward button
right-click
middle-click (twice I think, not sure about that)
left-click (to close the window)

[3469212,377] wl_pointer@3.button(525, 67512481, 272, 0)
[3469299,341] wl_pointer@3.button(530, 67512587, 272, 1)
[3469393,563] wl_pointer@3.button(531, 67512677, 272, 0)
[3471709,076] wl_pointer@3.button(534, 67514996, 272, 1)
[3471783,917] wl_pointer@3.button(535, 67515072, 272, 0)
[3471896,873] wl_pointer@3.button(538, 67515178, 272, 1)
[3471978,416] wl_pointer@3.button(539, 67515264, 272, 0)
[3484457,411] wl_pointer@3.button(540, 67527737, 275, 1)
[3484670,222] wl_pointer@3.button(541, 67527959, 275, 0)
[3486778,118] wl_pointer@3.button(542, 67530052, 276, 1)
[3486995,055] wl_pointer@3.button(543, 67530284, 276, 0)
[3489604,895] wl_pointer@3.button(546, 67532893, 273, 1)
[3489777,779] wl_pointer@3.button(547, 67533053, 273, 0)
[3491571,269] wl_pointer@3.button(548, 67534860, 274, 1)
[3491729,145] wl_pointer@3.button(549, 67535018, 274, 0)
[3491853,457] wl_pointer@3.button(550, 67535142, 274, 1)
[3492027,900] wl_pointer@3.button(551, 67535312, 274, 0)
[3493984,365] wl_pointer@3.button(554, 67537273, 272, 1)
[3494086,095] wl_pointer@3.button(555, 67537375, 272, 0)

I ran some more tests (reversed button press order) and it looks like this is the correct matching:
272 left (primary)
273 right (secondary)
274 middle (tertiary, scroll wheel press)
275 back
276 forward

left, right and middle mouse buttons work fine as you wrote above. back and forward don't work in weston either. My mouse doesn't have more buttons that work on a computer.

Oh and by the way the behavior is identical with or weston and with gnome-shell+mutter.

(In reply to Christian Stadelmann from comment #2)
> Ok, I ran
> WAYLAND_DEBUG=1 nautilus > logfile 2&>1
> and filtered the log file to match only .button() events.
> 
> mouse buttons I used:
> left-click (two double-clicks. It seems like nautilus didn't get both of
> them completely because weston's terminal interfered for a very short time)
> scroll up (I don't think this counts as button)
> scroll down
> back button
> forward button
> right-click
> middle-click (twice I think, not sure about that)
> left-click (to close the window)
> 
> [3469212,377] wl_pointer@3.button(525, 67512481, 272, 0)
> [3469299,341] wl_pointer@3.button(530, 67512587, 272, 1)
> [3469393,563] wl_pointer@3.button(531, 67512677, 272, 0)
> [3471709,076] wl_pointer@3.button(534, 67514996, 272, 1)
> [3471783,917] wl_pointer@3.button(535, 67515072, 272, 0)
> [3471896,873] wl_pointer@3.button(538, 67515178, 272, 1)
> [3471978,416] wl_pointer@3.button(539, 67515264, 272, 0)
> [3484457,411] wl_pointer@3.button(540, 67527737, 275, 1)
> [3484670,222] wl_pointer@3.button(541, 67527959, 275, 0)
> [3486778,118] wl_pointer@3.button(542, 67530052, 276, 1)
> [3486995,055] wl_pointer@3.button(543, 67530284, 276, 0)
> [3489604,895] wl_pointer@3.button(546, 67532893, 273, 1)
> [3489777,779] wl_pointer@3.button(547, 67533053, 273, 0)
> [3491571,269] wl_pointer@3.button(548, 67534860, 274, 1)
> [3491729,145] wl_pointer@3.button(549, 67535018, 274, 0)
> [3491853,457] wl_pointer@3.button(550, 67535142, 274, 1)
> [3492027,900] wl_pointer@3.button(551, 67535312, 274, 0)
> [3493984,365] wl_pointer@3.button(554, 67537273, 272, 1)
> [3494086,095] wl_pointer@3.button(555, 67537375, 272, 0)

Thanks! that's the info I was looking for.

> 
> I ran some more tests (reversed button press order) and it looks like this
> is the correct matching:
> 272 left (primary)
> 273 right (secondary)
> 274 middle (tertiary, scroll wheel press)
> 275 back
> 276 forward

Ok, that goes as expected with left/right/middle, however the hexadecimal of those back/forward codes correspond to:

#define BTN_SIDE                0x113
#define BTN_EXTRA               0x114

in /usr/include/linux/input.h, I would have expected those to be:

#define BTN_FORWARD             0x115
#define BTN_BACK                0x116

defined right after.

I think there's 3 bugs here:

1) For other mouse buttons than 1/2/3, clutter uses the following to determine the 1-indexed button used in ClutterEvents:

    button_nr = evcode - BTN_LEFT + 1;

That gives us buttons 4 and 5 for your back/forward buttons. IMO this is wrong in itself as 4-7 were usually reserved for scroll buttons, I don't think we can ignore that and remain compatible.

2) Mutter takes those ClutterEventButton events with 1-indexed buttons, and transforms those back into evcodes in order to pass these in wl_pointer events, the conversion goes:

    evcode = button_nr + BTN_LEFT - 1;

This is effectively the inverse of what Clutter is doing, so it seems one bug defeats each other (and explains that you get the same results on weston and gnome-shell). However, this seems really flaky, IMO it would be preferable to get rid of this double conversion and pass the evcodes around directly.

3) On the gtk+ app side, we again do:

    button_nr = evcode - BTN_LEFT + 1

Which again gives us buttons 4-5, and is again wrong.

In X11, back/forward buttons get pushed after the regular 4-7 scrolling buttons, so those get 8-9 instead. I'm still unsure though about which mapping would get BTN_FORWARD/BACK on X11, although AFAIK 8-9 is common for back/forward buttons, so I tend to think that (oddly) BTN_SIDE/EXTRA are more common in mice.

I'll leave this bug for the gtk+ side and open new ones to Clutter/Mutter.

I totally forgot there are scroll buttons. My mouse has two of them (mouse to the right, mouse to the left). I tried them (both in gnome-shell and weston) and they are not recognized as wl_pointer@3.button() event. Instead they show up as 

[3357558,566] wl_pointer@3.axis(73135468, 1, -10,000000)
[3357558,719] wl_pointer@3.axis(73135468, 1, -10,000000)
[…]
[3359141,661] wl_pointer@3.axis(73137051, 1, 10,000000)
[3359141,751] wl_pointer@3.axis(73137051, 1, 10,000000)

I pressed the scroll wheel to the left and then to the right. Pressing the wheel left or right is no scroll operation obviously. Is this behavior intended? Is it just some old "we always did it that way" setting? "Normal" Scroll operations look like this instead:

[3502459,180] wl_pointer@8.axis(73280365, 1, -10,000000)
[3502459,308] wl_pointer@8.axis(73280365, 1, -10,000000)
[…]
[3675916,860] wl_pointer@8.axis(73453826, 0, 10,000000)
[3675917,001] wl_pointer@8.axis(73453826, 0, 10,000000)

Button codes 8 and 9 are what nautilus usually has set in org.gnome.nautilus.preferences.mouse-back-button (8) and org.gnome.nautilus.preferences.mouse-forward-button (9) using dconf/GSettings. This is correct for my setup under X11.

Created attachment 315761 [details] [review]
wayland: Move additional pointer buttons after the old 4-7 scrolling ones

We were using that range for the extra buttons after left/right/middle,
while this is harmless for clients not handling extra buttons (we
used to translate those button events into scroll events in x11 anyway)
this will be unexpected for clients that do handle additional mouse
buttons themselves (eg. back/forward buttons present in some mice).

In order to remain compatible with X11, those need to be assigned from
button 8 onwards.

Also, include input.h, and stop using magic numbers here.

I've filed the following bugs for the Clutter/Mutter bits:

https://bugzilla.gnome.org/show_bug.cgi?id=758237
https://bugzilla.gnome.org/show_bug.cgi?id=758238
https://bugzilla.gnome.org/show_bug.cgi?id=758239

Review of attachment 315761 [details] [review]:

Looks fine to me.

Attachment 315761 [details] pushed as 4c49c0a - wayland: Move additional pointer buttons after the old 4-7 scrolling ones