GNOME Bugzilla – Bug 789756
Color picking interface
Last modified: 2021-07-05 14:25:06 UTC
Hi, With the move to Wayland, color picking stopped working in GNOME Shell (/mutter) for a variety of applications such as GIMP[1] and my own, gcolor3[2]. Through reaching out by one of my users to the gnome-apps-devel mailing list it was suggested I open this bug report to discuss the addition of a color picking interface to GNOME Shell. The DBus interface suggested fits my needs perfectly (with the addition of coordinates for the pixel to pick): ``` org.gnome.Shell.ColorPicker pick(x, y) → (success: boolean, color: ai) ``` where the `pick(x, y)` method would tell the compositor to initiate the selection of the color at pixel `(x,y)`. The color return value would contain an array of 3 integers, with the values of the RGB channels. I am not sure if compositor-specific interfaces such as the one suggested are the best solution: ideally, there would be one such protocol in Wayland upstream that would then be implemented (or not) by Wayland compositors. This would make it much easier for general applications such as GIMP and my own to work in all compositors, without having to special-case different implementations. Since both approaches are new to me, I decided to open this bug report in the hopes to discuss both approaches and decide on the best one. If the outcome is that a protocol to Wayland is favorable, I will look into how this is done and write a patch. Likewise, I can also look into writing the patch for GNOME Shell if it is determined that this is the best approach. Thanks, Jente [1]: https://bugzilla.gnome.org/show_bug.cgi?id=780375 [2]: https://github.com/Hjdskes/gcolor3/issues/38
We should consider trying to expose 'sandbox:able' APIs only, i.e. use org.freedesktop.portal.*. Also I don't think it should to be Wayland protocol, as it has nothing to do with Wayland, also as it doesn't fit well with how portals work, as we can't have a portal process acting as a proxy. FWIW, portals work not only for sandboxed applications, but for non-sandboxed as well, which I think is a nice bonus. Then, whether the screenshot API is "good enough" is another question. It does make the color picking a bit more awkward, as one first needs to share the screenshot, then do the picking.
About the API itself, (x, y) wouldn't be available on Wayland, as all pointer positions are relative to the surface, and the global surface position not known. You also need to take pixel format into consideration. For example 3 integers between 0 and 255 does not work well if the format of the desktop is RGB10.
> We should consider trying to expose 'sandbox:able' APIs only, i.e. use org.freedesktop.portal.*. Are you saying this shouldn't be part of GNOME Shell but rather be implemented by a portal? Is there somewhere I can read more about these portals? I am not sure I follow "also as it doesn't fit well with how portals work, as we can't have a portal process acting as a proxy.". > Also I don't think it should to be Wayland protocol, as it has nothing to do with Wayland. It has something to do with Wayland insofar that picking colors works on X11 but not on Wayland. I understand why, but it is a feature regression that perhaps should be a new protocol in wayland-protocols to bring this more in-line with X11. > About the API itself I might have been a bit too forward with a specific API example -- let's first focus on the correct way to implement this and then discuss an API.
(In reply to Jente Hidskes from comment #3) > > We should consider trying to expose 'sandbox:able' APIs only, i.e. use org.freedesktop.portal.*. > > Are you saying this shouldn't be part of GNOME Shell but rather be > implemented by a portal? Is there somewhere I can read more about these > portals? I am not sure I follow "also as it doesn't fit well with how > portals work, as we can't have a portal process acting as a proxy.". With sandboxing systems like flatpak, a way for applications to get access to the outside world, for example to get screenshots and read/write files in arbitrary places on the filesystem, is to expose a portal API via D-Bus under org.freedesktop.portal providing the needed functionality. A portal backend will implement the functionality in a certain way depending on what desktop environment it runs under outside of the sandbox (could be using a private mutter/gnome-shell API for example). The portal will take care of access control for the certain I/O the portal provides, and provides a UI if needed. See https://github.com/flatpak/xdg-desktop-portal/ for more info. > > > Also I don't think it should to be Wayland protocol, as it has nothing to do with Wayland. > > It has something to do with Wayland insofar that picking colors works on X11 > but not on Wayland. I understand why, but it is a feature regression that > perhaps should be a new protocol in wayland-protocols to bring this more > in-line with X11. Sure, but it is more of a general IPC issue than a Wayland vs X11 issue.
Hi! I don't understand. Why would we want to have 2 APIs, one for Wayland and one as a sandbox portal?! This just makes things more complicated since we have to implement every API. Why not just have a single API which works as a portal in a flatpak (or other sandbox) and which also works in non-sandbox environment. > pick(x, y) → (success: boolean, color: ai) Also for the API itself, I guess we'd need in particular a "SelectPoint" API (similar to the "SelectArea") which will return coordinates of the cursor with user interaction (change of pointer and click). Because with this pick(x, y) API, the problem is: how did we get the x and y data (since as I understand, an application don't have access to screen-relative coordinates)?! Finally in GIMP, we have this "Sample average" feature, which basically returned an average color in a square centered around the cursor. This is only used for our color picker tool, which works only within the opened images so Wayland is not a problem there. Our generic picker (which works anywhere on the screen) does not have such a feature. Yet this is quite useful and I could imagine we'd may be interested into implementing it at some point. So an additional "circumradius" parameter to both pick and SelectPoint may be useful to obtain a matrix of pixels.
(In reply to Jehan from comment #5) > Hi! > > I don't understand. Why would we want to have 2 APIs, one for Wayland and > one as a sandbox portal?! This just makes things more complicated since we > have to implement every API. Why not just have a single API which works as a > portal in a flatpak (or other sandbox) and which also works in non-sandbox > environment. Right. And I think it'd be best to add that as a portal, assuming taking a screenshot via the screenshot portal is not enough. Portals should work just fine from outside a sandbox. > > > pick(x, y) → (success: boolean, color: ai) > > Also for the API itself, I guess we'd need in particular a "SelectPoint" API > (similar to the "SelectArea") which will return coordinates of the cursor > with user interaction (change of pointer and click). > Because with this pick(x, y) API, the problem is: how did we get the x and y > data (since as I understand, an application don't have access to > screen-relative coordinates)?! I think the API should rather be: pick(options) -> color (ignoring error handling), where the portal is responsible of changing the cursor to the color-picking-cursor and actually doing the color picking. The application would just get the color back, without knowing where it came from. > > Finally in GIMP, we have this "Sample average" feature, which basically > returned an average color in a square centered around the cursor. This is > only used for our color picker tool, which works only within the opened > images so Wayland is not a problem there. Our generic picker (which works > anywhere on the screen) does not have such a feature. Yet this is quite > useful and I could imagine we'd may be interested into implementing it at > some point. So an additional "circumradius" parameter to both pick and > SelectPoint may be useful to obtain a matrix of pixels. I suppose one could make the API expose the ability to grab the colors around the cursor, like a mini screenshot thing.
Right, so we implement this functionality in a portal. That sounds good to me, assuming these portals are implemented in other desktops/compositors as well. I suppose that is another matter, though, and not the focus of this bug report. > assuming taking a screenshot via the screenshot portal is not enough. My preference would be that this complexity is hidden in the portal such that not every client that makes use of this API has to implement this logic itself. > I think the API should rather be: pick(options) -> color (ignoring error handling), where the portal is responsible of changing the cursor to the color-picking-cursor and actually doing the color picking. The application would just get the color back, without knowing where it came from. This sounds like it's the way to go, and has my vote. We can easily extend this API with other methods (such as the proposed "mini screenshot"), if required.
Also please, the color pick interface needs to be color-managed. This is very important for graphics software. GIMP in particular now is fully color-managed and these features need to be as well to provide accurate color representation. See: https://bugzilla.gnome.org/show_bug.cgi?id=779942#c5
GNOME is going to shut down bugzilla.gnome.org in favor of gitlab.gnome.org. As part of that, we are mass-closing older open tickets in bugzilla.gnome.org which have not seen updates for a longer time (resources are unfortunately quite limited so not every ticket can get handled). If you can still reproduce the situation described in this ticket in a recent and supported software version, then please follow https://wiki.gnome.org/GettingInTouch/BugReportingGuidelines and create a new ticket at https://gitlab.gnome.org/GNOME/gnome-shell/-/issues/ Thank you for your understanding and your help.