GNOME Bugzilla – Bug 784199
Add screen cast and remote desktop support
Last modified: 2019-02-06 23:53:12 UTC
These patches indirectly add screen sharing support to mutter. The main patch (Add remote desktop and screencast functionality) introduces two new D-Bus APIs: org.gnome.Mutter.ScreenCast - org.gnome.Mutter.ScreenCast.Session - org.gnome.Mutter.ScreenCast.Stream and org.gnome.Mutter.RemoteDesktop - org.gnome.Mutter.RemoteDesktop.Session In short, the screen cast interfaces are used to create PipeWire[0][1] streams of desktop content, while the remote desktop interfaces are used to inject input events for remote controlling. These two APIs together with an external tool can be used to implement screen sharing and remote desktop application. One such application is "gnome-remote-desktop", currently residing at <https://github.com/jadahl/gnome-remote-desktop>. See individual patches for more details.
Created attachment 354472 [details] [review] ClutterVirtualInputDevice: Allow passing CLUTTER_CURRENT_TIME If CLUTTER_CURRENT_TIME is passed, let the backend find an appropriate time stamp representing the current time in the clock that is used by that backend.
Created attachment 354473 [details] [review] clutter/evdev: Move scroll notification to ClutterSeatEvdev This refactors the scroll event emitting code, moving it behind a clutter_seat_evdev_ helper.
Created attachment 354474 [details] [review] ClutterVirtualInputDevice: Add discrete scrolling Add emitting of discrete scrolling events (such as mouse scroll wheel-like events).
Created attachment 354475 [details] [review] clutter/stage: Add capture_into API Add API similar to clutter_stage_capture() but that draws into externally allocated memory. It is assumed that the pixel format is ARGB32, and the memory is structured in a way that the width of the passed rectangle is identical to the stride divided by 4.
Created attachment 354476 [details] [review] wayland/xdg-foreign: Move out ID generation helper to util.c It'll be used to generate ID in the same way in other places later.
Created attachment 354477 [details] [review] monitor-manager: Add helper for getting monitor from connector This will be used later.
Created attachment 354478 [details] [review] monitor: Add API to get regenerated monitor from old instance When monitors changed, previous monitor instances are defunct, and any reference holder should drop its reference. Sometimes they will want to continue having a reference to the same monitor, so add this function to make it possible to find it. Currently the output and crtc references are invalid, as they are not yet reference counted, so this can only look at cached fields.
Created attachment 354479 [details] [review] Add MetaFraction for dealing with fractions Add MetaFraction, which consists of two integers, the numerator an the denominator. The utility function to convert a double to a MetaFraction comes from gstreamer.
Created attachment 354480 [details] [review] Add remote desktop and screencast functionality This commit adds basic screen casting and remote desktoping functionalty. This works by exposing two D-Bus API services: org.gnome.Mutter.ScreenCast and org.gnome.Mutter.RemoteDesktop. The remote desktop API is used to create remote desktop sessions. For each session, a D-Bus object is created, and an application can manage the session by sending messages to the session object. A remote desktop session the user to emit input events using the D-Bus methods on the session object. To get framebuffer content, the application should create an associated screen cast session. The screen cast API is used to create screen cast sessions. One can so far either create stand-alone screen cast sessions, or a screen cast session associated with a remote desktop session. A remote desktop associated screen cast session is managed by the remote desktop session. So far only remote desktop managed screen cast sessions are implemented. Each screen cast session may have one or more streams. A screen cast stream is a stream of buffers of some part of the compositor content. So far API exists for creating streams of monitors and windows, but only monitor streams are implemented. When a screen cast session is started, the one PipeWire stream is created for each screen cast stream created for the session. When this has happened, a PipeWireStreamAdded signal is emitted on the stream object, passing a unique identifier. The application may use this identifier to find the associated stream being advertised by the PipeWire daemon. The remote desktop and screen cast functionality must be explicitly be enabled at ./configure time by passing --enable-remote-desktop to ./configure. Doing this will build both screen cast and remote desktop support. To actually enable the screen casting and remote desktop, the user must enable the experimental feature. See org.gnome.mutter.experimental-features.
Created attachment 354481 [details] [review] screen-cast: Handle PipeWire errors When the PipeWire context or stream ends up in an error state, signal that the source has closed. This then triggers the stream and finally the session to be closed too.
Created attachment 354482 [details] [review] screen-cast-stream-src: Only try to record frames when streaming Only when the PipeWire stream state is 'PW_STREAM_STATE STREAMING' should the signal be connected causing the src to maybe record a frame.
Missed to include the link to PipeWire: https://cgit.freedesktop.org/~wtay/pipewire To try out the patches, you must first install PipeWire and pass --enable-remote-desktop to mutters ./configure.
Created attachment 354609 [details] [review] ScreenCast: Pass PipeWire stream node ID directly As of commit 5f5ef3de2cdc816dab82cb7eb5d7171bee0ad2c5 in pipewire the stream creator can find out the node ID of the stream it created. So instead of using a special purpose entry to the info property box to let the application discover stream by monitoring added nodes searching for the given special purpose entry, just pass the node directly.
Leaving open, as it's still disabled by default. Attachment 354472 [details] pushed as 1e7628a - ClutterVirtualInputDevice: Allow passing CLUTTER_CURRENT_TIME Attachment 354473 [details] pushed as d3c559a - clutter/evdev: Move scroll notification to ClutterSeatEvdev Attachment 354474 [details] pushed as a10ad57 - ClutterVirtualInputDevice: Add discrete scrolling Attachment 354475 [details] pushed as 12710dc - clutter/stage: Add capture_into API Attachment 354476 [details] pushed as 34e7134 - wayland/xdg-foreign: Move out ID generation helper to util.c Attachment 354477 [details] pushed as 3005a40 - monitor-manager: Add helper for getting monitor from connector Attachment 354478 [details] pushed as 3ce3a5a - monitor: Add API to compare with old instance Attachment 354479 [details] pushed as 38235bc - Add MetaFraction for dealing with fractions Attachment 354480 [details] pushed as 97f2c7c - Add remote desktop and screencast functionality - none Attachment 354481 [details] pushed as 9d89227 - screen-cast: Handle PipeWire errors Attachment 354482 [details] pushed as 920541f - screen-cast-stream-src: Only try to record frames when streaming Attachment 354609 [details] pushed as 53175e8 - ScreenCast: Pass PipeWire stream node ID directly
Created attachment 359154 [details] [review] screen-cast-stream: Don't broadcast PipeWireStreamAdded signal The helper function from gdbus-codegen broadcasts the signal emission, but we really only care about sending it to the specific peer that created the session. Thus, only emit the signal to the particular peer that owns the session.
Review of attachment 359154 [details] [review]: Would it make sense for the session to expose something like meta_screen_cast_session_emit_pipewire_stream_added (session, object_path, node_id) instead of connection and peer-name?
(In reply to Florian Müllner from comment #16) > Review of attachment 359154 [details] [review] [review]: > > Would it make sense for the session to expose something like > > meta_screen_cast_session_emit_pipewire_stream_added (session, object_path, > node_id) > > instead of connection and peer-name? I don't think so. "PipeWireStreamAdded" is a signal on the MetaScreenCastStream object/interface, which is implemented in meta-screen-cast-stream.c.
Any progress with this issue?
(In reply to Jonas Ådahl from comment #17) > (In reply to Florian Müllner from comment #16) > > Review of attachment 359154 [details] [review] [review] [review]: > > > > Would it make sense for the session to expose something like > > > > meta_screen_cast_session_emit_pipewire_stream_added (session, object_path, > > node_id) > > > > instead of connection and peer-name? > > I don't think so. "PipeWireStreamAdded" is a signal on the > MetaScreenCastStream object/interface, which is implemented in > meta-screen-cast-stream.c. It has been a year since the last comment by Jonas above, has there been an movement/progress on this? Google Hangouts are used a lot in the private and commercial sectors and this is a total show stopper. It would be nice to see that this issue has some traction and is moving.
(In reply to David Dreggors from comment #19) > It has been a year since the last comment by Jonas above, has there been an > movement/progress on this? See comment #14 - the feature itself has landed (since 3.26) but is still disabled by default. There is some movement to get browsers to support the API, so that hangouts and co. start working on wayland (but that's outside the scope of this bug).
Browsers etc are not really using (and shouldn't use) this API, they are using org.freedesktop.portal.ScreenCast/RemoteDesktop. While it's disabled by default build time, it's enabled by default if built in now with 3.30. I think we can close this bug actually, but I'll open a MR for the left over patch.