GNOME Bugzilla – Bug 745410
Adding named pipe abstraction
Last modified: 2018-05-24 17:32:50 UTC
At NICE we've been working on a named pipe abstraction that works in a similar way to GSocketService/Client. I'd like to know if there is an interest on getting this into glib. Right now the current code lives in: https://github.com/nice-software/nice-mit/blob/master/dcv/dcv-win32-pipe-service.h https://github.com/nice-software/nice-mit/blob/master/dcv/dcv-win32-pipe-service.c https://github.com/nice-software/nice-mit/blob/master/dcv/dcv-win32-pipe-client.h https://github.com/nice-software/nice-mit/blob/master/dcv/dcv-win32-pipe-client.c If there is some interest let me know, so we can relicense the code to be LGPL and to fix the issues on the API and whatever that you propose us to change.
spice-gtk also has a gio named-pipe implementation (LGPL): http://cgit.freedesktop.org/spice/spice-gtk/tree/gtk/controller/namedpipe.h http://cgit.freedesktop.org/spice/spice-gtk/tree/gtk/controller/namedpipe.c http://cgit.freedesktop.org/spice/spice-gtk/tree/gtk/controller/namedpipeconnection.h http://cgit.freedesktop.org/spice/spice-gtk/tree/gtk/controller/namedpipeconnection.c http://cgit.freedesktop.org/spice/spice-gtk/tree/gtk/controller/namedpipelistener.h http://cgit.freedesktop.org/spice/spice-gtk/tree/gtk/controller/namedpipelistener.c
well, it sounds like it passes the "at least two users of the API" test
We definitely want this. It makes me think that maybe we got the class hierarchy a bit with GSocketService -- would have been nice to add other arbitrary types of listener objects into that and get the advantages of (for example) GThreadedSocketService. I also think that it's definitely interesting to use this as the basis for GApplication uniqueness (based on a pipe named by app id). In that case, we could speak some modified form of DBus over the pipe so that all the usual things (action invocations, etc.) still worked properly. Eventually we may end up able to make this look very 'natural' from the standpoint of the user: on the primary side they have a single GDBusConnection object that can speak to any connected client, and on the remote side they have a GDBusConnection which is connected to the primary as if it were the message bus.
This implementation makes heavy use of threads (one thread per pipe name). This may or may not be necessary. Also, g(unix|win32)iostream (which is what this API will provide) is blocking and not selectable/pollable, as far as i can see. I don't appreciate things like this. It's here for a reason, but i don't get it, so don't ask my opinion.
(In reply to LRN from comment #4) > This implementation makes heavy use of threads (one thread per pipe name). > This may or may not be necessary. No threads in spice implementation. > Also, g(unix|win32)iostream (which is what this API will provide) is > blocking and not selectable/pollable, as far as i can see. g_win32_{input,output}_stream is not pollable atm, however the spice namedpipe object as implicit requirement on "overlapped" namedpipe, so it should be used non-blocking.
I've only read the NICE implementation, and my previous comment only covers that implementation. I have no desire to read the Spice implementation, so i'll just take your word for it. Overlappiness is great (more great if XP support is dropped, as AFAIR post-XP Windowses have APIs for enabling overlappiness on existing handles), but stream I/O API exposed by the base stream I/O classes does not seem to have any provisions for polling or async operations - in fact, IIRC, GUnixInputStream implements read() as a loop that polls the fd and reads it - i.e. it restricts itself to a blocking interface, even though the fd supports asynchronous I/O. You could probably grab the handle from the stream I/O object and then use it as you wish, but that kind of kills the point of high-level glibby API. So while less-thready implementation is good, the whole API around GIOStream seems to be incompatible with polling (meaning g_poll), which means it's incompatible with main loop (meaning GMainLoop), which means it's incompatible with GUI applications (meaning GTK). Well, if you add more threads and more code, you can turn blocking I/O into asynchronous I/O, but often that "more threads and more code" is, in fact, a lot of threads and a lot of code.
(In reply to LRN from comment #6) > So while less-thready implementation is good, the whole API around GIOStream > seems to be incompatible with polling (meaning g_poll), which means it's > incompatible with main loop (meaning GMainLoop), which means it's > incompatible with GUI applications (meaning GTK). Well, if you add more > threads and more code, you can turn blocking I/O into asynchronous I/O, but > often that "more threads and more code" is, in fact, a lot of threads and a > lot of code. It has been a while I haven't looked at this code, refreshing my memories. The win32 streams are GIO stream "via threads": {read,write}_async() use a thread to call the sync read/write. It's not pollable, yet you can do non-blocking IO. There isn't more code needed than what is already provided by glib base class. Isn't this compatible with GUI app for you? However, the spice NP listener doesn't use threads.
GInputStream supports async I/O, it just does it using a callback-based API rather than a polling-based API; you call g_input_stream_read_async(), passing a buffer and a callback, and then the callback gets invoked after the data has been read into the buffer. IIRC, it was designed this way in part because Windows doesn't generally provide polling-based APIs, and there's no way you could implement a polling-based API at the gio level if you only have a callback-based API underneath. (Whereas the reverse--implementing a callback-based API on top of an underlying polling-based API--is easy.) But anyway, later on GPollableInputStream/GPollableOutputStream were added, which allow you to do polling-style I/O, but those interfaces are only implemented by certain stream types (such as GUnixInput/OutputStream, but not GWin32Input/OutputStream).
Well, for an I/O-intensive application callbacks-implemented-as-threads-that-do-blocking-I/O mean that you need one thread per I/O primitive. I've looked at GInputStream, and you're right, while the OS-specifig GWin32InputStream only implements synchronous I/O, GInputStream implements high-level async I/O on top of it by running a sync I/O call inside a GTask (a thread, i presume). I guess this eliminates the "lots of code" problem, as glib already does this for you (still quite a bit of code and abstraction layers, but it's glib's problem, not user's), leaving the "lots of threads" in place. There's also the fact that glib main loop is based on g_poll, which on Windows can only poll up to 64 objects. How do I/O threads notify the main (loop) thread when they've completed an operation, which in turn invokes the callback? If each thread uses its own pollable proxy object (an event, a semaphore, etc) for that, this will quickly exhaust the 64 objects limit. If there's a single shared object, then it's not so bad (lots of threads is still kind of bad, but not immediately fatal).
I'm not so worry about whether we will use threads on the implementation but about the api right now. One thing to have into account is that GSimpleIOStream was merged in this cycle in glib so it is something we can use to have something similar to GSocketConnection, which is what NICE's implememntation currently does. Clearly we went too far down on the implementation and made the service directly. Probably we should take a step back and make something more similar to the structure of GSocketListener and then implement the service on top of it and maybe have a Threaded version as well. This makes me wonder if we should have an interface implemented by GSocket variants and by the Pipe ones. Thoughts?
Created attachment 322228 [details] [review] Add named pipe high level api It provides a GSocketListener/Client like api using named pipes
So this is a first version of the api based on both the spice and the nice implementations. Here are some questions though: - should we create a PipeConnection class instead of using gsimpleiostream? This way one could detect what kind of connection is being used. - should we use a namedpipe object instead of using the a pipe name string on the api? IMHO it is not really needed. How should we proceed? My next step will be to add unit tests for it.
(In reply to Ignacio Casal Quinteiro (nacho) from comment #12) > So this is a first version of the api based on both the spice and the nice > implementations. Here are some questions though: > - should we create a PipeConnection class instead of using gsimpleiostream? > This way one could detect what kind of connection is being used. Yes, and provide additional pipe-related method & properties. Imho it's neater to wrap low-level np HANDLE into a namedpipe GObject the way its done with GSocket and the way I did in spice-gtk/controller. > - should we use a namedpipe object instead of using the a pipe name string > on the api? IMHO it is not really needed. Imho, it's needed if you want to customize your pipe, or if you get a pipe handle from a different windows API. This is full of hardcoded properties that might be incompatible with your needs: 171 PIPE_ACCESS_DUPLEX | 172 FILE_FLAG_OVERLAPPED, 173 PIPE_TYPE_BYTE | 174 PIPE_READMODE_BYTE | 175 PIPE_WAIT, 176 PIPE_UNLIMITED_INSTANCES, 177 DEFAULT_PIPE_BUF_SIZE, 178 DEFAULT_PIPE_BUF_SIZE, > How should we proceed? My next step will be to add unit tests for it. Yes, unit tests would be welcome. Anyway thanks for your work
(In reply to Marc-Andre Lureau from comment #13) > (In reply to Ignacio Casal Quinteiro (nacho) from comment #12) > > So this is a first version of the api based on both the spice and the nice > > - should we use a namedpipe object instead of using the a pipe name string > > on the api? IMHO it is not really needed. > > Imho, it's needed if you want to customize your pipe, or if you get a pipe > handle from a different windows API. This is full of hardcoded properties > that might be incompatible with your needs: > I think we can keep the API leaner and avoid the extra object. What we can add (now or even later if needed) is a g_win32_named_pipe_listener_add_handle method: when you need full control you can use the low level windows api to create the handle and then hand it to the listener.
A more generic question is: do we call this stuff GWin32NamedPipe or do we call it GWin32Pipe? Do these object make sense with anonymous pipes?
Created attachment 322250 [details] [review] Add named pipe high level api It provides a GSocketListener/Client like api using named pipes
This is a new version adding a GWin32NamedPipeConnection.
Created attachment 322335 [details] [review] Add named pipe high level api It provides a GSocketListener/Client like api using named pipes
This already includes some tests which seem to pass properly. The code can also be found in the branch wip/namedpipes in case somebody wants to try it out.
Review of attachment 322335 [details] [review]: The patch looks good to me now. I also take back my question about whether this should be called generacally "GWin32Pipe" since anon pipe cannot do overlapped i/o.
This code now lives in https://git.gnome.org/browse/wing
-- GitLab Migration Automatic Message -- This bug has been migrated to GNOME's GitLab instance and has been closed from further activity. You can subscribe and participate further through the new bug through this link to our GitLab instance: https://gitlab.gnome.org/GNOME/glib/issues/999.