Created attachment 32891 [details]
Thread with Jeff Garzik

Here's a conversation I had with Jeff Garzik on this subject and
what would be involved in doing this.

On the priority problem:
epoll(4) says that epoll descriptors can be added to other epoll sets.
Maybe, a cascaded scheme can be implemented where each active priority level gets an epoll set for its fds, and the master epoll set contains epoll descriptors for the levels. A blocking epoll_wait is called on the master epoll descriptor, then the triggered level sets are processed with non-blocking epoll_wait accordingly to their priority. This will have acceptable performance when few priority levels are in effect, which was Oven's concern. Bizarre priority schemes will still work, too, at the cost of extra non-blocking system calls. When there is only one effective priority, the cascade can be bypassed altogether.

Sorry Owen, I mistyped your name.

Hmm. What might work well if possible, is use poll() to poll on
a set of one epoll descriptor per priority, in the case of
one priority, that could be optimized to a blocking epoll_wait().

Of course, the whole bug really does depend on someone having a GLib-using
app with enough file descriptors for epoll()/poll() to matter.

(In reply to comment #4)
> Hmm. What might work well if possible, is use poll() to poll on
> a set of one epoll descriptor per priority,

Choice between poll or epoll_wait on the master set is not principal. I'd prefer using the same call every time. Also, epoll_wait allows for easier descriptor set structures.

Unfortunately, GMainContext exposes poll-like semantics; at least g_main_context_set_poll_func/g_main_context_get_poll_func will have to be deprecated if epoll is implemented internally.

> Of course, the whole bug really does depend on someone having a GLib-using
> app with enough file descriptors for epoll()/poll() to matter.

Count me in: I want to do the unthinkable and drag GLib onto a c10k-ish server.
With GSlice, thread pools and other niceties, it starts to look attractive for that niche, too.
So, will you be interested if I submit a patch?

Created attachment 56316 [details]
Feasibility test for cascaded epoll

Here's a small test that demonstrates how epoll works in the proposed two-layered scheme. To put it short: an epoll over other epoll descriptors reports "read" events on those epolls that got any events to retrieve.

I'm going to implement GPollSet, an object that abstracts the polling mechanism preferable for the platform (this may include kqueue as well some day). It will also implement priority cutoff similarly to how g_main_context_query operates.
Then, GMainContext can be modified to use GPollSet internally.

Created attachment 56990 [details] [review]
Initial GPollSet implementation

Here is a patch that adds GPollSet, a waitable poll set abstraction, to GLib, and doesn't change anything else. It borrows heavily from GMainContext code and is currently implemented with epoll, poll, select, or Win32 wait functions, whichever is available and the most efficient. Instead of poll(2)-style event arrays, GPollSet uses callbacks to notify event consumers. While it can be shoehorned into existing g_main_context_query()/g_main_context_check() functions, doing so may actually hurt the main loop performance. Note that gratuitous add/remove/max priority change operations should be avoided because they may result in system calls being made.
The GPollSet operations are not MT-safe except the reference counting.

Known problems:
1. Behavior of g_pollset_wait() on an empty set is not implemented consistently; implementations rely on the underlying wait function's handling of an empty set. It should probably be specified and implemented so that g_pollset_wait() is equivalent to sleeping for the specified timeout (even infinite?) if the effective set is empty.
2. Return status of g_pollset_wait() should probably be more informative to be able to indicate timeouts in a distinct way from errors or event arrival.
3. The Win32 implementation is a heap of untested code that I didn't even try to build. However, it is thought out as well as the other implementations.
4. The test may lack coverage for a few corner cases; I wasn't especially thorough there.

Further ideas for GPollSet:

- Implement g_pollset_modify for efficient modification of a descriptor event mask, which may be useful to temporarily block/unblock attached sources. This function might be callable from inside GPollNotify callbacks, which is currently prohibited for add/remove operations.

- Add a "one shot" event flag that would translate to EPOLLONESHOT for epoll and generally have a descriptor automatically disabled (its event mask set to 0) when an event is reported for it. g_pollset_modify should be used to enable the descriptor back.

- Separation of epoll- and non-epoll implementations is made squeaky clean for the initial code. In real life (e.g. distribution packages), epoll may turn out to be unavailable at runtime. So, the epoll code should probably be blended with the basic POSIX implementation so that when epoll_create returns ENOSYS, it falls back to the poll-based implementation.

*** Bug 386350 has been marked as a duplicate of this bug. ***

You could take a look at GstPoll[0], it abstracts poll() and similar things in a platform independent way.

[0] http://gstreamer.freedesktop.org/data/doc/gstreamer/head/gstreamer/html/gstreamer-GstPoll.html

adding myself to cc.

An idea update for anyone who might have time and motivation to implement this:
I think the priority-based poll set management is more trouble than it's worth; it only has a negligible runtime cost with traditional poll/select because the poll set has to be resubmitted for each poll iteration anyway. All descriptors should be uniformly polled in an epoll-friendly replacement for GMainContext, making the priorities only meaningful for sources triggered besides the file descriptor polling.

It looks like we may get some epoll love after all:
http://mail.gnome.org/archives/gtk-devel-list/2011-August/msg00059.html

See #bug 699132 for a more ambitious proposal, which should support epoll as well as any other event loop backends generally conforming to the same model.

An experimental epoll backend is implemented as part of the work on bug #699132:

https://github.com/mzabaluev/glib/commits/pluggable-mainloop

(In reply to comment #16)
> An experimental epoll backend is implemented as part of the work on bug
> #699132:
> 
> https://github.com/mzabaluev/glib/commits/pluggable-mainloop

And now for new and improved:

https://github.com/mzabaluev/glib/commits/pollers

Since this covers much of the same ground as bug #699132, let’s close it as a duplicate.

*** This bug has been marked as a duplicate of bug 699132 ***