GNOME Bugzilla – Bug 126266
g_io_channel_read_line and nonblocking IO
Last modified: 2018-05-24 10:30:18 UTC
We are trying to read lines from a nonblocking stream and we are encountering 100% CPU time usage. We created a watch using g_io_channel_create_watch(G_IO_IN) and then attached a read callback which calls g_io_channel_read_line. When a line without newline is present in the GIOChannel's buffer the poll loop maxes out the processor until a complete line is available. The problem seems to be that g_io_unix_prepare assumes readability when _any_ characters are present in the buffer and then g_io_channel_read_line returns G_IO_STATUS_AGAIN as no complete lines are available. The solution would be a new option to GIOChannel (or Watch) that we want line based polling.
Can you provide a) a testcase and b) an api proposal ?
*** Bug 321053 has been marked as a duplicate of this bug. ***
Hm... I am being affected by this problem as well :-/
Zenity is hitting this problem too; see bug 541001. Supporting line-based polling is one option. However, an alternative that is easier to implement and potentially more general is to recognize a third mode of return from a GIOFunc: "there's nothing more I can do with the input in the buffer, so don't call me again until new input is available from the fd". When a GIOFunc returns this way, a flag is set in the GIOUnixWatch that causes the g_io_unix_* functions to ignore the buffer condition; this flag is cleared when g_io_unix_check finds input on the fd. (I'm less sure of how to implement this for Windows.) The question is how to indicate the third mode. We shouldn't change the return type of GIOFunc since that would break existing code, and neither should we introduce a new type GIOFunc2 and a corresponding function g_io_add_watch2 because the meaning of g_source_set_callback would become unclear. As ugly as it might be, I think the approach least likely to break stuff is to add a function g_io_watch_set_buffer_exhausted(GSource *, gboolean) that sets the flag. (The capability to reset the flag may be useful in case some other event changes a watch's behavior in such a way that it may be able to make progress with the data currently in the buffer.) Unfortunately, the GSource is not passed to GIOFuncs, so they would have to get it through the data argument. An alternative would be to keep the flag on the channel (which is passed to GIOFuncs) instead of the source; that might break when there are multiple watches on the same channel, but I can't think of a good example where multiple watches on the same channel would be useful.
Created attachment 113804 [details] [review] Draft of proposed API from comment #4 Here's one way to add the API. The patch does work (I made a modified zenity based on it that handled partial lines correctly), but it could use some serious review.
Created attachment 113805 [details] [review] Modified zenity for testing proposed API Here, for the purpose of demonstrating and testing the proposed API, is a patch that produces the modified zenity I mentioned. To test, run the following in the zenity/src dir, making sure LD_LIBRARY_PATH is set if necessary to use the modified glib: (echo -n 'Hello'; sleep 15; echo ' world!') | ./zenity --list --column=Message "Hello world!" should appear in the list box after 15 seconds. During the 15-second wait, zenity should be responsive and should not use 100% CPU, and clicking "Cancel" should close the dialog immediately. Unmodified zenity uses 100% CPU and won't quit before the end of the 15 seconds no matter how many times "Cancel" is clicked.
The patch looks sane to me, and seems to solve the problem. As I would again try to use GIOChannel in a project which needs to read lines from a non-blocking channels I'd try to use this patch, if its fate is to be included in mainline. Any chance we could get a review by one of the glib developers? Thanks.
There is also another situation in which this problem arises. When encoding is in place and there is an incomplete multi byte sequence in the GIOChannel buffer, i.e. you get the error G_CONVERT_ERROR_PARTIAL_INPUT when reading, the callback will be called continously as the buffer is not empty while you would like to wait for more data (to complete the sequence).
Hello, For the zenity case (bug #541001), I was able to fix the error, using the g_usleep in the while. Modify the tree.c line 131: do { status = g_io_channel_read_line_string (channel, string, NULL, &error); while (gtk_events_pending ()) gtk_main_iteration (); g_usleep(10000); } while (status == G_IO_STATUS_AGAIN); With this, zenity doesn't use 100% cpu. I'm not sure if this is the best way to handle this, so I'm asking here if I can commit this little workaround on Zenity or if it will impact in glib in some way.
-- GitLab Migration Automatic Message -- This bug has been migrated to GNOME's GitLab instance and has been closed from further activity. You can subscribe and participate further through the new bug through this link to our GitLab instance: https://gitlab.gnome.org/GNOME/glib/issues/15.