GNOME Bugzilla – Bug 309117
filesink: non-fatal out of disk space handling
Last modified: 2018-11-03 12:12:16 UTC
What would be really nice to have in filesink (and gnomevfssink as well probably if possible) is non-fatal out-of-disk-space handling. File sink could fire an out-of-diskspace signal and put the buffer it wanted to write in an internal queue. The application can pause the pipeline at this point and show a warning to the user. Once the user has freed some space, the application can continue doing what it was doing before, without a single byte lost. This would be a very neat function to have for (non-realtime) encoders, transcoders, sound-juicer IMHO. Cheers -Tim
You don't even need a pause. Way better, you can attach to the signal handler, and show a dialog in the callback (possible with a GCond to stream back into the main thread). When the signal handler returns, the user has clicked yes or no, and some stuff may have changed. If you provide no filename, but a GnomeVFSHandle ("handle" property), you can just set a new handle and you'll be fine already.
*** Bug 318673 has been marked as a duplicate of this bug. ***
Additionally, it would be nice to pre-set the amount of disk space the sink element may use. I propose adding a property specifying the maximum file size in bytes. When the specified amount of bytes has been written, the same out-of-diskspace signal can be fired.
Created attachment 104486 [details] [review] out-of-space signal and max-size property Add "out-of-space" signal handler to filesink. Add "max-size" property to filesink to signal "out-of-space" when the specified amount of bytes have been written.
Created attachment 104491 [details] [review] out-of-space signal and max-size property Add "out-of-space" signal handler to filesink. Add "max-size" property to filesink to signal "out-of-space" when the specified amount of bytes have been written. This patch differs from the previous in that the signal handler should return TRUE if it wants the operation retried.
Created attachment 104545 [details] [review] out-of-space signal and max-size property Add "out-of-space" signal handler to filesink. Add "max-size" property to filesink to signal "out-of-space" when the specified amount of bytes have been written.
The previous patch did not work because GST_FLOW_RESEND doesn't work. I have this working in a real-life scenario. I register an out-of-space handler that * sets the max-size property to -1 (no more limits), * stops my mainloop, * returns TRUE to indicate that the operation should be retried. When the mainloop stops, in the main thread, I send an EOS signal on the bus and restart the mainloop. This allows me to cleanly finish processing of a stream until I reach a certain file threshold. The same mechanism can be used for integration with a GUI that can stop or pause a pipeline when the system is out of disk space.
IMHO this is a bit awkward API-wise. We should avoid signals from the streaming thread wherever possible, even more so complicated things where the app has to handle blocking in the callback. Couldn't we just post an element message of some sorts on the bus (if requested, e.g. via setting a property) and block the writing thread on a cond and then have the app do something (e.g. via g_signal_emit (sink, "continue-writing", ..) or an interface+method) to make the writing thread unblock and retry? Not quite as elegant as the signal since the app would actively need to tell us that it wants the new behaviour, but easier for apps to implement.
Created attachment 104548 [details] [review] out-of-space signal and max-size property Add "out-of-space" signal handler to filesink. Add "max-size" property to filesink to signal "out-of-space" when the specified amount of bytes have been written. There was a bug in the previous patch; the return type of the signal handler was still declared as void.
Created attachment 126735 [details] [review] Warning before disk is full. The other patches fall into warn when the disk is full and give the application a chance to do something, or notify when X bytes have been written. In my case I need to know before the disk is full, so that an avi file can be properly closed, or it isn't playable on many players. The problem with either of the above solutions is, it presumes you can do something about the disk being full, nope disk full is too late, or that nothing else is writing to the disk. How about looking at the remaining storage space? Here is a patch adds a reserved-space propertly to, "Send no space left warning when fewer than reserved-space bytes remain". It does nothing unless it is non-zero. To avoid checking the available disk space after every write it will only check again when half of the available space has been written. In my case the application will receive the warning, clear out the buffers, and shutdown the pipeline all before the storage space actually runs empty.
What should we do with this bug? Both patches need to be at least rebased to latest git master, additionally the first one with the signal is not ideal because it's a signal from the streaming thread that has to be handled by the application. I think overall I prefer the approach of the second patch.
Review of attachment 126735 [details] [review]: ::: gstfilesink.c @@ +45,3 @@ #include <sys/types.h> +#include <sys/stat.h> +#include <sys/statfs.h> Need to check for this with configure @@ +177,3 @@ + g_object_class_install_property (gobject_class, PROP_RESERVED_SPACE, + g_param_spec_uint ("reserved-space", "Reserved Storage Space", Should be guint64 property @@ +276,3 @@ break; + case PROP_RESERVED_SPACE: + if(G_VALUE_HOLDS_UINT64(value)) That will never happen, just always make this a guint64 property @@ +599,3 @@ + /* Each time the reserved check halfs, update and possibly send an error, the + * current buffer was written, so return flow ok. */ + if(filesink->reserved_space && filesink->next_space_check < 0) { Conditionally do that if the required API is available... some #ifdef around here
> What should we do with this bug? I think this is a valid bug/feature request that we really should somehow be able to handle gracefully. What we need is some kind of async-response-from-the-app message that the sink can block on. This is something that is useful in general, not just in this one case. Wim had some ideas how to implement that, but I don't remember details.
There's a GCond in GstMessage that could maybe be used for that :) It's used in GstBus if the sync handler returns ASYNC though, not sure what exactly that does.
-- GitLab Migration Automatic Message -- This bug has been migrated to freedesktop.org's GitLab instance and has been closed from further activity. You can subscribe and participate further through the new bug through this link to our GitLab instance: https://gitlab.freedesktop.org/gstreamer/gstreamer/issues/3.