GNOME Bugzilla – Bug 741602
Make file / directory creation work on database-backed storages
Last modified: 2015-08-20 16:42:49 UTC
Database-backed storages like Google Drive represent each file / directory as a unique, server-defined, persistent ID. This ID is human unreadable and not fit to be exposed to the user. Therefore a GVfs backend (see bug 739008) for such a storage would use the ID as standard::name and the human readable title as standard::display-name. This is convenient because then paths like /abcd123/pqrs456 can be easily mapped back to the actual server-side objects. Also, the standard::copy-name conveniently maps it back to a sane string when downloading from Drive to the local disk. However, when creating directories (and files) g_file_make_directory is called with a GFile pointing to /abcd123/Untitled Folder and there is no mechanism for the GVfs backend to say that the directory was created as /abcd123/wxyz789. This is unlike the set_display_name operation in GVfs where one can use g_vfs_job_set_display_name_set_new_path to return the actual path via g_file_set_display_name. MTP is similar to this case because it also uses IDs internally to represent files. However it does not expose the IDs as standard::name. Instead it has its own cache to map the human readable names to their internal IDs. I would like to avoid this because online storages like Drive are inherently more race-prone than MTP devices. Users might be changing their Drive out of band from a web browser or a second computer, unlike a phone where GVfs is probably the only thing touching it. Being able to use the server-defined ID appears to be an easy way to ensure consistency of the data.
Created attachment 292843 [details] [review] gfile: Add new g_file_resolve_created_path() API Alex suggested that we can add a g_file_resolve_created_path API to let the backend specify the actual path on the storage to the client application. I have only added the sync variant of the API for the moment. If this looks like a sane thing to do, then I will add the async variants too.
So, this is kind of a weird API, but I don't think we can necessarily do much better with all the existing APIs and UI expectations that we have. The one thing that is a bit unclear to me is the exact lifetime semantics of the "temporary" unresolved paths. We're kind of handwaving this with "they are cached", but what does this mean? Do the "fake" files exist in one process only, or is it cross-process? Do they persist over unmount? reboot? For how long? What happens if you use a fake name for a file and then by chance a real file appears with that name? I don't know the exact answer to the above, but we should probably figure it out, so that app authors have some idea of the semantics that are guaranteed. Another thing we could do is have some kind of flag on a filesystem indicating that it has these kinds of semantics. Something from g_file_query_filesystem_info() probably. We also need to go over the gio/gtk apis with a fine-toothed comb to figure out exactly which operations may need to use this. Here is a quick minimal list: g_file_append_to, g_file_create, g_file_replace, g_file_create_readwrite, g_file_replace_readwrite, g_file_copy, g_file_move, g_file_make_directory, g_file_make_symbolic_link g_file_replace in particular is tricky, with its handling of pre-existing files, atomicity and backups. We need to go through all of these and make sure the semantics we come up with make sense for them. Anyway, i don't think we can land this before we have also worked on the gdrive side and made sure it works in some apps (say e.g. nautilus and gedit).
(In reply to comment #2) > Anyway, i don't think we can land this before we have also worked on the gdrive > side and made sure it works in some apps (say e.g. nautilus and gedit). I am also slightly uncomfortable basing this only on Google Drive. Ideally, we would have another similar backend to validate our assumptions.
What about SkyDrive/OneDrive, does it have object-store like semantics?
(In reply to comment #0) > MTP is similar to this case because it also uses IDs internally to represent > files. However it does not expose the IDs as standard::name. Instead it has its > own cache to map the human readable names to their internal IDs. I would like > to avoid this because online storages like Drive are inherently more race-prone > than MTP devices. Users might be changing their Drive out of band from a web > browser or a second computer, unlike a phone where GVfs is probably the only > thing touching it. Being able to use the server-defined ID appears to be an > easy way to ensure consistency of the data. The mtp backend is also vulnerable to these races because changes can be made by apps running on the device while the backend is mounted (this is a reason to use mtp rather than usb mass storage). However, it is a fairly small race because there is a separate thread which receives change events from the device to invalidate ids.
(In reply to comment #4) > What about SkyDrive/OneDrive, does it have object-store like semantics? Yes, it does, but I have not looked at it as closely as I have looked at Drive. I will write a OneDrive backend and see if we hit the same issues or whether they are slightly different.
I wonder if we can do this without adding a new method by adding a GFileInfo info that has the "real" name. I.e. we can expose the "fake" pathname as if it was the real file, except we add an extra set of info like "is-volatile-path" and "non-volatile-path". Then you could use g_file_io_stream_query_info() on the stream you get from g_file_replace() to get the real filename during save.
Created attachment 306074 [details] [review] fileinfo: Add a G_FILE_ATTRIBUTE_STANDARD_IS_VOLATILE attribute
Review of attachment 306074 [details] [review]: This looks good to me, but lets keep it out until we have the rest done and are sure this approach is working out.
Review of attachment 292843 [details] [review]: We have pretty much decided in favour of the GFileInfo attribute.
From #gtk+ on GIMPNet: 16:37 <rishi> alex: mclasen: Unless anything unexpected comes up, do you think https://bugzilla.gnome.org/show_bug.cgi?id=741602 can go in 3.17.91? It would be after the API freeze, but I don't know glib has an exception or not. 16:37 <mclasen> rishi: you can always ask for a freeze break 16:37 <alex> rishi: by "this" i suppose you meant the attribute patch only? if so, ok with me 16:38 <rishi> alex: Yes, the attribute. I marked the other one as "rejected". 16:38 <alex> rishi: in theory glib+gtk is not bound to the gnome schedule 16:38 <mclasen> and given that .90 is not even out yet, just push it... 16:38 <rishi> alex: Ok, to push as mclasen says? 16:39 <alex> yeah
Comment on attachment 306074 [details] [review] fileinfo: Add a G_FILE_ATTRIBUTE_STANDARD_IS_VOLATILE attribute Pushed to master.