GNOME Bugzilla – Bug 694455
7.3.128.4,7.3.136.1: File upload is slow
Last modified: 2014-06-15 07:59:17 UTC
Actually, the upload itself is not the problem. It's just that it takes Rygel 5 seconds to update the status of the upload to "COMPLETED" after the upload has actually finished. Here's a snippet of a mediaconsole session >>> root.upload_to_any("Test","/home/markus/DLNAContent/Image/JPEG_SM/Current-Content-Set/B-JPEG_S-14.jpg") Transfer ID: 1 Path: /com/intel/MediaServiceUPnP/server/0/3465323133396335366430636661616432623231616161626238383966333862 >>> root.get_upload_status(1) Status: IN_PROGRESS Length: 118652 Total: 118652 >>> root.get_upload_status(1) Status: IN_PROGRESS Length: 118652 Total: 118652 >>> root.get_upload_status(1) Status: IN_PROGRESS Length: 118652 Total: 118652 >>> root.get_upload_status(1) Status: IN_PROGRESS Length: 118652 Total: 118652 >>> root.get_upload_status(1) Status: IN_PROGRESS Length: 118652 Total: 118652 >>> root.get_upload_status(1) Status: IN_PROGRESS Length: 118652 Total: 118652 >>> root.get_upload_status(1) Status: IN_PROGRESS Length: 118652 Total: 118652 >>> root.get_upload_status(1) Status: IN_PROGRESS Length: 118652 Total: 118652 >>> root.get_upload_status(1) Status: IN_PROGRESS Length: 118652 Total: 118652 >>> root.get_upload_status(1) Status: IN_PROGRESS Length: 118652 Total: 118652 >>> root.get_upload_status(1) Status: IN_PROGRESS Length: 118652 Total: 118652 >>> root.get_upload_status(1) Status: IN_PROGRESS Length: 118652 Total: 118652 >>> root.get_upload_status(1) Status: IN_PROGRESS Length: 118652 Total: 118652 >>> root.get_upload_status(1) Status: IN_PROGRESS Length: 118652 Total: 118652 >>> root.get_upload_status(1) Status: COMPLETED Length: 118652 Total: 118652 Note that the upload actually finished before I called get_upload_status for the first time. It just took a long time to update the upload Status. Looking at the code in rygel-http-post.vala, Rygel is waiting for an update from the container to which the file has just been uploaded. If it doesn't get this notification after 5 seconds it times out. This corresponds to what I'm seeing in the Rygel logs. (rygel:20168): Rygel-DEBUG: Waiting for update signal from container '89ebff5f10f00aac21e05a90fd295773' after pushing content to its child item '2a81f8511875e84ab33ea76ac37f2cdb'.. (rygel:20168): Rygel-DEBUG: rygel-visual-item.vala:67: Failed to get thumbnail: No thumbnail available (rygel:20168): MediaExport-DEBUG: rygel-media-export-harvester.vala:260: Starting grace timer for harvesting file:///home/asasas/Pictures/36d05744-500e-4f8c-a6e3-e3dc6398e5e8-Test.jpg… (rygel:20168): MediaExport-WARNING **: Failed to get file info for file:///home/asasas/Pictures/.36d05744-500e-4f8c-a6e3-e3dc6398e5e8-Test.jpg (rygel:20168): Rygel-DEBUG: rygel-http-post.vala:136: Timeout on waiting for 'updated' signal on '89ebff5f10f00aac21e05a90fd295773'. I'm guessing the problem is with MediaExport for not generating the signal but I could be wrong. Perhaps we're connecting to the signal too late.
There's a timer that waits for file events to settle before it runs extraction on a file, maybe that conflicts with the 5s timeout
I mean to prevent extraction on incomplete files, causing crashes.
Just as a quick update. I changed the 5 second timer in wait_for_item to a 1 second timer on my local build and the uploads complete almost immediately. I realise this is not a fix but it does demonstrate that upload to media export is blocking on this timer. The real question is why?
http://git.gnome.org/browse/rygel/tree/src/plugins/media-export/rygel-media-export-harvester.vala#n270 The default grace period is 5s. As I said in comment 2, to prevent extracting on half-changed files; sometimes changes-done hint doesn't really work. Probably not the best thing to do if we aim for maximum certifiability of things, but definitely useful for the normal user.
I guess the problem here is that this 5 second delay will make +UPSYNC+ controllers really slow, e.g., it would take 10 minutes to sync 120 newly taken photos, assuming that the controller uploads serially. I don't really understand how the corruption could occur. If I understand the code correctly, all data received from the client is written to a temporary hidden file. Once the upload has completed, the file is renamed so that it will be seen by the plugin. MediaExport then waits 5 seconds before starting the meta data extraction. So what is the 5 second wait for? Is it simply to ensure all buffers are flushed before starting meta data extraction on the new file or is there something else I'm missing here?
Yes, we do that. But Transmission, Firefox or ... don't do that. This is for external download tools dumping their temporary file directly into a folder watched by Rygel. I'm not saying this is great and I totally understand your issue :)
Okay, I now understand the problem. Tricky stuff. So I guess to speed up upload, we would need to find a way to distinguish between files whose upload is managed by the DMS and files that are added to MediaExport's upload directories through some other mechanism.
On the other hand it doesn't prevent crashes in the extractor reliably, looking at eg. bug 691380
Also affects 7.3.128.4 as it tries an immediate browse after HTTP POST when the information isn't yet available.
And 7.3.136.1
Created attachment 278469 [details] [review] media-export: Make grace period configurable While it helps not to run discover on half-downloaded files, it slows down the server sync use-case and breaks several DLNA requirements. It can turned off now using monitor-grace-timeout=0 Signed-off-by: Jens Georg <mail@jensge.org>
Attachment 278469 [details] pushed as c70a2a3 - media-export: Make grace period configurable