GNOME Bugzilla – Bug 736495
Work around Android MTP stack's SendObject being extremely slow for large files
Last modified: 2017-05-10 14:01:03 UTC
When I originally verified Android's support for SendObject with > 4GB files on a Nexus 4, it seemed to work fine, but i've recently tested with a Nexus 7 and a Nexus 5 (Android 4.4) and it's significantly slower than for < 4GB files. On the Nexus 7, I see large files are sent at about 50% of the rate of a small file, and on the Nexus 5 it's 99.99% slower (yes, really - 20Kbps vs 20MBps). I don't know if this is a bug in Android 4.4 or in the hardware - my original Nexus 4 test was on Android 4.3 and I'm not in a position to reproduce it. So, my thought is simply to not use SendObject on Android. The Android direct I/O extensions allow file transfers to be done as normal file I/O and I've verified that the performance here is identical for large and small files (direct I/O is more CPU intensive but can still hit peak transfer rates). It doesn't seem worth trying to detect more specifically than that.
Created attachment 285942 [details] [review] Don't use SendObject for large files if Direct I/O is available
Review of attachment 285942 [details] [review]: Looks good, although I'm not in a position to verify the speeds.
Hmm. I did more testing - both the Nexus 7 and the Nexus 4 (I managed to clear out enough space) seem to be able to transfer just fine - so this is looking like a Nexus 5 specific problem, so I should try and come up with a Nexus 5 specific solution. That will be interesting, given that the USB IDs are identical for all three devices...
Hmm, any chance it's something particular with your device? It seems a bit nasty to start putting in workarounds for particular devices...
I'm going to try and find someone with another Nexus 5 to try with, but I'd be surprised if it was device specific. The fact that it's linked to large files specifically means it's clearly software related - and I'm not sure how that could be device specific. But we'll see.
I have two Android 4.4 devices and both seem to be about the same speed when using the extensions vs SendObject. However, the one device seems rather flaky using SendObject for large files so maybe it's a good idea to use the Android extensions for large files...
I've finally had a chance to check with a different Nexus 5 and the behaviour is identical, so I'm pretty confident that this is Nexus 5 specific - though I couldn't tell you why. Ross, can you tell me what model(s) you tested with and what OS flavour they were running? (Stock, CyanogenMod, Samsung craziness, etc)? I'd like to just turn this on for all (google mtp stack) Android devices but I did see somewhat lower transfer rates on the Nexus 4 and 7, so I don't want to unnecessarily penalize anyone.
The first device I tested was a iFive X3 tablet running a stock version of Android 4.4.2. Both methods seemed pretty similar speed-wise. The second device I tested was a Huawei Ascend G300 running CyanogenMod 11.0, based on Android 4.4.2. Both methods seemed pretty similar speed-wise, but using SendObject for > 4GB files did not work properly (it worked fine using the Android extensions).
Good to know. Another related thought. Using Direct I/O has the additional advantages of allowing users to do other operations on the device (such as browsing around in Nautilus) and cancelling the file transfer. This is possible because the transfer is broken down into the small SendPartialObject calls which can be interleaved, and when you cancel, it just stops making the calls. SendObject (and GetObject) block the device completely for the duration, and cancelling them doesn't cancel them on the device, so it stays blocked until they complete (without visual feedback)
(In reply to comment #9) > Another related thought. Using Direct I/O has the additional advantages of > allowing users to do other operations on the device (such as browsing around in > Nautilus) and cancelling the file transfer. This is possible because the > transfer is broken down into the small SendPartialObject calls which can be > interleaved, and when you cancel, it just stops making the calls. Yes, that's a good point. > > SendObject (and GetObject) block the device completely for the duration, and > cancelling them doesn't cancel them on the device, so it stays blocked until > they complete (without visual feedback) You're probably hitting bug 720058. With that patch applied, cancelling with SendObject works properly, the cancellation stops the transfer and I can immediately start transferring something else.
Interestingly, I discovered that downloading files from the device to windows is super slow as well. I assume the windows mtp client code uses GetObject too, so that must mean there's a 'right way' and a 'wrong way' to use it, and we happen to use the right way. From that, one might conclude there's a right way to use SendObject as well, but obviously, we can't see the windows source to spot the differences.
This patch actually seems necessary for uploading files larger than 4 GB to my ASUS TF701T (stock Asus ROM; Android 4.4.2). Initially, it looked like the SendObject approach works as well, but it turned out that this is due to libmtp 1.1.6 not signalling > 4 GB size properly (so it actually transferred only 3.4 GB of 7.2 GB file). With libmtp 1.1.8, SendObjectInfo fails immediately, same as when trying to transfer that file from Windows. Regarding the differences in Windows mtp client, perhaps those could be found by comparing USB traces during a large file transfer?
Rok, does GetObject work on your device for large files? At this point, I'm feeling comfortable pushing this change - we're seeing enough bad behaviour with SendObject that it's reasonable to want to avoid it where possible. But I do want to get some sense about whether we need a matching change for GetObject.
Yeah, GetObject does seem to work; I can download a large file from tablet without a problem. Actually, it seems I need to take a closer look at my problems with SendObject, because they do not seem to be entirely consistent between linux, my Windows VM and Windows laptop... Will check the timings for both download and upload while I am at it.
Change pushed as is. If we need to expand usage of Direct I/O, we can down the line.
I've now had a chance to test with Lollipop and it looks like they've fixed it. Not sure if we should react to that or not.
The new Marshmallow devices (5x and 6p) definitely have this fixed. I don't have a nexus 5 or nexus 6 any more to check if marshallow is fixed on them. There is no way to detect versions at the mtp level, so we have to make a global decision on which path to use...
Created attachment 326140 [details] [review] Use a gsetting to control the workaround So, it seems like a gsetting toggle is the only way to accommodate devices with and without the bug. Neither MTP nor USB offer a way to detect the android version on the device.
Ondrej, can you take a look at this.
Created attachment 326142 [details] [review] Use a gsetting to control the workaround v2 Forgot to include the schema file in the first diff
Review of attachment 326142 [details] [review]: It will never be perfect and truly I don't like this approach at all. How many people is affected by this bug (i.e. how many people use files bigger than 4GB over MTP and have some of the problematic devices)? And how many people do you expect will use this gsettings option? I don't think this apply on many people. I think we have to choose one preferred solution, which works on the most devices (even though it is for example slower)... AFAIK libmtp has a database for devices and it uses different hacks for different devices. So I think this should be handled by the libmtp library itself, not by the clients of the libmtp library...
I haven't found any way to distinguish the devices. The USB vid and pid are the same for all nexus devices (although the vendor/product strings are different) and the mtp level info is the same (always says android 1.0). So there's no way to know what version of android is installed, except in so far as we know what the oldest possible version is (based on what was on the phone when it was released).
Created attachment 351315 [details] [review] Remove work-around It's time to move on. Let's just remove the work-around completely. This bug was fixed in Android 6.0. We shouldn't be optimizing for the old devices.
Review of attachment 351315 [details] [review]: Thanks, it makes sense. Just please add the bug URI in the description...
It seems we are done here...
Comment on attachment 351315 [details] [review] Remove work-around https://git.gnome.org/browse/gvfs/commit/?id=3738d918a69d79dc2687d13c45989d4995041adc