GNOME Bugzilla – Bug 732663
excessive memory usage when importing large videos
Last modified: 2017-03-09 20:28:26 UTC
If I use the 'import from camera' feature to import a large (e.g. >=500MB) video, then shotwell will use a tremendous amount of anonymous memory during the import. The amount of extra memory used seems to be roughly equal to the size of the video being imported. For example, the attached 'shotwell-1GB-video-import-pmap-log.txt' is a recording of memory usage summaries (from 'pmap -X $(pidof shotwell) | tail -n1') taken every 5 seconds while I imported a ~1 GB video from a Canon PowerShot A570. Notice that the anonymous memory usage gradually increases until it's about 1 GB greater than the pre-import level. Note, too, that my system had to swap stuff out while this was happening. Fortunately, memory usage drops back down to normal levels about 2/3rds of the way though the import. A similar pattern appears when I import a ~500 MB video, except the memory usage increase is "only" about 500 MB. This problem also occurs when importing from my laptop's SD card reader. However, it does not occur if I just drop the video into ~/Pictures and let the library watcher handle it. Notably, the gphoto2 command line tool does not suffer from this bug; its memory usage stays flat throughout the transfer. This was previously reported by Matthijs ten Kate against the Ubuntu package: https://bugs.launchpad.net/ubuntu/+source/shotwell/+bug/912964 In that report, Matthijs discussed both the memory usage and the inadequate progress feedback during large video imports. The progress feedback part was upstreamed as bug 718494; this bug is for the memory usage part.
Created attachment 279816 [details] shotwell-1GB-video-import-pmap-log.txt
Hmm, now that I think about it, maybe that wasn't the right way file this bug. My issue is not the memory usage per se but rather the impact on my system's responsiveness during large video imports. Importing a 1 GB video with shotwell on a computer with 1 GB of RAM takes about 5-6 minutes, and for the last 2-3 minutes or so, the computer is too slow to be usable. Gnome-shell becomes painfully laggy, and vmstat reports that the system is swapping a lot. Latencies become particularly bad once the camera (according to its status indicator) finishes transmitting the video and shotwell starts copying it out to the filesystem. By contrast, 'gphoto2 --get-file 1' fetches the same video in about 2 minutes with a little performance impact. Note that the 1 GB video is only about 9 minutes long, and this camera is capable of producing videos much longer than that. (Or a least it would be if I could find my high-capacity SD card. :P) So I fear that this bug could affect memory-rich users as well. My original report focused on the memory usage because I strongly suspect that it triggers the slowdowns, and UI latency is harder to measure. ;) But the performance impact is a real problem here, especially since it can effectively DoS the computer for minutes at a time.
If I understand the GPhoto handling code in shotwell correctly, we are shoving the whole file through memory. Switching to fd-based should fix this.
Created attachment 347575 [details] [review] Use gp_file_new_from_fd for getting files Previously, we would use a memory file and load the whole file into memory and write it to disk afterwards. That was probably ok for (small) photos, but totally killed memory usage with videos. Signed-off-by: Jens Georg <mail@jensge.org>
Attachment 347575 [details] pushed as eceb40c - Use gp_file_new_from_fd for getting files