GNOME Bugzilla – Bug 721402
Extremely slow when starting with a clean DB
Last modified: 2014-01-23 12:05:13 UTC
Created attachment 265225 [details] top when gnome-photos was first opened, then 10 minutes later. GNOME photos is unusable. As soon as I open it up, it pegs my CPU and my fan starts kicking in. It's fairly unresponsive... if I press one of the tab buttons in it it takes about 20 seconds to respond. The full usage of my CPU doesn't stop until I kill gnome-photos from a terminal. I have attached screenshots showing top over a ten minute period as an example. I do have a flickr account, but I never set it up with gnome photos. I guess it's taking the information from the GNOME accounts dialog. I have had a Flickr account since maybe 2005, maybe earlier. I have probably 6000-7000 photos in my account. I'm guessing a lot of people have old Flickr accounts and have a substantial number of photos in their account. I do not want all of my Flickr photos on this system, because my hard drive space is limited since I have a laptop with a $$ SSD).
(In reply to comment #0) This is known. I am assuming that you are ran the application for the first time after you had added the Flickr account. > I do have a flickr account, but I never set it up with gnome photos. I guess > it's taking the information from the GNOME accounts dialog. Yes. > I have had a Flickr > account since maybe 2005, maybe earlier. I have probably 6000-7000 photos in my > account. The photos are only downloaded and cached when you want to view one. Since it is a cache, it should be deleted when you start running out of disk space. Otherwise it only fetches the metadata.
Is it actually possible to run gnome-photos with a Flickr account as old as mine then? Should I leave it running overnight or something? Could it maybe just fetch metadata for a reasonable number of photos and use the 'load more photos' button on the bottom to progressively grab more on user request?
(In reply to comment #2) > Is it actually possible to run gnome-photos with a Flickr account as old as > mine then? Should I leave it running overnight or something? You could. I have noticed that the performance is not that bad on subsequent runs, but I should stop procrastinating on fixing this. You clearly have more photos than I have in my account (I got 1000 odd). It is an ideal test case for debugging this. I have two suspects -- the thumbnailing code, and the way we react to new metadata being found (more on that later). Lets first look at thumbnailing since it is easier to investigate. Clear your thumbnail cache (rm -rf ~/.cache/thumbnails/*) before starting the application. This should simulate the "just added an account" scenario. Launch the application and once it starts chewing the CPU, try to grab a series to back traces to see what is going on (gstack $(pidof gnome-photos)). > Could it maybe just fetch metadata for a reasonable number of photos and use > the 'load more photos' button on the bottom to progressively grab more on user > request? Funnily enough, fetching the metadata itself is not the problem. The metadata is fetched by the gom-flickr-miner process that you can see in your initial top screen, but while gnome-photos continues to hog 100% CPU the miner goes away. However, it is possible that gnome-photos is reacting very inefficiently to signals emitted by the miner as a result of new metadata being added to the database. PS: Most (if not all) of this is relevant to gnome-documents as well, but the performance issues are ameliorated by the fact that most people have a lot more photos than documents. If you had 6000+ documents in your Google account, I would not be surprised if gnome-documents also showed similar issues. Even though I do not have that many documents, I still experience some slowness on the initial run.
Created attachment 265404 [details] [review] change-monitor, change-event: Be smarter when resolving new item IDs This tries to optimize the way we react to changes in metadata.
Created attachment 265405 [details] [review] change-monitor: Avoid a copy
Created attachment 265410 [details] backtrace 1
Created attachment 265411 [details] backtrace 2
Created attachment 265412 [details] backtrace 3
Created attachment 265413 [details] backtrace 4
Okay, something I noticed just now is that my laptop is burning my leg even though I closed gnome-photos. gnome-photos is totally gone, not running, but gom-flickr-miner is still running. top says 100% cpu. So I took some backtraces of that too. I had to manually kill it to get my laptop to cool off. gom-flickr-miner backtrace attachments to follow...
Created attachment 265414 [details] gom-flickr-miner backtrace #1
Created attachment 265415 [details] gom-flickr-miner backtrace #2
Created attachment 265416 [details] gom-flickr-miner backtrace #3
Created attachment 265417 [details] gom-flickr-miner backtrace #4
Comment on attachment 265415 [details] gom-flickr-miner backtrace #2 That is the Flickr miner waiting for the server to respond. This is OK.
Comment on attachment 265416 [details] gom-flickr-miner backtrace #3 Ditto.
Comment on attachment 265414 [details] gom-flickr-miner backtrace #1 What version of tracker do you have? I want to make sure that you have the patch for bug 710413
Comment on attachment 265417 [details] gom-flickr-miner backtrace #4 Ditto. If it is not bug 710413 then I wonder why the miner is still hitting sqlite after the application has died:
+ Trace 233025
All calls to tracker have a cancellable, so maybe we are not cancelling the cancellable? PS: This is one thing that I would not have noticed with the size of my own data set. So, thanks.
Hey, I have tracker-0.16.4-2.fc20.x86_64 I believe from bug 710413 comment 19 - that the patch is on my version of tracker. I'm glad the bug is helpful! :)
Created attachment 266128 [details] [review] change-monitor, change-event: Be smarter when resolving new item IDs
Created attachment 266129 [details] [review] change-monitor: Rename a variable for consistency
Comment on attachment 265410 [details] backtrace 1 This is waiting for a thumbnail to be downloaded from Flickr, which looks OK.
Comment on attachment 265412 [details] backtrace 3 Ditto.
I have filed bug 722278 for the miner.
Máirín, would you be able to test my patches from bug 722278 if I built an SRPM for you that you could rebuild locally on F20? Those are targetted at ensuring that gom-flickr-miner does not hog 100% CPU under any condition.
Hi Debrashi, of course I'm happy to help. Just post the SRPM here or email it to me.
Here you go: http://rishi.fedorapeople.org/gnome-online-miners-3.10.2-3.fc20.src.rpm To test: $ tracker-control -rs $ killall gom-flickr-miner $ gnome-photos It will basically hard reset your local metadata cache and force it to re-index your content. I am curious about gom-flickr-miner's performance. Does it still choke the CPU? Or is it better?
Created attachment 266711 [details] [review] view-model: Try to constrain the addition of new items
Created attachment 266721 [details] [review] view-model: Try to constrain the addition of new items
After running with these for a few days, I think this (and bug 722278) addresses the main performance issues. I am closing this for now. Please feel free to reopen if you have reason to believe otherwise. This problem has been fixed in the development version. The fix will be available in the next major software release. Thank you for your bug report.