GNOME Bugzilla – Bug 724475
Parallel database upgrade cripples desktop responsiveness
Last modified: 2014-03-05 19:08:12 UTC
I just spent waiting 50 minutes for 0.5.2. to finish the database upgrade. I would have worked on something else while it was running - but couldn't since it was killing my desktop's responsiveness for that whole time. Granted, with three accounts, all with a large number of messages, the database sizes are quite large: 1.3G, 800M and 2.1G each. However the upgrade shouldn't a) take that long and b) destroy my ability to use my desktop while running. It seems that the database upgrade process runs in parallel, since the journal files for all three were growing at the same time. Since I do not have an SSD, this probably explains the issue - trying to upgrade all three at once is likely killing read/write time and blowing up the I/O queue. For traditional disks, a serial database upgrade should reduce I/O pressure, let other processes get a bigger share of the IO queue and so reduce the impact on the user, and might even be faster overall.
Thanks for the info -- we'll try to take a look at this for 0.6.
I'll try to get some big mail databases rounded up and take a look at this next. I suspect we might be able to continue running them in parallel if we can get the upgrade process not to be so onerous (e.g. if it took a few seconds total, even on huge databases). It was a recent change that introduced a really slow upgrade, so I'll look at speeding that up first.
Turns out we were doing a lot of unnecessary work in one recent upgrade process. I've got a branch at wip/serial-db-724475 that fixes that issue. Unfortunately, I think the upgrade in question still has to do a lot of work, so Mike, I don't think this will fix the issue for you. We need to serialize all database upgrades so the desktop is usable while they're running, at least. I've been playing around with ways to do that, but don't have anything concrete yet. Jim, let's discuss on Monday.
Mike, we've gone a dead-simple route in our db upgrade code that should serialize the costly upgrades. This should make the desktop usable while the upgrades are happening, though they'll still chunk for quite a while. If you've already upgraded, don't have older copies of your database handy, and would like to test this new code, you can fire up your databases (under ~/.local/share/geary) in sqlite3 and run: pragma user_version = 17; to roll them back to a state where Geary will kick off the upgrade again. Currently the subsequent upgrades are fine to run multiple times, so there should be no harm, although as always you may want to back up your databases first. If the problem isn't as gone as we think it is, feel free to re-open this ticket. Thanks! Fixed in commit 85d1a97.
Charles, thanks for looking into this. I just tried it then using the pragma reset. I'm not sure however the mutex is working however, starting it up with -d shows the following printed immediately: > [deb] 16:06:21 1.676398 db-versioned-database.vala:75: VersionedDatabase.upgrade: current database schema for /home/mjg/.local/share/geary/michael@quuxo.com/geary.db: 17 > [deb] 16:06:21 0.060450 db-versioned-database.vala:115: Upgrading database to version 18 with /usr/share/geary/sql/version-018.sql > [deb] 16:06:21 0.189452 db-versioned-database.vala:75: VersionedDatabase.upgrade: current database schema for /home/mjg/.local/share/geary/mikeg@cse.unsw.edu.au/geary.db: 17 > [deb] 16:06:21 0.189455 db-versioned-database.vala:75: VersionedDatabase.upgrade: current database schema for /home/mjg/.local/share/geary/mike@vee.net/geary.db: 17 > [deb] 16:06:21 0.000057 db-versioned-database.vala:115: Upgrading database to version 18 with /usr/share/geary/sql/version-018.sql > [deb] 16:06:21 0.000061 db-versioned-database.vala:115: Upgrading database to version 18 with /usr/share/geary/sql/version-018.sql Which, if you look at src/engine/db/db-versioned-database.vala shouldn't be printing all three "Upgrading database..." messages at the same time since they're inside the guarded code path. I can see multiple journal files being created while it's running again, also.
Mike, did you pull from the latest master in git when testing? I can't reproduce what you're seeing. How are you building/installing Geary to test?
Ah, my bad - was inadvertently using the system binary. /me puts on a brown paper bag This makes the desktop much more responsive while the upgrade is going on. A trial just then using current master took ~45m under light load, so the total time was not very different, but it was nice to be able to listen to some music and switch windows instantly while that was going on. Thanks!
Great, glad to hear it's at least usable while we're churning through the upgrades. Thanks for your help!