GNOME Bugzilla – Bug 331876
Hang while trying to load large library
Last modified: 2018-05-24 11:26:25 UTC
Please describe the problem: When I try and load a large library of music (~80 GB) that has previously loaded (so metadata tags and the rhythmdb.xml is already created), I get a hang. The application doesn't crash, but it just spins: Steps to reproduce: 1. gdb shell/rhythmbox 2. (gdb) run -d > ~/rb.log 2>&1 3. Ctrl-C when application hangs Actual results: In step 1, interface comes up, lists in browser and songs start being populated, and status displays "Loading...", then it hangs, just after reaching (what I remember seems to be full count of songs). In step 2, looking at the rb.log, the last entry in the log is different from run to run, here's one: [0x9fdf130] [rhythmdb_process_events] rhythmdb.c:1664 (01:44:25): processing RHYTHMDB_EVENT_STAT [0x9fdf130] [rhythmdb_process_stat_event] rhythmdb.c:1454 (01:44:25): not modified: file:///mnt/data/iTunes/Everything%20But%20The%20Girl/Missing/03%20Missing%20(Rockin'%20Blue%20Mix).mp3 In step 3:, here's the backtrace: Program received signal SIGINT, Interrupt.
+ Trace 66348
Thread NaN (LWP 26665)
Expected results: No hang, application should load. Does this happen every time? Yes, with this library with current CVS HEAD. Previous versions have loaded successfully, but I have had problems like this in the past, intermittently, but a while ago now, so I thought they were fixed. Other information:
Here is what the process looks like: $ ps aux|grep rhythmbox alex 26664 1.6 4.2 48508 44208 pts/18 S+ 01:43 0:04 gdb ./shell/rhythmbox alex 26665 8.8 3.6 87004 38224 pts/18 Tl 01:43 0:26 /home/alex/src/remote-cvs/gnome.org/rhythmbox/shell/rhythmbox -d
I could also successfully load the track that the library was hanging on before, when starting up with a clean rhythmdb.xml: <?xml version="1.0" standalone="yes"?> <rhythmdb version="1.1"> <entry type="song"> <title>Missing (Rockin' Blue Mix)</title> <genre>Rock</genre> <artist>Everything But the Girl</artist> <album>Missing</album> <track-number>3</track-number> <duration>469</duration> <file-size>9383792</file-size> <location>file:///mnt/data/iTunes/Everything%20But%20The%20Girl/Missing/03%20Missing%20(Rockin'%20Blue%20Mix).mp3</location> <mountpoint>file:///mnt/data</mountpoint> <mtime>1072612244</mtime> <first-seen>1140429665</first-seen> <last-seen>1140429665</last-seen> <rating>1.000000</rating> <date>728294</date> <mimetype>application/x-id3</mimetype> </entry> </rhythmdb> Also, the problem also occurs when loading up a fresh library, and re-importing the songs.
We should probably add the vfs monitors asynchronously (or in another thread if there are no async versions). Jonathan has mentioned before that adding the monitors all at once can cause gnome-vfs to block for a long time.
(In reply to comment #3) > Jonathan has mentioned before that adding the monitors all at once can cause > gnome-vfs to block for a long time. Is that bug #325215?
I have actually seen it deadlock a couple of times. It's a problem with the communication between gnome-vfs and gamin, and I vaguely recall deciding that it can't be fixed without changing the FAM API. Since gnome-vfs 2.13.x can apparently use inotify directly, the problem should go away (for linux users, anyway) by itself. If you strace the rhythmbox and gam_server processes once the deadlock has occurred, you should see them both in a blocking write to the socket connecting them. Since neither side has a timeout on the write, they'll just sit there until one side is killed.
(In reply to comment #5) > I have actually seen it deadlock a couple of times. It's a problem with the > communication between gnome-vfs and gamin, and I vaguely recall deciding that > it can't be fixed without changing the FAM API. Since gnome-vfs 2.13.x can > apparently use inotify directly, the problem should go away (for linux users, > anyway) by itself. Is there no workaround for gnome-vfs < 2.13.x that we can implement?
As well as gnome-vfs < 2.13, it's also needed for non-linux systems, and situations when FAM gets used anyway (e.g. over NFS). A solution would be to add an async queue for monitor additions, and have a thread processing that which limits the rate (e.g. sleeping for a short time, or whatever).
(In reply to comment #5) > If you strace the rhythmbox and gam_server processes once the deadlock has > occurred, you should see them both in a blocking write to the socket connecting > them. Since neither side has a timeout on the write, they'll just sit there > until one side is killed. Yep, bingo, tracing gam_server: $ gdb --pid 8645() [...] (gdb) bt
+ Trace 66423
Both hang with the _write_nocancel. OK, so what to do?
I have the same problem with gam-server (gamin) and rhythmbox. The problem arose today after I decided to copy all my Music partition on another hard disk that would substitute the previous (external) hd. So, rhythmbox couldn't "see" the difference (they were both known as /dev/sda1). Anyway, rhythmbox has re-scanned the music folder (of about 90 gigas) but at the end of the process the cpu usage still is at 100% of its capacity. That is due to gam-server as I can see tabbing "$ top" by shell. Another thing is that after the first run and quit I can't run rhythmbox until next reboot, even by "$ shell" (but I can do it with root privilege). I use Rhythmbox 0.9.3.1 on my Debian Sid. What can I do? thnx, giopas
Created attachment 61596 [details] Screenshot rhythmbox & top here a screenshot of rhythmbox and top after a complete loading. As you can see, there is a "gam-server" process that occupies a huge amount of resources, even if there is no file to load anymore.
Slowing down the addition of directory monitors should be fairly simple to do. The question is how fast can we safely add them?
this is critical, but this isn't a blocker by definition. any news on this?
*** Bug 352494 has been marked as a duplicate of this bug. ***
Is there some hidden yet incredibly useful way to automatically refresh the library? 0.9.6 is faster on restarting, but I hate to do that every time I want to see something I updated in the library.
(In reply to comment #14) > Is there some hidden yet incredibly useful way to automatically refresh the > library? 0.9.6 is faster on restarting, but I hate to do that every time I > want to see something I updated in the library. Erm... obviously I meant "manually refresh the library", not "automatically".
Doesn't rhythmbox use gnomeVFS? If that's the case, wouldn't it be supposed to automatically detect changes almost immediately (as far as I know, it is what it does already). However, I believe having an option to manually refresh would be a great performance gain in the way that I would not have that library check run needlessly on startup.
(In reply to comment #15) > Erm... obviously I meant "manually refresh the library", not "automatically". If you know which directory changed, you could just reimport the directory. (It's a kludgy workaround, but that's what I do if I want to update tags that I've changed in another program and I've switched off library watch).
(In reply to comment #16) > Doesn't rhythmbox use gnomeVFS? If that's the case, wouldn't it be supposed to > automatically detect changes almost immediately (as far as I know, it is what > it does already). Yes, it does if you have library "watch" enabled. The problem, as I understand it, is that if you are using a gnomevfs source that doesn't support inotify (such as an NFS share) then adding it locks up adding directories to the gam server. > However, I believe having an option to manually refresh would be a great > performance gain in the way that I would not have that library check run > needlessly on startup. If the file system supports inotify directly (such as the local harddisk) then the check goes relatively fast.
I assume "relatively fast" is a keyphrase here :) because scanning my 8000 songs library necessitates 1 minute 20 seconds for the startup of rhythmbox (about 20-30 seconds without it, if I recall correctly), running 0.9.6 on my local reiserfs partition. It is "relatively" fast, but still really painful for me to wait 1 minute and a half before having a responsive UI.
(In reply to comment #19) > I assume "relatively fast" is a keyphrase here :) because scanning my 8000 > songs library necessitates 1 minute 20 seconds for the startup of rhythmbox > (about 20-30 seconds without it, if I recall correctly), running 0.9.6 on my > local reiserfs partition. It is "relatively" fast, but still really painful for > me to wait 1 minute and a half before having a responsive UI. I have to admit it was a while ago I checked the start-up with the library watch enabled. I actually don't use watch at all, but my ~3200 library takes about 3-4 seconds to load completely on an ext3 local partition.
(In reply to comment #19) > I assume "relatively fast" is a keyphrase here :) because scanning my 8000 > songs library necessitates 1 minute 20 seconds for the startup of rhythmbox > (about 20-30 seconds without it, if I recall correctly) Bug #360648 comment #6 probably addresses your situation (since your problem isn't a lock-up), Jonathon Matthew suggests that the patch on bug #325215 will probably help.
does gio instead of gnomevfs change anything here?
gio still uses fam when there's nothing better available, so it's probably still susceptible to this problem.
Hi, Any news on this crash? I've reported a bug on launchpad : https://bugs.launchpad.net/ubuntu/+source/rhythmbox/+bug/442434 is it the same bug?
I don't know how you decided that your problem resembles this bug. There is no useful information in your launchpad bug report.
Hi Jonathan, I think it's an automatic system which when seeing a bug url mark it as bug watch. I said "I think this bug looks like this one", it only looks like, it was just an hypothesis*. * I don't found translation of the French word "hypothèse". Thanks for your work,
(In reply to comment #26) > I think it's an automatic system which when seeing a bug url mark it as bug > watch. That is not true, and totally offtopic for this bug report.
-- GitLab Migration Automatic Message -- This bug has been migrated to GNOME's GitLab instance and has been closed from further activity. You can subscribe and participate further through the new bug through this link to our GitLab instance: https://gitlab.gnome.org/GNOME/rhythmbox/issues/156.