GNOME Bugzilla – Bug 384452
live.gnome.org wiki seizing up repeatedly
Last modified: 2008-02-06 15:37:21 UTC
Over the last day or two, I've had to restart the 'live.gnome.org' wiki python process several times. See http://live.gnome.org/GnomeWikis I noticed while investigating that we were running moinmoin 1.5.5a. On checking the moinmoin site, it seems they have since released 1.5.6, which fixes a couple of infinite loop bugs, which I thought might have been causing this. As I don't have the time right now (or much experience) to debug the python process running the moinmoin wiki, I have simply upgraded it to see if that would fix the problem. I haven't had to restart it since (touch wood), but it does still seem to run a lot slower than the other two wikis hosted on the server (www.pango.org and www.gnome-db.org), so I suspect that bugfix wasn't the magic bullet in this case.
Guessing... I think the following URL causes the hang in just on thread: http://live.gnome.org/LSR/ScratchPad/Braille?action=fullsearch&value=linkto%3A%22LSR/ScratchPad/Braille%22&context=180 This will stop just that thread from working. Eventually (because the user keeps refreshing), all threads stop working and the entire site is down. The URL I gave is the link when you click the 'Braille' text on: http://live.gnome.org/LSR/ScratchPad/Braille I've added the user requesting that page (201.242.56.107) to iptables, cause I'm not sure how to avoid it cleanly. If moinmoin stays alive, then above URL is the cause and it is time to file a bugreport.
Well there are different possibilties: a) surge protection limits: http://moinmaster.wikiwikiweb.de/HelpOnConfiguration/SurgeProtection b) Ip blocking with hosts_deny http://moinmaster.wikiwikiweb.de/HelpOnConfiguration (under Overview of configuration options) I think surge protection would be a more general solution that would work with different Ip adresses trying teh same thing. I have not played with it yet because I had not seen such problems in my wikis, yet. Hope this helps.
So you would have to set another value for 'fullsearch': (5, 60), I think.
Surge protection does not help. As I read it, this '5, 60' means 5 requests per 60 seconds. The problem is that just one requests disables the thread for a very long time (while maxing the CPU). Instead of something like hosts_deny I was thinking of a .htaccess, but due to the setup this isn't possible. Could've changed the apache server config, but ehr.. iptables was a bit easier (I like that the IP address is removed when someone reboots).
I suggest somebody that works on it comes to our IRC channel. Info at http://moinmoin.wikiwikiweb.de/MoinMoinChat. Maybe we can help there.
The upshot of a brief look at the problem on #moin is that we added the 'MoinMoin.util.thread_monitor' to the live.gnome.org python process, so next time it starts getting bogged down we can add a '?action=thread_monitor' to a request to generate a thread dump showing the state of the 101 processes and attach it to this bug report for review.
From an e-mail on gnome-infrastructure from Jeff Waugh: <quote who="Thilo Pfennig"> > > Is this running in twisted mode? It would be good if it is running with > > Python 2.5 Not yet -- when I set it up, I used the threadpool simple server, just as a simple starting point. We have a couple of choices: * Stick with the separate moin web server process but shift to the Twisted implementation (I'd prefer to keep running the app like this, personally, but twisted and new versions of python are going to be troublesome on our platform -- better to stick with what's packaged and supported) * Shift to mod_python -- does RHEL ship with mod_python? - Jeff
Is this still a problem?
Problem is gone. The default GNOME theme prevents people from doing these intensive searches. Further, mod_python is used.