The E2 MySql database seems to have an upper limit of ~300 queries/second, at least for the type of queries which we do. When queries are sent which take a large amount of sorting or handle large amounts of data (ie doing a user search on pseudo_intellectual) it takes longer than the 0.003 seconds each query is "allowed" - this means everyone else has to wait.

What you can do: Turn off nodelets that you don't care about -- many of them take a decent amount of code running behind them.
If you're using the Chatterbox most of the time, try using the Java Chatterbox and save Everything from having to generate the other 100lbs of HTML. If you're not around, turn off the Java Chatterbox so that it doesn't take system resources.
Use the "expensive" features on E2 sparingly -- don't reload your user search every minute to see if any Reps have changed. Don't sit and chatter on ENN or the cool archive.
Use Create a node rather than searching for a node you know want to create -- searching is a lot of string comparisons and sorting.

E2's web server currently runs 60 Apaches. That's roughly how many simultaineous connections it can handle. When request #61 comes in, it has to wait until there is one available to process the request. This is what causes the most severe lag (>30s pageloads), and can be triggered by database saturation as each process blocks until it's db transactions are complete.

If you start hitting extremely long load times, Please be patient or try again in a couple minutes. Don't hit reload over and over again, as that can compound the problem -- especially when everyone is doing it. A lagged-out web server is like a line at the DMV -- when you hit reload, you get out of line and go to the back again.

One perl script written without any sleep()s in it can take up more resources than 100 normal users. They can be auto-noders, auto-linkers, personal statistics managers, chatter-bots, etc etc. Also, a wide array of data-harvester bots have visited us, and the only one allowed to stay is Google's. Once detected, they're fairly easy to weed out -- via IP's, User Agents, query patterns, etc -- but until then they can ruin the party.

Please, if you write bots, join Edev and try to make them as friendly to the system as possible. Use generous sleep()s, use the XML interface, and the bot will play nice and you (and others who use your 'ware) won't get your IP logged and banned.

At roughly 7am server time, E2 begins purging nodes marked for death on node row. Deleting nodes is an expensive process, partly because of the table locking, and because of the softlink work required. The server's just going to be slower at that time, despite the fact that it's serving < 1/4 of the traffic during peak hours.

There are a few other jobs which the database takes care of which can slow down performance -- maintaining the "other users" list, updating the nodelets, or sending out the Everything Daily Report. Most of these jobs take a second or so, but monopolize the database while they're working. Sorry, you'll have to wait.

assuming, of course, you can get the page to load

--nate