Note that quake.wr had about 2/3 of the traffic that earthquake had, but it handled it adequately, while earthquake was flat on its back. This was seen in the recenteqs monitoring page snapshot Bob sent out. At 9:15, quake was responding fine, and the earthquake servers were all stone dead.
The two sites use basically identical servers: Dual-Xeon 2.4GHz, 1GB, running FreeBSD 5.4.
On the backend, quake.wr handled 737 hits/sec, or about 245 on each server. By comparison, earthquake fell over dead when it tried to do 460 hits/sec, and it didn't run adequately until the load was down to 115 hits/sec, or 38 on each server. Based on the number of requests being sent to the backend servers, we were short by about a factor of 20.
This illustrates the difference between a site of purely static pages, and a site with the semi-dynamic pages that earthquake has. It appears that by going to all-PHP on the site, we have lost performance by a factor of at least 6 and possibly up to a factor of 10.
My best guess is that if we'd been running the earthquake site with static pages, we probably still would have come up short, but only by a factor of 2. This is not great, but it's better than missing by a factor of 20.
One thing that might help would be to add something to cache the output from PHP. Possible packages are Cache Lite or Smarty. These will reportedly increase performance up to 100%. The downside is that they will require some additional progamming on any page that is going to use them. And a 100% improvement only doubles our speed, which doesn't make up for the order of magnitude we've lost.
For historical interest, I looked in my notes, and I found this item:
January 23, 2006:
Talked to Lisa and Jeremy about the new Program site and Akamai caching. I hadn't realized that the *whole site* is now PHP-based. When I saw '.php' on the end of the recenteqs pages, I felt a cold chill. I called Akamai, and they said they have no problem with caching a .php URL, but they will honor the expires headers of the page. By default, PHP sends stuff out with an immediate expiration time, so it will not be cached. So we need to explicitly defeat that with a separate header call in the PHP for each page. Our default expiration is 5 minutes.