M4.9 near Gilroy, CA - 13 May, 2002

Here is a quick rundown of web traffic generated by the M4.9 event that occurred at 22:00 PDT on May 13, 2002. This report is current as of 09:00 May 16. More information will be posted here as it becomes available.

Here is the traffic on the Earthquake Hazards Program web site, as reported by Akamai. The peak of 630/sec almost equals the record of 705/sec that was set after the M6.8 Nisqually event.

Here is the total hits and data rate for the Pasadena web site as reported by Akamai. Note that the hit rate peaks at about 360/sec and the data rate is about 25 Mbits/sec.

Here is the traffic seen on the Pasadena web server. This server sits behind the Akamai EdgeSuite servers, so the traffic seen here is only a small fraction of the true total. Still, it shows the classic spike, followed by an exponential decay. The peak rate is 43 hits/sec, which indicates that the Akamai service is amplifying the site by a factor of a bit more than 8.

Here is an analog report on the logs from the Pasadena servers.

Here is the graph of 'Did you feel it' questionnaires submitted for the first four hours after the event. Note the immediate spike after the event time, followed by a surge about 20 minutes later, when the special news pages on the main Earthquake Hazards Program server came on line. The illustrates the effect of making the questionnaire easier to find. Note that the peak value of about 480 questionnaires/minute works out to about 4/second for each of the two servers. The servers are Athlon PCs running FreeBSD and Apache with mod_perl. The servers can handle 20/second each, so we were within our capacity this time. For comparison, after the September 3, 2000 Yountville earthquake, our server was brought to its knees by a peak of 193 questionnaires per minute. So the new server configuration is working well.

Here is the total traffic on the Menlo Park web site. Note that the peak hit rate is nearly 3,000/sec, with a data rate of 85 Mbits/sec. This is clearly far beyond the capacity of our pre-Akamai server setup. Despite this, this site was observed to be slow in the first few hours after the Gilroy event. The explanation for this is below.

Here is the traffic seen by the back-end servers for the Northern California web site. Note that it does not display the classic vertical spike followed by an exponential decay. This is an indication that the server is sick. At the time, we observed that we could not log in to either of the two servers, and they were not responsive to anything more complicated than a simple ping.

Here is an analog report on the logs from the Menlo Park servers.

Here is the smoking gun. There is a CGI program called helicorder.pl that displays real-time seismograms from selected stations in the network. This is a Perl script running under the Apache server. Note that calls to this peaked at 430 per minute soon after the event. The servers are a Sun Netra and a Sun Ultra-10. From our experience after the Hector Mine event, a Sun Netra takes several seconds to fire up Perl to run something like this. Having hundreds of them running simultaneously quickly consumes all available system resources and renders the machine catatonic.

As a test, the Apache Benchmark program was run to send a set of 100 helicorder.pl requests to one of the servers. During the test, the load average on the machine climbed from 0.5 to over 7. The report generated by the program indicates that the machine was taking on the order of 2.2 seconds to process each request. The test was repeated with 1,000 requests, and the load average was observed to be continuously rising, going over 10 by the time the test was aborted after 300 requests. This indicates that the machine was in a death spiral at the time.