M5.2 offshore near San Diego, June 15, 2004

There was a M5.2 centered about 49 miles WSW of San Diego at 15:29PDT or 22:39GMT. It was widely felt in the San Diego area, and even felt as far north as some parts of Los Angeles. This event created a surge of traffic on the all of the California earthquake web servers.

Servers That Did Well

Here is the traffic on the Earthquake Hazards Program web site, the Western Region web site in Menlo Park, and the Pasadena web site, as reported by Akamai. The Program site peak is about 87.5 requests/sec, Menlo Park peaked at 310/sec, and Pasadena had 66/sec. On all sites, the peak occurred 5 minutes after the event. These three sites are all served through the Akamai EdgeSuite service, so none of these sites experienced any availability problems after this event.

Note that the Menlo Park graph displays the 'classic' curve of a rapid rise, followed by an exponential decay. The graphs for the Program and Pasadena sites are a bit different. The Program site traffic fall-off is slower because a special report about the event was linked off of all the earthquake web sites. This made the server a major secondary destination, keeping the traffic up for some time after the event.

In the case of the Pasadena server, the Community Internet Intensity Map was the secondary destination that drove traffic to the site after the event. There were nearly 7,000 questionnaires submitted for this event in the first hour, which works out to about 116 per minute on average, with a peak of 205 per minute. There were no problems processing this input. In addition, the Pasadena earthquake mailing lists gained 428 new subscribers in the first 8 hours after this event.

Servers That Did Less Well

These graphs show the traffic on the SCEDC, CISN, and Trinet web sites. In each case, the shape of the curve deviates from the 'classic', which indicates that the web server was unable to keep up with the load. In the case of the CISN site, it is unclear whether it had a problem or not, since the level of traffic on it should not have been a problem. However, the site was observed to be serving error messages shortly after the event. A check of the server configuration shows that 'MaxClients' is set to 200, which caps the number of simultaneous connections allowed. This is most likely the culprit. The CISN web server is a dual-Xeon, which is a very capable machine. From my experience with similar machines, a value of 2048 would be more appropriate. There is a hard-coded limit of 256 on this in Apache, so it will be necessary to recompile the web server in order to raise this value.

The SCEDC web site displays the same curve as after the San Simeon event. Note that in both cases, the server topped out at about 150 requests/second. The SCEDC web server hardware is slated for replacement in the near future, which should improve their situation somewhat.

The Trinet server peaked at about 380 requests/sec, and the shape of the curve on the graph indicates that the server was at its limit. This is about as expected, since in my report on the M4.1 and M4.3 near San Fernando in 2001, I estimated that a single Squid server such as is used on the Trinet site is good for about 400 requests/sec.

Finally, the GPS department web server is not the first that we 'in the business' think of for earthquake response, but it is clear that there are a lot of other people out there who do. The graph of activity on this server clearly shows a clipped peak, and the server was observed to be unresponsive during this time.

What to Make of This

As could be expected, the sites served through Akamai did not have problems coping with the increased traffic after the event, while the other sites did. This just reinforces the point that felt earthquakes can generate traffic surges larger than a single web server can handle. These surges come on quickly, and their size cannot be easily predicted.

Stan Schwarz
Honeywell Technical Services
Southern California Seismic Network Contract
Pasadena, California
June 17, 2004