M6.8 Seattle Event: Web Traffic Report
The M6.8 earthquake centered near Seattle, WA provided a good test of the USGS
Earthquake Hazards Program's web service capabilities. This is a look at how the
main program page and the Pasadena office web site fared after this event.
The earthquake occurred at 10:54 PST on Wednesday, February 28, 2001. It was reported
felt over a wide area of the Pacific Northwest, and as far away as Salt Lake City. Traffic on
all the USGS earthquake web servers began increasing almost immediately. Most servers
were quickly crushed by the avalanche of traffic, and were rendered unresponsive
for several hours following this event. The two exceptions were the main
Earthquake Hazards Program page at http://earthquake.usgs.gov and
the
USGS Pasadena web site at http://pasadena.wr.usgs.gov. The Pasadena
web site serves as the main center for the
Community Internet
Intensity Map web pages, also known as the 'Report an Earthquake' pages.
This site allows people to submit an online questionnaire to report the intensity of an
earthquake at their location. Data from the submitted questionnaires is used to build
a map showing the reported earthquake intensity. Both of these sites are
using Squid reverse-proxy servers
as http
accelerators. This effectively splits the load on the web server and
improves web service capacity by about an order of magnitude.
This proved decisive in the
aftermath of the Seattle earthquake. The story of how we arrived at this
configuration is in
Web Servers, Earthquakes, and the Slashdot Effect. This
article describes how our web server was flooded after the Hector Mine
earthquake, and our response to this event.
 |
 |
 |
earthquake.usgs.gov - total traffic
Figure 1 |
Traffic on earthquake.usgs.gov peaked at 710 hits/sec, and 12.8 Mb/sec data rate.
The curve shows the characteristic shape of an earthquake-driven traffic
spike. The rise is nearly vertical, followed by an exponential decay.
The earthquake.usgs.gov site is served off of a pair of Sun Solaris
servers located in Menlo Park
and Reston. There are three Squid servers acting as a front-end for the site.
Figure 1 shows the traffic on each of the three servers. Load sharing is done
with standard round-robin DNS, so the load balance is not perfect, but good
 |
earthquake.usgs.gov - total traffic
Figure 2 |
enough for this application. The highest traffic on a single Squid server was
248 hits/sec, and 4.5 Mb/sec data rate. Testing indicates that a single Squid
server of this type is capable of handling about 400 hits/sec and a data rate
of about 60 Mb/sec, so we did not exceed the limits of the system this time.
Figure 2 shows the aggregate traffic on all three servers.
 |
USGS Pasadena - total traffic
Figure 3 |
Ordinarily, earthquakes outside Southern California do not have a
significant impact
on the Pasadena Office web server. However, this changed with the
national rollout of the
Community Internet Intensity Map. This is an online facility for
people to submit their experiences in an earthquake. This data is used to
build a map showing the distribution of reported shaking.
Figure 3 shows the total traffic seen on the Pasadena web server.
 |
Traffic detail - Pasadena web site
Figure 4 |
Most of the people coming to the Pasadena site were referred from
earthquake.usgs.gov, which is why the peak is not as sharp as the
peak seen on that server.
The peak in Pasadena was 124 hits/sec, and 21.8 Mb/sec. Note that
the data rate in Pasadena was actually higher than the rate for
earthquake.usgs.gov. This is because the CIIM map page for an
event grows with the number of submitted reports. This event
generated an unprecedented volume of responses, so the map
page became quite large.
Figure 4 shows detailed traffic information for the Pasadena web site.
Note that traffic to the CIIM pages makes up almost all of the traffic.
Also note the purple line on the graph. This represents completed
questionnaires submitted for processing. The rate of incoming
questionnaires never went above 1/sec. The Pasadena server is
capable of processing 20/sec, so we had a comfortable safety margin.
This shows that the improvements we made to our web server after
the
September 3, 2000 Yountville earthquake were worthwhile.
 |
USGS Pasadena Squid Server - CPU Usage
Figure 5 |
The only dark spot for the Pasadena server is shown in Figure 5. This shows
CPU usage on the Squid server. Note that from about 12:30 to 15:00, the
server was 100% busy. Traffic was being served, and the site was accessible
during this time. However, the server was observed to be lagging slightly, and
was just a little slow.
The red line indicates CPU time spent on system
functions, typically disk IO. This indicates that the machine needs more
memory. Also, there may be some more kernel tuning that could improve
performance under heavy loads.
We are currently exploring options for upgrading this machine and improving
its performance.
Still, all things considered, the Pasadena and Earthquake Hazards Program
web servers performed quite well under the onslaught.
- Stan Schwarz
- Honeywell Technical Services
- Southern California Seismic Network Contract
- Pasadena, California
01 Mar, 2001 09:42 PST
