Weekly log #362 for week ending 01-Mar-2002

Swapped the hardware between Trinet-squid and Flint. For some unknown reason, Squid never ran well on the Athlon, and was actually slower than on the K6-2.
Fixed a deep paper jam in the 535 Lexmark printer.
Event 12458384 was missing from the Simpson map. Turned out it did not meet the criteria for generating a QDDS message, even though it generated an email and page.
Upgraded Flint to FreeBSD 4.5.
Built JDK 1.3 on Foreshock.
Put up an online RSVP page for the office open house.
Wrote up a brief report on the web traffic generated by the M5.7 on February 22. The report is http://bort.gps.caltech.edu/spikes/22feb2002/
Assembled the new racks for the computer room in 525.
Upgraded PHP on Bort in response to the file upload vulnerability.
Took a vacation day on Friday.

Weekly log #363 for week ending 08-Mar-2002

Made accounts for Anthony on Bigone and Iron.
Put replacement memory in Jet. At the same time, one of the old kits failed. Mandy contacted Digital Basics.
ITS blocked JD's computer when the MPAA contacted Caltech complaining of DMCA violations regarding pirated video clips on the computer. Removed the machine's network cable to prevent it from coming back on the network.
Matt's old machine, scfad, was hacked and being used as a spam relay. Turned it off.
Set up some scripts to keep a database of the DHCP address pool. The pool can be seen at http://bort.gps.caltech.edu/stan/net/66net/dhcp.php
Upgraded Foreshock and Pluton to FreeBSD 4.5.
Made a 'rock' account on Agent86 for Steve Rock. He is doing some work for Ned.
Vacation day on Friday.

Set up a mysql database for Eqexperts.
Helped Mike Watkins with configuration on the Monterey Park router.
Disk at 0,7 on Pluton failed on Monday. Power cycle brought it back.
Disk at 0,7 on Pluton failed on Tuesday. Power cycle brought it back.
Did some mix-and-match with SIMMs on Jet. Got it to come up with the full 8GB of memory working.
Talked to Kiril at Akamai about a problem with a router in Reston that was preventing some of their servers from seeing the NEIC origin servers.
Vacation day on Friday.
Jet crashed with memory errors at 17:20 on Friday.
Pluton crashed and hung at 21:57. Came in and did a manual fsck to get it back up.

Jet crashed with more memory errors at 18:18 and 19:56 on Saturday. Came in to remove the offending memory bank. Found that Busby had put the plexiglas door on the rack, which was causing the machine to overheat. Removed the memory and the door.
M4.6 offshore and M3.2 events on Saturday at 13:33 and 21:50. Both generated modest web traffic spikes.
Put the memory back in Jet on Monday. With the door off the rack, it did not log any more errors.
Pluton locked up hard at 21:30 on Monday.
Trinet-squid crashed and rebooted at 00:11, 00:44, 01:01, 01:28, 03:02, 06:03, and 06:14. Swapped out its motherboard for an identical one.
Jet crashed at 05:00 and 07:20. This may have been related to the fact that its system console is hooked up to the serial port on Trinet-squid.
Pluton locked up again at 11:00 on Tuesday. Had to power cycle to get it to reboot.
Pluton locked up again at 05:19 on Wednesday. After rebooting, it locked up again, and then it would not boot at all. It locked up during self-test.
Kimo set up an export from Martha to Hotspot for testing.
Built a new wavepool area on Martha for Hotspot to use for testing.
Office Open House on Friday.

Ehzsouth, Hotlava, Sangabriel, and Genie were all broken into at about 19:00 on Saturday. Discovered this on Sunday when Big Brother was complaining about the /var partition on Ehzsouth filling up.
Martha locked up at 15:17 on Sunday.
Researched the breakin. It appears to have been a published vulnerability in dtspcd. Most machines were patched on January 16, but at the time there was no patch for 2.7, and I just plain forgot about Ehzsouth.
Fixed Ehzsouth, Sangabriel, and Hotlava. Did a newfs and restore on /usr and /var.
Kimo fixed Martha, and Hotspot was able to reboot successfully on Thursday.
Took a vacation day on Friday.

***

Got the racks assembled and installed in the new computer room.
Set up a database to track usage of the USGS DHCP address pool. This keeps track of the computers using addresses from the pool, and maintains a list of which ones are on line at any given time.
Jet and Spring both experienced memory errors this month. Jet overheated on the weekend of March 16-17 and crashed several times with severe memory errors. Fixed the overheating problem and swapped SIMMs in both systems to eliminate the memory errors.
Set up a database for the USGS Earthquake Experts list. This includes a dynamic interface that allows searching the database for particular geographic areas or areas of expertise.
Several machines were broken into this month. Fixed the damage and closed up the hole that was used to get in.