- Put a text search facility into the new earthquake commentary and
station updates database interfaces.
- USGS staff meeting on Monday.
- M4.5 near Ludlow 06:16 Tuesday morning.
- Pages and emails about the event were delayed due to the Caltech campus
network being down from 06:16 to 06:55. The dialout modem on Pacific did
not work, so pages were delayed until the network came back up.
- The modem problem turned out to be a combination of the fact that the
modem wasn't plugged in to the phone line and that it was a different
kind of modem than the one the software was expecting to talk to. Swapped
the old modem back in so that the program would be happy.
- Moved Hebgen out of the Dark Room.
- The Lexmark Optra C had another obscure paper jam on Tuesday.
- 1/2 day vacation on Tuesday.
- Moved the EDIS serial line off of the old Lantronix LAT terminal server.
The old LAT server has been hanging a lot lately. Put it on port 16 on the
LRS dialup server. Called Ron Rosenow at OES to inquire about why we had
not been getting EDIS messages lately. He and Richard Osborne did something
to the satellite station in Sacramento and had me cycle power on our
receiver. Then messages started coming through.
- The AA RAID on Atlantic failed on Wednesday afternoon. It appears to
have had three disks go off line at once. The SDRV 1 and 3 arrays both
thought that they were using the hot spare disk. Talked to Eric Schultz
at Anacomp [nee nStor, nee Andataco]. He said, "It's not supposed to do
that. I've never seen it do that." Ended up having to delete and re-create
both those arrays to bring it back on line. The RAID was back on line
Thursday evening. Re-created the partitions and filesystems Thursday
night. Restored tape backups on Friday morning.
- Disk 11 on the Arena RAID on Pacific failed on Thursday night. Replaced it
with a spare. The unit was complaining about 'HDD 6 remap overflow'. Called
Electronix and they said that this was most likely due to some problem with
that disk and recommended that I replace it. Replaced it with another
spare and sent both bad disks back to Maxtor.
- Rebuilt and upgraded Sendmail on all systems on Monday in response to
a spam attack. Someone was using the two-part relay exploit to send
spam advertising the 'Tina Cam'.
- Fixed the AA RAID on Hotspot. The cabling was wrong after it had been
moved. Reinitialized the four disk arrays. Built a new wavepool area
on the RAID.
- Made a 'golden' account on Horst and Graben for Susan Rhea.
- The modem on Rift was broken. Replaced it with an old Hayes modem.
- Made the new database-driven earthquake commentary live on Thursday.
- Problems with the Sopac and AP connections. The LAT1 terminal server
was hung again.
- Put another 256M in Trinet-squid. Increased the size of the memory
file system for squid to use for caching files.
- Helped Karen with the fiber connection across the street. Found out
that the bottom pair of fibers do not work. Most likely this is due to
faulty termination. Dave Johnson said he would have a look at it and
replace the terminators.
- Moved Hebgen.
- The /home disk on Atlantic was full on Sunday morning. The evtdetect
log had eaten the entire disk. Talked to Mandy and she told me how to
stop and restart this process to clear the log file.
- Holiday on Monday.
- Network problems from 23:00 Sunday to 01:30 Monday. Routing problems
caused all the subnets to lose sight of each other.
- Net abuse allegations about the Bigquake mailing list. Some guy
claimed that the list server went berserk and sent him 900GB of junk.
Says his ISP is billing him $4000 for the bandwidth usage. This is
not possible, since 900GB over a four-day period is more than six times
the total bandwidth available to the bigquake mail server.
- Moved the Sopac and AP connections to ports 13 and 14 on the LRS server.
- Got back the two replacement disks from Maxtor on Tuesday.
- Atlantic hung on reboot on Tuesday. Turned out that Oracle can't deal
with the system having more than 10 IP addresses. Mandy simplified the
addressing to eliminate some of the private numbers.
- Asked Kimo to delete the MX record for bombay.
- Got and RMA for the failed disk from the Atlantic AA RAID.
- Got estimates for replacing the AA RAID with another Arena unit.
- Upgraded Solaris on Granite on Thursday.
- Mailed Charlene about creating the name service information about
opensha.org.
- Installed a new Openssh on Granite and Gneiss.
- M7.5 Kuril Islands event on Sunday. Generated several spurious events
on our network. Alarms all hung on sending the the EDIS message.
- Paul Friberg can't log in to Granite. Turns out it's the new OpenSSH.
It reads the system's hosts.allow file to decide who to allow to connect.
Added his IP address to the allow list.
- Upgraded Solaris on Pacific on Monday.
- Charlene set up DNS for opensha.org. Told Verisign about it.
- Cajon died. The SCSI B bus failed. Replaced it with the old Malibu to
run the tape drives.
- Upgrade Solaris on Magma on Wednesday. This turned out to be a huge
can of worms, since the OpenBoot PROM needed to be flashed with a new
version to run the new Solaris. The upgrade ended up taking most of the
day.
- Added Lisa's new home IP address to the allow lists on Sqehzmenlo, Horst,
and Graben.
- There was a problem with the analog data from Carizo on Monday afternoon.
Rebooted Carizo to try and clear it. The problem persisted overnight, due
to there being bad data in the analog stations' wavepool directories.
Mandy cleaned out the wavepool on Tuesday morning and everything started
working again.
- Figured out the reason why EDIS messages were hanging alarming. Part
of the EDIS procedure was to send a copy of the message over the AP
line. This portion of the script didn't know about how to talk to the
AP line, and was hanging while trying to talk to one of the serial ports
on the system.
- Built a new system to be a backup for Magma. It is an old Ultra-5 and
it will be called 'Pluton'.
- Thanksgiving Holiday on Thursday and Friday.
***
- The nStor RAID on Atlantic failed. Fixing it involved replacing one
disk, and completely reconfiguring and reinitializing the unit.
- Finished the online system OS upgrades.