Regular backups are your friend. Also, if you can’t normally back something up (i.e. data in open ldap’s backend) do a regular dump and back that up.

rsync/unison/scp data off your co-loc to a local machine, and back it up again, just for good measure.

The new drive is the drive most likely to die first.

Losing /var really fucking sucks.

Recreating most of the information in ldap out of mail logs is cool, though.

Drinking doesn’t solve sysadmin nightmares, but it makes you feel good while you’re having the nightmare.