While looking for the source of database backup errors I found it, the hard way. I have been running my database on a separate hard disk from everything else, and have been getting occassional errors from the cron script that does a nightly dump. I was under the impression that this was due to latency causing the script to time out. Not the case.
Looking at the dump it seemed like everything was being written ok, at least it looked that way yesterday. (I have not had much time for site maintenance, so this has been in the “put it off until later” pile.) While looking at the script this afternoon, and trying another run it timed out again (or so I thought). I figured I could put it off until this weekend, until I went to look at the site and got the big “Unable to contact the database” error. I went the server and fired up mysql on the command line and discovered that there were NO DATABASES! I tried to get a file listing of the /var/lib/mysql directory and got nothing. Nada. Since I don’t seem to be able to get anything off of that disc I did a quick modification of the fstab (to remove the line mounting that drive), rebuilt the dbs from the last (failed) backup, and here I am, missing two months worth of data.
Can I cry now?
In case you are wondering, the Margin vs Markup page is still available (as I made it a regular page as well as a post, it was my most popular ever).
EDIT: Ah the joys of using decade-old equipment.