Thursday, February 10, 2011

The worst of times, the best of times

There's an old saying "Just once I wish I could have the silver lining without the cloud?" well once again it wasn't my turn.

A week ago I mentioned how I was exhausted after cleaning up domains and sites. Well I continued cleaning and improving over the weekend and by Sunday night was ready for a distraction. I had an on-going problem that my VM was CentOS 4 and current versions of Mediawiki won't run on the XML library in there. I've been looking at upgrading to version 5 for a while but the documentation is scary and advises starting again from scratch. The problem is I didn't really have a good way of doing that, so I decided I needed a distraction. I decided to have a play with the b2evolution blogging software instead. What makes this interesting is that with some minor set-up in the control panel it can support multiple blogs in the one directory. Unlike its cousin (both descend from the now dead b2/cafelog) Wordpress they did this years ago with no add-ins, tweaks, etc. Just point both domains at the save virtual server & tell b2evolution about the blog and you're going.

Although not ideal, blog software will usually do service as CMS substitutes, and Wordpress is often used as a straight CMS so I think b2evolution should easily be able to be made to do the same.

Unfortunately my experimentation was cut short by the arrival of 9:37 PM. Suddenly my site was dead. Not just the domains on b2, but all of them. Dead, dead, dead. I went to my hosting company's control panel & the log-in was broken. It flashed up something about not being able to contact the machine my VM was on then reloaded itself ... in a loop. I logged onto their CRM and raised a support ticket. Just over half an hour later they let me know that the disk had crashed hard and they were creating a new VM for me on a different node, what operating system would I like. I decided to ask for CentOS 5 ... at least Mediawiki should work :)

The rebuild of the machine was finished 11:30 PM, but by then I'd gone to bed, deciding to get up at 5:30 AM. I started the restore before going to work and it ran all day (Slow ADSL) before I stopped it. I arranged to take Tuesday off and between Monday morning and early this morning I've been alternating between uploading backups and restoring the sites based on them. I still have some photos to upload for an on-line gallery, but that can run over-night.

The Good
I've got my upgrade to CentOS 5 so Mediawiki works again.

I got nearly everything back. My MediaWiki backups were corrupt, but the sites had been down for so long (see above about XML library) that the Installatron backups had everything.

I had a bit of a mess with sign-ons and distribution of sites against them on the machine. I've restructured.

Diana's business site and blog were recovered intact.

All our business sites and my mini-sites were recovered intact although I did have to massage a couple of database back-ups.

One piece of luck was that I had a hosting control panel with a full list of domains, domain aliases and related users open when the crash happened, I cut and pasted the HTML into a libreoffice document.

The Bad
After months of neglecting it, Tessa had just done two postings on her Unknown Chef blog. These were lost.

It's taken forever to upload data. My home ADSL set-up is too slow for using as a backup mechanism
Some of my database backups were corrupt. I will need to use a better method of backing up the tables.

I didn't restore things in the best order. For example, I develop them on my local machine here and I have a fully automated shell script that publishes Tessa's Sweet Expectations site and another that publishes my minisites. I didn't think to run them early Monday. Another mistake was that  HotDomains, one of my most important sites was down until this morning as I clean forgot to restore it.

Those Mediawiki sites are rubbish ... I so need to redevelop them.

The Strange
Once I restored the DNS on and similar, my blogs were unaffected ... of course that's because I host them on Blogspot. I host other people's Wordpress blogs and store my own on Blogspot. Go figure.

In the backups of the server user space was a directory containing my complete user space from my earlier hosting before I moved to my current hosting. Inside that was a complete backup of the system I migrated from to go there ... talk about carrying history. Other than that my suer space was badly littered with discarded files and archives.

I knew I had got my MX records right, 5 minutes later the spam started arriving again followed an hour or so later with real email. For once I was glad to see spam ... sort of like a canary in a coal mine (or rats in the drain) to show the air is breathable.

7:30 tonight I decided to draw a line under the recovery effort by declaring success. I know I need to finish uploading photos, and I need to have a post mortem on the whole experience. I need better backups, I need a plan for restoring so I don't forget important sites and so-forth; but I'll look into that another day.

Tonight I've spent blogging and watching TV.

... and Zebedee said "Time for bed"

Update 16-Feb
I've since discovered that I haven't restored a few minor sites, not sure if I'll even bother with them I also missed recreating some name server records so quite a few parked domains stopped working for a week (and in some cases two weeks). The whole mess has put back deployment of mini-sites by over a week, luckily the next large batch of developed mini-sites wasn't due until later this week.

No comments: