Wednesday, 1st June, 2005

Reconstructive Surgery

I'm sorry, just what did I do to deserve that?

Yesterday morning, I check my emailand stuff just before I leave for Leeds. It was about 7am. Everything was fine. By the time I get to Leeds though, I try and ssh my way back home ... and nothing

Now, my network has been playing silly-biggers with me for the past couple of days (AirPorts turning themselves off and the like) so I figured "Ack, I'll be home by four, I'll just hit the reset button on the AirPort and it'll all be fine again"

Thing is though, when I do get home, Erin is making the most hideous crunchy noise, and it seems to be coming from her hard disk.

A few weeks ago, something similar happened: I'd written myself a cron job that backed up the databases to a folder and named them sequentially — but I'd set it to run every hour, and so soon enough my hard disk was full of .sql files, all 4MB each. That really screwed things up — all my mailboxes got corrupted and I lost 2 years worth of email.

I changed the script to run once a day very soon after that

So I thought something similar must have happened. After all, the noise was similar. Rebooting her didn't help though, and it became apparent quite quickly that there was something seriously wrong.

Something about a disk1s I/O error

Anyway, to cut a long story short, I've stuck a new hard disk in her, and I've spent the past twelve hours or so trying to salvage as much data from the borked disk as I can. I've completely lost siobhansplace.co.uk, along with my boy-site, all my spamd stuff is missing, but I have managed to get (obviously) Apache running properly again, along with Postfix and BIND.

And fortunately, all my MySQL databases were intact.

There's a lot of clean-up to do, so things will be shakey for a bit, but at least I didn't lose everything

You know what this is though don't you? This is bloody karma for what I said the other day about "computers will never be intelligent" ... "Fuck you, Siobhan," says Erin, "I'll show you just how intelligent I can be"

...

Here's the stuff from my little temporary blog BTW — I must confess, that using pure text files was a rather pleasant and refreshing change from struggling with a MySQL way of doing things. I might have to switch things over to that over the next week or so...

Tuesday, 31st May, 2005
================================================================

There is a very VERY big problem >
----------------------------------------------------------------

4.20pm:  Erin's hard disk is making crunching noises

4.44pm:  Scrabbling round the boxes in the bedroom to find 
         a disk with "MacOSX Server Installation" written on it.

5.08pm:  Main hard disk is dead — trying to pull what I can 
         from it.

5.09pm:  Not a good time to use words like "back up"

5.30pm:  Think I'm going to be able to pull most of the stuff 
         intact. I'll have to reinstall on a new disk though

6.04pm:  Mail all backed up ... Webserver likewise. Just 
         preparing the new disk for clean install.

         (I really didn't need this today BTW. Good job I'm 
         on holiday eh?)

6.07pm:  Hoping *desperately* that the bit of the disk with my 
         Postfix configuration isn't borked.

         (Kinda cute this isn't it? It's like a mini-emergency
         blog. I'm being rather calm considering about two
         hours ago I thought I'd lost every web page that I've
         ever written)

6.22pm:  Damm. The httpd logs and the system logs are corrupt
         meaning I've (a) lost a year's worth of webserver
         stats, and (b) I'm not going to be able to try and 
         work out what went wrong today

7.04pm:  Finished getting everything ready on the spare disk.
         Just going to get a few important bits and pieces off 
         the main disk (or as much as I can really), then 
         reinstal the server. I reckon as long as I can get the 
         webserver and the mailserver up and running tonight, 
         then that'll be enough to earn me a drink 

7.14pm:  Oh yeah — *and* DNS as well

7.39pm:  A problem: The drive that I'm about to Erase And 
         Install on, is the one I've been using to keep all my 
         MP3s and Fonts on. Copying the MP3s was easy — but 
         copying the fonts from the command-line hasn't worked, 
         because they're all Macintosh *forked* files. :-/ 
         Time for a rethink...

8.13pm:  hdiutil to the rescue  Created a disk image of the 
         FONTS folder, checked it to make sure, now all set to 
         install.

         (Started on the wine already. Probably a bad move)

8.59pm:  Erin is back up and running, albeit as a fresh server. 
         What I've got to do now is restore all the Apache, 
         Postfix, MySQL and BIND stuff...

         Weirdly, when I tried to log in, she would only let 
         me do so as root.

9.17pm:  Wish I'd backed up my NetInfo stuff — it would have 
         saved me having to give everyone new passwords. (of 
         course, *actually* I wish I'd backed up the *whole 
         server* — but that's hindsight for you I guess)

9.25pm:  Time to eat I guess. Erin is Software-Updating 
         herself to 10.3.9, so I'm twiddling my thumbs for a 
         bit.

         Tell you what though — I'm liking this *using Text-
         Edit to write your blog* malarky  Using a MySQL 
         database has all *sorts* of problems (or at least it 
         does when you're as inept at using it as I am) After 
         I get things back to normal, I *must* look into 
         switching to a text-file driven blog. Much, *much* 
         easier to maintain...

10.02pm: It's looking rather too optimistic to get Postfix 
         *and* Apache *and* BIND running tonight : The 
         imporant one is Postfix — so I'll concentrate on 
         that.

         (See? I *knew* I should have laid off the wine until 
         I was all done)

10.18pm: Dear 80.220.226.139,

         Will you *please* stop asking for my favicon?! It 
         doesn't exist at the moment. How many 404s do you 
         need to convince you?

11.12pm: I've had enough of this for one night >

         It's not that there's anything wrong with the server 
         now — it's just that half the files from the old hard 
         disk are corrupt. I've lost (potentially) an entire 
         dark-purple website, and all the .php files from a 
         blue one. All the light-purple files are OK, but some 
         of the stuff is spread across three different domain 
         names.

         Plus, even though the mailserver's up and running 
         again, I need to pull off all my DNS configurations 
         from the fscked disk. And to do that means putting it 
         *back* into the machine — which I don't think I 
         should do right now.***

         Dammit, I *wish* I knew what happened this morning. 
         It must have been while I was on the way to work. It 
         was fine when I left, but screwed when I got to the 
         office.

         Apologies to all the people I host stuff for — but 
         well, if yous were paying me for it, then you could 
         shout at me 

         ***Just to elaborate on that for a second. While I was
         installing, I set the domain name of my server to what
         it used to be. But I hadn't grabbed the DNS settings
         off the old hard disk befre I took it out. All I need
         to do is copy them off that (fingers crossed that
         they're still in one piece) — but I'll have to open her 
         up and slot the HD in to do that, and well, I'm a bit 
         drunk. So I probably shouldn't

1.32am   Against my better judgement...

         1.     DNS is working (sort of). 
         2.     Webserver back online (but not live — databases 
                not restored yet)
         3.     Email working again. 
         4.     Webmistress is very very drunk.

         This *would*, I think, be a lot easier if I knew what I 
         was doing...

Well, I was thinking that all I use the database for is (a) storing the entries, and (b) the searching. So if convert the whole db to flat files, to make it easy for me to edit, then have a script that pulls out stuff from the flat files and shoves it in the db if the file was modified in the last 24 hours, then that might be a cunning way around it — essentially indexing the files

Whoops

And your boy name is lurking there...

In a commented-out link no less! I'd forgotten that was there

p.s. things happen it three's

Oh gee — thanks for that

No, don't do that — she's fully of worthiness. If you want sheer unadulterated trannie-whinging then I think you're really got to come to it's source.

That reminds me, I noticed someone else had got themselves a weblog

See? She's rubbish — at least I know the right smiley-code to use to storm out properly

No no no, I'm a tinkerer, and tinkerers take backups even less than pros