Downtime and EXT3 undelete
September 17, 2009
The other day, the server I share with the other bicycle/computer nerd Will (yes, there are two of us) went down.
I called my friend Paul who basically runs the datacenter the box lives in to give him a heads up I’d be heading over. Paul was already having a rough work day, but still carved out a little time to help me out – thanks again P.
Anyways, we hooked up a crash cart, and here’s what we saw:
Ruh-roh. We could discern that it was asking for the root password for maintenance, and that was about all we could glean from the garbled screen. Unfortunately neither Paul nor Will knew the password, and if you tried to skip it, it would just reboot. This meant I had to pull it and work on it elsewhere.
I went to boot from a cd, but the server had none, so I tried the next step – putting a bootable disc image onto a USB thumb drive. That didn’t work as the hardware is old enough to not be able to detect bootable USB devices.
I hadn’t yet opened up the case (which I should have done earlier), so I went the only route that I had available to me at the moment: floppy boot disks.
Holy crap y’all, I made linux boot floppies. I know, I know, I as stunned as you are. I was not only surprised to find that I still had some floppies, but that they were still good! As it turned out, it too was all for naught – when booted up, I couldn’t really do anything for one reason or another.
I finally opened the damn case, and to my exultant joy, I spied an IDE port. I plugged in a CD-ROM and booted up a Gentoo install disc. After fixing the screen garble (which was due to a splashimage being loaded by Grub), I was able to see and address the actual problem: somehow the EXT2/3/4 fsck had gone missing.
Before I actually fixed the issue, I figured it would be a good time to back some stuff, and due to an unfortunate timing of keystrokes, I managed to delete the entire home directory, residing on its own partition. Dumb.
After immediately unmounting the partition and some serious googlation of various terms including EXT3 undelete, EXT3 restore, oh holy hell what did I just do, linux undelete, etc… I had pretty much lost all hope of restoring this data. Luckily, at the bottom of some forum thread, someone had posted a link to an app called ext3grep. This page and application, considering how completely amazing it is, has an extremely poor google listing. Hopefully this may help. I doubt it, but you never know.
Anyways, after crapping myself, here’s all it took to restore the deleted files:
umount /home
cd /usr/src
wget http://ext3grep.googlecode.com/files/ext3grep-0.10.1.tar.gz
tar -zxvf ext3grep-0.10.1.tar.gz
cd ext3grep-0.10.1
./configure; make; make install
ext3grep /dev/sda8 –restore-all -after=1252972801
There’s actually a bit more, but that’s superfluous things like changing to a directory with enough free disk space. But the pure simplicity of this app is that once downloaded / compiled / etc, is that with a single command, you can restore files deleted after a certain timestamp.
If for no other reason that finding a working, powerful, and simple ext3 undelete utiliy, the headache was all worth it.
Here’s a quick photo of the server on my desk at work:
I still can’t believe I deleted the /home directory – sometimes you just hit that Enter key a mere moment before you actually hit the tab key a second time. At least I learned from this mistake.
The Joys of Home Ownership
September 14, 2009
I know most people say the phrase “Ah, the joys of home ownership” in a sarcastic tone, but I really enjoy it. I work in computers, and for the most part, don’t get to literally get my hands dirty. There’s just something enjoyable about working with and building/fixing tangible items when the bulk of what you build and work with is software.
There have been house projects that I quite honestly haven’t been a fan of, and some I have. One of them, more recently, has been the middle roof. When I say middle, I mean that from the front to the back, there are three different pitches. The front is the highest (covering the master bedroom/bathroom and office), the middle pitch covers the kitchen, den, and part of the back room, and the third and lowest/least steep pitch covers the remainder of the back room and the back porch.
To help facilitate, I spend some time geeking out this afternoon and built a basic 3D model of my house – it’s actually pretty damn close to actual measurements* – maybe off by an inch here, an inch there. Here’s the view from the rear:
And a photo where the perspective does the length of roof no justice:
That dark / wet looking section is actually about 11′ long – the entire middle pitch is about 32′ long. The rear is 25′, and the front is almost 47′ long.
What aggravates me is the design of the middle. It doesn’t extend the entire width of the house, so there is a flat area. Usually, I guess that wouldn’t be a big deal, except there’s a little wall there! What the hell is it doing there?
As far as I can tell, it’s a water-retention “feature”. And by feature, I mean SNAFU. What happens is that water just builds and as the level rises, it goes underneath the singles, back into the house. Great job there, whoever you are.
So, I’ve been “sealing” the base of the shingles with roofing cement (basically tar). I don’t really mind doing it, I just wish I didn’t have to do this patch.
*If you said “hey, the roof peaks aren’t centered – they don’t line up. You messed up!”, then you’re wrong. The roof peaks aren’t centered. Again, I don’t know why. It’s not really a big deal.