Downtime and EXT3 undelete

The other day, the server I share with the other bicycle/computer nerd Will (yes, there are two of us) went down.

I called my friend Paul who basically runs the datacenter the box lives in to give him a heads up I’d be heading over. Paul was already having a rough work day, but still carved out a little time to help me out – thanks again P.

Anyways, we hooked up a crash cart, and here’s what we saw:

Garble

 
Ruh-roh. We could discern that it was asking for the root password for maintenance, and that was about all we could glean from the garbled screen. Unfortunately neither Paul nor Will knew the password, and if you tried to skip it, it would just reboot. This meant I had to pull it and work on it elsewhere.

I went to boot from a cd, but the server had none, so I tried the next step – putting a bootable disc image onto a USB thumb drive. That didn’t work as the hardware is old enough to not be able to detect bootable USB devices.

I hadn’t yet opened up the case (which I should have done earlier), so I went the only route that I had available to me at the moment: floppy boot disks.

Holy crap y’all, I made linux boot floppies. I know, I know, I as stunned as you are. I was not only surprised to find that I still had some floppies, but that they were still good! As it turned out, it too was all for naught – when booted up, I couldn’t really do anything for one reason or another.

I finally opened the damn case, and to my exultant joy, I spied an IDE port. I plugged in a CD-ROM and booted up a Gentoo install disc. After fixing the screen garble (which was due to a splashimage being loaded by Grub), I was able to see and address the actual problem: somehow the EXT2/3/4 fsck had gone missing.

Before I actually fixed the issue, I figured it would be a good time to back some stuff, and due to an unfortunate timing of keystrokes, I managed to delete the entire home directory, residing on its own partition. Dumb.

After immediately unmounting the partition and some serious googlation of various terms including EXT3 undelete, EXT3 restore, oh holy hell what did I just do, linux undelete, etc… I had pretty much lost all hope of restoring this data. Luckily, at the bottom of some forum thread, someone had posted a link to an app called ext3grep. This page and application, considering how completely amazing it is, has an extremely poor google listing. Hopefully this may help. I doubt it, but you never know.

Anyways, after crapping myself, here’s all it took to restore the deleted files:

umount /home
cd /usr/src
wget http://ext3grep.googlecode.com/files/ext3grep-0.10.1.tar.gz
tar -zxvf ext3grep-0.10.1.tar.gz
cd ext3grep-0.10.1
./configure; make; make install
ext3grep /dev/sda8 –restore-all -after=1252972801

There’s actually a bit more, but that’s superfluous things like changing to a directory with enough free disk space. But the pure simplicity of this app is that once downloaded / compiled / etc, is that with a single command, you can restore files deleted after a certain timestamp.

If for no other reason that finding a working, powerful, and simple ext3 undelete utiliy, the headache was all worth it.

Here’s a quick photo of the server on my desk at work:

Patient on the Operating Table

 
I still can’t believe I deleted the /home directory – sometimes you just hit that Enter key a mere moment before you actually hit the tab key a second time. At least I learned from this mistake.

4 responses to “Downtime and EXT3 undelete”

  1. Matt says:

    Good find! Adding to my “oh shit” box.

  2. Brock Tice says:

    Neat utility! Always a harrowing experience when that kind of thing happens, but part of me finds it a little fun too. Only a little.

  3. Fred says:

    Be blessed. You save my night, and one of my fellow’s 2 weeks of hard work.

    Finally we perhaps have a chance to deliver on schedule.

    No kidding, it’s a SHAME this tool is not better advertised.

  4. Sophmom says:

    I find most of my typss are a result of my right hand getting ahead of my left hand. Glad this had a happy ending.

Leave a Reply

Your email address will not be published. Required fields are marked *