One does not simply SSH into Mordor

Today has been apeshit crazy, and it’s only noon. Besides the usual tidying/dog walking/plant watering I tend to do in the morning before work, I had to rush out some code for a project and get a braketag.

Except, it never seems to be that easy. I’ve been having issues with my reverse lights lately. It seems they work fine right between when the car starts and before you put it into drive. Luckily at the brake tag station, reverse lights were tested before I had to move the car.

I also have a headlight with water in it, and it isn’t secured to the frame very well. It won’t fall out or anything, but it’s loose. The brake tag guy said that he couldn’t give me a tag for that, but if I went home, drilled some small holes so the headlight would drain now and in the future, and made an effort to secure the headlight, all would be good. I drilled the holes, and showed that I had run the suggested bead of caulk (which I don’t thing will work long term, but that’s what he suggested), and drove over. When the tag hit the window I got the hell out, and met up with Amy Jett so she could take the truck for the weekend to move.

Also, I made this for you Linux users and fans of The Lord of the Rings. It’s not hard to imagine the Venn diagram for that – just draw a circle.

One does not simply shell into Mordor

 
I’m sure this isn’t an original idea, the “One does not simply _____ into Mordor” being an established meme and all, but I have yet to see an SSH implementation – of course I haven’t looked either.

Downtime and EXT3 undelete

The other day, the server I share with the other bicycle/computer nerd Will (yes, there are two of us) went down.

I called my friend Paul who basically runs the datacenter the box lives in to give him a heads up I’d be heading over. Paul was already having a rough work day, but still carved out a little time to help me out – thanks again P.

Anyways, we hooked up a crash cart, and here’s what we saw:

Garble

 
Ruh-roh. We could discern that it was asking for the root password for maintenance, and that was about all we could glean from the garbled screen. Unfortunately neither Paul nor Will knew the password, and if you tried to skip it, it would just reboot. This meant I had to pull it and work on it elsewhere.

I went to boot from a cd, but the server had none, so I tried the next step – putting a bootable disc image onto a USB thumb drive. That didn’t work as the hardware is old enough to not be able to detect bootable USB devices.

I hadn’t yet opened up the case (which I should have done earlier), so I went the only route that I had available to me at the moment: floppy boot disks.

Holy crap y’all, I made linux boot floppies. I know, I know, I as stunned as you are. I was not only surprised to find that I still had some floppies, but that they were still good! As it turned out, it too was all for naught – when booted up, I couldn’t really do anything for one reason or another.

I finally opened the damn case, and to my exultant joy, I spied an IDE port. I plugged in a CD-ROM and booted up a Gentoo install disc. After fixing the screen garble (which was due to a splashimage being loaded by Grub), I was able to see and address the actual problem: somehow the EXT2/3/4 fsck had gone missing.

Before I actually fixed the issue, I figured it would be a good time to back some stuff, and due to an unfortunate timing of keystrokes, I managed to delete the entire home directory, residing on its own partition. Dumb.

After immediately unmounting the partition and some serious googlation of various terms including EXT3 undelete, EXT3 restore, oh holy hell what did I just do, linux undelete, etc… I had pretty much lost all hope of restoring this data. Luckily, at the bottom of some forum thread, someone had posted a link to an app called ext3grep. This page and application, considering how completely amazing it is, has an extremely poor google listing. Hopefully this may help. I doubt it, but you never know.

Anyways, after crapping myself, here’s all it took to restore the deleted files:

umount /home
cd /usr/src
wget http://ext3grep.googlecode.com/files/ext3grep-0.10.1.tar.gz
tar -zxvf ext3grep-0.10.1.tar.gz
cd ext3grep-0.10.1
./configure; make; make install
ext3grep /dev/sda8 –restore-all -after=1252972801

There’s actually a bit more, but that’s superfluous things like changing to a directory with enough free disk space. But the pure simplicity of this app is that once downloaded / compiled / etc, is that with a single command, you can restore files deleted after a certain timestamp.

If for no other reason that finding a working, powerful, and simple ext3 undelete utiliy, the headache was all worth it.

Here’s a quick photo of the server on my desk at work:

Patient on the Operating Table

 
I still can’t believe I deleted the /home directory – sometimes you just hit that Enter key a mere moment before you actually hit the tab key a second time. At least I learned from this mistake.