Embracing Chaos

It took a long time, but EmbracingChaos has finally escaped form Typepad.  About a year ago (just before the end of the previous billing cycle) I started trying to move this blog to blogger.  I like Google’s pace of development and wanted to hop on the blogger train and get automatic upgrades for everything they do.  But ultimately I didn’t because I couldn’t make blogger meet all of my requirements for migration:

• Keep all blog posts and comments
• Keep all posts at their original URLs
• Maintain all category pages at the same URLs

The first one’s easy.  Google released some migration tools which cover that quite well.  But, at least when importing from typepad / movable type, they don’t preserve permalink URL’s.  So anybody who followed a linked to a specific page on my site would get a 404 page.  Weak.

I spent a lot of time on this.  Basic problem is that Typepad doesn’t include URL information in their export file format. It would be very easy for them to do this, but then why would they want to make it easy for you to leave?  Actually the answer there is easy.  Because by trying to lock in users, they create angry vocal opponents of their service.  I’m not angry, but I would advise against anybody considering Typepad as a blog host, specifically because of their tendency to lock people in.

<rant> Don’t keep my data hostage.  It’s my content.  I created it.  You’re just delivering it.  Do not try to lock me into using you as a service provider.  You might get some more money out of me, but every dollar I give you after I want to leave will contribute to my dis-liking you.  As the internet matures and consumers become more sophisticated and better able to share their experiences with each other, they will increasingly choose the service providers who are open.  (Echoing Jonathan Rosenberg’s recent diatribe on openness.)  I really appreciate Google’s commitment to Data Liberation.  My current provider, Dreamhost, also does a splendid job of giving me control over my data.</rant>

It turns out that getting a full-fidelity export out of typepad is possible with some work.  I followed these instructions from FolioVision which provides a custom export template that does include URL’s.  If your blog has more than 100 posts, then you need to change the first line to

<MTEntries lastn="100">

<MTEntries lastn="100" offset="100">

<MTEntries lastn="100" offset="200">

etc. and merge all these files together into one big export file that has URL’s.  Then I tried to get blogger to honor the import file with permalinks but I couldn’t.  I do believe blogger is capable of doing this, but what ultimately turned me away from it was that it doesn’t seem to offer any way to honor links like www.embracingchaos.com/humor for category listings.  Which I like and get a lot of visitors on.  So I went with wordpress.

FolioVision helpfully posted a custom wordpress import plugin to match their typepad output template, which makes it all go.  Once that’s done, you have to move all the attachments hosted at typepad, and then there’s a bunch of wordpress configuration, and moving your analytics and favicons and finally switching DNS.

So here we are.  Please tell me if you notice anything amiss with the new site.

I recently blogged about my efforts to move from TypePad to Blogger.  My friend Brian pointed out that Google just announced a set of tools to convert to or from popular blog export formats, including MovableType which uses the same file formats as my TypePad.  The converters are open source, distributed under the Apache license, so you can download the code and run them on your local machine.  Or, if your blog isn’t too big, you can run the code hosted on AppEngine by going to

• http://movabletype2blogger.appspot.com/
• http://wordpress2blogger.appspot.com/
• http://livejournal2blogger.appspot.com/

Well I tried this with my blog, and the resulting file it spit out was almost empty.  I think my blog is just a bit too large, since when I ran it on my local machine it came out to 1.03 megs.  So if your blog is smaller than mine you can probably use the online tool.

After a couple of bX-xji785 errors, the file imported into blogger about as well as could be expected, which is to say okay but not great.  The blog is mostly there.  Feel free to take a peak at http://leo-embracingchaos.blogspot.com/ but please don’t make any permanent links to that URL as it’s really just for testing.  The posts and drafts all made it with the right dates and times, along with the comments and tags.  But as previously noted, the TypePad export format does not include URLs.  So if I were to actually use this conversion, all the inbound links to pages other than the homepage of my blog would break, which is totally unacceptable for me.

I started a thread on the discussion group if you’d like to follow along.

For a while now I’ve wanted to move this blog from Typepad to something else like Blogger.  I keep finding more reasons to do this as Blogger improves and Typepad stagnates.  Some reasons include:

• Better WYSIWYG editing in Blogger.  (I can’t change any font characteristics in Typepad without going into raw HTML. Ugh.)
• Typepad’s lack of support for Chrome, Google’s awesome new browser
• The awkward and limiting way Typepad assigns human-readable URLs to posts
• Blogger is free, while Typepad costs >$100 per year
• Cool new features being added to Blogger like Followers
• Adding new features in Typepad is a p.i.t.a. (i.e. I’m too old to be messing with perl-based templating languages.  Did more than my share of that in 1996.)

Typepad has long supported an export feature, which dumps out all the posts, drafts, comments, etc into a text file.  And now blogger has an import feature, which takes an XML file that looks kinda like an Atom feed.  I looked around for a tool to convert between the two and found only others asking the same question. 

So I thought here’s a chance for me to contribute something to
society and help myself at the same time.  Converting one text file to
another shouldn’t be that hard.  So I rolled up my sleeves and started
playing with it.  I realized one problem fairly quickly — the typepad export format
doesn’t include the URL for each post.  I really don’t want to break
all the previous inbound links because
that’s how people get to my content.  But I’m going to need to crawl
the old blog to get those old links.

Since the blogger format looks like a regular Atom feed, I thought I’d try to just grab the Atom feed off the blog and import it into Blogger.  Well to that, blogger said:

Blogger does not currently support import files generated by TypePad.

Fine.  But what are the differences?  The blogger format includes a bunch of fake "entry" elements which are really configuration — like a giant "layout" node and a bunch of other settings.  So I tried grafting the two together — just taking the legitimate "entry" nodes from the typepad feed and putting them into the blogger export that includes all the layout and settings.  By converting everything to look like the blogger format, it imports, but that also loses information.  So I set about looking for the minimal set of changes that will work.

One that definitely breaks it is a side-efect of being hooked-up to feedburner.  There’s extra stuff in there like the feedburner:origLink tags:

    <feedburner:origLink>http://www.embracingchaos.com/2008/12/repairing-a-deg.html</feedburner:origLink>

These break the import, which shouldn’t be much of a surprise since the feedburner: namespace isn’t defined in the Blogger export.

I gave up looking for today because blogger was having too many server errors on import.  They’re intermittent, so I was happy to refresh through them for a while.  But at some point 90% of my imports gave me funny error codes like bX-gxxw6w so I’m giving up for now. 

I wrote a stub of a python program to do the merge.  I called it typepad2blogger.py and happily donate it to the public domain.  It’s definitely not done and is useful only insofar as it shows one way to approach the problem and has enough stuff in place that somebody else should be able to get a running start.  I’ll hopefully continue work on this at some point.  But even a couple hours of work on it has triggered my RSI so I’ll have to take a break.  If you want to pick this up, let me know and we can coordinate efforts.

For everybody who has been waiting with baited breath to hear whether or not the repair of the RAID array worked or not, it did. It took several days, but since we were away on vacation seeing my dad receive the Fleming Medal from the American Geophysical Union, the waiting was pretty easy.

To convince myself that the repair was successful, I unplugged one of the previously functional drives, and saw that all my files were still there when the array was running just on the new drive and the other previous drive. I recommend this to anybody who thinks they’re running a RAID system — until you’ve seen the RAID array work with a drive removed, how can you be sure it’s really working? If your system is set up better than mine is, you’ll get some kind of warning message too.

A while back, lightning scrambled one of the disks in my home RAID 5 array.  I figured out how to recover it.  And I got the critical data off.   Here I describe the steps I took to add a new drive and get it working with the old RAID array.  I share this with the net in hopes it will make it easier for somebody else who has to go through this process themselves, and selfishly as notes for me to refer to.  It’s a testament to the power of EVMS and a warning to anybody who thinks it might be fun to run their own open-source software RAID server at home. 

My advice for people seeking reliable storage: go with a hosted solution.  Understanding the arcane nuances of these software systems is an extremely specific skill that doesn’t translate well to many real-life necessities.  If you’re smart, you can figure it out, but it doesn’t teach you much of anything except how to do exactly that.  Each person who understands this stuff should be keeping petabytes of data happy, rather than one couple’s pictures and music collections.  I hear Microsoft’s "home server" actually makes this pretty easy, but I can’t recommend anybody willingly lock themselves into Microsoft’s business model.

Background

So I bought a new drive, following my own advice about picking drives from different manufacturers when building a raid array, and plugged it in to the mobo and booted the machine.  After futzing with /etc/fstab to get it to find the boot disk and load up, I logged into evms and got these messages:

MDRaid5RegMgr: RAID5 array md/md1 is miissing the member  with RAID index 0.  The array is running in degrade mode.

and

MDRaid5RegMgr: Region md/md1 is currently in degraded mode.  To bring it back to normal state, add 1 new spare device to replace the faulty or missing device.

Conceptually easy.  I’ve got a new 500 GB drive in the system.  Linux sees it.  It didn’t take me too long to figure out it’s called /dev/sda, while the previous 2 disks in the array are sdb and sdc, with a small boot drive at sdd.  Now the fun part is figuring out EVMS terminology enough to tell it to use the new disk.

The hierarchy of the array in EVMS land seems to be as follows:

• Logical Volume teraraid (contains)
• Region md/md1 (which contains)
• Segments sdb1 and sdc1 (which are built on)
• Logical disks sdb, sdc.

What I tried, and what seems to have worked

I see that logical disk sda has no segments.  So I try Action -> Create -> Segment.  It only gives me one choice for "Segment Manager" which is "GPT Segment Manager."  But when I choose it, it doesn’t let me make a segment on sda.  Only the tiny free space on sdb and sdc.  So sda needs something else done to it before we can use it.  What?

sda also shows up in the list of Logical Volumes, next to Teraraid and the formatted boot partition.  Hmmm.

Well I tried converting it to an EVMS Volume.  It complained that sda does not have a File System Interface Module (FSIM) associated with it, but it made the new logical volume anyway.  This wasn’t getting me anywhere.  So I erased it.

Next I tried "Add" -> "Segment Manager to Storage Object".  I noticed that all of the Disk Segments associated with the array were listed as using "Plug-in" "GptSegM" and this gave me the choice of adding Gpt Segment Manager to sda.  W00t.  I said "No" to make this a system disk.  This seems to be working.  Now I see a bunch of Disk Segments starting with sda, including a big one (465 GB) labelled sda_freespace1. 

Now when I tried to Create -> Segment, it let me use GPT Segment Manager on sda_freespace1 and allocate a 450 GB disk segment to match the others.  (I left 15 GB off each disk with the idea I could put a boot segment in that space, but I’ve never gotten around to it.)

Now in "Available Objects" there is sda1 with 450.0 ready for me.  Alrighty we’re getting there.

Now I look at "Storage Regions" and in the context menu for md/md1 I see an option that says "Add spare to fix degraded array…"  I didn’t see it there before — it might have not shown up when there weren’t any spares, or maybe I was just being thick.  In any case, selecting it now gives me a menu with one choice — sda1. 

Now in details of md/md1 it shows:

 Na┌──────────────────── Detailed Information - md/md1 ─────────────────────┐ ──│                                                                        │── lv│     Name                    Value                                      │ lv│ ────────────────────────────────────────────────────────────────────── │ lv│     Major Number            9                                          │ md│     Minor Number            1                                          │   │     Name                    md/md1                                     │   │     State                   Discovered, Degraded, Active               │   │     Personality             RAID5                                      │   │ +   Working SuperBlock                                                 │   │     Number of disks         3                                          │   │ +   Disk 1                  sdb1                                       │   │ +   Disk 2                  sdc1                                       │   │     Number of stale disks   1                                          │   │ +   Stale disk 0            sda1                                       │   │                                                                        │   │                                                                        │   │                                                                        │   │                                                                        │   │                                                                        │   │    Use spacebar on fields marked with "+" to view more information     │   │                                                                        │   │ [Help]                                                          [OK]   │   │                                                                        │   └────────────────────────────────────────────────────────────────────────┘

That last line about the Stale disk is new.

Actions -> Save commits these changes to disk.  Now looking at Detailed information for md/md1 shows

 Na┌──────────────────── Detailed Information - md/md1 ─────────────────────┐ ──│                                                                        │── lv│     Name                    Value                                      │ lv│ ────────────────────────────────────────────────────────────────────── │ lv│     Major Number            9                                          │ md│     Minor Number            1                                          │   │     Name                    md/md1                                     │   │     State                   Discovered, Degraded, Active, Syncing =  0 │   │     Personality             RAID5                                      │   │ +   Working SuperBlock                                                 │   │     Number of disks         3                                          │   │ +   Disk 1                  sdb1                                       │   │ +   Disk 2                  sdc1                                       │   │     Number of stale disks   1                                          │   │ +   Stale disk 0            sda1                                       │   │                                                                        │   │                                                                        │   │                                                                        │   │                                                                        │   │                                                                        │   │    Use spacebar on fields marked with "+" to view more information     │   │                                                                        │   │ [Help]                                                          [OK]   │   │                                                                        │   └────────────────────────────────────────────────────────────────────────┘

Emotionally I feel like I should be done now.  But I don’t hear the thrashing noise of a half-terabyte of of checksums being unwound and copied onto a fresh disk.  And it says "Syncing = 0".  Hmmm.

I quit evmsn and reload it to see two new messages.  One familiar:

MDRaid5RegMgr: Region md/md1 is currently in degraded mode.  To bring it
back to normal state, add 1 new spare device to replace the faulty or missing device.

And one novel:

MDRaid5RegMgr: RAID5 array md/md1 is missing the member  with RAID index 0.  The array is running in degrade mode.  The MD recovery process is running, please wait…

But this novel message saying it’s recovering is "Number 0" implying that it came before the other message (Number 1) which tells me I need to take action for it to fix itself.  And the drives are not thrashing.  Again I look at the details for md/md1 and now I see:

 Na┌──────────────────── Detailed Information - md/md1 ─────────────────────┐ ──│                                                                        │── lv│     Name                 Value                                         │ lv│ ────────────────────────────────────────────────────────────────────── │ lv│     Major Number         9                                             │ md│     Minor Number         1                                             │   │     Name                 md/md1                                        │   │     State                Discovered, Degraded, Active, Syncing =  0.3% │   │     Personality          RAID5                                         │   │ +   Working SuperBlock                                                 │   │     Number of disks      3                                             │   │ +   Disk 1               sdb1                                          │   │ +   Disk 2               sdc1                                          │   │ +   Disk 3               sda1                                          │   │                                                                        │   │                                                                        │   │                                                                        │   │                                                                        │   │                                                                        │   │                                                                        │   │    Use spacebar on fields marked with "+" to view more information     │   │                                                                        │   │ [Help]                                                          [OK]   │   │                                                                        │   └────────────────────────────────────────────────────────────────────────┘

Which really seems to say its doing its thing.  Maybe I don’t hear the disks because it’s formating the disk first, which is a linear process.  Or maybe the whole copy process is very linear and I won’t hear it thrashing.  Its progress implies it’s going to take a couple/few days to finish, which is what I’d expect.  So maybe it’s working. I’ll let it run for a while and see what happens to the array if I try to unplug one of the previously working drives.

Pretty cool that I didn’t even need to unmount the array to do this.

Now if I could just figure out why my laser printer periodically decides it needs to print it internal test page, I’d be even happier.

Last week I went to a different kind of Dork Bot meeting here in Seattle called the Catastrophic Cacophony Workshop.  It was a maker-style event where we broke up into teams of people and in 90 minutes had to build a robotic musical instrument out of one supplied motor and whatever other parts we could scrounge together. 

Our team started with the basic idea of a repetitive percussive instrument with multiple hammers hitting multiple objects.  So we constructed a platform to hold the motor and used tennis rackets to make a rotating frame that held a series of arms that struck objects as they went around.  We then built frames to hold up things like pot lids or copper pipes to be struck, and attached everything from small chains to bundles of spaghetti to the rotating frame to strike them.  The real enabling insight came from the women on the team who realized that by spacing the arms un-evenly we could create an interesting rhythm.  The one-two-and-three rhythm across a variety of instruments ended up sounding much like a drum circle that actually got people dancing.

Thanks to Espressobuzz for capturing our creation on video.  Thanks to Josh Kopel for bringing the event to Seattle, and Shelly Farnham for organizing.  And to my wonderful teammates whose energy and creativity made the Spinning Spaghetti Monster possible.

I love my Toyota Prius.  I’m very grateful to Google for giving me
a financial incentive to buy a brand new car which does everything I
want.  It’s totally big and practical, it’s very high-tech, it both
supports and projects my values,
etc.  I could sing its virtues for pages.  But for now, I want to share
some non-obvious things about it in a series I’m calling "Stupid Prius
Tricks."  (With a tip of a hat to David Letterman.)  Several of these articles (like this one)
are complaints, but this must be taken in the context that overall I love the car.

The Prius’s High-Tech Safety Features

The Prius has at least a couple of smart traction control systems which use technology
to help maintain control of the car in difficult situations.  Of course
there’s anti-lock brakes (ABS).  In addition to ABS the braking system
does things called Electronic Brake-force Distribution (EBD) and Brake
Assist.  Beyond that, it’s got Traction Control, and in all but the
stripped-down cars something called Vehicle Stability Control (VSC).
What each of these features does, or how they differ from each other I
don’t know and don’t frankly care.  They help the car do what I’d want
it to do under good driving conditions when the road is slick or uneven
or I’m driving too fast or what have you.  All this is great.  But
there’s one subtle and very bad flaw in the implementation of at least
one of these systems.

The Stupid UX for these Features

When the Prius needs to engage one of its TLA control systems, an
orange light appears on the otherwise dark dashboard.  Sometimes a
sound will beep.  This is presumably done to let you know that the
fancy feature that you paid for is working on your behalf.  Call it a
marketing feature if you will, because there’s nothing you can do with
this information, or way to react to it within the vehicle’s controls.
The stupid thing about this feature is that it’s distracting to have
lights flashing and sounds beeping.  When you’re having trouble
controlling the car is the last time you want something distracting
you.  So the engineers at Toyota have done something completely moronic
– a complex system senses when you’re having trouble controlling your
car, and chooses that time to throw more distractions at you.  I’ve
observed the light and beeping  on when driving over bumpy roads, or
down
something very steep that has a rut in it, or when the back-right wheel
accidentally jumps up the curb.  None of these situations I’ve gotten
into have been serious or very dangerous.  But even still, they were a
bit confusing, and the confusion was confounded by having the car meow
at me demanding attention.  When the car is not doing what you expect it to, the last thing you want is additional distractions.  Bad choice, Toyota.  These features should be absolutely silent.

What you can do about it

If anybody knows a cheat code to disable this light or beeping, please
post it in a comment.  Until then, you can help a little bit by putting
a small piece of electrical tape over the orange light with the symbol
for "I think you’re losing control of the car."

The EVMS RAID 5 array in my linux fileserver crashed recently due to a lightning storm, and I thought I’d lost everything.  But with some luck and intuition I was able to recover all my files.  I’ll tell you how I did it, so hopefully others who run into similar problems can recover their data too.  But first, a little background.

Last week Seattle had some crazy electrical storms.  In recent years’ storms, my block has done better than most with respect to power failures making me think we’re either lucky or in a particularly robust section of the grid.  So I was a little surprised to find my whole house offline on Wednesday morning.  After a bit of debugging I figured out that the small UPS that runs all my networking gear got toasted, and for some reason the file server was down.

I left it alone for several days, and when I got around to turning it back on, I was happy that the whole stack through the samba server came up by itself.  (It doesn’t always!)  But when I started looking around I quickly realized things were amiss.  The media/video directory normally has 4 subdirectories: movies, episodic TV, imake and other.  But today it listed:

leo@elephant:/raid/shares/media/video$ lsdpisndic TV  hmakd  movies  nther

WTF!?  A few bits had been scrambled in the directory names.  This sounds really bad.  Moreover, even though the first couple levels of the directory hierarchy were there, but no files were to be found.  Definitely a problem.

Step 1: As soon as you suspect your RAID array has a problem, stop writing to it until you know what’s going on.  Writing changes can make things worse.  Stop the bleeding.   

I unmounted the drive from my mac, not trusting Finder or Spotlight to sprinkle damaging meta-files over the array.  Once I remembered how to ssh into the box, I stopped the samba daemon,

leo@elephant:/$ sudo /etc/init.d/samba stop

unmounted the filesystem

leo@elephant:/$ sudo umount /raid

and changed fstab so it would be read-only when it comes back, and that it wouldn’t come back without me asking.

leo@elephant:/$ sudo vi /etc/fstab

changing

/dev/evms/teraraid500 /raid ext3 defaults  0 0

to

/dev/evms/teraraid500 /raid ext3 ro,noauto  0 0

I tried poking around in EVMS by running

leo@elephant:/$ evmsn

But it hung during initialization with blue dialog saying "Discovering segments…"  I’m thinking EVMS can’t help me.  After a bit of googling I thought I should try e2fsck or some such.  First, I tried to mount it again read-only and see what’s there.

mount: wrong fs type, bad option, bad superblock on /dev/evms/teraraid500,       missing codepage or other error       In some cases useful info is found in syslog - try       dmesg | tail  or so

Bad superblock.  Uh oh.  Well this guy managed to recover a drive with a bad superblock.  Lots of things were pushing me in this direction — fix the filesystem.  But I realized that was a mistake.

Step 2: Do not make changes at the filesystem level until you’re confident that the RAID array is working properly.  You set up RAID for a reason.  You’ve still got a chance to recover everything, but if you start
making changes to it in a broken state, you’re almost certainly going
to make things worse.

Me to self: Think about it.  EVMS is confused.  Linux is confused.  Ext2 and ext3 are messed up complaining about bad superblocks.  The problem was caused by lightning.  When the drive was mounted there were wierd bit-level corruptions in the data that were still there.  Maybe one of the drives in the array got data scrambled, but didn’t get totally fragged so it went offline.  RAID 5 is designed to survive total loss of a single drive.  But if a drive gets corrupted, who knows what will happen.  So I came up with this plan:

Step 3: Try physically disconnecting the drives in your array, one at a time.  If only one of them is scrambled, disconnecting it should restore all the data in the array.

Having followed my own advice, it’s easy for me to tell the drives in my array apart since each drive in the RAID array is from a different manufacturer (which makes array failure due to manufacturing defects far less likely). 

This plan actually worked perfectly!  Removing a drive caused a bit of a hassle in getting the machine back up, because when I booted it couldn’t find the /boot partition complaining

 * Starting Enterprise Volume Management System...[42949392.340000] raid5: raid level 5 set md1 active with 2 out of 3 devices, algorithm 0

 * Checking all filesystems...fsck.ext3: No such file or directory while trying to open /dev/sdd5/dev/sdd5:The superblock could not be read or does not describe a correct ext2 filesystem.  If the device is valid and it really contains an ext2 filesystem (and not swap or ufs or something else), then the superblock is corrupt, and you might try running e2fsck with an alternate superblock:    e2fsck -b 8193 <device>

Notice the complaint about the superblock again — don’t trust it, and don’t do what it says!  What really happened was that the boot drive letter had been changed from /dev/sdd to /dev/sdc, so I had to change /etc/fstab to mount /boot from  /dev/sdc5 instead of /dev/sdd5.  In my system, I boot off a non-RAID disk attached to the mobo, which for some annoying reason gets the last drive letter after all the drives no the SATA card.

But once I got past this, it quickly turned out that the Samsung drive was the culprit.  With it removed, the software RAID kicked in and plugged all the whole.  Everything the array looked
completely normal again.  All the directories.  All the files.  Hooray!

XKCD is a rad web-comic.  Today’s makes passing yo-mama jokes while discussing the concept of the "28-hour day." 

It’s an interesting idea.  Six 28-hour days make up a week.  6 x 28 = 7 x 24 = 168 hours.  {math corrected} If you sleep 8 hours out of 28, then you’re getting 48 hours of sleep a week, which is equivalent to just under 7 hours of sleep a night on a regular schedule.  (6 hours 51 minutes).  Personally, I’m optimal at about 7 hours 25 minutes a night on a long-term basis, but I can go 6 hours a night for quite a while and stay upright.

To see what this might be like with my schedule, I put XKCD’s schedule into a public Google calendar.  It doesn’t fit with my school schedule next quarter, so I won’t be trying it.  But if you give it a try, let me know how it works.

Or here’s the ical version.

Here’s a little hacker tid-bit for those of you with phones running WM5.  It’s the closest thing to task manager that I’ve found without installing a 3rd party app.  If you want to get a list of running tasks and actually stop one as opposed to push it into the background, follow these steps:

• Press Start
• Select Settings
• Choose the System tab
• Run Memory
• Choose the Running Programs tab

Happy terminating!