To convince myself that the repair was successful, I unplugged one of the previously functional drives, and saw that all my files were still there when the array was running just on the new drive and the other previous drive. I recommend this to anybody who thinks they’re running a RAID system — until you’ve seen the RAID array work with a drive removed, how can you be sure it’s really working? If your system is set up better than mine is, you’ll get some kind of warning message too.

My advice for people seeking reliable storage: go with a hosted solution.  Understanding the arcane nuances of these software systems is an extremely specific skill that doesn’t translate well to many real-life necessities.  If you’re smart, you can figure it out, but it doesn’t teach you much of anything except how to do exactly that.  Each person who understands this stuff should be keeping petabytes of data happy, rather than one couple’s pictures and music collections.  I hear Microsoft’s "home server" actually makes this pretty easy, but I can’t recommend anybody willingly lock themselves into Microsoft’s business model.

Background

So I bought a new drive, following my own advice about picking drives from different manufacturers when building a raid array, and plugged it in to the mobo and booted the machine.  After futzing with /etc/fstab to get it to find the boot disk and load up, I logged into evms and got these messages:

MDRaid5RegMgr: RAID5 array md/md1 is miissing the member  with RAID index 0.  The array is running in degrade mode.

and

MDRaid5RegMgr: Region md/md1 is currently in degraded mode.  To bring it back to normal state, add 1 new spare device to replace the faulty or missing device.

Conceptually easy.  I’ve got a new 500 GB drive in the system.  Linux sees it.  It didn’t take me too long to figure out it’s called /dev/sda, while the previous 2 disks in the array are sdb and sdc, with a small boot drive at sdd.  Now the fun part is figuring out EVMS terminology enough to tell it to use the new disk.

The hierarchy of the array in EVMS land seems to be as follows:

• Logical Volume teraraid (contains)
• Region md/md1 (which contains)
• Segments sdb1 and sdc1 (which are built on)
• Logical disks sdb, sdc.

What I tried, and what seems to have worked

I see that logical disk sda has no segments.  So I try Action -> Create -> Segment.  It only gives me one choice for "Segment Manager" which is "GPT Segment Manager."  But when I choose it, it doesn’t let me make a segment on sda.  Only the tiny free space on sdb and sdc.  So sda needs something else done to it before we can use it.  What?

sda also shows up in the list of Logical Volumes, next to Teraraid and the formatted boot partition.  Hmmm.

Well I tried converting it to an EVMS Volume.  It complained that sda does not have a File System Interface Module (FSIM) associated with it, but it made the new logical volume anyway.  This wasn’t getting me anywhere.  So I erased it.

Next I tried "Add" -> "Segment Manager to Storage Object".  I noticed that all of the Disk Segments associated with the array were listed as using "Plug-in" "GptSegM" and this gave me the choice of adding Gpt Segment Manager to sda.  W00t.  I said "No" to make this a system disk.  This seems to be working.  Now I see a bunch of Disk Segments starting with sda, including a big one (465 GB) labelled sda_freespace1. 

Now when I tried to Create -> Segment, it let me use GPT Segment Manager on sda_freespace1 and allocate a 450 GB disk segment to match the others.  (I left 15 GB off each disk with the idea I could put a boot segment in that space, but I’ve never gotten around to it.)

Now in "Available Objects" there is sda1 with 450.0 ready for me.  Alrighty we’re getting there.

Now I look at "Storage Regions" and in the context menu for md/md1 I see an option that says "Add spare to fix degraded array…"  I didn’t see it there before — it might have not shown up when there weren’t any spares, or maybe I was just being thick.  In any case, selecting it now gives me a menu with one choice — sda1. 

Now in details of md/md1 it shows:

 Na┌──────────────────── Detailed Information - md/md1 ─────────────────────┐ ──│                                                                        │── lv│     Name                    Value                                      │ lv│ ────────────────────────────────────────────────────────────────────── │ lv│     Major Number            9                                          │ md│     Minor Number            1                                          │   │     Name                    md/md1                                     │   │     State                   Discovered, Degraded, Active               │   │     Personality             RAID5                                      │   │ +   Working SuperBlock                                                 │   │     Number of disks         3                                          │   │ +   Disk 1                  sdb1                                       │   │ +   Disk 2                  sdc1                                       │   │     Number of stale disks   1                                          │   │ +   Stale disk 0            sda1                                       │   │                                                                        │   │                                                                        │   │                                                                        │   │                                                                        │   │                                                                        │   │    Use spacebar on fields marked with "+" to view more information     │   │                                                                        │   │ [Help]                                                          [OK]   │   │                                                                        │   └────────────────────────────────────────────────────────────────────────┘

That last line about the Stale disk is new.

Actions -> Save commits these changes to disk.  Now looking at Detailed information for md/md1 shows

 Na┌──────────────────── Detailed Information - md/md1 ─────────────────────┐ ──│                                                                        │── lv│     Name                    Value                                      │ lv│ ────────────────────────────────────────────────────────────────────── │ lv│     Major Number            9                                          │ md│     Minor Number            1                                          │   │     Name                    md/md1                                     │   │     State                   Discovered, Degraded, Active, Syncing =  0 │   │     Personality             RAID5                                      │   │ +   Working SuperBlock                                                 │   │     Number of disks         3                                          │   │ +   Disk 1                  sdb1                                       │   │ +   Disk 2                  sdc1                                       │   │     Number of stale disks   1                                          │   │ +   Stale disk 0            sda1                                       │   │                                                                        │   │                                                                        │   │                                                                        │   │                                                                        │   │                                                                        │   │    Use spacebar on fields marked with "+" to view more information     │   │                                                                        │   │ [Help]                                                          [OK]   │   │                                                                        │   └────────────────────────────────────────────────────────────────────────┘

Emotionally I feel like I should be done now.  But I don’t hear the thrashing noise of a half-terabyte of of checksums being unwound and copied onto a fresh disk.  And it says "Syncing = 0".  Hmmm.

I quit evmsn and reload it to see two new messages.  One familiar:

MDRaid5RegMgr: Region md/md1 is currently in degraded mode.  To bring it
back to normal state, add 1 new spare device to replace the faulty or missing device.

And one novel:

MDRaid5RegMgr: RAID5 array md/md1 is missing the member  with RAID index 0.  The array is running in degrade mode.  The MD recovery process is running, please wait…

But this novel message saying it’s recovering is "Number 0" implying that it came before the other message (Number 1) which tells me I need to take action for it to fix itself.  And the drives are not thrashing.  Again I look at the details for md/md1 and now I see:

 Na┌──────────────────── Detailed Information - md/md1 ─────────────────────┐ ──│                                                                        │── lv│     Name                 Value                                         │ lv│ ────────────────────────────────────────────────────────────────────── │ lv│     Major Number         9                                             │ md│     Minor Number         1                                             │   │     Name                 md/md1                                        │   │     State                Discovered, Degraded, Active, Syncing =  0.3% │   │     Personality          RAID5                                         │   │ +   Working SuperBlock                                                 │   │     Number of disks      3                                             │   │ +   Disk 1               sdb1                                          │   │ +   Disk 2               sdc1                                          │   │ +   Disk 3               sda1                                          │   │                                                                        │   │                                                                        │   │                                                                        │   │                                                                        │   │                                                                        │   │                                                                        │   │    Use spacebar on fields marked with "+" to view more information     │   │                                                                        │   │ [Help]                                                          [OK]   │   │                                                                        │   └────────────────────────────────────────────────────────────────────────┘

Which really seems to say its doing its thing.  Maybe I don’t hear the disks because it’s formating the disk first, which is a linear process.  Or maybe the whole copy process is very linear and I won’t hear it thrashing.  Its progress implies it’s going to take a couple/few days to finish, which is what I’d expect.  So maybe it’s working. I’ll let it run for a while and see what happens to the array if I try to unplug one of the previously working drives.

Pretty cool that I didn’t even need to unmount the array to do this.

Now if I could just figure out why my laser printer periodically decides it needs to print it internal test page, I’d be even happier.

Diversifying from hardware

Since Jobs pulled the company out of the doldrums, Apple has been a manufacturer of high-end hardware.  This is a very nice place to be. Their margins are very high.  If you compare the hardware bits that go into a Mac to those in any PC, Apple’s prices are much higher.  PC manufacturers squabble over low single digit margins because their product is almost completely commoditized. HP, Dell and IBM struggle to differentiate themselves in the market.
Meanwhile Apple can charge a hefty premium for good industrial design and software that is slicker than windows.  iPod followed in this tradition of high margins by setting the bar for usability in portable
media devices and following up with fantastic marketing.

Being a niche retailer of high-end products is a comfortable and stable place to be.  However, having a single line of revenue isn’t good for a large company, so diversifying makes a lot of sense.  Thus Apple’s current push into services.

The iTunes music store was an important pre-cursor to the current push into services.  Even though iTunes moves a massive volume of music, if you work through the accounting confusion, they’re not actually making much money there.  Because the established music industry giants are completely fucked and have no idea what to do, they drive the margins of all big online retailers down to a pittance.  But iTunes did something very important for Apple: it established a billing relationship with customers. Cellular operators have this kind of ongoing billing relationship with their customers which enables them to push high margin, low utility products like ring-tones. Similarly, people are used to spending small quantities of money in iTunes to get music. So iTunes is the perfect precursor for an App Store.  In this way, Apple doesn’t even need to rely on the carrier’s billing relationship
to build a services business.  This will be very important for Apple in coming years as carriers increasingly become just another provider of wireless bandwidth.  In the coming decades, the value will not come from piping bits around.  It will come from the services built on those pipes. 

MobileMe is a very Apple service

MobileMe is a cloud-hosted email, contact and calendaring solution. There’s nothing revolutionary about this.  Google offers all of these services for free.  By charging for these services, Apple is implicitly promising to provide a better solution.  Considering their vertical integration into popular hardware, it’s not hard to imagine that they will succeed at this.  Google will probably remain committed to supporting open standards for working with hardware.  By using proprietary protocols, Apple can provide a higher-quality product and support it better on the few platforms they care about.  It’s a classic story we’ve seen in this industry before.

MobileMe’s launch was a disaster.  This isn’t surprising as Apple is yet to demonstrated great skill in online services.  Personally I believe they’ll figure it out, because it’s important to their long-term objectives, and they’re a smart company, and the skills to do this well are getting easier and easier to find. But from a marketing perspective, MobileMe is nothing new for Apple.  Because MobileMe is a premium service for which there is a very good free alternative, MobileMe still targets Apple’s classic market segment. They’re still targeting people who are willing to pay extra to have something really polished.  They can stick to their classic bag of tricks, like the ads that make fun of people who aren’t as cool as the mac devotees.

App Store is new: a platform play

App Store on the other hand is going to be much harder for them to pull off.  For App Store to succeed, their primary challenge is not to attract paying customers, but developers.  Because independent software developers (ISVs) are the ones who are actually creating value in the App Store.  App is just a distributor taking a cut on that.  So what attracts developers to the App Store?  Customers do.

It’s a real chicken and egg problem.  Nobody’s going to build software unless there are customers to buy it, and it’s hard to get customers without cool apps.  Well right now Apple has the whole farm, but it’s a very small farm.  There are millions of iPhones out there, representing potential customers.  Moreover, the iPhone SDK is very rich and capable when compared to its competitors.  Qualcomm’s BREW, Windows Mobile, Palm OS, Blackberry and Symbian have all faltered for one reason or several.  Google’s Android holds much promise and hope, but at this
point it’s complete vapor-ware.  So for now, Apple has almost all the mindshare of mobile application developers.

But how long will this dominance last?  Software platforms are natural monopolies, meaning the economics tend to be winner-take-all.  The chicken and egg “problem” can easily turn into a virtuous cycle, pushing a winner to the top. Application developers are fickle and will code to whatever platform has the best distribution.  iPhones are very popular, but they are still only used by a small fraction of all mobile subscribers.  Until the distribution of iPhones reaches a critical mass, their dominance as a mobile application platform is very shaky.

The mobile app challenger is HTML

But the laundry list of alternatives shows that the competition is fragmented.  What could unseat Apple?  IMHO it’s not another application platform, open or proprietary.  It’s the web.  Every high-end phone can display web pages, and increasingly they’re using high-quality javascript engines that can run real web applications. Webkit, the super-fast open-source HTML/JS engine behind Safari is showing up in Symbian devices, Blackberries and Windows Mobile phones.

Apple’s challenge is to make the proprieray iPhone SDK compelling to developers.  The alternative is to build a web application that works well on webkit, and works on every high-end phone.  They can attract developers in two basic ways — make the native features of their SDK more compelling, and provide a large market for distribution of the applications.

Charging for distribution of these applications is a gimick that won’t last long.  Soon all the interesting applications will be free, but tied to cloud services that have their own business models independent of the mobile client.  Premium applications will start to seem a lot like premium ringtones pretty fast.  Still, it will help bootstrap this market for Apple so long as there are no serious competitors.

To stay on top, iPhones need distribution quickly

One thing that ties all these points together is that Apple’s continued success with App Store hinges on having wide distribution of iPhones. They are currently throttling the distribution of iPhones for some reason, possibly
because of software glitches.  But the aggressive $199 pricing is clearly aimed at attracting a new larger customer base that will help maintain their dominance in the mobile application space.

Long term they might be happy getting by offering premium versions of applications that are freely available on the web.  But something tells me they’re actually trying to break open the mass market on this one. This is Jobs’ big play.  It’ll be really interesting to see how it works out once Android hits the streets.

New iPhones are hard to get

The blogosphere is full of speculation about whether or not Apple deliberately made the iPhone scarce on opening day and since then.  Most assume that this is deliberate on Apple’s part for a variety of reasons, mostly to attract more attention, increase demand, etc.  I assume most of these rants are from bloggers who want their new iPhones but haven’t overcome the barriers to obtain one yet.

But if Apple’s goal was purely to meter out their distribution, why not sell them online?  To get a phone you need to place an order for one, wait a week or two, and then you can get it.  This seems reasonable in conditions of scarcity.  But to get an iPhone 3G, you need to walk into an at&t store to place your order, and then walk into the store again to pick it up.  Think about this.  If the limitation was purely lack of supply then there are several ways this could be easier for customers:

1. You could order a phone online to be delivered to your house.
2. You could order a phone to be delivered to your nearest at&t store.
3. You could call the nearest at&t store to place your order, but still have to walk in to pick it up.

Try asking them why you can’t do any of these things and they will answer with one word: policy.  Clearly Apple & at&t have gone out of their way to make it difficult for people to get their hands on a phone.   This goes above and beyond just preserving a limited supply.  You have to work to get an iPhone 3G.

New iPhones have Issues

From all the reports I’ve read, the problems with the new iPhone are in the software not the hardware.  I conclude this because my friends with first generation iPhones are experiencing the same problems as those with the new 3G iPhones.  Moreover everybody seems to agree that these problems only showed up after they upgraded their iPhone software.  Problems include:

• Frequent crashes of applications, especially Safari
• Increased lag in common operations
• Significant problems with large contact lists (>200 contacts)
• Extended delays before placing a call

Apple is legendary for their high quality software.  People buy Macs because they "just work."  It’s really not like Apple to release a buggy piece of software.  But it sure seems that they did in this case.  Why?  Obvious answers of fierce competition for high-end smartphones.  The more interesting question for me is "How did they get away with it?"  Which it sure seems they are.

Escalation of Commitment: The Hush-factor

There’s a well-document psychological principal at play which prevents people from objectively critiquing things that they are personally invested in.  Sometimes called escalation of commitment, or irrational escalation, the idea is the same.  If somebody works really hard to obtain something, they will blind themselves to its faults.  Imagine this conversation:

"Dude, I can’t believe you waited in line for hours to get that phone.  What do you think of it?"

"Actually, it’s just okay.  The applications crash a lot.  And it’s not nearly as fast as I’d hoped it would be — sometimes it just hangs for like 10 seconds.  But at least it’s pretty."

Very few people have the objectivity to imply that their personal sacrifice was not worth while.  This effect is commonly observed in people who buy high-end items. 

The flip side of this effect is buyer’s remorse.  But since the phone itself is not actually at all expensive (when compared to the monthly fees), that’s unlikely.  Also, it has become a positional good, whereby it has value simply because other people don’t have one.  That fact remains regardless of how unreliable it is.

Speculative Conclusion

I posit that Apple knew about the software problems with the iPhone 3G before launch.  They did manage to iron out all the performance and stability problems they encountered before launch.  They felt they needed to launch it this summer to get ahead of other notable smartphones like the Blackberry Bold, HTC Touch, and Android which are hot on their heels.  So they rushed it out the door at sub-standard quality.

In order to partially cover for this mistake, they have made this device especially hard to get.  This covers their tracks in two ways: people make even more noise about scarcity.  And those who do jump through the whoops to obtain one are far less likely to complain about it.

Last week Seattle had some crazy electrical storms.  In recent years’ storms, my block has done better than most with respect to power failures making me think we’re either lucky or in a particularly robust section of the grid.  So I was a little surprised to find my whole house offline on Wednesday morning.  After a bit of debugging I figured out that the small UPS that runs all my networking gear got toasted, and for some reason the file server was down.

I left it alone for several days, and when I got around to turning it back on, I was happy that the whole stack through the samba server came up by itself.  (It doesn’t always!)  But when I started looking around I quickly realized things were amiss.  The media/video directory normally has 4 subdirectories: movies, episodic TV, imake and other.  But today it listed:

leo@elephant:/raid/shares/media/video$ lsdpisndic TV  hmakd  movies  nther

WTF!?  A few bits had been scrambled in the directory names.  This sounds really bad.  Moreover, even though the first couple levels of the directory hierarchy were there, but no files were to be found.  Definitely a problem.

Step 1: As soon as you suspect your RAID array has a problem, stop writing to it until you know what’s going on.  Writing changes can make things worse.  Stop the bleeding.   

I unmounted the drive from my mac, not trusting Finder or Spotlight to sprinkle damaging meta-files over the array.  Once I remembered how to ssh into the box, I stopped the samba daemon,

leo@elephant:/$ sudo /etc/init.d/samba stop

unmounted the filesystem

leo@elephant:/$ sudo umount /raid

and changed fstab so it would be read-only when it comes back, and that it wouldn’t come back without me asking.

leo@elephant:/$ sudo vi /etc/fstab

changing

/dev/evms/teraraid500 /raid ext3 defaults  0 0

to

/dev/evms/teraraid500 /raid ext3 ro,noauto  0 0

I tried poking around in EVMS by running

leo@elephant:/$ evmsn

But it hung during initialization with blue dialog saying "Discovering segments…"  I’m thinking EVMS can’t help me.  After a bit of googling I thought I should try e2fsck or some such.  First, I tried to mount it again read-only and see what’s there.

mount: wrong fs type, bad option, bad superblock on /dev/evms/teraraid500,       missing codepage or other error       In some cases useful info is found in syslog - try       dmesg | tail  or so

Bad superblock.  Uh oh.  Well this guy managed to recover a drive with a bad superblock.  Lots of things were pushing me in this direction — fix the filesystem.  But I realized that was a mistake.

Step 2: Do not make changes at the filesystem level until you’re confident that the RAID array is working properly.  You set up RAID for a reason.  You’ve still got a chance to recover everything, but if you start
making changes to it in a broken state, you’re almost certainly going
to make things worse.

Me to self: Think about it.  EVMS is confused.  Linux is confused.  Ext2 and ext3 are messed up complaining about bad superblocks.  The problem was caused by lightning.  When the drive was mounted there were wierd bit-level corruptions in the data that were still there.  Maybe one of the drives in the array got data scrambled, but didn’t get totally fragged so it went offline.  RAID 5 is designed to survive total loss of a single drive.  But if a drive gets corrupted, who knows what will happen.  So I came up with this plan:

Step 3: Try physically disconnecting the drives in your array, one at a time.  If only one of them is scrambled, disconnecting it should restore all the data in the array.

Having followed my own advice, it’s easy for me to tell the drives in my array apart since each drive in the RAID array is from a different manufacturer (which makes array failure due to manufacturing defects far less likely). 

This plan actually worked perfectly!  Removing a drive caused a bit of a hassle in getting the machine back up, because when I booted it couldn’t find the /boot partition complaining

 * Starting Enterprise Volume Management System...[42949392.340000] raid5: raid level 5 set md1 active with 2 out of 3 devices, algorithm 0

 * Checking all filesystems...fsck.ext3: No such file or directory while trying to open /dev/sdd5/dev/sdd5:The superblock could not be read or does not describe a correct ext2 filesystem.  If the device is valid and it really contains an ext2 filesystem (and not swap or ufs or something else), then the superblock is corrupt, and you might try running e2fsck with an alternate superblock:    e2fsck -b 8193 <device>

Notice the complaint about the superblock again — don’t trust it, and don’t do what it says!  What really happened was that the boot drive letter had been changed from /dev/sdd to /dev/sdc, so I had to change /etc/fstab to mount /boot from  /dev/sdc5 instead of /dev/sdd5.  In my system, I boot off a non-RAID disk attached to the mobo, which for some annoying reason gets the last drive letter after all the drives no the SATA card.

But once I got past this, it quickly turned out that the Samsung drive was the culprit.  With it removed, the software RAID kicked in and plugged all the whole.  Everything the array looked
completely normal again.  All the directories.  All the files.  Hooray!

When I sobered up the next afternoon it occured to me that I’d pulled my file server off the orange DMZ network for performance and simplicity, and that the other server box had long since been virtualized into the file server.  I moved my local public wifi off the blue network onto the red to make its security brain-dead simple.  So despite all the pretty color-coded cables and corresponding hubs, all I really had was a big loud NAT box with a few key port holes in it.  And since I’ve switched from outlook to Gmail, I never even RAS into my home XP boxes any more.  And since I do all my personal development on EC2 or some other host, I never use my home dev servers any more.  So in fact, I don’t need to tunnel home for anything.  Cloud computing.  For real.  All this stuff I used to need I don’t any more.  I could replace that old linux box with a cheap low-power firewall.

But that got me thinking.  There’s this li’l XP box sitting next to the printer that I have configured never to go to sleep because otherwise I can’t print from my laptops.  Print servers are similarly small and low-power and sometimes come in the same box as the firewall.  Then my eye turned to the terabyte file server in the corner and next thing you know I’ve got an Apple Time Capsule in the mail to replace all three permanently powered-on PCs in my house.

Happy BEarthday, Miller!

But think about the longevity of these devices.  Computers always slow down.  In a few years, any laptop is going to feel really slow, no matter how fast it feels today.  But if it’s a light, small laptop, then you’ll have something which is slow, but at least nice and portable.  Some of my friends’ house has this ancient Pentium II Viao laptop kicking around the living room — it barely runs a browser.  But it’s so small and portable that it’s still a reasonable computing device today.  If your laptop is heavy to start with, then in a few years when it slows down you’re stuck with a heavy, slow laptop, which nobody nobody wants.

Another good reason is that whole batches of drives sometimes get manufacturing defects.  So if you get 3 or 4 drives from the same batch with sequential serial numbers, they might all have the same defect.  So the odds of 2 drives crashing at once are much lower if they came from different factories.  Most RAID schemes can survive a single drive failure, but few can survive multiple.

That’s all.