Embracing Chaos

Here I propose an alternative explanation for iPhone scarcity: the difficulty in obtaining a new iPhone keeps people from complaining about problems with it.  I will explore this sophisticated marketing technique that Apple may or may not be employing to cover up quality problems with the new iPhone 3G.  Even if Apple is not doing this deliberately, I assert that it is a valid and potentially very useful technique if your product is lucky enough to have the prerequisites.

New iPhones are hard to get

The blogosphere is full of speculation about whether or not Apple deliberately made the iPhone scarce on opening day and since then.  Most assume that this is deliberate on Apple’s part for a variety of reasons, mostly to attract more attention, increase demand, etc.  I assume most of these rants are from bloggers who want their new iPhones but haven’t overcome the barriers to obtain one yet.

But if Apple’s goal was purely to meter out their distribution, why not sell them online?  To get a phone you need to place an order for one, wait a week or two, and then you can get it.  This seems reasonable in conditions of scarcity.  But to get an iPhone 3G, you need to walk into an at&t store to place your order, and then walk into the store again to pick it up.  Think about this.  If the limitation was purely lack of supply then there are several ways this could be easier for customers:

1. You could order a phone online to be delivered to your house.
2. You could order a phone to be delivered to your nearest at&t store.
3. You could call the nearest at&t store to place your order, but still have to walk in to pick it up.

Try asking them why you can’t do any of these things and they will answer with one word: policy.  Clearly Apple & at&t have gone out of their way to make it difficult for people to get their hands on a phone.   This goes above and beyond just preserving a limited supply.  You have to work to get an iPhone 3G.

New iPhones have Issues

From all the reports I’ve read, the problems with the new iPhone are in the software not the hardware.  I conclude this because my friends with first generation iPhones are experiencing the same problems as those with the new 3G iPhones.  Moreover everybody seems to agree that these problems only showed up after they upgraded their iPhone software.  Problems include:

• Frequent crashes of applications, especially Safari
• Increased lag in common operations
• Significant problems with large contact lists (>200 contacts)
• Extended delays before placing a call

Apple is legendary for their high quality software.  People buy Macs because they "just work."  It’s really not like Apple to release a buggy piece of software.  But it sure seems that they did in this case.  Why?  Obvious answers of fierce competition for high-end smartphones.  The more interesting question for me is "How did they get away with it?"  Which it sure seems they are.

Escalation of Commitment: The Hush-factor

There’s a well-document psychological principal at play which prevents people from objectively critiquing things that they are personally invested in.  Sometimes called escalation of commitment, or irrational escalation, the idea is the same.  If somebody works really hard to obtain something, they will blind themselves to its faults.  Imagine this conversation:

"Dude, I can’t believe you waited in line for hours to get that phone.  What do you think of it?"

"Actually, it’s just okay.  The applications crash a lot.  And it’s not nearly as fast as I’d hoped it would be — sometimes it just hangs for like 10 seconds.  But at least it’s pretty."

Very few people have the objectivity to imply that their personal sacrifice was not worth while.  This effect is commonly observed in people who buy high-end items. 

The flip side of this effect is buyer’s remorse.  But since the phone itself is not actually at all expensive (when compared to the monthly fees), that’s unlikely.  Also, it has become a positional good, whereby it has value simply because other people don’t have one.  That fact remains regardless of how unreliable it is.

Speculative Conclusion

I posit that Apple knew about the software problems with the iPhone 3G before launch.  They did manage to iron out all the performance and stability problems they encountered before launch.  They felt they needed to launch it this summer to get ahead of other notable smartphones like the Blackberry Bold, HTC Touch, and Android which are hot on their heels.  So they rushed it out the door at sub-standard quality.

In order to partially cover for this mistake, they have made this device especially hard to get.  This covers their tracks in two ways: people make even more noise about scarcity.  And those who do jump through the whoops to obtain one are far less likely to complain about it.

I love my Toyota Prius.  I’m very grateful to Google for giving me
a financial incentive to buy a brand new car which does everything I
want.  It’s totally big and practical, it’s very high-tech, it both
supports and projects my values,
etc.  I could sing its virtues for pages.  But for now, I want to share
some non-obvious things about it in a series I’m calling "Stupid Prius
Tricks."  (With a tip of a hat to David Letterman.)  Several of these articles (like this one)
are complaints, but this must be taken in the context that overall I love the car.

The Prius’s High-Tech Safety Features

The Prius has at least a couple of smart traction control systems which use technology
to help maintain control of the car in difficult situations.  Of course
there’s anti-lock brakes (ABS).  In addition to ABS the braking system
does things called Electronic Brake-force Distribution (EBD) and Brake
Assist.  Beyond that, it’s got Traction Control, and in all but the
stripped-down cars something called Vehicle Stability Control (VSC).
What each of these features does, or how they differ from each other I
don’t know and don’t frankly care.  They help the car do what I’d want
it to do under good driving conditions when the road is slick or uneven
or I’m driving too fast or what have you.  All this is great.  But
there’s one subtle and very bad flaw in the implementation of at least
one of these systems.

The Stupid UX for these Features

When the Prius needs to engage one of its TLA control systems, an
orange light appears on the otherwise dark dashboard.  Sometimes a
sound will beep.  This is presumably done to let you know that the
fancy feature that you paid for is working on your behalf.  Call it a
marketing feature if you will, because there’s nothing you can do with
this information, or way to react to it within the vehicle’s controls.
The stupid thing about this feature is that it’s distracting to have
lights flashing and sounds beeping.  When you’re having trouble
controlling the car is the last time you want something distracting
you.  So the engineers at Toyota have done something completely moronic
– a complex system senses when you’re having trouble controlling your
car, and chooses that time to throw more distractions at you.  I’ve
observed the light and beeping  on when driving over bumpy roads, or
down
something very steep that has a rut in it, or when the back-right wheel
accidentally jumps up the curb.  None of these situations I’ve gotten
into have been serious or very dangerous.  But even still, they were a
bit confusing, and the confusion was confounded by having the car meow
at me demanding attention.  When the car is not doing what you expect it to, the last thing you want is additional distractions.  Bad choice, Toyota.  These features should be absolutely silent.

What you can do about it

If anybody knows a cheat code to disable this light or beeping, please
post it in a comment.  Until then, you can help a little bit by putting
a small piece of electrical tape over the orange light with the symbol
for "I think you’re losing control of the car."

A couple weeks ago, Dan Kaminsky found a flaw in DNS.  Without getting into details, this flaw enables a malicious attacker to fool your web browser into connecting to the wrong computer to get your web pages.  So when you type www.facebook.com into your browser, you might actually go to Joe Hacker’s site, even though your browser says http://www.facebook.com/ in its address bar just like it should.

Dan, being a "good guy", tried to keep the details of this hack quiet for long enough for network operators to patch their systems and close the loophole.  He wanted everybody running a DNS server to do this before the "bad guys" figured out what the bug is and started to take advantage of it.  He was hoping for 30 days of time to prepare, but somebody spilled the beans after 13 days, and now the hackers are off and running.

It’s a jungle out there

You might be asking, So what? What are the dangers of being directed to the wrong website?  Of course, you could read incorrect news and that’s not great.  More likely you’re going to have your password stolen for whatever site you log into.

The obvious attacks are to sites like paypal or banks, but they’re actually safe from such attacks if you use your browser properly.  Any financial site will use a secure connection.  You can tell because of the https:// at the beginning of their address.  These sites use a digital certificate that your browser checks to verify their authenticity.  All this happens independently of the DNS system.

But you can still connect to a hacked site with https.  Your browser will probably warn you saying something about a certificate not matching.  More often than not these errors occur because of a lazy sysadmin or something.  But right now, I strongly advise you to take all HTTPS warnings seriously.

Protect yourself

If you want to be sure you’re safe, manually connect your machine to OpenDNS, as Dan recommends.  We know they’re patched and can take the traffic.  I’ll give you the steps to do this on Windows:

1. Start menu
2. Control Panel
3. Network connections  (might have to switch to "classic view")
4. Select the one you’re actually using.  It’s likely called "local area connection".
5. Click Properties on the status dialog
6. Scroll down in the list of checkboxes, and select "Internet Protocl (TCP/IP)" so that it’s highlighted.  (Leave it checked!)
7. Click Properties
8. In the first General tab, change the second radio-button from "Obtain DNS server address automatically" to "Use the following DNS server addresses:"
9. For Preferred DNS server, enter: 208.67.222.222
10. For Alternate DNS server, enter: 208.67.220.220

That’s mostly it, but to be safe, you should reboot, restart your browsers, and/or:

11. (Windows key+R).  In the dialog type  "ipconfig /flushdns" (without the quotes) and hit okay.

I spent the day yesterday at XMPP Summit #5 alongside OSCON in Portland.  It was a great chance to catch up with old friends and meet a few new ones.  But my favorite part was the break-out discussion of XMPP PubSub as it relates to micro-blogging.  We discussed what hopefully will emerge as a standard way to associate an existing Atom/RSS feed with an XMPP PubSub Node.  First some background on the relevant technologies.  Feel free to skip ahead if you understand this stuff.

PubSub 101: Push vs Pull

PubSub is short for "publish subscribe" which is a common design pattern describing a way to distribute information to interested parties.  The publisher tells a server about new information, and the server fans the information out to everybody who has registered interest in that topic or channel.  Data consumers find out about the new information very quickly, with relatively little load on the whole system, since the pubsub mechanism provides a means to "push" data to them. 

By contrast, almost all of the web today follows uses a "pull model" where a data consumer only finds out about new information when it gets around to checking if there is something new.  This data distribution model is significantly simpler because the server only needs to keep track of the content, not who is interested in knowing about it.  Modern networks are optimized for this kind of query-based traffic where data consumers (clients, web browsers) initiate connections to servers, such that it’s often impossible for servers to initiate conncetions to clients because of firewalls or NAT.

The downside of the pull model is that the only way a data consumer can find out if thanything is new on the server is to "check back frequently" or "poll" the server for changes.  If you want to know within 15 minutes if anything new has been posted, you have to ask the server at least every 15 minutes "anything new?"  No.  "How about now?"  No.  "Got anything yet?"  No.  Mulitply this by potentially millions of interested data consumers and you end up spending a lot of network bandwidth and server resources doing very little.  Even worse, the problem scales horribly.  If clients want to know about changes within 5 minutes instead of 15, that puts 3 times the load on the server.  Want to know within a few seconds?  Forget it — the servers would crash.  There’s an intrinsic delay in distributing information in this model, and reducing that delay is very expensive.

XMPP as an alternative to polling

XMPP is the protocol used for Instant Messaging by Google Talk and Jabber and a large number of small servers.  In order to deliver instant messages, XMPP systems maintain persistent connections between all machines that allow packets of data to be pushed with very low latency — IM messages are typically delivered within a second of sending them.  So it’s natural to want to use this infrastructure to deliver other data more efficiently than through polling HTTP.

The XMPP PubSub spec known as XEP-0060 describes how to do exactly this at the protocol level.  But for a variety of reasons, this standard has not gained wide adoption.  IMHO the biggest reason is that there isn’t a very pressing need.  The current system is horribly inefficient, but it works.  Moreover, it puts the burden of inefficiency squarely in the hands of the information publishers.  Popular publishers are just expected to shell up for necessary hardware to meet the demands of their readers, and with advertising they can typically recoup the necessary investment.

Another way to state that is that pubsub hasn’t found its niche yet.  IMHO this is partly because the mechanism is so useful it can be applied to almost anything.  Not just breaking news, but everything from e-mail mailing lists to doorbell chimes get used as examples of how XMPP pubsub technology could be applied.  Not wanting to exclude any of these potentially interesting uses, the protocol remains very generic.

Micro-blogging, Atom and Yesterday’s Realization

One place where the current HTTP model breaks down is micro-blogging, which is the generic term for services like Twitter or Facebook’s udpates.  Here, the payload of actual content is very small, so the overhead of checking far outweighs the "useful data" which is delivered.  Also, because the information is social (i.e. "Heading to Broadway for a bite — wanna come?") consumers demand it be delivered quickly.  Nonetheless, current micro-blogging services still rely on polling clients, and their servers suffer as a result.

Yesterday, a group of us including Blaine Cook, Anders Conbere, Evan Prodromou, and XEP-0060 co-author Ralph Meijer were discussing how to apply XMPP PubSub to micro-blogging.  This was likely obvious to many there already, but during the discussion I had a realization.  We aren’t solving this problem from whole cloth.  RSS and Atom feeds already describe all the information we need.  We just need to find a way to substitute XMPP for the assumed transport HTTP.

So we discussed mechanisms for mapping an Atom URL to an XMPP PubSub Node.  (We pretty much ignored RSS because RSS isn’t as cool for reasons I really don’t understand.)  We talked about putting a link-rel tag in the feed to point to the XMPP PubSub node.  This would look something like 

   <link rel="alternate" type="xmpp/pubsub" href="xmpp:twitter.com?;node=users/leopd" />


Even better, the URL for these nodes should be guessable from the URL for the HTTP feed.   The above node would be the normal place to look for a the pubsub version of http://twitter.com/leopd.  Even though it’s not as generic and robust to have a standard mapping like this, I think it’s an important way to speed adoption of a new standard.  The code to do a bit of string manipulation is vastly easier than fetching the URL and looking for a link-rel tag.  And developers are intrinsically lazy (for good reasons!) so making things easier for them means they’ll succeed a lot more.

Ever pragmatic, Blaine pointed out that we should use HTTP for things it is good at, and not re-invent them in XMPP.  I wholeheartedly agree.  Re-transmission is a key example.  What happens if a client is offline when a new post happens, and so never hears about it?  Answer: The clients should fetch the historic archive of the feed over HTTP.  These feeds exist today — no need to improve on them.  If all the posts have sequence numbers on them, then it’s easy to figure out if you’ve missed one.  So all the posts from a user should have sequence numbers.  I don’t think this is standard in Atom feeds today.

The story unfolds…

There’s a lot more to be worked out and standardized here.  And clearly many more people need to voice their opinions before we can reach consensus.  Sadly I can’t be down in Portland today to continue the discussion, so this post will have to take my place as I return to my regular daily commitments.  If you’d like to stay tuned as the story unfolds, you’ll have to poll this site, as I can’t yet give you a PubSub node to subscribe to for updates.  If I could it would probably be something like xmpp:embracingchaos.com?;node=xmpp — try it.  By the time you read this, it might be working!

Evolution-like processes exist in many places beyond genetic adaptation of biological species.  We see similar processes in a great many aspects of modern life, generally running many orders of magnitude faster.  Much of economics and business is governed by processes that select for the most successful product or business model or manufacturing process or organizational structure.  Successful practices thrive and out-compete ones which are less effective at meeting human needs and desires.  Warfare has very obvious parallels.  In computer science, user interfaces, programming languages and system architectures all evolve by analogous processes.  Similar effects can be found in governments, religions, cell phone design or city planning, just to name a few more.  The basic idea that human choices lead to faster propagation and increased presence of BETTER STUFF can be seen almost everywhere.  Except in our refrigerators.

Open your fridge.  If you’ve lived with that fridge for a while, there’s a good chance it looks something like mine does.  Shelf upon shelf of half-used bottles and jars of long-lasting meta-foods.  Condiments, salad dressings, jellies, beverages, chutneys, nut butters, salsas, pickled vegetables, etc.  We expect our fridges to be full of food, so this doesn’t in itself challenge the evolutionary principal of selection.  But taking an inventory shows that there is a strong bias towards foods we don’t actually like.  In fact, the typical selection process for foods in our refrigerators tends to concentrate foods we don’t like, thus running backwards to what should intuitively evolve towards a selection of our favorite foodstuffs.  But for a couple very understandable reasons, that just doesn’t happen.

Consider salad dressings.  Most of us like to have some choices when we’re topping our raw vegetables.  So when we’re at the store, we don’t just buy the one salad dressing we like, but will often try a new variety.  There’s a documented psychological principal called Variety Seeking that encourages diversity in buying because people want to explore different choices.  But what happens when we buy a variety we don’t particularly enjoy?  Like that orange blossom vinaigrette or the honey mustard that’s just a bit too thick and sweet.  We try it once, form an opinion, and the next time we have salad we go for the old-reliable Goddess dressing.  So it lingers.  But we don’t throw it away.  Because there’s nothing WRONG with it.  Besides, one day when we have guests over they might prefer a syrupy honey-mustard dressing.  Or maybe we could dip chicken knuckles into it or something.  Plus the combination of preservatives, low-temperature and food that doesn’t promote bacterial growth in the first place means it can stay edible for years.  So their continued presence provides some small marginal benefit of choice.  The only real alternative is throwing them away  (which makes us feel guilty) since there’s no secondary market for used condiments.

Beyond choice, they do provide marginal benefit in terms of ballast for heat capacity.  Refrigerators run more efficiently when they’re full since there’s a larger thermal mass which is more stable.  But this assumes the fridge has ample space for the food that is being cycled through and consumed.  In many households the need to find space for food you’re actually going to eat creates a selection pressure to remove such undesirable foods.  But the door of the fridge is a niche environment that isn’t very well suited to large, short-lived main courses and thus things like eleven different varieties of mustard tend to thrive.

What’s the take-home lesson here?  How do we fight this scourge on our pallets?  Actually I don’t think it’s that big of a problem.  When we need space in the fridge, we find it.  But otherwise we collect things like Mang Thomas All Purpose Sauce, and pickled cherry peppers.   If clutter bothers you, resist the temptation to try something new and stick with something you know you’ll use.  Heck, get a really big bottle.  Or look for similar reverse-evolutionary processes in your medicine cabinet, liquor shelf, or office supplies, and be conscious that you have the power to change things.  Or just accept that sometimes human nature tends to concentrate our surroundings with things we don’t actually like.

The EVMS RAID 5 array in my linux fileserver crashed recently due to a lightning storm, and I thought I’d lost everything.  But with some luck and intuition I was able to recover all my files.  I’ll tell you how I did it, so hopefully others who run into similar problems can recover their data too.  But first, a little background.

Last week Seattle had some crazy electrical storms.  In recent years’ storms, my block has done better than most with respect to power failures making me think we’re either lucky or in a particularly robust section of the grid.  So I was a little surprised to find my whole house offline on Wednesday morning.  After a bit of debugging I figured out that the small UPS that runs all my networking gear got toasted, and for some reason the file server was down.

I left it alone for several days, and when I got around to turning it back on, I was happy that the whole stack through the samba server came up by itself.  (It doesn’t always!)  But when I started looking around I quickly realized things were amiss.  The media/video directory normally has 4 subdirectories: movies, episodic TV, imake and other.  But today it listed:

leo@elephant:/raid/shares/media/video$ lsdpisndic TV  hmakd  movies  nther

WTF!?  A few bits had been scrambled in the directory names.  This sounds really bad.  Moreover, even though the first couple levels of the directory hierarchy were there, but no files were to be found.  Definitely a problem.

Step 1: As soon as you suspect your RAID array has a problem, stop writing to it until you know what’s going on.  Writing changes can make things worse.  Stop the bleeding.   

I unmounted the drive from my mac, not trusting Finder or Spotlight to sprinkle damaging meta-files over the array.  Once I remembered how to ssh into the box, I stopped the samba daemon,

leo@elephant:/$ sudo /etc/init.d/samba stop

unmounted the filesystem

leo@elephant:/$ sudo umount /raid

and changed fstab so it would be read-only when it comes back, and that it wouldn’t come back without me asking.

leo@elephant:/$ sudo vi /etc/fstab

changing

/dev/evms/teraraid500 /raid ext3 defaults  0 0

to

/dev/evms/teraraid500 /raid ext3 ro,noauto  0 0

I tried poking around in EVMS by running

leo@elephant:/$ evmsn

But it hung during initialization with blue dialog saying "Discovering segments…"  I’m thinking EVMS can’t help me.  After a bit of googling I thought I should try e2fsck or some such.  First, I tried to mount it again read-only and see what’s there.

mount: wrong fs type, bad option, bad superblock on /dev/evms/teraraid500,       missing codepage or other error       In some cases useful info is found in syslog - try       dmesg | tail  or so

Bad superblock.  Uh oh.  Well this guy managed to recover a drive with a bad superblock.  Lots of things were pushing me in this direction — fix the filesystem.  But I realized that was a mistake.

Step 2: Do not make changes at the filesystem level until you’re confident that the RAID array is working properly.  You set up RAID for a reason.  You’ve still got a chance to recover everything, but if you start
making changes to it in a broken state, you’re almost certainly going
to make things worse.

Me to self: Think about it.  EVMS is confused.  Linux is confused.  Ext2 and ext3 are messed up complaining about bad superblocks.  The problem was caused by lightning.  When the drive was mounted there were wierd bit-level corruptions in the data that were still there.  Maybe one of the drives in the array got data scrambled, but didn’t get totally fragged so it went offline.  RAID 5 is designed to survive total loss of a single drive.  But if a drive gets corrupted, who knows what will happen.  So I came up with this plan:

Step 3: Try physically disconnecting the drives in your array, one at a time.  If only one of them is scrambled, disconnecting it should restore all the data in the array.

Having followed my own advice, it’s easy for me to tell the drives in my array apart since each drive in the RAID array is from a different manufacturer (which makes array failure due to manufacturing defects far less likely). 

This plan actually worked perfectly!  Removing a drive caused a bit of a hassle in getting the machine back up, because when I booted it couldn’t find the /boot partition complaining

 * Starting Enterprise Volume Management System...[42949392.340000] raid5: raid level 5 set md1 active with 2 out of 3 devices, algorithm 0

 * Checking all filesystems...fsck.ext3: No such file or directory while trying to open /dev/sdd5/dev/sdd5:The superblock could not be read or does not describe a correct ext2 filesystem.  If the device is valid and it really contains an ext2 filesystem (and not swap or ufs or something else), then the superblock is corrupt, and you might try running e2fsck with an alternate superblock:    e2fsck -b 8193 <device>

Notice the complaint about the superblock again — don’t trust it, and don’t do what it says!  What really happened was that the boot drive letter had been changed from /dev/sdd to /dev/sdc, so I had to change /etc/fstab to mount /boot from  /dev/sdc5 instead of /dev/sdd5.  In my system, I boot off a non-RAID disk attached to the mobo, which for some annoying reason gets the last drive letter after all the drives no the SATA card.

But once I got past this, it quickly turned out that the Samsung drive was the culprit.  With it removed, the software RAID kicked in and plugged all the whole.  Everything the array looked
completely normal again.  All the directories.  All the files.  Hooray!

This last week there was a lot of news coverage of a "daring hostage rescue in Columbia."  Fifteen people were freed from the FARC.  Many had been held captive for years, including politician Ingrid Betancourt, and three Americans.  The press has been celebrating the victory along several lines.  How wonderful it is for these people to be set free after years of captivity.  How the US military helped plan and support the operation.  How the guerrillas were fooled into giving the hostages up without firing a single shot.  (Aren’t we smart!  Aren’t they stuipd?)

But there’s a dark side to this rescue that I haven’t seen anybody discuss.  The reason the guerrillas allowed those hostages to get on that helicopter without firing a shot because they thought it was operated by a humanitarian group.  It’s true that the operation relied on intercepted communications and a spy in the FARC’s command structure.  But the operation relied on a having military helicopter painted white and its crew claiming to be apolitical.  The press even describes the acting lessons the soldiers took to pretend to be NGO workers.  Oh those foolish rebels who fell for such a simple trick by trusting aid workers.  What dupes!

Now look at this from another angle.  Imagine you really are an NGO worker, trying to provide some kind of support service to remote Columbia.  How does knowledge of an operation like this make you feel?  Scared, probably.  From now on, rebels are going to doubt the legitimacy of all NGO workers.  They might think you’re in the Columbian military trying to take advantage of them again.  They might even start shooting down Red Cross helicopters.  The negative externality of this rescue is that all legitimate humanitarian work in the area has just gotten a lot more difficult and dangerous.

So as Santos brags that this rescue "will go down in history for its audaciousness and effectiveness" he ignores the fact that he just cashed in a bunch of good will to make this happen.  This stuff doesn’t grow easily like coca plants.  I’m glad those people have their lives back, but I am in no way convinced it was worth the sacrifice.  What’s going to happen next time there’s a public health crisis in the area?  The moral calculus is undoubtedly complex.  But ask yourself, would you trade the freedom of a dozen captives (including three Americans) for risking the well-being of many thousands of needy individuals?  How about for the lives of a half dozen International Red Cross workers murdered by suspicious rebels?

Wonder what I’ve been up to at work lately?  Here’s a tiny glimpse.

Google just launched another way to access the Google Talk network.  It’s a web-based instant messaging chat client optimized for the iPhone browser.  It’s not my primary project or my secondary or tertiary, but I did write a blog post about it and made sure the whole thing got out the door today.

If you have an iPhone, try it out at www.google.com/talk.  Warning: Non-iPhone browsers will be directed away.