It was quite a chore to get Apple to admit that the cause was a hardware problem, and fix it.  But I finally succeeded, so I thought I’d share some of my experiences.  I’ll explain what a Kernel Panic is, how they sometimes can be caused by faulty software but often indicate hardware problems, how they differ from other kinds of crashes, and provide a guide on how to read a Mac OS X kernel panic report.

Dealing with the “Genius” Bar staff

“Genius” is what Apple calls its first tier of technical support.  I find the brand unfortunate and insulting for everybody involved.  There is no intelligence test required to work as a “genius” — just some minimal training on how to follow Apple customer service scripts like an obedient robot.  Knowing Apple, I wouldn’t be surprised if the “Genius” staff are required to follow these scripts verbatim and face not only termination but punitive lawsuits for deviating from the party line.  Keep this in mind when dealing with them.  Also know that they have some discretion in the outcome of your visit, but the discretion exists within guidelines that they cannot control.

Some tips on getting past the “genius” from my limited experience.  Print out your kernel panic reports and bring them in.  The more the better.  Highlight the relevant parts.  I’m not sure if bringing a bad attitude with you helps or not — they want to make their customers happy, but they don’t like their “genius” title challenged with logic.  I also recommend persistence.  Following their stupid advice and showing them that it did no good will help.  I’m not sure if understanding what’s going on will or not.  But if you’d like to understand more about why your Mac is crashing, read on…

Kernel panics and hardware failures vs regular software failures

There are two basic ways your Mac can crash.  First, an application might lock up on you and become unresponsive.  You get the spinning beachball of death, and eventually have to Force Quit your application, losing whatever work you hadn’t saved.  This kind of user mode failure is very common with buggy software.  If the beachball is getting you down, the problem is almost certainly caused by bad software, not by a hardware problem.  In OS 9 and before, this kind of failure could have taken down your entire machine, but since the introduction of the BSD kernel in OS X, the system is designed to allow one application to fail while protecting all the other applications.

Sometimes though your entire Mac will crash hard.  Without warning your system displays a full-screen message saying “You need to restart your computer. Hold down the Power button for several seconds or press the Restart button.” in several languages.  This is OS X’s last ditch attempt to tell you something about what happened before it goes completely teets up.  It’s formally known as a kernel panic.  Sometimes the system is so screwed it can’t even get that error message onto the screen before it dies.

Kernel panics indicate a serious problem, either with the computer’s hardware, or the low-level software in the operating system. In fact there are only three things that can cause a kernel panic:

1. Faulty hardware causes a problem that the OS doesn’t know how to deal with
2. A bug in OS X itself
3. A bug in an OS plugin called a kernel extension or kext

Firstly, if the hardware itself has problems, then kernel panics are a common way they manifest themselves.  Similarly, if the operating system itself has any bugs, they could take down the entire system.  The third option could be caused by third-party software, while the first two are entirely Apple’s responsibility.  So when it comes to dealing with the “Genius” behind the bar, the first two are fairly straightforward.  If you’re seeing this problem a lot, and nobody else is, then it’s probably a hardware problem, and they should replace your hardware. Here’s a thought experiment I tried unsuccessfully with the Apple “geniuses” I had to deal with: Imagine you have a hundred Macs all running the same software, and one of them crashes periodically, but the other 99 don’t.  Would you classify that Mac as having a hardware problem or a software problem?  In my case, the genius insisted that it was a software problem.  In fact he claimed he was certain that if I uninstalled Adobe Flash, the problem would be fixed.  Read on, and you’ll learn how the kernel panic reports themselves show that this explanation is impossible.

Understanding and interpreting Kernel Panic reports

First a bit about what a Kernel Panic is.  Very simply, it’s when something unexpected goes wrong in the operating system kernel.  What’s the kernel?  The kernel is the lowest level of the operating system — the part that’s closest to the hardware.  In modern operating systems, there’s a fairly arbitrary line between what functionality lives in the kernel and what functionality lives in the user space.  The key difference is that when something goes wrong with software in the user space, you get a beachball on the app, but the system survives.  When something goes wrong in the kernel, you get a kernel panic, and the whole system goes bye bye fast.  So it’s critical that any code running in the kernel space be ultra reliable.  You don’t change kernel code quickly or lightly, and you test the hell out of it before you release it.  But code runs faster in the kernel, so most modern operating systems put important things like networking and graphics into the kernel.  The BSD kernel which powers OS X allows the installation of “kernel extensions” or “kexts” which add functionality.  More about these soon.  But suffice to say that when anything goes wrong with any kext, it’s a big deal problem because there’s nothing to fall back on (e.g. can’t display an error dialog if the problem is with the display system), so the system’s reaction is called a panic.  Thus “kernel panic.”

Immediately after a KP, your computer does two things: it stores a bunch of information to help diagnose what caused the problem, and puts up the error screen, if it can.  When you reboot, your computer asks if you want to send the KP report to Apple.  You should do this.  The smarter of the “genius” staff can look these reports up and see that your Mac is actually crashing, but they’ll admit that the contents are too technical for a mere “genius” to understand.  Well I’m going to explain to you what the reports contain and what it means about what’s wrong with your computer.

Here’s a typical crash report from my computer.  In my case, these panics weren’t even accompanied by the “restart your computer message” because as I’ll explain, the problem originated in the graphics system.  My computer just suddenly went black and non-responsive.  I’ve highlighted a few key sections for explanation below.

Interval Since Last Panic Report:  420 sec
Panics Since Last Report:          1
Anonymous UUID:                    8A09F455-1039-4696-8479-xxxxxxxxxxxx
Thu Apr 21 09:00:51 2011
panic(cpu 3 caller 0x9cdc8f): NVRM[0/1:0:0]: Read Error 0x00000100: CFG 0xffffffff 0xffffffff 0xffffffff, BAR0 0xc0000000 0xa734e000 0x0a5480a2, D0, P2/4
Backtrace (CPU 3), Frame : Return Address (4 potential args on stack)
0xbc001728 : 0x21b510 (0x5d9514 0xbc00175c 0x223978 0x0)
0xbc001778 : 0x9cdc8f (0xbe323c 0xc53840 0xbf23cc 0x0)
0xbc001818 : 0xae85d3 (0xe0cfc04 0xe5c9004 0x100 0xb83de000)
0xbc001868 : 0xadf5cc (0xe5c9004 0x100 0xbc001898 0x9bd76c)
0xbc001898 : 0x16c8965 (0xe5c9004 0x100 0x438004ee 0x28)
0xbc0019d8 : 0xb07250 (0xe5c9004 0xe5ca004 0x0 0x0)
0xbc001a18 : 0x9d6e23 (0xe5c9004 0xe5ca004 0x0 0x0)
0xbc001ab8 : 0x9d3502 (0x0 0x9 0x0 0x0)
0xbc001c68 : 0x9d4aa0 (0x0 0x600d600d 0x704a 0xbc001c98)
0xbc001d38 : 0xc89217 (0xbc001d58 0x0 0x98 0x2a358d)
0xbc001df8 : 0xc8ec1d (0xe8e5404 0x0 0x98 0x45e8d022)
0xbc001f18 : 0xc8f0b4 (0xe8e5404 0x124b6204 0x6d39d1c0 0x0)
0xbc001f78 : 0xc8f39f (0xe8e5404 0x124b6204 0x6d39d1c0 0xbc0021e0)
0xbc002028 : 0xca3691 (0xe8e5404 0x1f80d8e8 0xbc00239c 0xbc0021e0)
0xbc002298 : 0xc84d09 (0x6d0b7000 0x1f80d8e8 0xbc00239c 0x0)
0xbc0023f8 : 0xc84f47 (0x6d0c6000 0x1f80d800 0x1 0x0)
0xbc002428 : 0xc87a04 (0x6d0c6000 0x1f80d800 0x0 0x97c6c4fc)
0xbc002468 : 0xca9d40 (0x6d0c6000 0x1f80d800 0x6d09f274 0x140)
0xbc0024f8 : 0xc9b5a9 (0xde94bc0 0x1f80d800 0x0 0x1)
0xbc002558 : 0xc9b810 (0x6d09f000 0x6d09f77c 0x1f80d800 0x0)
0xbc0025a8 : 0xc9bce4 (0x6d09f000 0x6d09f77c 0xbc0028cc 0xbc00286c)
0xbc0028e8 : 0xc98aaf (0x6d09f000 0x6d09f77c 0x1 0x0)
0xbc002908 : 0xc605a1 (0x6d09f000 0x6d09f77c 0x1956a580 0x0)
0xbc002938 : 0xc9a572 (0x6d09f000 0xbc002a7c 0xbc002968 0x5046b1)
0xbc002978 : 0xc648de (0x6d09f000 0xbc002a7c 0x0 0xc000401)
0xbc002ab8 : 0xc9dee6 (0x6d09f000 0x0 0xbc002bcc 0xbc002bc8)
0xbc002b68 : 0xc60c93 (0x6d09f000 0x0 0xbc002bcc 0xbc002bc8)
0xbc002be8 : 0x56a738 (0x6d09f000 0x0 0xbc002e3c 0xbc002c74)
0xbc002c38 : 0x56afd7 (0xcef020 0x6d09f000 0x129bab88 0x1)
0xbc002c88 : 0x56b88b (0x6d09f000 0x10 0xbc002cd0 0x0)
0xbc002da8 : 0x285be0 (0x6d09f000 0x10 0x129bab88 0x1)
0xbc003e58 : 0x21d8be (0x129bab60 0x1ec235a0 0x1fd7e8 0x5f43)
      Backtrace continues...

      Kernel Extensions in backtrace (with dependencies):
         com.apple.GeForce(6.2.6)@0xc55000->0xd0afff
            dependency: com.apple.NVDAResman(6.2.6)@0x967000
            dependency: com.apple.iokit.IONDRVSupport(2.2)@0x95a000
            dependency: com.apple.iokit.IOPCIFamily(2.6)@0x927000
            dependency: com.apple.iokit.IOGraphicsFamily(2.2)@0x938000
         com.apple.nvidia.nv50hal(6.2.6)@0x1592000->0x19a6fff
            dependency: com.apple.NVDAResman(6.2.6)@0x967000
         com.apple.NVDAResman(6.2.6)@0x967000->0xc54fff
            dependency: com.apple.iokit.IOPCIFamily(2.6)@0x927000
            dependency: com.apple.iokit.IONDRVSupport(2.2)@0x95a000
            dependency: com.apple.iokit.IOGraphicsFamily(2.2)@0x938000

BSD process name corresponding to current thread: kernel_task

Mac OS version:
10J869
Kernel version:
Darwin Kernel Version 10.7.0: Sat Jan 29 15:17:16 PST 2011; root:xnu-1504.9.37~1/RELEASE_I386
System model name: MacBookPro6,2 (Mac-F22586C8)
System uptime in nanoseconds: 35829130822125

unloaded kexts:
com.apple.filesystems.msdosfs 1.6.3 (addr 0xbc1e5000, size 0x53248) - last unloaded 12216461868115

loaded kexts:
com.parallels.kext.prl_vnic 6.0 11992.625164
com.parallels.kext.prl_netbridge 6.0 11992.625164
com.parallels.kext.prl_usb_connect 6.0 11992.625164
com.parallels.kext.prl_hid_hook 6.0 11992.625164
com.parallels.kext.prl_hypervisor 6.0 11992.625164
com.apple.filesystems.smbfs 1.6.6 - last loaded 12151022138289
com.apple.driver.AppleHWSensor 1.9.3d0
com.apple.driver.AGPM 100.12.19
com.apple.driver.AppleMikeyHIDDriver 1.2.0
com.apple.driver.AppleHDA 1.9.9f12
com.apple.driver.AppleUpstreamUserClient 3.5.4
com.apple.driver.AppleMCCSControl 1.0.17
com.apple.driver.AppleMikeyDriver 1.9.9f12
com.apple.driver.AudioAUUC 1.54
com.apple.driver.AppleIntelHDGraphics 6.2.6
com.apple.driver.AppleIntelHDGraphicsFB 6.2.6
com.apple.driver.SMCMotionSensor 3.0.0d4
com.apple.kext.AppleSMCLMU 1.5.0d3
com.apple.Dont_Steal_Mac_OS_X 7.0.0
com.apple.iokit.CHUDUtils 201
com.apple.iokit.CHUDProf 216
com.apple.driver.AudioIPCDriver 1.1.6
com.apple.driver.AppleGraphicsControl 2.8.68
com.apple.driver.ACPI_SMC_PlatformPlugin 4.5.0d5
com.apple.GeForce 6.2.6
com.apple.driver.AppleLPC 1.4.12
com.apple.filesystems.autofs 2.1.0
com.apple.driver.AppleUSBTCButtons 200.3.2
com.apple.driver.AppleUSBTCKeyboard 200.3.2
com.apple.driver.AppleIRController 303.8
com.apple.driver.AppleUSBCardReader 2.5.8
com.apple.iokit.SCSITaskUserClient 2.6.5
com.apple.BootCache 31
com.apple.AppleFSCompression.AppleFSCompressionTypeZlib 1.0.0d1
com.apple.iokit.IOAHCIBlockStorage 1.6.3
com.apple.driver.AppleUSBHub 4.1.7
com.apple.driver.AppleFWOHCI 4.7.1
com.apple.driver.AirPortBrcm43224 427.36.9
com.apple.iokit.AppleBCM5701Ethernet 2.3.9b6
com.apple.driver.AppleEFINVRAM 1.4.0
com.apple.driver.AppleSmartBatteryManager 160.0.0
com.apple.driver.AppleUSBEHCI 4.1.8
com.apple.driver.AppleAHCIPort 2.1.5
com.apple.driver.AppleACPIButtons 1.3.5
com.apple.driver.AppleRTC 1.3.1
com.apple.driver.AppleHPET 1.5
com.apple.driver.AppleSMBIOS 1.6
com.apple.driver.AppleACPIEC 1.3.5
com.apple.driver.AppleAPIC 1.4
com.apple.driver.AppleIntelCPUPowerManagementClient 105.13.0
com.apple.security.sandbox 1
com.apple.security.quarantine 0
com.apple.nke.applicationfirewall 2.1.11
com.apple.driver.AppleIntelCPUPowerManagement 105.13.0
com.apple.driver.DspFuncLib 1.9.9f12
com.apple.driver.AppleProfileReadCounterAction 17
com.apple.driver.AppleProfileTimestampAction 10
com.apple.driver.AppleProfileThreadInfoAction 14
com.apple.driver.AppleProfileRegisterStateAction 10
com.apple.driver.AppleProfileKEventAction 10
com.apple.driver.AppleProfileCallstackAction 20
com.apple.driver.AppleSMBusController 1.0.8d0
com.apple.iokit.IOFireWireIP 2.0.3
com.apple.iokit.IOSurface 74.2
com.apple.iokit.IOBluetoothSerialManager 2.4.0f1
com.apple.iokit.IOSerialFamily 10.0.3
com.apple.iokit.CHUDKernLib 208
com.apple.iokit.IOAudioFamily 1.8.0fc1
com.apple.kext.OSvKernDSPLib 1.3
com.apple.driver.AppleHDAController 1.9.9f12
com.apple.iokit.IOHDAFamily 1.9.9f12
com.apple.iokit.AppleProfileFamily 41
com.apple.driver.AppleSMC 3.1.0d3
com.apple.driver.IOPlatformPluginFamily 4.5.0d5
com.apple.driver.AppleSMBusPCI 1.0.8d0
com.apple.nvidia.nv50hal 6.2.6
com.apple.NVDAResman 6.2.6
com.apple.iokit.IONDRVSupport 2.2
com.apple.iokit.IOGraphicsFamily 2.2
com.apple.driver.BroadcomUSBBluetoothHCIController 2.4.0f1
com.apple.driver.AppleUSBBluetoothHCIController 2.4.0f1
com.apple.iokit.IOBluetoothFamily 2.4.0f1
com.apple.driver.AppleUSBMultitouch 206.6
com.apple.iokit.IOUSBHIDDriver 4.1.5
com.apple.iokit.IOSCSIBlockCommandsDevice 2.6.5
com.apple.iokit.IOUSBMassStorageClass 2.6.5
com.apple.driver.AppleUSBMergeNub 4.1.8
com.apple.driver.AppleUSBComposite 3.9.0
com.apple.iokit.IOSCSIMultimediaCommandsDevice 2.6.5
com.apple.iokit.IOBDStorageFamily 1.6
com.apple.iokit.IODVDStorageFamily 1.6
com.apple.iokit.IOCDStorageFamily 1.6
com.apple.driver.XsanFilter 402.1
com.apple.iokit.IOAHCISerialATAPI 1.2.5
com.apple.iokit.IOSCSIArchitectureModelFamily 2.6.5
com.apple.iokit.IOUSBUserClient 4.1.5
com.apple.iokit.IOFireWireFamily 4.2.6
com.apple.iokit.IO80211Family 314.1.1
com.apple.iokit.IONetworkingFamily 1.10
com.apple.iokit.IOUSBFamily 4.1.8
com.apple.iokit.IOAHCIFamily 2.0.4
com.apple.driver.AppleEFIRuntime 1.4.0
com.apple.iokit.IOHIDFamily 1.6.5
com.apple.iokit.IOSMBusFamily 1.1
com.apple.kext.AppleMatch 1.0.0d1
com.apple.security.TMSafetyNet 6
com.apple.driver.DiskImages 289
com.apple.iokit.IOStorageFamily 1.6.2
com.apple.driver.AppleACPIPlatform 1.3.5
com.apple.iokit.IOPCIFamily 2.6
com.apple.iokit.IOACPIFamily 1.3.0

The first line is fairly clear — how long has your system been running since its last crash?  If this is less than an hour, as it was for my computer, then your machine is completely FUBAR.  Less than a day and you’ve still got a seriously unstable computer.  (Hint for any “genius” that might be reading this article: take the number of seconds, divide it by 60 using the Calculator app on your store-issued-iPad, and that will give you the number of minutes.  Divide that new smaller number by 60 again to get an even smaller number which is hours.  If you can figure out how to get to number of days by yourself, it’s time to apply for the “Genius Lead” job.)

The Anonymous UUID is an effectively random code that allows Apple to lookup the crash reports for your computer when you go into the store.  Then there’s the date.  Straightforward.

The line which starts “panic” is the closest thing you’ll find to a concise explanation of what went wrong. In all likelihood this will be a jumble of words and numbers that make no sense, but it’s a great string to Google.  If you’re having a hardware problem, this message will probably stay about the same with each KP.  Googling my error message “NVRM[0/1:0:0]: Read Error 0×00000100” turns up a bunch of people with similar problems — computer going black without warning, often while playing World of Warcraft.

The next section titled “backtrace” is worthless unless you’re actually diving into the source code that caused the problem.  Skip over it.  But the section after it is extremely interesting and relatively easy to interpret.

The section titled “Kernel Extensions in backtrace (with dependencies)” actually tells you what part of the system failed.  Read this one closely and try to make sense of it. In the case of my example, there are three kernel extensions involved with the crash.  They are called “com.apple.GeForce” and “com.apple.nvidia.nv50hal” and “com.apple.NVDAResman”.  The first one is fairly obvious — GeForce is the kind of graphics chip in the macbook.  The second one is also pretty clear — NVidia is the company that makes GeForce, and nv50hal I would guess means “NVidia 5.0 Hardware Abstraction Layer” or something similar.  I’m not sure what NVDAResman is but looking down a bit I see it’s related to “IOGraphicsFamily”.  This paints a really clear picture that the failure is in the graphics system.  Moreover, since every line here starts with “com.apple” we know the failure is entirely in code written by Apple.  There is no third-party software involved in this crash.

For my particular crash, it’s important to know something about the graphics hardware of these MacBooks, since all evidence points to the graphics hardware.  This generation of macbooks have two graphics chips — a faster one from Nvidia, and a more battery-friendly one from Intel.  The nvidia chip which is apparently having problems is always used when the computer has an external monitor plugged in, or when something fancy is happening on the built-in screen.  A nice utility called gfxCardStatus can help you understand this complexity, and will definitely give you a leg up on the “genius.”

The following line starting with “BSD process name” can also be important.  This will sometimes tell you which user-level app originated the call into the kernel which failed.  In my case it was “kernel_task” which provides no additional information.

The next section gives some basic info about the Mac — hardware and OS versions.  What follows is a complete list of kernel extensions (kexts) installed.  This gives you a bit more ammo in dealing with the “genius” who is probably ignoring you at this point anyway.  You can look through this list and see everything that might possibly contribute to a kernel panic.  In my case, the only software modules that aren’t from Apple are some drivers from Parallels for running my Windows virtual machine.  So the only reasons my Mac might kernel panic are because of a hardware problem, a bug in OS X itself, or something going wrong with Parallels.  Understanding this should, in theory, be very helpful when talking to your local neighborhood “genius” but unfortunately they are simple bots that only run scripts authored in Cupertino and are not permitted to listen to logic.

Apple’s Propaganda about Flash

When the “genius” told me my Mac’s problem was that I had Adobe Flash installed, I just laughed at first.  Flash is installed on something like 97% of desktop computers, and very few of them regularly turn themselves off for no reason.   Moreover, the kernel panic report lists every piece of software that could possibly contribute to the kernel panic, and neither the word “flash” nor “adobe” appear anywhere in the list.  But then I realized he wasn’t joking.

Apple’s ongoing arguments with Adobe over Flash are well publicized.  The root of the issue, in very brief summary, is that Apple sees Adobe’s Flash as a strategic threat to their incredibly profitable iPhone platform.  The poor “genius” I’m stuck with has become a pawn in Apple’s PR battle, throwing himself on the grenade of propaganda just to spread FUD about Flash.  I tried reasoning with him, explaining that Adobe’s software doesn’t run in the kernel, and therefore cannot cause a kernel panic.  The job of the kernel is to protect users from badly written software crashing the whole machine. But he would not budge.  I imagined a “genius” script which read as follows:

Mac is crashing…

1. Run hardware diagnostic tests.

2. Address any identified hardware problems.

3. If hardware tests come back clean, tell customer that the problem (whatever it is) is caused by Flash.  Tell them to uninstall it, and see if that helps.

Here I imagine the Dantesque trap of the rare “genius” who actually understands how OS X works.  I’m telling the customer something which is impossible on its face, and he knows it.  He’s arguing with me telling me I’m being stupid.  But I signed a contract with Apple saying I would defame Adobe, and deviation from this contract will bring the wrath of Steve’s legal team on me.  I just have to smile and say things like “yeah, that’s the really strange thing about this particular software problem — it only affects certain computers.  But it’s definitely caused by Flash.”

One might reason that Flash could cause kernel panics because it makes more extensive use of the graphics system than other applications.  But in this case, Flash isn’t the actual problem.  Flash is exposing the underlying problem, as would any software which works the graphics system hard.  Thus lots of people with the same problem as me who play World of Warcraft.  If the “genius” advice ever works, it’s just because Flash is the most graphics intensive software that many people use on their Macs.  The actual problem is still either a bug in OS X, or a hardware problem.

Consider the advice not to use Flash on your Mac in analogy to a car.  (A high-end MacBook actually costs as much as some cars.)  Imagine that your car sometimes just turned its engine off while you were in the middle of driving it – catastrophic failure with no warning or apparent reason.  You go to the dealership and they can’t find anything wrong with it, but ask if you ever listen to electronic music?  Well, yes, sometimes.  That’s the problem!  It’s the electronic music which is causing your car to malfunction.  So stop listening to it, and the problem will be fixed.  Umm, what?  The closest thing to the truth, by analogy, would be that any bass-heavy music (graphics-intensive application) is stressing out some weak connection in the electronics.  But because the car dealership is owned by the local philharmonic, they’re blaming it on that awful music the kids listen to.   Using your misfortune and their incompetence to push an unrelated political agenda.

It’s an interesting glimpse into how Apple is using their retail presence to advance a strategic PR goal.  Evidence that Apple has grown up as a company to the point where their own motives are more important than doing what actually helps customers.  *sigh*  At least I got my MacBook fixed.

To get across the chasm, your technology really needs to work well.  Apple seems to have done that with FaceTime.  But there’s more than just having it work — you also need to explain to the public why it’s important.  Here, Apple is just paying for that with traditional advertising.  They’re putting out lots of touching, heartwarming commercials showing the value of video chat.  Apple is spending millions of dollars to explain to people why they’ll like video chat. Primary demand stimulation.  They’re working to overcome people’s biases against  the technology or the idea — that it’s clunky, or the extremely common “I don’t want to see the people I’m talking to” reaction, which is really pretty funny when you think about it.  Of course, there will be some times when you will prefer audio only, but that’s going to be the exception when the technology is good enough.

It’s all playing out in a textbook fashion.  The geeks are all crying that there’s nothing new here, that this technology has been around forever, and they don’t understand why it’s such a big deal.  But it is a big deal, because video chat is finally entering the mainstream.

Some side-effects of this that I pointed out before, but are perhaps more clear now as the story unfolds, are that this will benefit existing established video chat vendors.  Apple is explaining to people that video chat matters.  This will help Skype and Google and Cisco with their products.

Speaking of Cisco, there’s another prediction coming true: Cisco is pushing into consumer video chat.  I had guessed 2012, but barely more than a week after my last post, they announce Cius, a video chat terminal.  Kinda like a Flip phone fused with a linksys router, but running Android and in a pretty nice looking case.

Video chat is in a similar place today to where mp3 players were 10 years ago.  There are lots of video chat solutions out there on the market.  Skype is the most well known.  I helped launch Google’s video chat system across Gmaill, iGoogle and Orkut during my tenure there.  It definitely is one of the best on the market, and it’s still only appealing to early adopters.  I mean “early adopters” in the classic Everett Rogers sense, which is to say folks for whom the extra value of the new technology outweighs the hassles of using it.  This is a step beyond the “innovators” category, who are willing to bend over backwards debugging a brand new product just because they understand that it will be important later.  Video chat has been available to innovators for a great many decades.

With iPhone 4, Apple will push video chat to the early majority category. Apple has a history of sitting on potential technologies until all the bugs are worked out, which is fundamentally what’s needed to appeal to more than just early adopters.  I’m pretty sure FaceTime will be no exception.  In 6 months, video chat won’t be this geeky thing that people put up with out of desperation.  It will begin to be integrated into normal culture.  We’ll start to see television dramas and movies incorporate it as just a way some people communicate, rather than as a way to demonstrate how high-tech somebody is.  People who aren’t geeks will start to use video chat.

Video chat really matters

Those of us who have lived deeply with video chat understand its value.  There is a ton of additional content transmitted in video that helps communication on many levels.  It allows for a more nuanced informational discussion, but more importantly IMHO, it allows distant communication to be much more personal and emotional.  Anybody who has tried professional collaboration with another team that is thousands of miles away knows that this level of communication is at least as important for business uses as it is for social communication.  The first time you meet your collaborators in person, they become more real, more trustworthy, easier to talk to, especially about difficult subjects like problems that might arise in a project *gasp*.  Video chat is certainly not as good as meeting people in person, but it is a huge step above email, IM or phone.  (Getting drunk together I believe represents the highest professionally-accepted level of humanization.)

Human-to-human communication has always been the killer feature of computer technology. Video chat makes synchronous communication fundamentally better, and as such will become a major part of everybody’s life in the developed world in the years to come.

What does this mean for everybody else?

The history of technology innovation tells us that a couple things typically happen when an emerging technology pushes into the majority segments.  First, established players will all get a boost.  Apple will be doing a huge favor to Skype and Google Video Chat by removing the veil of geekiness from their products.  Apple’s huge investment in making this product work well will make all consumers more willing to try alternatives.

Another common side-effect is that the space will get more difficult for new entrants.  This usually happens as the technology standardizes.  There becomes a “normal way of doing things” that people start “to get.”  Before a technology can reach the majority, it will typically bounce around dozens of different modalities as everybody tries to find a way of doing it that resonates with the market.  This uncertainty represents a clear opportunity for start-ups and the subsequent standardization is the closing of that opportunity.  Another reason the space usually gets harder for startups is that economies of scale start to kick in as production levels ramp up to meet the larger demand.  This naturally favors large companies, since it raises the amount of investment needed to compete.

Which Social Graph?

Another reason the space might get harder for newcomers is the natural monopoly of social graphs — consumers are better off if there is a single definitive place to keep track of their contacts rather than having to replicate and maintain a different list for each service.  As such, social graphs are important assets to anybody in this space.  But if Tim O’Reilly gets his way (which I hope he does) and we end up with a loosely-coupled internet OS, this won’t be a problem for startups, as they’ll just be able to draw from an openly available graph, say from a Google or Facebook.

I’m really not sure how this aspect will play out.  My guess is that Apple will rely on the de-centralized social graph which is the contact list built into every iPhone.  It’s a less useful corporate asset than if it were properly cloud-hosted, which will make it harder for them to expand the service to OS X machines.  Perhaps they’ll make something useful out of mobileme here, but I have my doubts.  But given the revenue they get from App Store sales, it’s not clear that the OS X machine is even a major part of their consumer strategy going forwards.  If so, this would likely be a strategic shift for them, as the inclusion of web-cams on essentially every OS X machine for years was probably done in anticipation of making a major push into video chat at some point.

Don’t forget Cisco

In addition to the obvious players like Google and Skype, this is also incredibly important for Cisco too.  Cisco has long been interested in video chat.  Why?  The same underlying reason Intel has been investing in multi-media since the late 1980’s.  Multimedia on PCs needs lots of CPU power, and video chat needs lots of bandwidth.  It’s called “primary demand stimulation.”  To be very clear: Cisco wants everybody to use video chat because video chat uses lots of bandwidth, and when people are using lots of bandwidth, Cisco sells more big routers.

Cisco is in the final phases of buying Tandberg, who is the biggest supplier of video-chat hardware for businesses.  Their aquisition of Flip last year seems strategically odd in isolation, but in this context makes perfect sense.  They are building (buying) expertise in consumer-electronics which can handle high quality video.  Take a Flip camera, add a network (like a linksys wifi box) and you’ve got a video chat terminal.  I predict we’ll see such a toy out of Cisco in about 2012, as video chat fills the early majority segment and edges against the late majority.

In Walmart’s case, it’s generally suppliers who get squeezed.  Walmart demands that manufacturers of goods produce them at the lowest possible price so that Walmart can charge the lowest prices in their stores.  They really do try hard to pass the savings on to you.  Another case that is less well known is with so-called “interchange” fees for debit and credit cards, charged by the card networks like Visa and Mastercard.  Back in 2003, Walmart pushed hard on Visa and Mastercard to charge less for debit card transactions since they are both lower risk (because of pin-code use) and cheaper to process (verifying signatures is expensive).  The cynical will point out that with lower fees, Walmart just gets to keep more profit.  Which is true.  But they are genuinely motivated to lower prices for consumers, since that’s their main selling point.  So it’s a win-win – Wal-Mart’s motivations to lower costs are closely aligned with consumer’s desires to pay less.

Apple has similar desires for their network-connected gadgets like iPhones and iPads.  Apples wants people to be able to connect their devices to the network for as little as possible. Apple has clearly negotiated very hard with AT&T to demand low monthly rates on data plans for these devices.  Next month you’ll be able to buy an iPad with a 3G data plan for just $15 / month.  That is basically unheard of in the US.  For people on a limited budget, the iPad is the cheapest way to get online.  Compare this to other data plans available from major U.S. carriers:

The Apple / AT&T rates are the lowest in each of their categories, except Verizon’s smartphone data plan which ties the AT&T iPhone plan.  The iPad rates are extremely low compared to data plans for laptops, and also when when you consider that tethering plans or phone data plans require paying an extra $30/mo – $50/mo for a voice plan.  The unlimited iPad plan is literally half what it costs to get 3G on any other laptop, and it doesn’t come with the 5 GB limit that other plans do.  You might argue that the iPad can’t do as much as a full laptop, which is true.  So you might then argue that iPad won’t tax the network as much as a laptop, which I doubt considering the propensity to consume video on such a device.  So you can’t trade torrents on an iPad, which from an Intellectual Property perspective is just fine with me.

My guess (and this is pure speculation) is that Apple negotiated these rates by offering AT&T a share of the revenues generated through App Store purchases.

Again, the cynical will point out that Apple is just trying to grab the lion’s share of economic surplus for itself, which is true.  But nonetheless, this is a case where Apple’s desires and our desires as consumers line up well.  In a very real way, Apple is fighting on our behalf for lower prices from AT&T.

Apple products are expensive. Apple gets high margins on its hardware, allowing it to recoup large investments in NRE (non-recurring engineering) to design the hardware and its accompanying software.  This is a great place to be from a competitive standpoint, because as a company they don’t need to squabble over the cheapest parts to try to deliver the best prices to consumers.  So long as they can maintain a sufficiently large customer base to support the practice, it is an easy place to defend against competition from.  Certainly a lot easier than being Dell or HP, who struggle with operational efficiency to compete on price, and try to innovate within a very narrow window defined by their platform.

Apple’s success at selling high-end products has secondary benefits for the rest of the ecosystem.  Because the products are expensive, they tend to be purchased by people with more disposable income. So the segment of the computer market which buys Apple products self-selects to be very attractive demographic for many other reasons.  Advertisers love to get their products in front of people who are more-willing-than-most to buy something expensive / unnecessary / fun.

Similarly, app developers know that if they write an app for iPhone / iPad, the people who are able to buy it are much more likely to be willing to pay a couple bucks for something silly than, say, somebody who bought the cheapest smartphone they could afford because they felt they really need that functionality.  I had previously speculated that Apple’s platform play required a very large distribution base to attract developers, which is not quite correct.  The strategy is successful even with a relatively small market, provided that the market is segmented properly.  Which in this case it clearly is.

Scores of developers have had iPad’s for weeks now. They’ve had to sign non-disclosure agreements, and have the iPad locked in a separate room that random employees couldn’t access. And even that wasn’t enough. The iPads are literally chained to the desk with steel cable and a lock. Apple comes by the office with a suitcase, installs the iPad in a bolted case, chains it to the desk and locks it there. And they they do occasional surprise visits just to make sure it’s still there.

What’s more, Apple has told developers that they are monitoring the location of the device as well.

Apple.  Gotta love them.

Mike says he can type 50 wpm on it.  That’s really quite cool.

iSlate is rumored to have a bunch of new content associated with it.  Particularly print content.  Print publishers will probably want consumers to sign up for subscriptions.  So Apple’s probably going to be introducing people to the concept of content subscriptions on their portable devices, likely with iPhone OS 4.0 which probably will run the iSlate and old iPhones and iPod touches too.  So I wouldn’t be surprised if you can get an all-you-can-eat music subscription service available too.  We’ll see.  It’s pure speculation, but it would make sense.  I’d be particularly tickled if my off-the-cuff prediction of dates from 2007 turned out to be right.

Unless Apple has come up with a better way to do soft keyboards, that is.  When I say “soft keyboard” I mean the kind of keyboard that appears on a touch screen and has no physical keys.  I’ve complained about the iPhone’s keyboard for a while.  While it’s true that people do get better at using these, I still don’t think I’ll ever be nearly as fast or accurate (even with smart correction) with a soft keyboard as I was with my blackberry.  I think that’s probably true on average for most people.  The basic reason is the lack of tactile feedback.  With a physical keyboard, if my fingers are slightly off target, they are guided to the right place by feel.

I’ve argued for some time now that the way to solve this is by figuring out how to make a touch-screen display with tactile feedback.  How would such a device work?  Physically I couldn’t tell you.  But what we’d need would be a way to electronically manipulate texture in a clear material.  A plastic with a matrix of cells that could expand or contract under electronic control.  So the software could create bumps where each of the keys are.  This would allow a software-reconfigurable gadget that could be almost as usable as a dedicated-purpose device.

This is very different from what is commonly referred to as “haptic feedback” on some of today’s gadgets like the Nexus One.  Here, the phone’s vibrator pulses a bit when you press a soft key.  This is a kind of feedback which is tactile in that you feel it, and it gives you information about your interaction with the device without having to look at the screen.  It certainly helps.  But it is not going to improve basic typing for a critical reason — it can’t help guide fingers to the right place.  The basic act of positioning fingers on controls is still basically open loop, feed forward, without guidance.  What I’m referring to as tactile feedback helps the fingers find the right spots to press without looking.  Today’s haptic feedback can’t do that.

To be clear, true tactile feedback like this almost certainly doesn’t exist yet.  This kind of pure technological innovation basically always starts in universities or government run labs.  The ROI on pure research into unproven technology is so low that it doesn’t make sense for companies to invest there.  Even if a company proved this was possible (which AFAIK hasn’t been done yet) they’d need to figure out how to manufacture it at scale before they could sell a device with it.  Last time I predicted it would be about 2012 before we saw these.  Even though Jobs almost certainly foresees the value of such a system, Apple’s expertise is not in material science.  Wired speculated about such a keyboard based on Apple’s patent filings, but what they describe seems a bit too sci-fi for me to believe.

If they have come up with something new and cool, it’s going to be a smarter way to use basically existing hardware.  I’m gonna guess it’s probably something like a touch screen which is pressure sensitive, so you can rest your fingers on it without indicating a “button press”, making typing more natural.  You could combine this with fixed, transparent dimples on the screen under the positions where the keys are, and you’d do pretty well.  Restrict the keyboard to only work in landscape mode and you only need one set of dimples.  This would be a huge improvement in usability and the biggest technological breakthrough would be the ability to distinguish a soft push from a hard push on a capacitive touch-screen.  Like by how much surface of your finger is on it.

Regardless of what Apple’s actually managed to achieve, I wish them the best.  They’re really pushing the envelope on human-computer interactions.  If they’ve done anything significant to improve soft keyboards, they will have once again done something that the entire rest of the industry will want to emulate, and I’ll tip my hat to them.

Diversifying from hardware

Since Jobs pulled the company out of the doldrums, Apple has been a manufacturer of high-end hardware.  This is a very nice place to be. Their margins are very high.  If you compare the hardware bits that go into a Mac to those in any PC, Apple’s prices are much higher.  PC manufacturers squabble over low single digit margins because their product is almost completely commoditized. HP, Dell and IBM struggle to differentiate themselves in the market.
Meanwhile Apple can charge a hefty premium for good industrial design and software that is slicker than windows.  iPod followed in this tradition of high margins by setting the bar for usability in portable
media devices and following up with fantastic marketing.

Being a niche retailer of high-end products is a comfortable and stable place to be.  However, having a single line of revenue isn’t good for a large company, so diversifying makes a lot of sense.  Thus Apple’s current push into services.

The iTunes music store was an important pre-cursor to the current push into services.  Even though iTunes moves a massive volume of music, if you work through the accounting confusion, they’re not actually making much money there.  Because the established music industry giants are completely fucked and have no idea what to do, they drive the margins of all big online retailers down to a pittance.  But iTunes did something very important for Apple: it established a billing relationship with customers. Cellular operators have this kind of ongoing billing relationship with their customers which enables them to push high margin, low utility products like ring-tones. Similarly, people are used to spending small quantities of money in iTunes to get music. So iTunes is the perfect precursor for an App Store.  In this way, Apple doesn’t even need to rely on the carrier’s billing relationship
to build a services business.  This will be very important for Apple in coming years as carriers increasingly become just another provider of wireless bandwidth.  In the coming decades, the value will not come from piping bits around.  It will come from the services built on those pipes. 

MobileMe is a very Apple service

MobileMe is a cloud-hosted email, contact and calendaring solution. There’s nothing revolutionary about this.  Google offers all of these services for free.  By charging for these services, Apple is implicitly promising to provide a better solution.  Considering their vertical integration into popular hardware, it’s not hard to imagine that they will succeed at this.  Google will probably remain committed to supporting open standards for working with hardware.  By using proprietary protocols, Apple can provide a higher-quality product and support it better on the few platforms they care about.  It’s a classic story we’ve seen in this industry before.

MobileMe’s launch was a disaster.  This isn’t surprising as Apple is yet to demonstrated great skill in online services.  Personally I believe they’ll figure it out, because it’s important to their long-term objectives, and they’re a smart company, and the skills to do this well are getting easier and easier to find. But from a marketing perspective, MobileMe is nothing new for Apple.  Because MobileMe is a premium service for which there is a very good free alternative, MobileMe still targets Apple’s classic market segment. They’re still targeting people who are willing to pay extra to have something really polished.  They can stick to their classic bag of tricks, like the ads that make fun of people who aren’t as cool as the mac devotees.

App Store is new: a platform play

App Store on the other hand is going to be much harder for them to pull off.  For App Store to succeed, their primary challenge is not to attract paying customers, but developers.  Because independent software developers (ISVs) are the ones who are actually creating value in the App Store.  App is just a distributor taking a cut on that.  So what attracts developers to the App Store?  Customers do.

It’s a real chicken and egg problem.  Nobody’s going to build software unless there are customers to buy it, and it’s hard to get customers without cool apps.  Well right now Apple has the whole farm, but it’s a very small farm.  There are millions of iPhones out there, representing potential customers.  Moreover, the iPhone SDK is very rich and capable when compared to its competitors.  Qualcomm’s BREW, Windows Mobile, Palm OS, Blackberry and Symbian have all faltered for one reason or several.  Google’s Android holds much promise and hope, but at this
point it’s complete vapor-ware.  So for now, Apple has almost all the mindshare of mobile application developers.

But how long will this dominance last?  Software platforms are natural monopolies, meaning the economics tend to be winner-take-all.  The chicken and egg “problem” can easily turn into a virtuous cycle, pushing a winner to the top. Application developers are fickle and will code to whatever platform has the best distribution.  iPhones are very popular, but they are still only used by a small fraction of all mobile subscribers.  Until the distribution of iPhones reaches a critical mass, their dominance as a mobile application platform is very shaky.

The mobile app challenger is HTML

But the laundry list of alternatives shows that the competition is fragmented.  What could unseat Apple?  IMHO it’s not another application platform, open or proprietary.  It’s the web.  Every high-end phone can display web pages, and increasingly they’re using high-quality javascript engines that can run real web applications. Webkit, the super-fast open-source HTML/JS engine behind Safari is showing up in Symbian devices, Blackberries and Windows Mobile phones.

Apple’s challenge is to make the proprieray iPhone SDK compelling to developers.  The alternative is to build a web application that works well on webkit, and works on every high-end phone.  They can attract developers in two basic ways — make the native features of their SDK more compelling, and provide a large market for distribution of the applications.

Charging for distribution of these applications is a gimick that won’t last long.  Soon all the interesting applications will be free, but tied to cloud services that have their own business models independent of the mobile client.  Premium applications will start to seem a lot like premium ringtones pretty fast.  Still, it will help bootstrap this market for Apple so long as there are no serious competitors.

To stay on top, iPhones need distribution quickly

One thing that ties all these points together is that Apple’s continued success with App Store hinges on having wide distribution of iPhones. They are currently throttling the distribution of iPhones for some reason, possibly
because of software glitches.  But the aggressive $199 pricing is clearly aimed at attracting a new larger customer base that will help maintain their dominance in the mobile application space.

Long term they might be happy getting by offering premium versions of applications that are freely available on the web.  But something tells me they’re actually trying to break open the mass market on this one. This is Jobs’ big play.  It’ll be really interesting to see how it works out once Android hits the streets.

New iPhones are hard to get

The blogosphere is full of speculation about whether or not Apple deliberately made the iPhone scarce on opening day and since then.  Most assume that this is deliberate on Apple’s part for a variety of reasons, mostly to attract more attention, increase demand, etc.  I assume most of these rants are from bloggers who want their new iPhones but haven’t overcome the barriers to obtain one yet.

But if Apple’s goal was purely to meter out their distribution, why not sell them online?  To get a phone you need to place an order for one, wait a week or two, and then you can get it.  This seems reasonable in conditions of scarcity.  But to get an iPhone 3G, you need to walk into an at&t store to place your order, and then walk into the store again to pick it up.  Think about this.  If the limitation was purely lack of supply then there are several ways this could be easier for customers:

1. You could order a phone online to be delivered to your house.
2. You could order a phone to be delivered to your nearest at&t store.
3. You could call the nearest at&t store to place your order, but still have to walk in to pick it up.

Try asking them why you can’t do any of these things and they will answer with one word: policy.  Clearly Apple & at&t have gone out of their way to make it difficult for people to get their hands on a phone.   This goes above and beyond just preserving a limited supply.  You have to work to get an iPhone 3G.

New iPhones have Issues

From all the reports I’ve read, the problems with the new iPhone are in the software not the hardware.  I conclude this because my friends with first generation iPhones are experiencing the same problems as those with the new 3G iPhones.  Moreover everybody seems to agree that these problems only showed up after they upgraded their iPhone software.  Problems include:

• Frequent crashes of applications, especially Safari
• Increased lag in common operations
• Significant problems with large contact lists (>200 contacts)
• Extended delays before placing a call

Apple is legendary for their high quality software.  People buy Macs because they "just work."  It’s really not like Apple to release a buggy piece of software.  But it sure seems that they did in this case.  Why?  Obvious answers of fierce competition for high-end smartphones.  The more interesting question for me is "How did they get away with it?"  Which it sure seems they are.

Escalation of Commitment: The Hush-factor

There’s a well-document psychological principal at play which prevents people from objectively critiquing things that they are personally invested in.  Sometimes called escalation of commitment, or irrational escalation, the idea is the same.  If somebody works really hard to obtain something, they will blind themselves to its faults.  Imagine this conversation:

"Dude, I can’t believe you waited in line for hours to get that phone.  What do you think of it?"

"Actually, it’s just okay.  The applications crash a lot.  And it’s not nearly as fast as I’d hoped it would be — sometimes it just hangs for like 10 seconds.  But at least it’s pretty."

Very few people have the objectivity to imply that their personal sacrifice was not worth while.  This effect is commonly observed in people who buy high-end items. 

The flip side of this effect is buyer’s remorse.  But since the phone itself is not actually at all expensive (when compared to the monthly fees), that’s unlikely.  Also, it has become a positional good, whereby it has value simply because other people don’t have one.  That fact remains regardless of how unreliable it is.

Speculative Conclusion

I posit that Apple knew about the software problems with the iPhone 3G before launch.  They did manage to iron out all the performance and stability problems they encountered before launch.  They felt they needed to launch it this summer to get ahead of other notable smartphones like the Blackberry Bold, HTC Touch, and Android which are hot on their heels.  So they rushed it out the door at sub-standard quality.

In order to partially cover for this mistake, they have made this device especially hard to get.  This covers their tracks in two ways: people make even more noise about scarcity.  And those who do jump through the whoops to obtain one are far less likely to complain about it.

First off, I am confident Apple will launch a subscription music service.   As every Rhapsody fan and many industry analysts agree, subscription services are the best way to consume music.  Just like Hotmail moved email into the sky, and Google Docs are doing the same for office productivity applications, music can and will go the same way.  Being tied to a specific piece of hardware to enjoy your information services is so 20th century.  The reason we’re not there yet is that it’s not easy to provide a great experience.  And considering people’s long-standing investments in legacy music media like CD’s, non-hosted music services actually provide a smoother transition.

When I worked for Real people generally spoke of Apple launching a subscription service with fear.  I argued that it would actually be one of the best things for the company.  The reason being that even modern electronic music consumers don’t understand what a music subscription service is.  If Apple started spending their quarter-billion dollar per year marketing budget to explain this to consumers, it would do wonders for Rhapsody.  Especially considering the low-quality, poorly-funded advertising campaigns Real has traditionally engaged in.  I wish I could find some of the infomercial-style TV ads they used to run.  Glaser built Real Player without advertising and still believes all internet services should be able to bootstrap themselves.  Maybe the alliance with MTV will help there. 

Also, managing a multi-million song library is not easy.  Rhapsody does a pretty great job of it.  Although they’re going to get obsoleted unless they can figure out how to democratize the music editorial process.  But they’re still way better at it than Apple, who has frankly never been very skilled at online services.  So if Apple were to start spending their huge marketing budget tomorrow to explain why it’s not important to own your own music, it would be a huge boost to Rhapsody.

It won’t happen tomorrow though.  My guess is that within 5 years iTunes will offer all-you-can-eat music for a recurring monthly fee.  The timing depends on a couple of key factors:

• Uptake of network-enabled iPods
• Availability and quality of wireless net access

Before the iPhone, Apple could not launch a subscription music service for one simple reason.  If you stop paying your monthly fee, your subscription tracks need to be disabled from your portable device.  Otherwise somebody could pay the fee for a single month, go on a shopping spree and load up their device with all the music they’ve ever wanted, and never pay another dime.  So even though DRM is going away for track purchases, it has to stick around for subscription models, at least until many other things change.  How does this limit Apple’s ability to launch a subscription service?  As anybody who has used a portable music device with a subscription service can tell you, it is incredibly frustrating to pull your mp3 player off the shelf only to see a message that says it won’t play any of your music because your licenses expired and you need to plug it into a computer to verify that you have been paying your bills.  Even if you are paying, you need to constantly tend to your device or else it bricks itself after a few weeks, by design!  Steve Jobs would never allow his iPods to do this.  The solution is to enable the device to check your subscription entitlement itself — wirelessly, in the background, automatically. 

That’s exactly what the iPhone and iPod touch can do with their built-in networking stacks.  Even a slow network like AT&T’s EDGE network is good enough to verify that the monthly fee has been paid up.  Or for the wifi-only Touch, at least once per month you need to pass by an open hotspot or be in your house where it knows how to connect and it keeps working.  Not a serious burden.

So once there is a sufficiently large installed base of connected iPods, Apple will start selling a subscription service.  If I had more motivation to figure out the timing of when this would happen, I’d look at adoption/saturation curves for iPods and typical turn-over rates for such consumer electronic devices.  Other factors include the financial and market success of competing services.  I leave all this as an exercise to the reader for those of you working in this challenging industry.  My gut says it’ll be in 2010.

Safari does have some innovative UI features which will win them some
fans.  Without IP protection I’m guessing it won’t take long for the
Mozilla kids to replicate them.  (If there’s one thing open source is
good at it’s building a project to match somebody else’s spec.)  I
really doubt Apple will be able to keep innovating useful features fast
enough to keep a meaningful lead for long.

The real reason they’re doing this is of course to make it a more attractive platform for people to write applications for the iPhone.  It’s a close analogy to how Microsoft supported IE for Mac — the only reason is to have a reasonable cross platform story.  This is another sign that Apple’s betting big on the iPhone.  The fact that it won’t be easy to text while driving will limit its usefulness for some people.  It’s a symptom of the phenomenon that even with a great touch-screen like the iPhone has, generic UI’s aren’t as good as single-purpose UI’s.

Attracting web developers is really what Apple needs to do.  Listening to my coworkers, they’ve got an uphill battle ahead of them.  Safari claims to be based on standards, but I hear it’s a hassle to get things to work properly in it.  Right now, the subtle HTML/CSS rendering differences aren’t nearly as important as differences in javascript behavior.  For a long time Safari has been missing some a few key XML parsing APIs for AJAX.  In the near future, the key differentiator for browsers is going to be the availability of good libraries for off-line web applications, like Google Gears.  Unless they talk Google into release Gears for Safari, I predict this isn’t going to go very well for them.  I hope we don’t see too much industry fragmentation here.