It was quite a chore to get Apple to admit that the cause was a hardware problem, and fix it.  But I finally succeeded, so I thought I’d share some of my experiences.  I’ll explain what a Kernel Panic is, how they sometimes can be caused by faulty software but often indicate hardware problems, how they differ from other kinds of crashes, and provide a guide on how to read a Mac OS X kernel panic report.

Dealing with the “Genius” Bar staff

“Genius” is what Apple calls its first tier of technical support.  I find the brand unfortunate and insulting for everybody involved.  There is no intelligence test required to work as a “genius” — just some minimal training on how to follow Apple customer service scripts like an obedient robot.  Knowing Apple, I wouldn’t be surprised if the “Genius” staff are required to follow these scripts verbatim and face not only termination but punitive lawsuits for deviating from the party line.  Keep this in mind when dealing with them.  Also know that they have some discretion in the outcome of your visit, but the discretion exists within guidelines that they cannot control.

Some tips on getting past the “genius” from my limited experience.  Print out your kernel panic reports and bring them in.  The more the better.  Highlight the relevant parts.  I’m not sure if bringing a bad attitude with you helps or not — they want to make their customers happy, but they don’t like their “genius” title challenged with logic.  I also recommend persistence.  Following their stupid advice and showing them that it did no good will help.  I’m not sure if understanding what’s going on will or not.  But if you’d like to understand more about why your Mac is crashing, read on…

Kernel panics and hardware failures vs regular software failures

There are two basic ways your Mac can crash.  First, an application might lock up on you and become unresponsive.  You get the spinning beachball of death, and eventually have to Force Quit your application, losing whatever work you hadn’t saved.  This kind of user mode failure is very common with buggy software.  If the beachball is getting you down, the problem is almost certainly caused by bad software, not by a hardware problem.  In OS 9 and before, this kind of failure could have taken down your entire machine, but since the introduction of the BSD kernel in OS X, the system is designed to allow one application to fail while protecting all the other applications.

Sometimes though your entire Mac will crash hard.  Without warning your system displays a full-screen message saying “You need to restart your computer. Hold down the Power button for several seconds or press the Restart button.” in several languages.  This is OS X’s last ditch attempt to tell you something about what happened before it goes completely teets up.  It’s formally known as a kernel panic.  Sometimes the system is so screwed it can’t even get that error message onto the screen before it dies.

Kernel panics indicate a serious problem, either with the computer’s hardware, or the low-level software in the operating system. In fact there are only three things that can cause a kernel panic:

1. Faulty hardware causes a problem that the OS doesn’t know how to deal with
2. A bug in OS X itself
3. A bug in an OS plugin called a kernel extension or kext

Firstly, if the hardware itself has problems, then kernel panics are a common way they manifest themselves.  Similarly, if the operating system itself has any bugs, they could take down the entire system.  The third option could be caused by third-party software, while the first two are entirely Apple’s responsibility.  So when it comes to dealing with the “Genius” behind the bar, the first two are fairly straightforward.  If you’re seeing this problem a lot, and nobody else is, then it’s probably a hardware problem, and they should replace your hardware. Here’s a thought experiment I tried unsuccessfully with the Apple “geniuses” I had to deal with: Imagine you have a hundred Macs all running the same software, and one of them crashes periodically, but the other 99 don’t.  Would you classify that Mac as having a hardware problem or a software problem?  In my case, the genius insisted that it was a software problem.  In fact he claimed he was certain that if I uninstalled Adobe Flash, the problem would be fixed.  Read on, and you’ll learn how the kernel panic reports themselves show that this explanation is impossible.

Understanding and interpreting Kernel Panic reports

First a bit about what a Kernel Panic is.  Very simply, it’s when something unexpected goes wrong in the operating system kernel.  What’s the kernel?  The kernel is the lowest level of the operating system — the part that’s closest to the hardware.  In modern operating systems, there’s a fairly arbitrary line between what functionality lives in the kernel and what functionality lives in the user space.  The key difference is that when something goes wrong with software in the user space, you get a beachball on the app, but the system survives.  When something goes wrong in the kernel, you get a kernel panic, and the whole system goes bye bye fast.  So it’s critical that any code running in the kernel space be ultra reliable.  You don’t change kernel code quickly or lightly, and you test the hell out of it before you release it.  But code runs faster in the kernel, so most modern operating systems put important things like networking and graphics into the kernel.  The BSD kernel which powers OS X allows the installation of “kernel extensions” or “kexts” which add functionality.  More about these soon.  But suffice to say that when anything goes wrong with any kext, it’s a big deal problem because there’s nothing to fall back on (e.g. can’t display an error dialog if the problem is with the display system), so the system’s reaction is called a panic.  Thus “kernel panic.”

Immediately after a KP, your computer does two things: it stores a bunch of information to help diagnose what caused the problem, and puts up the error screen, if it can.  When you reboot, your computer asks if you want to send the KP report to Apple.  You should do this.  The smarter of the “genius” staff can look these reports up and see that your Mac is actually crashing, but they’ll admit that the contents are too technical for a mere “genius” to understand.  Well I’m going to explain to you what the reports contain and what it means about what’s wrong with your computer.

Here’s a typical crash report from my computer.  In my case, these panics weren’t even accompanied by the “restart your computer message” because as I’ll explain, the problem originated in the graphics system.  My computer just suddenly went black and non-responsive.  I’ve highlighted a few key sections for explanation below.

Interval Since Last Panic Report:  420 sec
Panics Since Last Report:          1
Anonymous UUID:                    8A09F455-1039-4696-8479-xxxxxxxxxxxx
Thu Apr 21 09:00:51 2011
panic(cpu 3 caller 0x9cdc8f): NVRM[0/1:0:0]: Read Error 0x00000100: CFG 0xffffffff 0xffffffff 0xffffffff, BAR0 0xc0000000 0xa734e000 0x0a5480a2, D0, P2/4
Backtrace (CPU 3), Frame : Return Address (4 potential args on stack)
0xbc001728 : 0x21b510 (0x5d9514 0xbc00175c 0x223978 0x0)
0xbc001778 : 0x9cdc8f (0xbe323c 0xc53840 0xbf23cc 0x0)
0xbc001818 : 0xae85d3 (0xe0cfc04 0xe5c9004 0x100 0xb83de000)
0xbc001868 : 0xadf5cc (0xe5c9004 0x100 0xbc001898 0x9bd76c)
0xbc001898 : 0x16c8965 (0xe5c9004 0x100 0x438004ee 0x28)
0xbc0019d8 : 0xb07250 (0xe5c9004 0xe5ca004 0x0 0x0)
0xbc001a18 : 0x9d6e23 (0xe5c9004 0xe5ca004 0x0 0x0)
0xbc001ab8 : 0x9d3502 (0x0 0x9 0x0 0x0)
0xbc001c68 : 0x9d4aa0 (0x0 0x600d600d 0x704a 0xbc001c98)
0xbc001d38 : 0xc89217 (0xbc001d58 0x0 0x98 0x2a358d)
0xbc001df8 : 0xc8ec1d (0xe8e5404 0x0 0x98 0x45e8d022)
0xbc001f18 : 0xc8f0b4 (0xe8e5404 0x124b6204 0x6d39d1c0 0x0)
0xbc001f78 : 0xc8f39f (0xe8e5404 0x124b6204 0x6d39d1c0 0xbc0021e0)
0xbc002028 : 0xca3691 (0xe8e5404 0x1f80d8e8 0xbc00239c 0xbc0021e0)
0xbc002298 : 0xc84d09 (0x6d0b7000 0x1f80d8e8 0xbc00239c 0x0)
0xbc0023f8 : 0xc84f47 (0x6d0c6000 0x1f80d800 0x1 0x0)
0xbc002428 : 0xc87a04 (0x6d0c6000 0x1f80d800 0x0 0x97c6c4fc)
0xbc002468 : 0xca9d40 (0x6d0c6000 0x1f80d800 0x6d09f274 0x140)
0xbc0024f8 : 0xc9b5a9 (0xde94bc0 0x1f80d800 0x0 0x1)
0xbc002558 : 0xc9b810 (0x6d09f000 0x6d09f77c 0x1f80d800 0x0)
0xbc0025a8 : 0xc9bce4 (0x6d09f000 0x6d09f77c 0xbc0028cc 0xbc00286c)
0xbc0028e8 : 0xc98aaf (0x6d09f000 0x6d09f77c 0x1 0x0)
0xbc002908 : 0xc605a1 (0x6d09f000 0x6d09f77c 0x1956a580 0x0)
0xbc002938 : 0xc9a572 (0x6d09f000 0xbc002a7c 0xbc002968 0x5046b1)
0xbc002978 : 0xc648de (0x6d09f000 0xbc002a7c 0x0 0xc000401)
0xbc002ab8 : 0xc9dee6 (0x6d09f000 0x0 0xbc002bcc 0xbc002bc8)
0xbc002b68 : 0xc60c93 (0x6d09f000 0x0 0xbc002bcc 0xbc002bc8)
0xbc002be8 : 0x56a738 (0x6d09f000 0x0 0xbc002e3c 0xbc002c74)
0xbc002c38 : 0x56afd7 (0xcef020 0x6d09f000 0x129bab88 0x1)
0xbc002c88 : 0x56b88b (0x6d09f000 0x10 0xbc002cd0 0x0)
0xbc002da8 : 0x285be0 (0x6d09f000 0x10 0x129bab88 0x1)
0xbc003e58 : 0x21d8be (0x129bab60 0x1ec235a0 0x1fd7e8 0x5f43)
      Backtrace continues...

      Kernel Extensions in backtrace (with dependencies):
         com.apple.GeForce(6.2.6)@0xc55000->0xd0afff
            dependency: com.apple.NVDAResman(6.2.6)@0x967000
            dependency: com.apple.iokit.IONDRVSupport(2.2)@0x95a000
            dependency: com.apple.iokit.IOPCIFamily(2.6)@0x927000
            dependency: com.apple.iokit.IOGraphicsFamily(2.2)@0x938000
         com.apple.nvidia.nv50hal(6.2.6)@0x1592000->0x19a6fff
            dependency: com.apple.NVDAResman(6.2.6)@0x967000
         com.apple.NVDAResman(6.2.6)@0x967000->0xc54fff
            dependency: com.apple.iokit.IOPCIFamily(2.6)@0x927000
            dependency: com.apple.iokit.IONDRVSupport(2.2)@0x95a000
            dependency: com.apple.iokit.IOGraphicsFamily(2.2)@0x938000

BSD process name corresponding to current thread: kernel_task

Mac OS version:
10J869
Kernel version:
Darwin Kernel Version 10.7.0: Sat Jan 29 15:17:16 PST 2011; root:xnu-1504.9.37~1/RELEASE_I386
System model name: MacBookPro6,2 (Mac-F22586C8)
System uptime in nanoseconds: 35829130822125

unloaded kexts:
com.apple.filesystems.msdosfs 1.6.3 (addr 0xbc1e5000, size 0x53248) - last unloaded 12216461868115

loaded kexts:
com.parallels.kext.prl_vnic 6.0 11992.625164
com.parallels.kext.prl_netbridge 6.0 11992.625164
com.parallels.kext.prl_usb_connect 6.0 11992.625164
com.parallels.kext.prl_hid_hook 6.0 11992.625164
com.parallels.kext.prl_hypervisor 6.0 11992.625164
com.apple.filesystems.smbfs 1.6.6 - last loaded 12151022138289
com.apple.driver.AppleHWSensor 1.9.3d0
com.apple.driver.AGPM 100.12.19
com.apple.driver.AppleMikeyHIDDriver 1.2.0
com.apple.driver.AppleHDA 1.9.9f12
com.apple.driver.AppleUpstreamUserClient 3.5.4
com.apple.driver.AppleMCCSControl 1.0.17
com.apple.driver.AppleMikeyDriver 1.9.9f12
com.apple.driver.AudioAUUC 1.54
com.apple.driver.AppleIntelHDGraphics 6.2.6
com.apple.driver.AppleIntelHDGraphicsFB 6.2.6
com.apple.driver.SMCMotionSensor 3.0.0d4
com.apple.kext.AppleSMCLMU 1.5.0d3
com.apple.Dont_Steal_Mac_OS_X 7.0.0
com.apple.iokit.CHUDUtils 201
com.apple.iokit.CHUDProf 216
com.apple.driver.AudioIPCDriver 1.1.6
com.apple.driver.AppleGraphicsControl 2.8.68
com.apple.driver.ACPI_SMC_PlatformPlugin 4.5.0d5
com.apple.GeForce 6.2.6
com.apple.driver.AppleLPC 1.4.12
com.apple.filesystems.autofs 2.1.0
com.apple.driver.AppleUSBTCButtons 200.3.2
com.apple.driver.AppleUSBTCKeyboard 200.3.2
com.apple.driver.AppleIRController 303.8
com.apple.driver.AppleUSBCardReader 2.5.8
com.apple.iokit.SCSITaskUserClient 2.6.5
com.apple.BootCache 31
com.apple.AppleFSCompression.AppleFSCompressionTypeZlib 1.0.0d1
com.apple.iokit.IOAHCIBlockStorage 1.6.3
com.apple.driver.AppleUSBHub 4.1.7
com.apple.driver.AppleFWOHCI 4.7.1
com.apple.driver.AirPortBrcm43224 427.36.9
com.apple.iokit.AppleBCM5701Ethernet 2.3.9b6
com.apple.driver.AppleEFINVRAM 1.4.0
com.apple.driver.AppleSmartBatteryManager 160.0.0
com.apple.driver.AppleUSBEHCI 4.1.8
com.apple.driver.AppleAHCIPort 2.1.5
com.apple.driver.AppleACPIButtons 1.3.5
com.apple.driver.AppleRTC 1.3.1
com.apple.driver.AppleHPET 1.5
com.apple.driver.AppleSMBIOS 1.6
com.apple.driver.AppleACPIEC 1.3.5
com.apple.driver.AppleAPIC 1.4
com.apple.driver.AppleIntelCPUPowerManagementClient 105.13.0
com.apple.security.sandbox 1
com.apple.security.quarantine 0
com.apple.nke.applicationfirewall 2.1.11
com.apple.driver.AppleIntelCPUPowerManagement 105.13.0
com.apple.driver.DspFuncLib 1.9.9f12
com.apple.driver.AppleProfileReadCounterAction 17
com.apple.driver.AppleProfileTimestampAction 10
com.apple.driver.AppleProfileThreadInfoAction 14
com.apple.driver.AppleProfileRegisterStateAction 10
com.apple.driver.AppleProfileKEventAction 10
com.apple.driver.AppleProfileCallstackAction 20
com.apple.driver.AppleSMBusController 1.0.8d0
com.apple.iokit.IOFireWireIP 2.0.3
com.apple.iokit.IOSurface 74.2
com.apple.iokit.IOBluetoothSerialManager 2.4.0f1
com.apple.iokit.IOSerialFamily 10.0.3
com.apple.iokit.CHUDKernLib 208
com.apple.iokit.IOAudioFamily 1.8.0fc1
com.apple.kext.OSvKernDSPLib 1.3
com.apple.driver.AppleHDAController 1.9.9f12
com.apple.iokit.IOHDAFamily 1.9.9f12
com.apple.iokit.AppleProfileFamily 41
com.apple.driver.AppleSMC 3.1.0d3
com.apple.driver.IOPlatformPluginFamily 4.5.0d5
com.apple.driver.AppleSMBusPCI 1.0.8d0
com.apple.nvidia.nv50hal 6.2.6
com.apple.NVDAResman 6.2.6
com.apple.iokit.IONDRVSupport 2.2
com.apple.iokit.IOGraphicsFamily 2.2
com.apple.driver.BroadcomUSBBluetoothHCIController 2.4.0f1
com.apple.driver.AppleUSBBluetoothHCIController 2.4.0f1
com.apple.iokit.IOBluetoothFamily 2.4.0f1
com.apple.driver.AppleUSBMultitouch 206.6
com.apple.iokit.IOUSBHIDDriver 4.1.5
com.apple.iokit.IOSCSIBlockCommandsDevice 2.6.5
com.apple.iokit.IOUSBMassStorageClass 2.6.5
com.apple.driver.AppleUSBMergeNub 4.1.8
com.apple.driver.AppleUSBComposite 3.9.0
com.apple.iokit.IOSCSIMultimediaCommandsDevice 2.6.5
com.apple.iokit.IOBDStorageFamily 1.6
com.apple.iokit.IODVDStorageFamily 1.6
com.apple.iokit.IOCDStorageFamily 1.6
com.apple.driver.XsanFilter 402.1
com.apple.iokit.IOAHCISerialATAPI 1.2.5
com.apple.iokit.IOSCSIArchitectureModelFamily 2.6.5
com.apple.iokit.IOUSBUserClient 4.1.5
com.apple.iokit.IOFireWireFamily 4.2.6
com.apple.iokit.IO80211Family 314.1.1
com.apple.iokit.IONetworkingFamily 1.10
com.apple.iokit.IOUSBFamily 4.1.8
com.apple.iokit.IOAHCIFamily 2.0.4
com.apple.driver.AppleEFIRuntime 1.4.0
com.apple.iokit.IOHIDFamily 1.6.5
com.apple.iokit.IOSMBusFamily 1.1
com.apple.kext.AppleMatch 1.0.0d1
com.apple.security.TMSafetyNet 6
com.apple.driver.DiskImages 289
com.apple.iokit.IOStorageFamily 1.6.2
com.apple.driver.AppleACPIPlatform 1.3.5
com.apple.iokit.IOPCIFamily 2.6
com.apple.iokit.IOACPIFamily 1.3.0

The first line is fairly clear — how long has your system been running since its last crash?  If this is less than an hour, as it was for my computer, then your machine is completely FUBAR.  Less than a day and you’ve still got a seriously unstable computer.  (Hint for any “genius” that might be reading this article: take the number of seconds, divide it by 60 using the Calculator app on your store-issued-iPad, and that will give you the number of minutes.  Divide that new smaller number by 60 again to get an even smaller number which is hours.  If you can figure out how to get to number of days by yourself, it’s time to apply for the “Genius Lead” job.)

The Anonymous UUID is an effectively random code that allows Apple to lookup the crash reports for your computer when you go into the store.  Then there’s the date.  Straightforward.

The line which starts “panic” is the closest thing you’ll find to a concise explanation of what went wrong. In all likelihood this will be a jumble of words and numbers that make no sense, but it’s a great string to Google.  If you’re having a hardware problem, this message will probably stay about the same with each KP.  Googling my error message “NVRM[0/1:0:0]: Read Error 0×00000100” turns up a bunch of people with similar problems — computer going black without warning, often while playing World of Warcraft.

The next section titled “backtrace” is worthless unless you’re actually diving into the source code that caused the problem.  Skip over it.  But the section after it is extremely interesting and relatively easy to interpret.

The section titled “Kernel Extensions in backtrace (with dependencies)” actually tells you what part of the system failed.  Read this one closely and try to make sense of it. In the case of my example, there are three kernel extensions involved with the crash.  They are called “com.apple.GeForce” and “com.apple.nvidia.nv50hal” and “com.apple.NVDAResman”.  The first one is fairly obvious — GeForce is the kind of graphics chip in the macbook.  The second one is also pretty clear — NVidia is the company that makes GeForce, and nv50hal I would guess means “NVidia 5.0 Hardware Abstraction Layer” or something similar.  I’m not sure what NVDAResman is but looking down a bit I see it’s related to “IOGraphicsFamily”.  This paints a really clear picture that the failure is in the graphics system.  Moreover, since every line here starts with “com.apple” we know the failure is entirely in code written by Apple.  There is no third-party software involved in this crash.

For my particular crash, it’s important to know something about the graphics hardware of these MacBooks, since all evidence points to the graphics hardware.  This generation of macbooks have two graphics chips — a faster one from Nvidia, and a more battery-friendly one from Intel.  The nvidia chip which is apparently having problems is always used when the computer has an external monitor plugged in, or when something fancy is happening on the built-in screen.  A nice utility called gfxCardStatus can help you understand this complexity, and will definitely give you a leg up on the “genius.”

The following line starting with “BSD process name” can also be important.  This will sometimes tell you which user-level app originated the call into the kernel which failed.  In my case it was “kernel_task” which provides no additional information.

The next section gives some basic info about the Mac — hardware and OS versions.  What follows is a complete list of kernel extensions (kexts) installed.  This gives you a bit more ammo in dealing with the “genius” who is probably ignoring you at this point anyway.  You can look through this list and see everything that might possibly contribute to a kernel panic.  In my case, the only software modules that aren’t from Apple are some drivers from Parallels for running my Windows virtual machine.  So the only reasons my Mac might kernel panic are because of a hardware problem, a bug in OS X itself, or something going wrong with Parallels.  Understanding this should, in theory, be very helpful when talking to your local neighborhood “genius” but unfortunately they are simple bots that only run scripts authored in Cupertino and are not permitted to listen to logic.

Apple’s Propaganda about Flash

When the “genius” told me my Mac’s problem was that I had Adobe Flash installed, I just laughed at first.  Flash is installed on something like 97% of desktop computers, and very few of them regularly turn themselves off for no reason.   Moreover, the kernel panic report lists every piece of software that could possibly contribute to the kernel panic, and neither the word “flash” nor “adobe” appear anywhere in the list.  But then I realized he wasn’t joking.

Apple’s ongoing arguments with Adobe over Flash are well publicized.  The root of the issue, in very brief summary, is that Apple sees Adobe’s Flash as a strategic threat to their incredibly profitable iPhone platform.  The poor “genius” I’m stuck with has become a pawn in Apple’s PR battle, throwing himself on the grenade of propaganda just to spread FUD about Flash.  I tried reasoning with him, explaining that Adobe’s software doesn’t run in the kernel, and therefore cannot cause a kernel panic.  The job of the kernel is to protect users from badly written software crashing the whole machine. But he would not budge.  I imagined a “genius” script which read as follows:

Mac is crashing…

1. Run hardware diagnostic tests.

2. Address any identified hardware problems.

3. If hardware tests come back clean, tell customer that the problem (whatever it is) is caused by Flash.  Tell them to uninstall it, and see if that helps.

Here I imagine the Dantesque trap of the rare “genius” who actually understands how OS X works.  I’m telling the customer something which is impossible on its face, and he knows it.  He’s arguing with me telling me I’m being stupid.  But I signed a contract with Apple saying I would defame Adobe, and deviation from this contract will bring the wrath of Steve’s legal team on me.  I just have to smile and say things like “yeah, that’s the really strange thing about this particular software problem — it only affects certain computers.  But it’s definitely caused by Flash.”

One might reason that Flash could cause kernel panics because it makes more extensive use of the graphics system than other applications.  But in this case, Flash isn’t the actual problem.  Flash is exposing the underlying problem, as would any software which works the graphics system hard.  Thus lots of people with the same problem as me who play World of Warcraft.  If the “genius” advice ever works, it’s just because Flash is the most graphics intensive software that many people use on their Macs.  The actual problem is still either a bug in OS X, or a hardware problem.

Consider the advice not to use Flash on your Mac in analogy to a car.  (A high-end MacBook actually costs as much as some cars.)  Imagine that your car sometimes just turned its engine off while you were in the middle of driving it – catastrophic failure with no warning or apparent reason.  You go to the dealership and they can’t find anything wrong with it, but ask if you ever listen to electronic music?  Well, yes, sometimes.  That’s the problem!  It’s the electronic music which is causing your car to malfunction.  So stop listening to it, and the problem will be fixed.  Umm, what?  The closest thing to the truth, by analogy, would be that any bass-heavy music (graphics-intensive application) is stressing out some weak connection in the electronics.  But because the car dealership is owned by the local philharmonic, they’re blaming it on that awful music the kids listen to.   Using your misfortune and their incompetence to push an unrelated political agenda.

It’s an interesting glimpse into how Apple is using their retail presence to advance a strategic PR goal.  Evidence that Apple has grown up as a company to the point where their own motives are more important than doing what actually helps customers.  *sigh*  At least I got my MacBook fixed.

Please democratize SSL certificates.  The ability to serve HTTPS:// pages without scaring users is currently controlled by a handful of “trusted authorities” whose business is to make it difficult to secure web communications.  Google, you have the ability to disrupt this oligarchy and empower individuals to make the web safer.

The web is a safer place when information passed between browsers and web servers is encrypted — that is when URLs start with HTTPS instead of HTTP.  The recent introduction of FireSheep demonstrated to the world just how insecure normal (HTTP) web communications are — anybody on your network with a simple browser plugin can impersonate you.  In fact, FireSheep democratized the ability to steal session authentication by bundling it up in a manner that is easily used by the masses.  Google’s own proposed SPDY protocol, whose primary goal is to make the web faster, is willing to slow down in the name of security.  “Although SSL does introduce a latency penalty, we believe that the long-term future of the web depends on a secure network connection.”  We all want a safer web, so please help us achieve that by making it easier to set up HTTPS on our web servers.

There is no technical challenge here.  All modern browsers and servers are capable of safely encrypting the information passed between them.  Encryption protects users against eavesdropping and session hijacking a la firesheep.  Today’s challenge to secure web communications does not lie in the encryption, but the authentication.  The HTTPS protocol begins with the server presenting its security “certificate” which is meant to assure the user they have not reached an imposter web site.  This assurance is provided courtesy of the oligarchy of trusted certificate authorities, for a fee and a hassle.  Alternately, servers can present a “self-signed certificate” which provides equally good encryption, but no assurance that the server is who it claims to be.  But instead of recognizing self-signed certificates as being safer than no security at all, today’s popular browsers do their best to terrify and/or inconvenience users when visiting sites with self-signed certificates.  Certainly there is some value in authenticating the web server, but is that value worth the cost of allowing eaves-dropping and session hi-jacking on the vast majority of web sites?  I think not.

The current standard practice is backwards.  An HTTPS request to a server using a self-signed certificate offers encryption but not authentication.  This is clearly safer than a plain-text HTTP request, which offers neither encryption nor authentication.  But browsers tell users that self-signed certs are worse than unsecured communications.  (Chrome is actually worse than others.)  Deploying SSL on a commercial scale is also complicated by shared IP addresses for multiple sites, which again interferes with authentication, but not encryption.  The certificate verification UI already demonstrates varying levels of trust as shown below.  But self-signed certificates which offer encryption without authentication are incorrectly indicated.  Let’s remove the simple barriers which are preventing encrypted web communications.

The best technical path to fix this mess is immaterial here — many options exist.  Changing browser behavior to make self-signed certs less scary is one path, although it’s not a complete solution because of the legacy of every installed browser.  A new free service that signed anybody’s certificate with a trusted cert would work, provided that company had sufficient clout to get their root cert recognized.  (Google, you can do this.)  Empowering any domain registrar to sign SSL certs also makes sense since they’re the ones ultimately authenticating who owns a domain.  This choice wouldn’t immediately bring certificate prices to zero, but would greatly accelerate the trend we already see of lowering prices.  Perhaps a bloom-filter algorithm similar to what Chrome uses to identify malware sites could differentiate those sites whose identity has actually been verified through stricter measures, where self-signing should not be trusted.  A deeper technical analysis is needed to determine the best tactics, but clearly Google has both the necessary skills and level of influence needed to effect this change.

Additionally, Google uniquely has the motivation to make the web safer.  Google long ago recognized the value of primary demand stimulation — more web use means more web searches which means more advertising revenue for Google.  Open standards do not advance without leadership from selfishly interested parties.  The state of SSL certificates mirrors a political situation that desperately needs legislative intervention — a special interest group (the root certificate authorities) has a strong financial incentive to maintain status quo, even though every individual marginally benefits from the change.  Google is the company that stands to benefit the most from a safer web.  So please Google, act now to bring democracy to the safe exchange of information on the web by enabling anybody to freely secure their web traffic.

Obviously there are huge differences.  Chernobyl was a massive disaster that killed thousands of people, the only accident to ever reach level 7 on the International Nuclear Event Scale (INES).  When I started writing this article, Fukushima was classified as level 4, although that was before the containment building at reactor 3 exploded, and trouble really started in reactor 2.  I had written that it was likely to be re-classified as level 5, and now lots of people are saying they think it might end up as level 6.  I had written that I think it’s extremely unlikely to reach level 7 where thousands of people die from radiation poisoning, but the way things are going, I’m not so confident of that any more.  :(

For a decent explanation of the defense-in-depth strategies of the Fukushima reactors, read this overly-optimistic article.  This article has been widely distributed and republished because its “you’re all over-reacting” message is a nice one to hear and it comes from a seemingly credible source, a scientist at MIT.  But the article has an interesting past, originally including a major technical confusion, mixing up moderators which speed up nuclear reactions with control rods which slow them down.  This mistake was fixed fairly quickly, and then article moved to a new location hosted by MIT, along the way shedding its re-assurances that nobody would get any more radiation than from “a long distance flight”.  Clearly things are worse than that.  Nonetheless, Fukushima was built with many layers of protection, making a Chernobyl-scale disaster much less likely.  But things just keep getting worse there.

Fukushima faces the same problem Chernobyl was trying to fix

As we’ve all probably heard, the Chernobyl reactor exploded while performing an experiment.  The causes of the disaster are many, but most fundamentally the reactor design was unstable.  Relying on cooling water as a nuclear damping material gave the RBMK-style reactors a positive void coefficient meaning that as the water boiled from liquid to gaseous state, the nuclear reaction accelerated.  This is fundamentally unstable since it can create a positive feedback cycle, as it did during their fateful experiment.  The reactor heats up, which boils water, and since steam is less dense than liquid water there is now less nuclear damping material to slow the reaction, so it goes faster.  (Modern reactors don’t do this.)  In fact just 36 seconds after operators started the experiment, somebody hit the “Oh Shit” button (which unfortunately due to even worse design actually exacerbated the problem), and seconds later the reactor core tragically exploded.  Chernobyl’s core didn’t have time to melt — it just exploded.  Then large amounts of radioactive graphite burned in a hot fire which carried toxic ash high into the atmosphere.  Thousands got sick and died.

Despite what the Soviets wanted everybody to think afterwards (and even convinced the IAEA for 7 years), the motivation for the experiment at Chernobyl was wise and well-intentioned.  The operators were not insane, stupid, nor psychotic.  They knew that their reactor relied on the external power grid to run its cooling systems.  Of course they had backup diesel generators on site in case the power grid failed, but they also knew these generators could take up to a full minute to kick in.  That seemed like too long of a gap, so they were trying something creative — using the momentum in the plant’s own steam turbine to power the cooling pumps as the turbine was coasting down, unpowered.  They were thinking to themselves “hey, we’ve got this great power source, why don’t we use it to run the cooling pumps instead of relying on the external grid.”  Great idea.  They’d tried the experiment a couple times before.  It hadn’t worked.  This time it really didn’t work.  But because the reactor was so unstable when the experiment started that a slight decrease in cooling caused it to explode, not because the idea was flawed.

The heart of Fukushima’s problems are the same — the electrical grid around them was taken out by the earthquake.  They shut down their own reactions almost instantly after the quake, and thus were no longer producing their own electricity.  So to power the cooling pumps they needed to switch to backup power.  Unfortunately the backup generators failed, most agree due to the tsunami.

So Fukushima has this ironic problem.  They have an incredibly hot thing.  Even 48 hours after stopping the fission reaction, the core is still producing megawatts of decay heat.  Enough heat to boil 20 tons of water each hour.  They need electricity to run the pumps to cool down this incredibly hot thing.  But they don’t have any electricity.  There’s an electrical power plant (a device to turn heat into electricity) with tons of heat coming off of it, but they don’t have any power to run the cooling pumps, so it overheats.  Ironic, no? This irony was at the core of the experiment that Chernobyl was attempting — use the energy of the offline plant to run the cooling systems.

Safer designs are possible

In principal it seems you should be able to design a reactor that uses this vast quantity of heat (which is power — heat equals power) to run the systems needed to cool the thing off.  Fundamentally this is just an engineering problem.  Shouldn’t we be able to design something that can keep itself cool using its own energy even when disconnected from the grid?  Happily the answer is yes.  But sadly the answer was not yes in the 1970’s when these plants were built.  Not quite at least.

In fact, these old GE Mark I reactors do have emergency core cooling systems designed to help with this, but were never meant to be a complete solution, and clearly didn’t work.  New experimental designs achieve cooling completely passively without any need for active pumping.  But AFAIK these designs have never made it to commercial scale.

A major lesson of Fukushima is clear: extremely unlikely disaster events are highly correlated with each other.  So safety systems should not have external dependencies.   I believe nuclear power has an important place in our path away from fossil fuels towards renewables, but to get there, we need safer designs.

Here are the slides from the talk.

I start out talking about the need for keeping a web page up to date. I talk about polling as a natural but expensive solution to this. Then I talk about how COMET works, a.k.a. hanging GET or long polling. Then I talk about the difficulties of building a COMET stack from scratch and why you shouldn’t. Then I talk about moving to a higher level of abstraction with hookbox and what a publish/subscribe pattern is. Then I build a demo app using hookbox for a simple web chat. The source code for the web chat example is at https://github.com/leopd/hookbox-demo.

The simple way to counter buffer bloat is simply to reduce the size of the transmit buffer in each piece of your network gear.  Most linux systems default to a transmit buffer of 1,000 packets, each of which can be 1.5 kilobytes, meaning that 1.5 megabytes of data can get queued up waiting for a chance to go across the network.  Any application that is trying to move a lot of data through a clogged network will fill this buffer.  That’s fine for the buffer-filling application, but any other application will suffer.  So, for example, if you’re watching youtube and your roommate is trying to surf the net, your roommate’s web page requests will suffer very long latency, because their small web pages must get in this megabyte-long line along with your youtube video before they can be delivered.  If your DSL line runs at say 10 mbps, then it’ll take 1.2 seconds for that 1.5 MB buffer to work fit through your pipe.  Since it takes at least 2 round-trips to get a web page that means your roommate’s web page will take at least 2.4 seconds to show up, no matter how small it is!

Gettys quotes Kleinrock that the ideal size of a network buffer is (bandwidth) x (latency).  Say your bandwidth is 10 mbps.  Latency to any web page you’re likely to visit in the US should be less than 100ms, so let’s use that.  This puts your ideal buffer size 125k.  Buffer sizes are usually configured in terms of maximum number of packets.  Typically the maximum packet size (MTU) is 1500 bytes, resulting in the ideal theoretical buffer size of 83 packets for a typical fastish home network line.  Please redo these calculations yourself and experiment with how different numbers affect your system. (Be careful not to set your buffer size to zero as it could lock up the device’s network.)  Remember that linux (which is likely what your wifi router is running) defaults to 1,000 packets!

[Update shortly after posting: a reader suggested I try setting my buffer to be much smaller still.  So I went down to just 2 packets, and noticed that my ping times are much more reliable now when the network has more than one thing.  His caveat which I will echo is that this will mess with your system if your router is trying to do any kind of traffic shaping, i.e. QoS.  But otherwise protocols like TCP will keep everything running fine.]

If your home wifi access points are using DD-WRT as mine are, here’s how you set them to use a more sane buffer size:

1. Log in to your router’s admin web page.

2. Select the Administration tab and the Commands sub-tab

3. Type in the following commands into the box:

ifconfig eth0 txqueuelen 2

ifconfig eth1 txqueuelen 2

4. Click the “Save Startup” button at the bottom.

There — you’re done!  For alternate techniques to configure your dd-wrt router for this kind of thing, see the wiki page on Startup Scripts.

I’m sharing this information because it took me a while to figure out.  This problem is not well documented.  I’m trying this out now on my house’s network now.  In some controlled tests it seems like it might be somewhat better.  But my tests have not been able to replicate the really horrible situations I’ve seen on our network which I suspect come from lots of simultaneous users.  So it’ll be a while before we know for sure if this was a good change.  To be clear, I don’t know if this advice is good or not. It could reduce your network’s maximum effective bandwidth, but hopefully it will do so by reducing the maximum latency, which is often a very good trade-off.  This advice is consistent with the advice Gettys offers in terms of optimizing buffer sizes, and make sense to me.  YMMV.  If you try it out, please leave a comment on whether or not it helps you.

It gives an introduction to South, which is the most popular database migration tool for Django. If you’re using Django on any kind of mature project and you’re not using South, you should really take a look at this. Without a framework like South you’re probably very scared of changing your model schema and making your life a whole lot harder than it needs to be. Without further ado…

When: Tue, Dec 14th 6:00pm – 7:30pm  (Meetings are on 2nd Tuesday of each month)

Where: Web Collective, 1402 3rd Avenue – Suite 925 – Seattle

(3rd and Union, 9th Floor, Vance Building, kitty corner to Benaroya Hall)

Please try to be here *by 5:55pm*, since the doors lock and we have to let you in
after 6.

My talk will be an introduction to South Migrations in Django and how you can use them to change your database schema on the fly. You don’t have to be afraid of changing your models after you’ve started using them.

I’m looking forwards to seeing everybody there!

Incidentally, if you’re consider rails for a new project, my post on whether Django or Rails is more popular has been getting a lot of interest lately.

Here’s the preso.

I set about answering this question with data.   I quickly found a list of oil spills on wikipedia.  A quick pass through google spreadsheets and a few regexs later, and I’ve got the data in a form that it can be graphed with protovis, a wonderful web-based visualization package.  An initial look shows some interesting trends. (Sorry IE users – modern browser required.)

First, it’s important to emphasize that this is a log-scale graph.  Given the dynamic range of the input, it’s the only reasonable way to visualize what’s there, but if you’re not used to reading log-scale graphs, the data will be deceptive.  In short, being a little higher on the graph means that the spill is a lot larger.  In fact if would be very reasonable to only include the spills near the top of the graph when thinking about “big spills.”  But I wanted to present the entire data set for completeness and analysis.

It’s interesting to note the general downward trend at the bottom of the graph.  I believe this is not a real effect at all, but a result of selective memory.  The smallest spill on this graph was only a couple months ago — the Great Barrier Reef spill in April.  Are we to believe that in the preceeding 100 years of oil exploration there had never been a spill of less than 10 tons of oil, and only a single other spill of less than 100 tons?  Of course not.  I bet spills of this size have happened dozens if not hundreds of times, but 50 or 100 years ago nobody bothered writing them down.  Or if they did write it down, the event has been filtered out of our collective historical memory before making it into wikipedia.  The Lakeview gusher in 1909 is another interesting example of this effect.  This certainly wasn’t the only oil production accident before 1930, but it was clearly an important major accident, and so has been remembered far better than others.

I’ve highlighted a few other spills because of their historical interest.  The Gulf War oil spill (purple dot) of 1991 is exceptional in that it was not an accident, but a deliberate act of war.  As such, it should not be considered in answering the question of whether oil exploration has been getting safer.  The Exxon Valdez spill (light blue dot) in 1989 is large in our memory, but in context we can see that it was not at all a large spill by historic standards.  But clearly the Deepwater Horizon spill (green dot) is huge, ranking as one of the largest spills ever and certainly the largest spill in quite some time.  But aside from this current mess, there does seem to be a real trend towards increased safety in oil exploration.

Again, the log-scale graph makes this somewhat hard to read intuitively.  Because the spills near the top are so much larger than the ones below them, a fair approximation of the sum of all spills can be found by simply considering the points along the top envelope, which is generally decreasing.  Looking just at the last several decades on a linear scale, this trend becomes more clear: since about 1980, serious oil spills have been getting smaller / less frequent. Now we see visually that the majority of spills listed are tiny compared to the few big ones.  I scaled the graph so only the bottom of the uncertainty bar for the gulf war oil spill.  Also note that I’ve kept the middle dots at the geometric average of the low and high estimates, which works visually on the log-scale graph, and makes logical sense given the nature of the problem.

Another factor to consider is that the total amount of oil being produced during this time period has been generally increasing.  I’ve overlaid data from the US Energy Information Administration on global oil production rate, scaled to the average amount produced each hour, to get it to show up on the same scale of this graph.   Another interesting comparison which I haven’t included is the average size of each well, or the number of wells being drilled per unit time.  My understanding is that oil exploration has been getting more difficult over time in that we’re having to drill deeper to get at relatively smaller oil deposits.  Again, this reinforces the idea that we have been getting better and safer — we’re spilling less even though we’re drilling more holes.  Except for the Deepwater Horizon.

Feel free to browse the javascript source code of the graphs for further details, inspiration, double-checking, or whatever.

Having been writing code for 25+ years, the differences between programming languages fade away in my mind.  I know that I, like any good software engineer, can be productive in basically any language.  Certainly within object-oriented languages, (which is where essentially all serious software engineering happens these days) the differences in the language itself are IMHO small compared with other factors in choosing a platform.  Other important factors I consider include:

• Quality and availability of libraries.
• Quality of tools like IDEs, debuggers, automation systems.
• Size and healthy of the active community using the framework.  (i.e. If I run into a problem, how easy is it to google the answer?)
• Ease of hiring people who already know the platform.

All of these considerations are IMHO more important than compiled vs. interpreted or run-time performance or whether the language is statically typed or dynamically typed, or even if it’s open source vs. proprietary technology.  But there is a single common factor which directly feeds into all four of the criteria I list above: How many people are actively using the platform?

The Candidates

The last time I was really writing much code was back in 2007.  At the time Ruby on Rails was the coolest thing since sliced bread.  I played around with it, was amazed by how easy it was to quickly put together simple database-driven websites, and got very frustrated with it when I wanted to color outside of its lines.  A major concern of mine was the level of “magic” that happens behind the scenes — this makes Rails beautiful and elegant when it works, but difficult to debug or extend.  I blogged about my experiences with rails and in particular my conclusion that at the time Rails was not ready for large, complex projects, partly because of a lack of good tools, libraries and sensible error messages, all of which can be fixed by more users.

Around then I started working for Google and stopped writing code.  The next year or so saw a couple big things happen for Ruby on Rails.  Twitter famously had trouble maintaining even 99% uptime (fail whale anybody?), and everybody knew they were running on Ruby on Rails.  Also, Google launched AppEngine, which supported Python, not Ruby, and pointed people like me to an alternative high-level web framework: Django.

So now, in 2010, I return to the fray, and I’m trying to decide between the two frameworks.  There are of course other alternatives, but for the purpose of brevity, I’ll leave out my process of reducing my choices to these two: django or rails?

Measuring community activity

I often use Google Trends to measure relative interest in technologies.  My time is very valuable, so even downloading something and reading its documentation is an investment I’d rather shortcut if I can.  This crude measure of relative search activity can actually be quite telling, and has saved me a bunch of time in choosing packages.

The relative values are hard to take literally since “django” as a single word search query will naturally be higher than a 3-word query like “ruby on rails”, but the search for “rails” by itself will clearly have lots of irrelevant searches.  Likewise, some people searching for “Django” won’t be looking for the web framework, but rather the guitarist or the movie or what-have you.

But it is clear that search activity for Ruby on Rails peaked in 2007-2008 and has been declining since then, while Django has been on a steady upwards trend.

Elsewhere I can find evidence that Rails is still a more used platform.  Stackoverflow has twice as many questions about Ruby on Rails vs Django: 13,882 versus 7,496.  To me this indicates pretty clearly that Rails is more active than Django.  Either that or Rails is more confusing and people ask more questions about it, but I doubt that.

The Tiobe Index attempts to objectively measure activity across programming languages.  By its easily manipulated measure, Python has twice the activity of Ruby, independent of the django / rails frameworks, with both languages in decline.

Meanwhile it’s easy to find comparisons on the net between the two.  Everything from content-free videos to people declaring rails dead, and everything in between.

On balance, I find the stackoverflow numbers the most compelling, unbiased indication that Ruby on Rails has more activity than Django / Python.  Even though I’ve been frustrated by it in the past, by my own objective criteria, that seems to make Rails a better choice for building a new web application.

What’s your opinion?

If you have experience with both frameworks, I’d love to hear your experiences.  Please leave a comment.

Simultaneously, technological advances have allowed advertising to progress along a different axis – to become more targeted.  Advertising used to only be broadcast widely through newspapers and television shows.  The best an advertiser could do to ensure their message reached the right kind of people was to select the aggregate demographics of everybody who read a particular magazine.  Now the internet allows ads to be targeted as precisely as you’d like.  Today, Google lets you get your message only in front of people who are about to buy a product like yours.  The ability to connect to people who have expressed an intention to “buy digital camera” is a literal gold-mine, making billionaires out of Larry, Sergey and Eric.  As effective as it is, targeted advertising won’t replace broadcast advertising, because there is still value in abstract brand-building.  Rather, the two will complement each other.

Enter Social Media

Social media has been all the buzz recently.  At its core it’s just a more convenient way for friends to communicate.  The “killer app” for computers has always been helping people communicate, and this is just another chapter in that book.  With this new communications medium comes a new opportunity for organizations to tell their stories.  In fact, I believe that social media will bring another tectonic shift in the entire marketing industry, possibly as important as search-based advertising.  As consumers have gotten more and more sophisticated at filtering out advertising from broadcast media, advertisers have gotten more and more desperate in their attempts to connect with people.  Social media marketing offers a new path – instead of hearing about products and services through ads, people can hear about products and services from their own friends.  Exactly how this will play out through Twitter/Facebook/Foursquare/whatever is not at all clear to me right now, but I fundamentally believe this change is coming, and it will take the entire marketing industry with it.  Klein and her fans are free to unplug from popular culture in order to avoid the onslaught of brand advertising, but they would be foolish to stop talking to their friends just because their friends are happy with things they’ve bought.

This vision is one of the main things that prompted me to jump off the comfy Google cruise liner and start paddling hard in Banyan Branch’s crowded dinghy.

Is marketing intrinsically evil?

I sometimes feel a need to justify this line of work to those who think that marketing is inherently dirty.  I admit that I’m more of a capitalist than many of my friends, but I certainly recognize that capitalism has its limits.  The vast majority of economic transactions are both consensual and mutually beneficial, and I will argue vigorously that there is nothing wrong with an economic system consisting of these transactions.  The biggest exception to this happens when transactions are not mutually beneficial because one party is not fully informed.  But what we’re doing is helping people share honest opinions and feedback about the things they buy and use.  By lubricating the flow of information between real people, I believe social media will reduce the effectiveness of deceptive marketing.  Moreover, it will help companies connect to their customers and hone their goods to people’s real concerns and desires.  It will help hold companies accountable for their mistakes, and enable companies to better make things that make people happy.

Additionally, I will point out that my employer represents no small amount of “pure good” for the world, including organizations such as The Bill & Melinda Gates Foundation and Vittana, helping them tell their stories.

Taking a chance on a startup

Why did I choose this opportunity out of the sea of possibilities?  I evaluated the landscape as an investor would, since I am investing no small chunk of my life in this effort.  From my entrepreneurial training and experience, I know that smart investors care more about the people than the specific business plan.  The plan will almost certainly change, but the key management will not.  Having known one of the founders of Banyan quite well for a number of years, I am certain that many key elements for success are in place.  The corporate culture and governance will be solid.  I will be working in an environment where I am supported, and where I can learn and grow as a manager and a technologist.

Exactly what will I be doing or building?  I admit I’m not sure yet, but I have some very interesting ideas that I won’t be sharing here anytime soon.  I am sure that my work is very well positioned to be a part of a major shift in an entire industry — a rare opportunity.  Whether or not my work will play a key role in this shift is somewhat out of my hands — these things are always a roll of the dice.  But in another sense, it’s entirely within my control, and this is what I love about working in a small company.  There’s almost nothing but work between me and effective execution of our ideas.  Many people tend to exaggerate the importance of the idea itself, forgetting that it is incredibly important to execute well on whatever ideas you have.  I’ve heard people say that they had the idea for YouTube years before YouTube did.  How quickly we forget the dozens of other companies all working on the same problem in 2006, which almost all fell by the wayside because they didn’t execute as well as YouTube did.  Ideas matter for sure.  But hard work is critical.

If you’d like to jump on this raft and start paddling too, get in touch with me.  I need a few key rock-star developers who are’t scared of chaos and can think creatively about business problems.

In Walmart’s case, it’s generally suppliers who get squeezed.  Walmart demands that manufacturers of goods produce them at the lowest possible price so that Walmart can charge the lowest prices in their stores.  They really do try hard to pass the savings on to you.  Another case that is less well known is with so-called “interchange” fees for debit and credit cards, charged by the card networks like Visa and Mastercard.  Back in 2003, Walmart pushed hard on Visa and Mastercard to charge less for debit card transactions since they are both lower risk (because of pin-code use) and cheaper to process (verifying signatures is expensive).  The cynical will point out that with lower fees, Walmart just gets to keep more profit.  Which is true.  But they are genuinely motivated to lower prices for consumers, since that’s their main selling point.  So it’s a win-win – Wal-Mart’s motivations to lower costs are closely aligned with consumer’s desires to pay less.

Apple has similar desires for their network-connected gadgets like iPhones and iPads.  Apples wants people to be able to connect their devices to the network for as little as possible. Apple has clearly negotiated very hard with AT&T to demand low monthly rates on data plans for these devices.  Next month you’ll be able to buy an iPad with a 3G data plan for just $15 / month.  That is basically unheard of in the US.  For people on a limited budget, the iPad is the cheapest way to get online.  Compare this to other data plans available from major U.S. carriers:

The Apple / AT&T rates are the lowest in each of their categories, except Verizon’s smartphone data plan which ties the AT&T iPhone plan.  The iPad rates are extremely low compared to data plans for laptops, and also when when you consider that tethering plans or phone data plans require paying an extra $30/mo – $50/mo for a voice plan.  The unlimited iPad plan is literally half what it costs to get 3G on any other laptop, and it doesn’t come with the 5 GB limit that other plans do.  You might argue that the iPad can’t do as much as a full laptop, which is true.  So you might then argue that iPad won’t tax the network as much as a laptop, which I doubt considering the propensity to consume video on such a device.  So you can’t trade torrents on an iPad, which from an Intellectual Property perspective is just fine with me.

My guess (and this is pure speculation) is that Apple negotiated these rates by offering AT&T a share of the revenues generated through App Store purchases.

Again, the cynical will point out that Apple is just trying to grab the lion’s share of economic surplus for itself, which is true.  But nonetheless, this is a case where Apple’s desires and our desires as consumers line up well.  In a very real way, Apple is fighting on our behalf for lower prices from AT&T.

Apple products are expensive. Apple gets high margins on its hardware, allowing it to recoup large investments in NRE (non-recurring engineering) to design the hardware and its accompanying software.  This is a great place to be from a competitive standpoint, because as a company they don’t need to squabble over the cheapest parts to try to deliver the best prices to consumers.  So long as they can maintain a sufficiently large customer base to support the practice, it is an easy place to defend against competition from.  Certainly a lot easier than being Dell or HP, who struggle with operational efficiency to compete on price, and try to innovate within a very narrow window defined by their platform.

Apple’s success at selling high-end products has secondary benefits for the rest of the ecosystem.  Because the products are expensive, they tend to be purchased by people with more disposable income. So the segment of the computer market which buys Apple products self-selects to be very attractive demographic for many other reasons.  Advertisers love to get their products in front of people who are more-willing-than-most to buy something expensive / unnecessary / fun.

Similarly, app developers know that if they write an app for iPhone / iPad, the people who are able to buy it are much more likely to be willing to pay a couple bucks for something silly than, say, somebody who bought the cheapest smartphone they could afford because they felt they really need that functionality.  I had previously speculated that Apple’s platform play required a very large distribution base to attract developers, which is not quite correct.  The strategy is successful even with a relatively small market, provided that the market is segmented properly.  Which in this case it clearly is.

This kind of information and much more is available from a cool web site called Space Weather.  For example, did you know that just a week ago an asteroid got as close to the Earth as the moon is?  It was just a 27 m in size, so not earth destroying.  But these encounters are happening all the time.  Browsing around SpaceWeather a bit more and you’ll find great pictures of the sun like this one.  Or the current interplanetary magnetic field measured in nanoTeslas.

All in all, fun stuff.

I don’t want to give too much away, but from watching the trailer you can tell it’s gonna be good.  The story explores “emoticlips” which are a way to digital encode and transmit emotions, like a podcast.  Drama heats up when a viagra ad shows up, something about blackmail.  And my favorite line asks if you’ve tried Googling “omnipotent self-aware botnets.”

I just saw a fun trailer for the play on YouTube, shared here for your convenience…

You can get your tickets now from Brown Paper.

Scores of developers have had iPad’s for weeks now. They’ve had to sign non-disclosure agreements, and have the iPad locked in a separate room that random employees couldn’t access. And even that wasn’t enough. The iPads are literally chained to the desk with steel cable and a lock. Apple comes by the office with a suitcase, installs the iPad in a bolted case, chains it to the desk and locks it there. And they they do occasional surprise visits just to make sure it’s still there.

What’s more, Apple has told developers that they are monitoring the location of the device as well.

Apple.  Gotta love them.

Mike says he can type 50 wpm on it.  That’s really quite cool.

UniXKCD command line console

My favorite webcomic, Randall Monroe’s brilliant XKCD, is running a command-line version of itself today.  A few commands you might want to try include:

• find
• wget http://xkcd.com/
• Make me a sandwich
• go west

Google renames itself to Topeka

In honor of Topeka, Kansas renaming itself Google in a bid to get ultra-high-speed broadband installed, Google has renamed itself Topeka today.  Although Google is well known for April Fool’s jokes I believe this is the first time any have been on the homepage.

YouTube’s TEXTp mode

YouTube has the option to render most any of its videos in ASCII by adding the &textp=fool parameter onto the URL.  Looking at bandwidth graphs I can’t tell if they’re actually sending ASCII over the wire, or doing the conversion client-side.  Fun trick though.

Bing’s funny cows

Bing has one of their defining pastoral pictures, this time literally bucolic, but with fake cows.  They’ve supposedly been bread to only make non-dairy creamer.  Glad you’re trying, folks.

Great job everybody!  It’s awesome to see what you can do without leaving your browser.

The big problem I have with the current variations of the CC license is the “who knows where else” part.  Folks are free to use my content, so long as they attribute it to me, without ever letting me know about it.  And that removes a big part of the fun for me — I get a huge kick when somebody tells me they want to publish my photo.  As a matter of politeness, I’ve taken to notifying content creators when I use their CC license, so they can get the joy.  But the license variations can’t require that.

What I’d like to see is a “Notification Required” variation of the Creative Commons License.  It would go alongside the current variations:

• Attribution
• Share-alike
• Non-commercial
• No derivative
• Notification (not yet implemented)

It would be executed similarly to the Attribution requirement — somewhere the content creator needs to explain to re-users how to attribute, and in this case how to notify.  In fact, I’ve updated the CC notice by my photos requesting notification.  I’m asking people to jump through some small hoops to contact me as a form Turing Test.  But if you understand English, they’re really pretty easy.

I know others would like to make similar requests when their content is re-used, so I think this should become a part of the standard array of options for CC licenses.

The author of the book is Arthur Benjamin.  He gave a demonstration of his mad skillz at TED a while back, which I’m embedding here because it’s awesome.

Migrating this blog has been fun because it’s forced me to look over a lot of the old content I’ve written.  A couple years ago I found Benjamin’s Ted talk, which has inspired all this craziness.  I think it’s good to keep the brain fresh by taxing skills that one might not have used in a while.

• Keep all blog posts and comments
• Keep all posts at their original URLs
• Maintain all category pages at the same URLs

The first one’s easy.  Google released some migration tools which cover that quite well.  But, at least when importing from typepad / movable type, they don’t preserve permalink URL’s.  So anybody who followed a linked to a specific page on my site would get a 404 page.  Weak.

I spent a lot of time on this.  Basic problem is that Typepad doesn’t include URL information in their export file format. It would be very easy for them to do this, but then why would they want to make it easy for you to leave?  Actually the answer there is easy.  Because by trying to lock in users, they create angry vocal opponents of their service.  I’m not angry, but I would advise against anybody considering Typepad as a blog host, specifically because of their tendency to lock people in.

<rant> Don’t keep my data hostage.  It’s my content.  I created it.  You’re just delivering it.  Do not try to lock me into using you as a service provider.  You might get some more money out of me, but every dollar I give you after I want to leave will contribute to my dis-liking you.  As the internet matures and consumers become more sophisticated and better able to share their experiences with each other, they will increasingly choose the service providers who are open.  (Echoing Jonathan Rosenberg’s recent diatribe on openness.)  I really appreciate Google’s commitment to Data Liberation.  My current provider, Dreamhost, also does a splendid job of giving me control over my data.</rant>

It turns out that getting a full-fidelity export out of typepad is possible with some work.  I followed these instructions from FolioVision which provides a custom export template that does include URL’s.  If your blog has more than 100 posts, then you need to change the first line to

<MTEntries lastn="100">

<MTEntries lastn="100" offset="100">

<MTEntries lastn="100" offset="200">

etc. and merge all these files together into one big export file that has URL’s.  Then I tried to get blogger to honor the import file with permalinks but I couldn’t.  I do believe blogger is capable of doing this, but what ultimately turned me away from it was that it doesn’t seem to offer any way to honor links like www.embracingchaos.com/humor for category listings.  Which I like and get a lot of visitors on.  So I went with wordpress.

FolioVision helpfully posted a custom wordpress import plugin to match their typepad output template, which makes it all go.  Once that’s done, you have to move all the attachments hosted at typepad, and then there’s a bunch of wordpress configuration, and moving your analytics and favicons and finally switching DNS.

So here we are.  Please tell me if you notice anything amiss with the new site.

Broadly, the concept of information democracy is that an increasingly large number of people are able to influence how information is aggregated.  Wikipedia is a clear and simple example of allowing anybody to contribute to what used to be authored by a select few — “The Encyclopedia.”  Google’s Pagerank algorithm democratized web search.  Today’s most successful software is democratizing the feature set by allowing users to vote on how they want to use it.  The general principal is that large numbers of individuals can together make better decisions than any small group.  Applying this principal to culture, we can predict that a cultural democracy will produce “better culture” than what was available before.

Information technology makes it cheaper and easier to both create and to distribute culture.  With the right software, any laptop today has all the power of a professional music or video studio.  Sure the quality won’t be as good without professional inputs (microphones, cameras, etc) but the cheap stuff is good enough for a lot of things.  Obviously the internet makes distribution of this content trivially easy, which is disrupting traditional media businesses.  Easy creation and distribution of cultural content is an important part of creating a cultural democracy, but it is not the critical enabling step.

The key to democratizing culture is in the editorial process.  If everybody is contributing cultural content that is easily distributed, but there’s still a small group deciding which pieces everybody watches, we’re still in a cultural dictatorship.  Enabling the mass public to “vote” on content is the democratizing step.  That enables the collective intelligence of all media consumers to help choose what should become part of mass culture.  So instead of some programming executive trying to guess what will be popular, the question almost becomes moot — whatever is popular becomes popular culture.  Actually making this work is not at all straightforward.  I’ll save a full description of the necessary ingredients for another post, but we can look at a couple examples.

Youtube does this quite well.  It blurs the line between sharing a video clip with your friends and publishing it as a piece of mass culture.  Any video that isn’t marked private is submitted into a kind of massive popularity contest.  Videos that get millions of views are undeniably bits of popular culture.  For music, last.fm does a good job of being inclusive, but hasn’t quite taken off.  When I started building social features into Rhapsody I hoped they could democratize the music editorial process but that hasn’t happened yet.  Like many things in social media there’s a chicken and egg problem with scale which Youtube has clearly gotten past, but music is still struggling with.

Cultural Democracy is “retro”?!

This post is inspired by a recent story by Heather Chaplin that NPR aired describing participatory culture in video games.  The surprising part of the story for me was the assertion that this trend is not modern but in fact “retro.”  The story points out that before analog broadcast media, most culture was participatory — singing, dancing, crafts, etc.  Analog technology created the possibility of cultural hegemonies, and digital technology is breaking them down. A fine point, implying that the 20th century will likely be unique as the only period in human history when popular culture was dictated by an elite group of editors.  Thanks for the interesting tidbit.

]]> http://www.embracingchaos.com/2009/12/participatory-culture-and-the-democratization-of-information.html/feed 1 http://www.embracingchaos.com/2009/03/dinocams-the.html http://www.embracingchaos.com/2009/03/dinocams-the.html#comments Sun, 01 Mar 2009 19:01:00 +0000 leodirac http://wp.embracingchaos.com/2009/03/dinocams-the.html DSLR cameras make very little sense today.  Modern imaging technology is rapidly turning them into dinosaurs.  The forces keeping them alive are a combination of a physical legacy in hunks of glass, and aspirational marketing.  I’ll explain, but first, what’s a DSLR and why don’t they make sense?

Background on SLRs and DSLRs

(If you what “f-stop” means, feel free to skip ahead to the next section.)

SLR stands for Single Lens Reflex.  Practically speaking it refers to a camera where you can change the lens.  You look through the same lens that actually takes the picture, letting you put any lens from an ultra-wide angle fisheye to a telescope-length zoom lens.  You can also put filters on the front like star filters or color shifters or polarizers.  Imagine a classic 35mm camera — like what a P.I. would carry to snap pictures of your wife having an affair — that’s an SLR.

SLR’s require a mirror that physically moves to divert the light into one of two places — your eye, or the film / CCD. The mirror was important when the only technology for capturing images was chemical film.  But nowadays we have various electronic devices like CCDs that digitize an image.  DSLR cameras use a CCD to get many of the benefits of digital imaging, but still have the same physical form factor as an old chemical-film SLR.  They can use the old lenses, which is one of their big appeals.  But so many things about these cameras just don’t make sense.

The problems with DSLR cameras

First there’s the noise. The sound of the mirror slapping against its stops as it switches positions is very recognizable. We used to accept sounds like that as a necessary part of taking
pictures.  Today it just annoys me.  Especially when I’m at a small
event and some photographer is there making loud clicking noises all
the time while I’m trying to enjoy whatever it is they’re digitizing
with their dinocam.  In 99% of all use cases, it’s totally unnecessary.  CCDs can continuously capture images and display them on a screen, creating a digital light path that doesn’t require loud expensive mechanical assemblies.  These displays aren’t as good as what a human eye can pick out, so this doesn’t work all the time.  But if you don’t need interchangeable lenses, then the camera can have a second optical path just for the eye, which doesn’t need to be as good.

One argument against a separate optical viewfinder is that youc can’t put filters in front of the lens.  This is very true, but filters are also obsolete.  With few exceptions, everything that a physical filter does can be done later in photoshop with more control and accuracy.  Color tinting, sparkle, gradients, soft, mist, etc — these all used to be rendered in physical glass out of necessity.  Polarizing filters are probably the most important exception to this — since CCD’s don’t record a light’s polarization state, it can’t be adjusted later.  But for the most part, filters aren’t necessary anymore, meaning you don’t need the whole single-lens thing.

But what about interchangeable lenses?  Isn’t it useful to have the same camera body and be able to change lenses?  (I hear you cry.)  Yes, sorta.  There are definitely situations where one lens won’t be able to do everything you want.  But those situations are getting rarer and rarer.  And in the few exception cases, I’ll argue that interchangeable lenses aren’t the right solution.  The reason these cases are getting less and less common is that zoom lenses are getting better.  When SLR cameras first came on the scene zoom lenses basically didn’t exist because they sucked when they did.  You needed a different lens for each amount of magnification you wanted, so people had lots of lenses.  But with computers to help us design the lenses, and vastly improved manufacturing processes, zoom lenses are getting better all the time.  Nowadays a lens with a huge 10x zoom can even win accolades from camera snobs.  And lenses as versatile as 26x cover every situation most of us would ever want, and at a quality we’ll be thrilled with.  So for almost all situations, a single zoom lens is good enough today.

What about the situations where that’s not quite good enough?  Where you need that 14mm fisheye that captures people standing immediately to the left or right side of the lens?  Or that 8000mm super- long telephoto telescope?  It turns out in either of these challenging cases, getting the lens to fit the standard SLR form factor becomes the hardest part.

Why SLR’s cripple even the extreme lens cases

With ultra-wide fisheye lenses, the problem is the space reserved for that stupid mirror.  In this case, the focal length is very short, so as a
lens designer, you’d naturally want the focal plane to be very close to
the glass.  (Like about 14mm.)  But the place where the lens attaches to the camera body necessarily needs to be a certain distance away from the imaging plane.  That distance was determined by the size of the mirror, which was determined by the size of your chemical film — 35mm, which is more than you’d really want for a 14mm lens.  Even on today’s 2009 DSLR cameras, that distance is exactly the same as it was a generation ago in order to ensure backwards compatibility with old lenses.  The literal tons of carefully polished glass represent a very real barrier to improvement since people have invested lots of money in them.

So if you really want a camera that’s good at taking super-wide angle pictures, you don’t want your lens to have to be that far away from the imaging plane.  You’re better off with a specially built camera.  The lens will be simpler, cheaper and higher quality.  But super-wide starts to look funny, no matter what.  Funny meaning
distorted, because if your eye is more than a couple of inches away
from the reproduced super-wide image, then it won’t look right.  And it’s not super useful to capture 360 degrees in one shot — you can shoot a dozen pictures and stitch them together later in software, and it’ll look more natural.  This is all why people don’t pay a lot of attention to how super-wide lenses get anymore.

On the super-telephoto side of things, the SLR legacy is even worse.  To get a super-long telephoto lens you need lots of big glass.  This gets expensive quickly simply because it’s a large mass of carefully manufactured stuff.  The amount of glass you need for a lens is proportional to the cube of the length of your imaging plane, which for legacy chemical-film is 35mm. But CCD’s just don’t need to be that big.  On almost every DSLR they’re only about 20mm across, and on high-quality non-SLR cameras are as typically about 6mm across.  So that size legacy means you would need literally 200x  the almost 40x the amount of physical glass to make a good telephoto lens for an SLR vs a non-SLR camera.  This ridiculous discrepency is just going to get worse.

CCD’s are silicon devices, so they share manufacturing improves along with CPU’s and follows a Moore’s law-like improvement curve for performance.  A key way they improve is in pixel density, but also by simply getting smaller.  As they get smaller, high-quality zoom lenses get smaller and cheaper too.  But only if the lenses are specifically designed for the new smaller CCD’s.  With an SLR system they can’t be — the size must be fixed in order to maintain backwards compatibility.  So while sensor technology improves at Moore’s law speed, lenses for non-SLR cameras improve as well, but SLR lenses do not.  Expensive zoom lenses for modern cameras just don’t need to be that big or expensive — It’s like having to build a cell-phone big enough to hold floppy disks.

To illustrate this point, consider the popular Canon SX10IS camera which does not feature interchangeable lenses.  It features a zoom lens that goes from pretty wide (28mm equivalent) to really very far zoom (560mm equivalent).  Because its CCD is only 6mm across, it can do all this for under $400 and weigh in under a pound for the whole camera.  For comparison, a comparable SLR lens weighs in at over 11lbs and costs upwards of $7,000, just for the lens.  No doubt this lens can take better pictures than the tiny Canon, but a smaller lens built for a modern CCD could take pictures that are every bit as good for a fraction the price.

I would be remiss if I didn’t mention the noise floor on these sensors.  When the scene is dark, you need more light to get a good image.  A bigger hunk of glass captures more light.  This all makes intuitive sense and is mostly accurate.  CCD sensors can take more accurate pictures in low light when they are bigger.  But the limits here are electronic noise, which is also improving.  At some point we’ll hit some other barrier like the thermal noise in the sensor, although a piezo cooler could work around that.  Ultimately there’s the the quantization of photons, but if you’re taking pictures in a scene that dark, you probably can’t see what you’re pointing at anyway.  My point is that while there are advantages in low light for larger glass and sensors, technology is erroding away at those too.  We’re seeing ISO equivalents of 6400 as fairly common in cameras these days, with an economic competitive pressure to improve that.

In summary, the problems with the SLR format are that it ties its owner to a physical legacy that denies them the advantages of advancing technology.  There are cases where specialized lenses are still important.  But those cases are dwindling.  Personally, I’m going to be happier carrying around a full featured small camera that can transform itself into whatever I want without needing interchangable parts than a bag full of bits that were standardized before email.

Creative Commons licenses can either allow somebody to make commercial use of your material, or not, at your discretion, assuming you’re the one who created the license.  Independently, you can allow anybody to modify, adapt, or remix the content.  Or not.  Or you can allow modification so long as the modified content shares the same license, a so-called “Share Alike” license.  Here’s a nice page that shows you the options and allows you to pick a license appropriate for your material.

A CC attribution license is in many senses more realistic in the modern world than an all-rights-reserved license.  It is practically impossible to stop people from using or distributing your work.  The all-rights-reserved license is a threat to take legal action to prevent somebody from using your work.  But suing somebody is such a hassle that it almost never happens for personal content.  Asking somebody to put a link to your website is a pretty reasonable thing and easy to accomplish.   An all rights reserved copyright is for most individuals a bluff.

CC content is also easier to use.  Negotiating terms of licensing under a traditional copyright is daunting.  It necessarily requires a back and forth with the author and probably a whole lot.  The underlying mindset is that content costs money, so if you’re going to use my content, then you’re going to sell it and I deserve some of that money.  As the music industry is slowly, painfully learning, in modern times this model doesn’t work so well.  Access to information is generally free, and those who are making money here are doing so by providing value-added services on top of merely distributing the information.  (Think ads by Google, or concerts for music.)  With a CC license the terms of use of the license are right there.  No need to negotiate.  Just follow the attribution instructions and do what you will.  Instead of requiring negotiation and payment in the traditional economy, this is payment in the nascent reputation economy.

• http://movabletype2blogger.appspot.com/
• http://wordpress2blogger.appspot.com/
• http://livejournal2blogger.appspot.com/

Well I tried this with my blog, and the resulting file it spit out was almost empty.  I think my blog is just a bit too large, since when I ran it on my local machine it came out to 1.03 megs.  So if your blog is smaller than mine you can probably use the online tool.

After a couple of bX-xji785 errors, the file imported into blogger about as well as could be expected, which is to say okay but not great.  The blog is mostly there.  Feel free to take a peak at http://leo-embracingchaos.blogspot.com/ but please don’t make any permanent links to that URL as it’s really just for testing.  The posts and drafts all made it with the right dates and times, along with the comments and tags.  But as previously noted, the TypePad export format does not include URLs.  So if I were to actually use this conversion, all the inbound links to pages other than the homepage of my blog would break, which is totally unacceptable for me.

I started a thread on the discussion group if you’d like to follow along.

I worked hard to make this feature possible when I was working at Real.  The fact that I couldn’t get it re-launched was a big motivator for me to move on to greener pastures.  I saw making Rhapsody social as an important evolution of the music catalog’s organizational schema.  It’s also an attempt to bring the product into what Tim O’Reilly would call Web 2.0.  Tim’s canonical essay is long-winded, but I really liked how he summarized it in a recent interview on NPR — basically the product gets better as people use it.  The millions of people who use Rhapsody are an asset that has been almost completely unused, except to take their money.  I saw it as a way to take on one of the product’s biggest shortcomings.

Rhapsody has tons of music.  TONS.  Rhapsody almost certainly has something you want to listen to right now, regardless of who you are or what your current mood or situation is.  It’s a strong statement, but there really is that much music.  The problem is figuring out what you want to listen to.  Rhapsody has a great categorical index of music, so if you know you want to listen to D&B or Emo or Vocal Jazz, no problem.  Or if you know specifically the name of something you want to listen, just search for it.  Other than that, you can take the homepage recommendations, browse the catalog manually, or sift through Playlist Central, a dumping ground for unvetted playlists that is a case study in how not to use user-generated-content (UGC) on a website.

Picking good music is difficult.  This is what DJ’s get paid for.  I originally wanted this feature to be called "DJ Pages."  The idea was to give a voice to the small fraction of Rhapsody users who are fanatical about the product.  People who are serious music buffs love Rhapsody, and if given a voice would and still might add tremendous value to the music catalog.  Right now the editorial voice in Rhapsody is controlled by a politburo of paid editors.  They’re really good, but they’re just a handful of hands.  DJ Pages would democratize the music editorial process so anybody with an opinion can contribute.  The social graph becomes the voting process to select who’s worth paying attention to, just like with pagerank.  What Tim calls Web 2.0, I like to refer to the democratization of information.  Partly because it’s fun to call people Communists when they cling to control of information, but mostly because the analogy is apt and helpful.

The Rhapsody team has made an important step in this direction of openness.  I hope they keep running with it.  If you want to see what’s been playing on my Sonos at home, check out my profile page.  But most importantly, I’d like to express my CONGRATULATIONS to everybody who made this possible again and the first time!!!!11!!1

• Better WYSIWYG editing in Blogger.  (I can’t change any font characteristics in Typepad without going into raw HTML. Ugh.)
• Typepad’s lack of support for Chrome, Google’s awesome new browser
• The awkward and limiting way Typepad assigns human-readable URLs to posts
• Blogger is free, while Typepad costs >$100 per year
• Cool new features being added to Blogger like Followers
• Adding new features in Typepad is a p.i.t.a. (i.e. I’m too old to be messing with perl-based templating languages.  Did more than my share of that in 1996.)

Typepad has long supported an export feature, which dumps out all the posts, drafts, comments, etc into a text file.  And now blogger has an import feature, which takes an XML file that looks kinda like an Atom feed.  I looked around for a tool to convert between the two and found only others asking the same question. 

So I thought here’s a chance for me to contribute something to
society and help myself at the same time.  Converting one text file to
another shouldn’t be that hard.  So I rolled up my sleeves and started
playing with it.  I realized one problem fairly quickly — the typepad export format
doesn’t include the URL for each post.  I really don’t want to break
all the previous inbound links because
that’s how people get to my content.  But I’m going to need to crawl
the old blog to get those old links.

Since the blogger format looks like a regular Atom feed, I thought I’d try to just grab the Atom feed off the blog and import it into Blogger.  Well to that, blogger said:

Blogger does not currently support import files generated by TypePad.

Fine.  But what are the differences?  The blogger format includes a bunch of fake "entry" elements which are really configuration — like a giant "layout" node and a bunch of other settings.  So I tried grafting the two together — just taking the legitimate "entry" nodes from the typepad feed and putting them into the blogger export that includes all the layout and settings.  By converting everything to look like the blogger format, it imports, but that also loses information.  So I set about looking for the minimal set of changes that will work.

One that definitely breaks it is a side-efect of being hooked-up to feedburner.  There’s extra stuff in there like the feedburner:origLink tags:

    <feedburner:origLink>http://www.embracingchaos.com/2008/12/repairing-a-deg.html</feedburner:origLink>

These break the import, which shouldn’t be much of a surprise since the feedburner: namespace isn’t defined in the Blogger export.

I gave up looking for today because blogger was having too many server errors on import.  They’re intermittent, so I was happy to refresh through them for a while.  But at some point 90% of my imports gave me funny error codes like bX-gxxw6w so I’m giving up for now. 

I wrote a stub of a python program to do the merge.  I called it typepad2blogger.py and happily donate it to the public domain.  It’s definitely not done and is useful only insofar as it shows one way to approach the problem and has enough stuff in place that somebody else should be able to get a running start.  I’ll hopefully continue work on this at some point.  But even a couple hours of work on it has triggered my RSI so I’ll have to take a break.  If you want to pick this up, let me know and we can coordinate efforts.

To convince myself that the repair was successful, I unplugged one of the previously functional drives, and saw that all my files were still there when the array was running just on the new drive and the other previous drive. I recommend this to anybody who thinks they’re running a RAID system — until you’ve seen the RAID array work with a drive removed, how can you be sure it’s really working? If your system is set up better than mine is, you’ll get some kind of warning message too.

My advice for people seeking reliable storage: go with a hosted solution.  Understanding the arcane nuances of these software systems is an extremely specific skill that doesn’t translate well to many real-life necessities.  If you’re smart, you can figure it out, but it doesn’t teach you much of anything except how to do exactly that.  Each person who understands this stuff should be keeping petabytes of data happy, rather than one couple’s pictures and music collections.  I hear Microsoft’s "home server" actually makes this pretty easy, but I can’t recommend anybody willingly lock themselves into Microsoft’s business model.

Background

So I bought a new drive, following my own advice about picking drives from different manufacturers when building a raid array, and plugged it in to the mobo and booted the machine.  After futzing with /etc/fstab to get it to find the boot disk and load up, I logged into evms and got these messages:

MDRaid5RegMgr: RAID5 array md/md1 is miissing the member  with RAID index 0.  The array is running in degrade mode.

and

MDRaid5RegMgr: Region md/md1 is currently in degraded mode.  To bring it back to normal state, add 1 new spare device to replace the faulty or missing device.

Conceptually easy.  I’ve got a new 500 GB drive in the system.  Linux sees it.  It didn’t take me too long to figure out it’s called /dev/sda, while the previous 2 disks in the array are sdb and sdc, with a small boot drive at sdd.  Now the fun part is figuring out EVMS terminology enough to tell it to use the new disk.

The hierarchy of the array in EVMS land seems to be as follows:

• Logical Volume teraraid (contains)
• Region md/md1 (which contains)
• Segments sdb1 and sdc1 (which are built on)
• Logical disks sdb, sdc.

What I tried, and what seems to have worked

I see that logical disk sda has no segments.  So I try Action -> Create -> Segment.  It only gives me one choice for "Segment Manager" which is "GPT Segment Manager."  But when I choose it, it doesn’t let me make a segment on sda.  Only the tiny free space on sdb and sdc.  So sda needs something else done to it before we can use it.  What?

sda also shows up in the list of Logical Volumes, next to Teraraid and the formatted boot partition.  Hmmm.

Well I tried converting it to an EVMS Volume.  It complained that sda does not have a File System Interface Module (FSIM) associated with it, but it made the new logical volume anyway.  This wasn’t getting me anywhere.  So I erased it.

Next I tried "Add" -> "Segment Manager to Storage Object".  I noticed that all of the Disk Segments associated with the array were listed as using "Plug-in" "GptSegM" and this gave me the choice of adding Gpt Segment Manager to sda.  W00t.  I said "No" to make this a system disk.  This seems to be working.  Now I see a bunch of Disk Segments starting with sda, including a big one (465 GB) labelled sda_freespace1. 

Now when I tried to Create -> Segment, it let me use GPT Segment Manager on sda_freespace1 and allocate a 450 GB disk segment to match the others.  (I left 15 GB off each disk with the idea I could put a boot segment in that space, but I’ve never gotten around to it.)

Now in "Available Objects" there is sda1 with 450.0 ready for me.  Alrighty we’re getting there.

Now I look at "Storage Regions" and in the context menu for md/md1 I see an option that says "Add spare to fix degraded array…"  I didn’t see it there before — it might have not shown up when there weren’t any spares, or maybe I was just being thick.  In any case, selecting it now gives me a menu with one choice — sda1. 

Now in details of md/md1 it shows:

 Na┌──────────────────── Detailed Information - md/md1 ─────────────────────┐ ──│                                                                        │── lv│     Name                    Value                                      │ lv│ ────────────────────────────────────────────────────────────────────── │ lv│     Major Number            9                                          │ md│     Minor Number            1                                          │   │     Name                    md/md1                                     │   │     State                   Discovered, Degraded, Active               │   │     Personality             RAID5                                      │   │ +   Working SuperBlock                                                 │   │     Number of disks         3                                          │   │ +   Disk 1                  sdb1                                       │   │ +   Disk 2                  sdc1                                       │   │     Number of stale disks   1                                          │   │ +   Stale disk 0            sda1                                       │   │                                                                        │   │                                                                        │   │                                                                        │   │                                                                        │   │                                                                        │   │    Use spacebar on fields marked with "+" to view more information     │   │                                                                        │   │ [Help]                                                          [OK]   │   │                                                                        │   └────────────────────────────────────────────────────────────────────────┘

That last line about the Stale disk is new.

Actions -> Save commits these changes to disk.  Now looking at Detailed information for md/md1 shows

 Na┌──────────────────── Detailed Information - md/md1 ─────────────────────┐ ──│                                                                        │── lv│     Name                    Value                                      │ lv│ ────────────────────────────────────────────────────────────────────── │ lv│     Major Number            9                                          │ md│     Minor Number            1                                          │   │     Name                    md/md1                                     │   │     State                   Discovered, Degraded, Active, Syncing =  0 │   │     Personality             RAID5                                      │   │ +   Working SuperBlock                                                 │   │     Number of disks         3                                          │   │ +   Disk 1                  sdb1                                       │   │ +   Disk 2                  sdc1                                       │   │     Number of stale disks   1                                          │   │ +   Stale disk 0            sda1                                       │   │                                                                        │   │                                                                        │   │                                                                        │   │                                                                        │   │                                                                        │   │    Use spacebar on fields marked with "+" to view more information     │   │                                                                        │   │ [Help]                                                          [OK]   │   │                                                                        │   └────────────────────────────────────────────────────────────────────────┘

Emotionally I feel like I should be done now.  But I don’t hear the thrashing noise of a half-terabyte of of checksums being unwound and copied onto a fresh disk.  And it says "Syncing = 0".  Hmmm.

I quit evmsn and reload it to see two new messages.  One familiar:

MDRaid5RegMgr: Region md/md1 is currently in degraded mode.  To bring it
back to normal state, add 1 new spare device to replace the faulty or missing device.

And one novel:

MDRaid5RegMgr: RAID5 array md/md1 is missing the member  with RAID index 0.  The array is running in degrade mode.  The MD recovery process is running, please wait…

But this novel message saying it’s recovering is "Number 0" implying that it came before the other message (Number 1) which tells me I need to take action for it to fix itself.  And the drives are not thrashing.  Again I look at the details for md/md1 and now I see:

 Na┌──────────────────── Detailed Information - md/md1 ─────────────────────┐ ──│                                                                        │── lv│     Name                 Value                                         │ lv│ ────────────────────────────────────────────────────────────────────── │ lv│     Major Number         9                                             │ md│     Minor Number         1                                             │   │     Name                 md/md1                                        │   │     State                Discovered, Degraded, Active, Syncing =  0.3% │   │     Personality          RAID5                                         │   │ +   Working SuperBlock                                                 │   │     Number of disks      3                                             │   │ +   Disk 1               sdb1                                          │   │ +   Disk 2               sdc1                                          │   │ +   Disk 3               sda1                                          │   │                                                                        │   │                                                                        │   │                                                                        │   │                                                                        │   │                                                                        │   │                                                                        │   │    Use spacebar on fields marked with "+" to view more information     │   │                                                                        │   │ [Help]                                                          [OK]   │   │                                                                        │   └────────────────────────────────────────────────────────────────────────┘

Which really seems to say its doing its thing.  Maybe I don’t hear the disks because it’s formating the disk first, which is a linear process.  Or maybe the whole copy process is very linear and I won’t hear it thrashing.  Its progress implies it’s going to take a couple/few days to finish, which is what I’d expect.  So maybe it’s working. I’ll let it run for a while and see what happens to the array if I try to unplug one of the previously working drives.

Pretty cool that I didn’t even need to unmount the array to do this.

Now if I could just figure out why my laser printer periodically decides it needs to print it internal test page, I’d be even happier.

Dan, being a "good guy", tried to keep the details of this hack quiet for long enough for network operators to patch their systems and close the loophole.  He wanted everybody running a DNS server to do this before the "bad guys" figured out what the bug is and started to take advantage of it.  He was hoping for 30 days of time to prepare, but somebody spilled the beans after 13 days, and now the hackers are off and running.

It’s a jungle out there

You might be asking, So what? What are the dangers of being directed to the wrong website?  Of course, you could read incorrect news and that’s not great.  More likely you’re going to have your password stolen for whatever site you log into.

The obvious attacks are to sites like paypal or banks, but they’re actually safe from such attacks if you use your browser properly.  Any financial site will use a secure connection.  You can tell because of the https:// at the beginning of their address.  These sites use a digital certificate that your browser checks to verify their authenticity.  All this happens independently of the DNS system.

But you can still connect to a hacked site with https.  Your browser will probably warn you saying something about a certificate not matching.  More often than not these errors occur because of a lazy sysadmin or something.  But right now, I strongly advise you to take all HTTPS warnings seriously.

Protect yourself

If you want to be sure you’re safe, manually connect your machine to OpenDNS, as Dan recommends.  We know they’re patched and can take the traffic.  I’ll give you the steps to do this on Windows:

1. Start menu
2. Control Panel
3. Network connections  (might have to switch to "classic view")
4. Select the one you’re actually using.  It’s likely called "local area connection".
5. Click Properties on the status dialog
6. Scroll down in the list of checkboxes, and select "Internet Protocl (TCP/IP)" so that it’s highlighted.  (Leave it checked!)
7. Click Properties
8. In the first General tab, change the second radio-button from "Obtain DNS server address automatically" to "Use the following DNS server addresses:"
9. For Preferred DNS server, enter: 208.67.222.222
10. For Alternate DNS server, enter: 208.67.220.220

That’s mostly it, but to be safe, you should reboot, restart your browsers, and/or:

11. (Windows key+R).  In the dialog type  "ipconfig /flushdns" (without the quotes) and hit okay.

PubSub 101: Push vs Pull

PubSub is short for "publish subscribe" which is a common design pattern describing a way to distribute information to interested parties.  The publisher tells a server about new information, and the server fans the information out to everybody who has registered interest in that topic or channel.  Data consumers find out about the new information very quickly, with relatively little load on the whole system, since the pubsub mechanism provides a means to "push" data to them. 

By contrast, almost all of the web today follows uses a "pull model" where a data consumer only finds out about new information when it gets around to checking if there is something new.  This data distribution model is significantly simpler because the server only needs to keep track of the content, not who is interested in knowing about it.  Modern networks are optimized for this kind of query-based traffic where data consumers (clients, web browsers) initiate connections to servers, such that it’s often impossible for servers to initiate conncetions to clients because of firewalls or NAT.

The downside of the pull model is that the only way a data consumer can find out if thanything is new on the server is to "check back frequently" or "poll" the server for changes.  If you want to know within 15 minutes if anything new has been posted, you have to ask the server at least every 15 minutes "anything new?"  No.  "How about now?"  No.  "Got anything yet?"  No.  Mulitply this by potentially millions of interested data consumers and you end up spending a lot of network bandwidth and server resources doing very little.  Even worse, the problem scales horribly.  If clients want to know about changes within 5 minutes instead of 15, that puts 3 times the load on the server.  Want to know within a few seconds?  Forget it — the servers would crash.  There’s an intrinsic delay in distributing information in this model, and reducing that delay is very expensive.

XMPP as an alternative to polling

XMPP is the protocol used for Instant Messaging by Google Talk and Jabber and a large number of small servers.  In order to deliver instant messages, XMPP systems maintain persistent connections between all machines that allow packets of data to be pushed with very low latency — IM messages are typically delivered within a second of sending them.  So it’s natural to want to use this infrastructure to deliver other data more efficiently than through polling HTTP.

The XMPP PubSub spec known as XEP-0060 describes how to do exactly this at the protocol level.  But for a variety of reasons, this standard has not gained wide adoption.  IMHO the biggest reason is that there isn’t a very pressing need.  The current system is horribly inefficient, but it works.  Moreover, it puts the burden of inefficiency squarely in the hands of the information publishers.  Popular publishers are just expected to shell up for necessary hardware to meet the demands of their readers, and with advertising they can typically recoup the necessary investment.

Another way to state that is that pubsub hasn’t found its niche yet.  IMHO this is partly because the mechanism is so useful it can be applied to almost anything.  Not just breaking news, but everything from e-mail mailing lists to doorbell chimes get used as examples of how XMPP pubsub technology could be applied.  Not wanting to exclude any of these potentially interesting uses, the protocol remains very generic.

Micro-blogging, Atom and Yesterday’s Realization

One place where the current HTTP model breaks down is micro-blogging, which is the generic term for services like Twitter or Facebook’s udpates.  Here, the payload of actual content is very small, so the overhead of checking far outweighs the "useful data" which is delivered.  Also, because the information is social (i.e. "Heading to Broadway for a bite — wanna come?") consumers demand it be delivered quickly.  Nonetheless, current micro-blogging services still rely on polling clients, and their servers suffer as a result.

Yesterday, a group of us including Blaine Cook, Anders Conbere, Evan Prodromou, and XEP-0060 co-author Ralph Meijer were discussing how to apply XMPP PubSub to micro-blogging.  This was likely obvious to many there already, but during the discussion I had a realization.  We aren’t solving this problem from whole cloth.  RSS and Atom feeds already describe all the information we need.  We just need to find a way to substitute XMPP for the assumed transport HTTP.

So we discussed mechanisms for mapping an Atom URL to an XMPP PubSub Node.  (We pretty much ignored RSS because RSS isn’t as cool for reasons I really don’t understand.)  We talked about putting a link-rel tag in the feed to point to the XMPP PubSub node.  This would look something like 

   <link rel="alternate" type="xmpp/pubsub" href="xmpp:twitter.com?;node=users/leopd" />


Even better, the URL for these nodes should be guessable from the URL for the HTTP feed.   The above node would be the normal place to look for a the pubsub version of http://twitter.com/leopd.  Even though it’s not as generic and robust to have a standard mapping like this, I think it’s an important way to speed adoption of a new standard.  The code to do a bit of string manipulation is vastly easier than fetching the URL and looking for a link-rel tag.  And developers are intrinsically lazy (for good reasons!) so making things easier for them means they’ll succeed a lot more.

Ever pragmatic, Blaine pointed out that we should use HTTP for things it is good at, and not re-invent them in XMPP.  I wholeheartedly agree.  Re-transmission is a key example.  What happens if a client is offline when a new post happens, and so never hears about it?  Answer: The clients should fetch the historic archive of the feed over HTTP.  These feeds exist today — no need to improve on them.  If all the posts have sequence numbers on them, then it’s easy to figure out if you’ve missed one.  So all the posts from a user should have sequence numbers.  I don’t think this is standard in Atom feeds today.

The story unfolds…

There’s a lot more to be worked out and standardized here.  And clearly many more people need to voice their opinions before we can reach consensus.  Sadly I can’t be down in Portland today to continue the discussion, so this post will have to take my place as I return to my regular daily commitments.  If you’d like to stay tuned as the story unfolds, you’ll have to poll this site, as I can’t yet give you a PubSub node to subscribe to for updates.  If I could it would probably be something like xmpp:embracingchaos.com?;node=xmpp — try it.  By the time you read this, it might be working!

Last week Seattle had some crazy electrical storms.  In recent years’ storms, my block has done better than most with respect to power failures making me think we’re either lucky or in a particularly robust section of the grid.  So I was a little surprised to find my whole house offline on Wednesday morning.  After a bit of debugging I figured out that the small UPS that runs all my networking gear got toasted, and for some reason the file server was down.

I left it alone for several days, and when I got around to turning it back on, I was happy that the whole stack through the samba server came up by itself.  (It doesn’t always!)  But when I started looking around I quickly realized things were amiss.  The media/video directory normally has 4 subdirectories: movies, episodic TV, imake and other.  But today it listed:

leo@elephant:/raid/shares/media/video$ lsdpisndic TV  hmakd  movies  nther

WTF!?  A few bits had been scrambled in the directory names.  This sounds really bad.  Moreover, even though the first couple levels of the directory hierarchy were there, but no files were to be found.  Definitely a problem.

Step 1: As soon as you suspect your RAID array has a problem, stop writing to it until you know what’s going on.  Writing changes can make things worse.  Stop the bleeding.   

I unmounted the drive from my mac, not trusting Finder or Spotlight to sprinkle damaging meta-files over the array.  Once I remembered how to ssh into the box, I stopped the samba daemon,

leo@elephant:/$ sudo /etc/init.d/samba stop

unmounted the filesystem

leo@elephant:/$ sudo umount /raid

and changed fstab so it would be read-only when it comes back, and that it wouldn’t come back without me asking.

leo@elephant:/$ sudo vi /etc/fstab

changing

/dev/evms/teraraid500 /raid ext3 defaults  0 0

to

/dev/evms/teraraid500 /raid ext3 ro,noauto  0 0

I tried poking around in EVMS by running

leo@elephant:/$ evmsn

But it hung during initialization with blue dialog saying "Discovering segments…"  I’m thinking EVMS can’t help me.  After a bit of googling I thought I should try e2fsck or some such.  First, I tried to mount it again read-only and see what’s there.

mount: wrong fs type, bad option, bad superblock on /dev/evms/teraraid500,       missing codepage or other error       In some cases useful info is found in syslog - try       dmesg | tail  or so

Bad superblock.  Uh oh.  Well this guy managed to recover a drive with a bad superblock.  Lots of things were pushing me in this direction — fix the filesystem.  But I realized that was a mistake.

Step 2: Do not make changes at the filesystem level until you’re confident that the RAID array is working properly.  You set up RAID for a reason.  You’ve still got a chance to recover everything, but if you start
making changes to it in a broken state, you’re almost certainly going
to make things worse.

Me to self: Think about it.  EVMS is confused.  Linux is confused.  Ext2 and ext3 are messed up complaining about bad superblocks.  The problem was caused by lightning.  When the drive was mounted there were wierd bit-level corruptions in the data that were still there.  Maybe one of the drives in the array got data scrambled, but didn’t get totally fragged so it went offline.  RAID 5 is designed to survive total loss of a single drive.  But if a drive gets corrupted, who knows what will happen.  So I came up with this plan:

Step 3: Try physically disconnecting the drives in your array, one at a time.  If only one of them is scrambled, disconnecting it should restore all the data in the array.

Having followed my own advice, it’s easy for me to tell the drives in my array apart since each drive in the RAID array is from a different manufacturer (which makes array failure due to manufacturing defects far less likely). 

This plan actually worked perfectly!  Removing a drive caused a bit of a hassle in getting the machine back up, because when I booted it couldn’t find the /boot partition complaining

 * Starting Enterprise Volume Management System...[42949392.340000] raid5: raid level 5 set md1 active with 2 out of 3 devices, algorithm 0

 * Checking all filesystems...fsck.ext3: No such file or directory while trying to open /dev/sdd5/dev/sdd5:The superblock could not be read or does not describe a correct ext2 filesystem.  If the device is valid and it really contains an ext2 filesystem (and not swap or ufs or something else), then the superblock is corrupt, and you might try running e2fsck with an alternate superblock:    e2fsck -b 8193 <device>

Notice the complaint about the superblock again — don’t trust it, and don’t do what it says!  What really happened was that the boot drive letter had been changed from /dev/sdd to /dev/sdc, so I had to change /etc/fstab to mount /boot from  /dev/sdc5 instead of /dev/sdd5.  In my system, I boot off a non-RAID disk attached to the mobo, which for some annoying reason gets the last drive letter after all the drives no the SATA card.

But once I got past this, it quickly turned out that the Samsung drive was the culprit.  With it removed, the software RAID kicked in and plugged all the whole.  Everything the array looked
completely normal again.  All the directories.  All the files.  Hooray!

He does this trick where he multiplies 5 digit numbers together in his head.  He was a nationally ranked backgammon player in his spare time.  #40 or so in the US at the time IIRC.  It turns out that serious backgammon is all about estimating your fractional odds of winning at any given point and raising the bet when you think you’re ahead.  So for him it was a “live probability lab.”

I was reminded of this when a friend was watching a video and sure enough it was one of my old math profs.  So here’s the video of him showing off at Ted.  You go!  Thanks for the great math.

Watch Professor Benjamin’s Mathemagics show at Ted

1. Buy the CD and rip it
2. Illegally download it through a peer-to-peer network or sketchy Russian service
3. Buy the DRM’d track legally

The first option sucked because it either involved driving to a brick and mortar store or waiting for somebody else to drive the CD to your house.  There’s no instant gratification.  Then there’s the hassle of converting the CD to electronic format.

The biggest problems with the second option are things like not knowing how to do it.  There’s also some risk of viruses, etc by trawling shady parts of the net as you try to figure it out.  And there’s a minimal risk of the RIAA suing you.  But once you overcome these fears and startup costs, you end up with exactly the product you want.

The third option kinda sucked because modern DRM systems don’t work very well.  I think there’s nothing intrinsically wrong with DRM.  Enforcing copyrights through technical means is a good part of a multi-layer security and incentive system to promote the creation of valuable intellectual property.  But modern DRM systems are incompatible with each other across brands and devices and often they just plain break.  So as a music consumer, if you do the morally correct thing and pay for your music, you end up with an inferior product.

Now with DRM-free track sales, consumers can do the right thing and get the best possible product.  How this fact has been lost on the RIAA for so long amazes me.

This is just one example of why the digital music industry is a horrible one to be in right now.  I wrote a thorough analysis of the industry for my strategy class at school will likely post more of it as time goes on, but I wanted to start here.

I spent my first week in Mountain View at the Googleplex.  My entering class of "Nooglers" were subjected to inane videos and boring HR discussions.  But a couple hours into it we powered up our laptops and within 15 minutes I’d found the internal wiki and started reading project plans for every internal initiative I found interesting and a bunch that I didn’t.  I devoured the information and didn’t unplug myself until wee hours of the morning.

I’ve been spending a lot of time over the last few years analyzing the industry and thinking about what opportunities exist for creating value by solving people’s problems on the net.  Many of those I’ve captured here.  Now I look at the world differently in terms of what problems are still left to be solved because I can see that Google is in the process of solving many of the problems I’d identified.  It’s a little difficult for me to remember what I thought before knowing Google’s plans.  A myriad of half-written blog posts help remind me.  I had been planning on finishing many of them but now I don’t feel so comfortable doing so.  For example, writing about security holes inside Gmail is fun target practice from the outside, but questionably ethical from the inside even though I’d identified it before joining.  The same applies to unexploited business opportunities.

I’ve been asked a lot about the best and strangest things encountered during my week in Mountain View.  The best thing was hands-down the food.  It’s amazing.  Almost every building has their own restaurant with a theme.  My building had a tapas-inspired restaurant featuring many small plates and often fabulous seafood.  The best hamachi sushi I’ve ever had was served there on a real shiso leaf with some light sauce I can only describe through the ecstasy I felt from it.  They served black cod, which I love love love.  (I’ve got a great recipe I need to post to addgarlic.)  Pumpkin bread pudding.  Fresh figs everywhere.  Chilled beet soup.  Even simple things like a ham and cheese sandwiches on fresh bread with arugula were fabulous.  Other cafes have themes like organic hippy foods, dishes prepared with a maximum of 5 ingredients, or everything grown within 150 miles.  It’s all amazing.  As a result I found myself drawn to campus in a predictably Pavlovian manner.

One thing I’ve said for a while that this trip reinforced was the idea that you can’t pay your corporate cafeteria’s chef too much money.  You can get a chef for $50k/yr or $150k/yr.  That extra $100k/yr will do so much more for employee satisfaction than pretty much any other way to spend the money.  Sure you’ll end up spending some more on ingredients or subsidies.  (Or else the chef will leave.)  But it’s worth it.  A couple years ago Real hired a new Chef, Ariel IIRC for their cafeteria and the food got so much better I started bragging to my friends about it.  A little while later a number of things happened at about the same time — Real’s stock dropped, Ariel moved on and life at Real wasn’t as much fun any more.  I won’t try to extract the causality relationships between those events here.

The oddest thing I saw was definitely the automatic toilets.  They’ve got butt-warmers, front and back washing sprays, dryers and more things that I never figured out.  I wonder if they weigh you and keep a high-score list for largest excretion.

Now I’ve got the fire-house turned on full bore and am trying to add value for my team from a position of relative ignorance and keep up with everything going on around me while finishing up a full load of business classes.  But I wanted to take a few minutes to share what’s been going on with you my dear readers.

But think about the longevity of these devices.  Computers always slow down.  In a few years, any laptop is going to feel really slow, no matter how fast it feels today.  But if it’s a light, small laptop, then you’ll have something which is slow, but at least nice and portable.  Some of my friends’ house has this ancient Pentium II Viao laptop kicking around the living room — it barely runs a browser.  But it’s so small and portable that it’s still a reasonable computing device today.  If your laptop is heavy to start with, then in a few years when it slows down you’re stuck with a heavy, slow laptop, which nobody nobody wants.

Up until now if you wanted to listen to an artist in Rhapsody that you hadn’t previously bookmarked, you would need to guess what top-level genre they were categorized under and then scroll through an enormous list to try to find the artist.  How many times have I scratched my head asking questions like "Is Pink Floyd Rock/Pop or Alternative/Punk?"  Much easier was to find a web browser, pull up http://www.rhapsody.com/pinkfloyd and bookmark music into your library.  That human-writable URL scheme is still one of my favorite accomplishments in the last several years.

I started beta testing this release last week.  As always, the update was fast, easy and works flawlessly.  My biggest complaint is that the search is not interactive.  Considering how fast results typically come back, I would much prefer to have a type-ahead style search where results start to appear as you type.  This would be especially useful considering the somewhat painful scroll-wheel-alphabet typing interface they provide.

Sonos is a great company that makes fabulous products.  They continue to advance the state-of-the-art in digital music systems.  By adding Napster support they have taken another step to commoditize Rhapsody’s music subscription product.  They’ve also released a new product called a ZoneBridge which acts as a WiFi range extender which would address one of my biggest complaints about the system.

The problem that’s bothering me today is that I have no idea what the new site is going to do with the password I give it.  Few things irk me more than seeing one of my good passwords sent to me in plain-text, say in a password reminder mail.  Why are you storing my password in plaintext!?  Some of my most valuable passwords I’ve never even seen written out, and the first time I do see them say in a username field it’s a little surprising and I feel like they’re a bit compromised.

I’d like to see sites post their password handling policies.  I’m going to start doing this on my own sites.  Here are some examples of levels of security to consider:

• Your password is stored in plaintext
  • Forgotten passwords will be emailed to you in plaintext
• Your password is stored using a one-way cryptographic hash form
  • Forgotten passwords cannot be recovered.  If you forget your
    password you will have to create a new password after we verify your
    identity.
  • using MD5 (not considered very strong cryptographically anymore)
  • using SHA1 (better)
• Your password is stored using a strong hash with a salt
  • Without a salt, somebody looking through the password database could see that two users have the same password since they hash to the same value.  Obtaining one password thus grants access to the other account.
• Your password is never transmitted or stored in the clear. 
  • All auth events happen over SSL
  • Your password will not appear in any server logs.  (Rails devs take note.)
• Your password is never stored in any form in a browser cookie.
  • Some sites might see it fitting to store a hashed password in a cookie for re-auth.  Ugh.

Responsible sites will follow the last three rules.  We could come up with a simplified grading system that explains in simple terms whether you can use a valuable password here or not.  We know we can trust the smart folks to do that.  But with so many features being launched as stand-alone sites, understanding these policies is important.

I haven’t seen anything work differently yet.  But that’s really the way it should be.  The Google Reader implementation of gears where you need to explicitly tell it when you’re online vs offline isn’t an ideal experience.  And the Gears API doesn’t require apps to be written that way either — it’s just easier.

I tried unplugging from the net and nothing interesting happened.  I couldn’t update my calendar.  I couldn’t load my calendar — I was hoping maybe they’re just caching the javascript in gears which would be a nice step.  But it doesn’t appear to actually be doing much yet.

It’s just a matter of time.  I predicted first half of 2008 and I’m sticking to that.

Original Web 1.0

Universal access to massive volumes of data.  Being able to search
through masses of data and find what you want.  Connecting people to
huge databases really well.  Key examples:

• Online telephone books
• Web search
• Huge e-commerce sites

But in all of these applications, the data set is static.  User activity will not change the data for anybody else.

Web 2.0.1: Democratizing use of the data

The users of these data make the data better.  They can collectively
organize the data.  (i.e. tags)  They can help filter good data from
junk.  (i.e. voting)  Or they can help you find the data that are most
interesting to you.  (i.e. collaborative filtering).  In other words,
you can interact with the data.

Any "Web 2.0 company" worth their salt has an API that federates out their raw data.  This enables other sites to use the data in new and novel ways.  But the primary problem with this paradigm is that anything built using these API’s is done from the ground up.  Using the gmail POP interface, it’s possible to build a better UI for gmail.  But to do so you need to first build an entire AJAX mail client — no small feat.  Better would be the ability to add features into the gmail UI itself.  But this is really the standard in web 2.0 API’s today.

Web 2.0.2: Democratizing the feature set

The next big trend will be enabling users to make more compelling ways to interact with
the data.  Users can change not just the data, but how other users see
and use the data.  Sometimes this means API’s with UI hooks.  Or other ways to enable new functionality into an existing site.  This kind of platform enables Independent Software Vendors to improve upon the UI’s that the original sites created.

Facebook is doing this by allowing ISV’s to add new communications features to their site.  Google Maps is doing this through maplets that allow developers to create new ways to interact with mapping data from within the fabulous Maps UI.  Right now these are the only two examples of web 2.0.2 platforms that I’m aware of.

Building this kind of API is very challenging.  There are several very different ways to go about doing it.  Here are a couple of ideas:

• Provide a server-server API that includes content generated from the ISV’s servers into the main experience.  Facebook style.
• Allow developers to author XML files that define new algorithms that are interpreted on the primary host’s servers.  Yahoo pipes is a service in this style, but they’re not doing anything to enhance an existing service so it doesn’t really meet my 2.0.2 criteria.
• Allow developers to author javascript plugins to run on the client machine.  Greasemonkey is essentially doing this.  This strategy has the best shot for a lot of applications in the long term, IMHO.  But it comes with some serious problems right now.

Doing this correctly would allow ISV’s to add new features to Gmail.  Think about it — if I wanted to change the way gmail messages were displayed, or how addressing happened, or whatever it was, this kind of platform could provide hooks for making gmail better in a way that a POP interface never would.  And even though a POP-style interface theoretically could do this, there would never be momentum because having a high-level base to build upon means that there are network effects from the extensions.  (Rails achieves a similar advantage over other web frameworks — just having a standard, any standard, means people will build upon that standard rather than argue over which library to use and extend none of them.)

3rd-party javascript

The big problem with this approach is security.  There is none.  You need to completely trust the ISV before you should allow their code to run in the context of your site.  The kind of editorial review required to do this today would completely kill the democratic goal of such a platform.

The world needs a security sandbox to run third-party javascript code inside.  This way primary site hosts could allow ISV’s to run their code on client machines safely.  Here are a few examples of places this kind of tool could be used.

ISV’s could…

• Add UI features to Gmail
• Create alternate ways to share and discuss images on Flickr
• Define new mathematical formulas to run client-side on a web spreadsheet
• Create new playlist selection / shuffling algorithms for Rhapsody

… and much more.  Even better, individual users (not developers) could pick which UI extensions they wanted to use.  Any site which provides such an API has democratized the feature development process in a very important way.  Not only does it provide a distributed mechanism to figure out which features are best, but it allows users to self-segment as to which features work for them.  Without such a mechanism the entire service must have the same features for everybody, which means product designers must play a political game where they’ll never make everybody happy.  Right now I think really only Facebook has solved this problem.

Building a security sandbox is an area that Microsoft could probably do best and fastest.  They are good at code API’s and layered security models,a nd they have a perfect place to do it with Silverlight and the CLR.  They’re trying to position Silverlight as a faster way to run DHTML, which is something else the world desperately needs right now.  But I just can’t imagine them doing anything this innovative or generally valuable.  Doesn’t sell more Office.  Doesn’t sell more Windows.  They don’t even really have many services that could use third party extensions, and they’ve lost touch with the ISV’s who might build such extensions too.  Google Gears could conceivably add such an extension.  There’s precedent there considering the javascript threading extensions they provide.

This will be a difficult problem to solve, I have no doubt.  But I hope somebody with the resources to leverage a solution takes it on, because I think it would really make the world a better place.

I spent Saturday hanging out with about a dozen hackers building Tagmindr: Remember the future.  Here’s the site’s self-description:

Put any bookmark in a time capsule and we’ll send it to your future self.

Give us your del.icio.us username and we’ll feed you anything that you’ve tagged as: "tagmindr" and "remind:YYYY-MM-DD". We’ll remind you via RSS, SMS, Email or IM, so long as it’s RSS.

(SMS, Email and IM coming later.)

The use case is that you find a page about a product or service that you’d like to look at sometime later so you tag it into del.icio.us thinking you’ll get back to it, but of course you never will.  With Tagmindr you can set a specific date when it will pop up in your feed reader so you will remember to check it out again.

Brian Dorsey came up with the idea and gathered a bunch of us together at his house with the goal of building a web 2.0 app in 6 hours.  We spent an hour or two setting up our dev environments and talking over the goals of what we were going to do.  Then we did a skills inventory, and divided up into teams to start doing the work.  I worked on the back-end team which was a ton of fun.  The project is written using Django, an MVC-based web application framework for Python, which is conceptually quite similar to Rails.  It’s got a few things that are way cooler than rails and a few things that are definitely lacking.  From noon to 6pm we coded, while others did graphic design, HTML layout, and wrote copy.  I had to leave fairly promptly but at the time it seemed we had slipped just a bit — there were still a few issues rendering the design on the production server, and the back-end code still had a couple of bugs.  It seemed like another hour or two’s work total.

Thanks and props to all the wonderful people I met and got to work with.  Special thanks to Anders for holding my hand through basic Python and Django to a level of minor productivity.  I gotta say that Python is really clean.  Makes me realize how much Ruby can look like incomprehensible Perl.