In the interest of keeping the internet safe, I’ll share what I found.  Dan Hill has a pretty good description of the problem on his blog, or at least a very similar one.  I know another friend who got hit in a similar manner.  They all have their sites hosted on dreamhost, as I do.  So it certainly could have been a result of the recent hacking there, but from what I saw, there are hints it is just an exploit of an insecure wordpress plugin.  In particular, the attack came in through Google Analytics for WordPress by joostdevalk (v 3.2.5).  Somehow the plugin directory had global-write (x777) permissions on it, and a couple rogue files were there including one called ainslieturing.php which is pure virus (as opposed to a modified file that was originally there and useful) and apparently the code which attaches the virus to all the other PHP files in the site.  The virus was triggered by a POST to the ainslieturing.php page from IP 176.65.163.29, which might be somewhere in Germany. Curiously, at the time of this writing, the exact phrase “ainslieturing.php” does not appear anywhere on the web, which is part of my motivation for documenting what happened.

Dissecting the ainslieturing file took a bit more work.  It was extra-obfuscated.  The code does the same thing of eval’ing a base64_decode’d string, but it does it in a way where the string “base64_decode” never shows up in the source (example source).  Presumably this is to make it harder  to detect when somebody is trying to clean up the mess.  For example, this avoids the simple sed fix posted on Dan Hill’s blog.  Additionally, the base64 encoded code appears written to avoid simple virus filters, because it is shuffled before evaluation by a key (143 in my case) which can be easily modified (example source).  The inner code is a PHP script which lets the attacker run arbitrary code on the server, or upload arbitrary files.  Interestingly, the whole thing is password protected, requiring the attacker to present a password with MD5 signature “ca3f717a5e53f4ce47b9062cfbfb2458″.   (Anybody feel like reversing that?)  If you want to check your  files to see if any of them have the double-obfuscated code, this will find them (and perhaps some false positives too):

grep ".x62.x61.x73.x65" * -R -l 2> /dev/null

Once ainslieturing was triggered, the rough symptoms were that a bunch of code got inserted at the top of many of wordpress’s PHP files which is lightly obfuscated through eval-base64-decode.  The virus code when de-obfuscated looks like this.  I haven’t bothered to fully understand it, but similar code has infected other people’s servers, with minor variations.  In particular, the code fetches some instructions from URLs which are doubly-obfuscated, but resolve to domains in Poland or Russia.  Many *.rr.nu domains with hoperjoper.ru as the nameserver and and in particular the throw-away domain ens122zzzddazz.com.  (Please be careful with these URLs — DO NOT JUST TYPE THEM INTO YOUR BROWSER.  Use wget and look at the files that come back.)  If you operate any blacklists, feel free to add these domains to them.

The …dazz domain in particular has a whois record which is not private:

Registrant:
Dan Brown admin@ens122zzzddazz.com +022.824460528 +022.824460528
ENO
Aleje Ujazdowskie 20-44
Warszawa,Warszawa,AF 00540

So, Dan, if you actually exist, you either have some explaining to do, or your domain has been completely taken over.  If any of my readers are traveling to Warsaw, Poland and feeling intrepid, feel free to drop by Dan’s office and let me know what you find.

That’s all for now.  If you have anything relevant to add to the situation, please leave a comment.

But this problem is not at all new.  Tuition has been outpacing inflation for decades.  The College Board‘s statistics show that tuition has increased faster than inflation almost every year going back to 1958.  On average it has outpaced inflation by about 2.8%.

(raw data)

With all the recent discussion about how unsustainable health care costs are, it’s very telling to note that the cost of higher education has been rising faster than health care for the last 30 years.  (Ref: freakonomics, seeking alpha.)

You might say this is all water under the bridge or sunk costs or what have you.  The important question is how fast will college tuition go up in the future? Of course, nobody knows for sure.  Past performance is no guarantee of future results, etc.  Some folks who pay attention to this think it will continue to go up about 6%/yr in the future, although long-term averages are more like 7%/yr or 8%/yr.  So something in that range is a reasonable guess.

Why is tuition going up so fast? That’s a great question which I won’t go into detail here.  But briefly, higher education is a good whose price is influenced strongly by market forces — supply and demand.  Demand must be increasing to keep up with the rising costs.  But another important factor is the unusual way that education is financed which distorts prices.  Also, many think we’re currently in a “bubble” in which higher education is overpriced.  I don’t subscribe to that point of view, but as I said more later…

To be clear, I understand that it is a wonderful feeling to find a home if you haven’t known one before.  In 1997 during my first visit to Burning Man, I felt like Gonzo in Muppets from Space when he (spoiler alert!) first meets his extended family.  His unique appearance had made him feel utterly alone, until a spaceship full of Gonzo-looking aliens landed on earth and explained that he was one of them.  The realization that he was not a freak outcast but part of a vibrant community is the same that many first experience at Burning Man.  I first experienced this sense of inclusion there, and it has undoubtedly transformed my life for the better.  It is a deeply powerful experience that continues to be extremely important for a great many people.  But why does it need to be rooted in a wasteland in Nevada?  Why not bring that feeling to your real home?

My challenge is this to everybody who considers Burning Man their home: How can you bring what you love about Burning Man into the other 51 weeks of your year? What is so immutable about your regular life that you can only feel comfortable 2% of the time?  Is that dusty dusty place really so special that you cannot bring its culture home in a sustainable way?  From personal experience, I think not.  It might take years, but you really can take the things you love about Burning Man back to your regular life. Let’s go through some of the features of BRC that many people find wonderful and discuss how to recreate them in the real world.

At Burning Man, I get to spending lots of time with my friends

One of the simplest pleasures of That Thing In The Desert is that you get to spend an entire week hanging out with your friends.  Vacations are great, right?  Well here’s an idea: go on a camping trip with your friends closer to home.  Or how about arranging a weekly gathering to play board games or cook dinner together?  Creating sustainable community activities is completely possible at home.

Or convince your friends to go somewhere new for a vacation.  Sure, BRC is a wonderfully amazingly different place (at least the first several times you go), but so is much of Africa or Asia.  It’s not like a trip to BRC is cheap either — on average people spend over $2,000 for the whole thing (ref: BRC Census).  Compare that to a plane ticket across the globe.

If you really like being around your friends all the time, how about actually moving into a house with them?  That’s what I did.  It’s called co-housing, and it’s awesome.  Every morning when I get up and every evening for dinner I see my good friends milling about living their lives, and we enrich each other.  I highly recommend it.  If that’s too intense for you, figure out how you and your closest can live within walking distance of each other.  It takes years for neighborhoods to coalesce, but when it works it’s wonderful.

At Burning Man, I’m surrounded by cool art

If this is an excuse for why you can’t feel at home in your regular home life, the irony is thick.  First tabulate how much time and money your camp expended on your last vacation in the desert.  Break that down into the part that was spent on personal comforts (i.e. making BRC more like home) and the part that was spent creating cool art for others to experience.  Now try harnessing all that creative brainpower which went into your project, and divert it towards doing something awesome for your local community.  A few quick ideas: a mural or sculpture in your neighborhood or a new community P-patch or a collective third place for your friends.

Sure it’s a different kind of challenge.  Most cities have more rules about modifying your surroundings than Black Rock City.  But as the years go on, the differences are shrinking.  BRC has strict fire codes and (less strict) building codes, and as the community expands, increasingly restrictive community decency standards.  You can always put up your own Jiffy Lube sculpture in your back yard.

At Burning Man, I can be myself

“Radical self expression” has been one of Burning Man’s philosophies from the beginning.  The ability to be yourself in your normal life seems on the surface like it really should be easy, but is often extremely hard.  What’s preventing you from being yourself?  Often it’s social inertia.  People who expect you to act a certain way — a way that maybe you’re tired of and want to move on from.  If this is the case for you, I’ll offer some bold advice: try spending less time with those people, and more time with people who reinforce the version of yourself you prefer.

If on the other hand you enjoy being somebody different only while you’re in the desert, then you have a harder choice to make.  Is that other person who you really want to be?  Perhaps they’re just a costume you enjoy wearing like for Halloween.  But if that other person has a real home, and you are living as an outsider, then this choice bears consideration.

At Burning Man, strangers are friendly and awesome

This one can be hard, especially for people living in certain cities.  After my first burn, my campmate and I decided to try to bring some of the playa attitude back to Los Angeles.  We attempted what we later termed “attack smiles” because their effect on sidewalk passersby was the exact opposite of what we hoped.  Within a year we both left LA for friendlier pastures.  So in the “tough choices” department, moving is always an option.  You might not feel at home because your home isn’t a very friendly place.  But I wouldn’t jump to that conclusion too quickly.

It might be cliche, but scientific research has shown that good moods spread through social networks.  Happiness is contagious.  Especially amongst friends.  So spend more time with your friends and friends of friends, and bring that same energy you bring to the desert.  Build community. (This is the simplest, strongest advice I can give.) Bring the cultural principles that you love into your 98%-of-the-year community.  It’ll take a lot of work over time.  But I bet your friends will be on board to help, and the end goal is absolutely worth the effort.

My housemate Ellery created these mac ‘n’ cheese cupcakes for dinner the other night. The frosting is mashed potatoes, and they’re topped with a cherry tomato. Inside is a meatless meaty macaroni and cheese combination surrounded by a savory dough. They were super fun and tasty.

I can take very little credit for these beyond the photo. And helping to consume them. But it’s a great example of why I love living with fun creative people! I’ve heard many requests for the recipe — stay tuned! It’s not mine to share, but when Ellery writes it down I’ll be sure to let you know. (And update this page.)

So you’ve found some folks you think you might want to live with.  Or maybe they’re awesome friends whom you’re super excited to live with.  Either way, before signing a lease (or a mortgage!) it’s important to do your due diligence and try to figure out how well you’ll get along living together.

If it’s somebody you don’t know very well, the need might seem obvious.  But if it’s an old friend, I posit it’s even more important to check your homie-compatibility index.  Being friends and being good housemates are not the same thing.  When considering co-housing, probably the most important thing is picking the right people to live with.  My very wise housemate Heater developed this list of discussion topics to go over with potential roommates.

• Communication style
• Occupancy dates
• Noise
• Guests
• Parties
• Food
• Regular meetings
• Use of the Common Spaces
• Substances
• Nudity
• Sex
• Scheduling use of space
• Cleanliness
• Utilities
• Methods of rent
• Parking and neighbors
• Rooms
• Pets
• Kids
• Temperature
• Decor
• Chores

We recommend scheduling 2-3 hours of uninterrupted time together to discuss everything on this list.  It takes a while to talk about everything!  Discuss each topic, and write down your expectations for how a household should work.  This forms an informal social contract that you can refer back to.  Make note of differences of opinion.  Decide how you’ll deal with them, or recognize that the barriers to a happy house are too large.

I’m going to give two examples of natural monopolies in high tech. They are not the perfect examples used in textbooks, but I think they are illustrative, and offer valuable lessons.

Natural Monopoly of Operating Systems

Operating systems are a good example of a natural monopoly. As much as we all value choice as a driver of innovation, the plain truth is that almost everybody is better off if there is a standard operating system upon which higher-level applications can be built. Application developers benefit because they have a single clear platform upon which to build. If there were two or three dominant operating systems, application vendors would need to build a separate version of their application for each one in order to reach consumers, which is considerably more effort. Similarly, the standard benefits consumers because they have a single choice which gives them the benefit of all the applications written on it.

Gates & Allen understood this long before most, which prompted them to drop out of school and pursue Microsoft with vigor. Windows succeeded in creating such a natural monopoly, enabling a rich ecosystem of third-party software vendors (ISVs in MS parlance) to create value for consumers without needing to worry about what chipset underlies the graphics card or network adapter their customers’ computers. In this way, Microsoft enabled the creation of value for PC customers and wealth for ISVs, and the monopoly persists in a form to this day.

But all is not rosy in this world. Other companies want to sell operating systems. People want choice. Once entrenched, the monopolist has a tendency to make choices which benefit the monopolist more than the consumer — Microsoft continues to exhibit this behavior even as their monopoly power fades. In classic natural monopolies like utilities, explicit regulation controls the monopolist’s abuse. With Windows, a combination of limited government intervention and competitive innovation ultimately limited their influence.

Social networks as natural monopolies

Online social networks also exhibit properties of a natural monopoly. A well built social networking service like Facebook creates tremendous economic opportunities. Particularly if the service exposes its valuable social graph data through an API that other services can use. Almost any online service can be made more compelling by incorporating social graph data. The existence of a publicly usable social graph dataset provides an economic boost to the entire tech sector.

This boost tends to create a winner-take-all situation.  When third-party services rely on a social API service, they reinforce consumer’s use of that service.  Third parties’ lives are easier when there is a single standard, because they only need to code to a single API in order to gain the benefits of the social graph.  Here the analogy to operating systems is clear.  The social network provides a platform upon which others can create value.  The value creation process is easier if there is a single standard social network upon which to build. These characteristics make the social networking monopoly natural.

A behavioral characteristic of social networking sites’ users also helps create a monopoly. People enjoy the benefits of having their social network defined online, but they do not enjoy the effort of defining it. Us geeks (everybody reading this and probably most of your friends) are willing to spend hours organizing our friends into circles or searching for people we know to connect with them. Some of us even enjoy it. But for most normal people this very quickly becomes a boring waste of time, especially if they’ve already done this once or twice on different websites.  Most people are not willing to maintain multiple social networks. Once they are invested in one, the barrier to switching is quite high.

Implications for Google+ in competing with Facebook

Facebook’s dominance is rapidly approaching monopoly levels.  They have crossed the tipping point where they are fast on their way to becoming the de-facto standard for social graph data, if they haven’t already.  The nature of social networks as supporting a natural monopoly means that Facebook’s rise will be supported more strongly than it would be otherwise.  When considering Facebook’s dominance, we readers must remember our place in the ecosystem as geeks.  We and our friends, are the innovators and early adopters who are far more willing to try the new thing, because we see intrinsic value in progress, and are far less perturbed by unrefined products.  The fact that recently Facebook’s fastest growing demographic was women over 55 shows that the service has crossed Moore’s chasm and now appeals to the majority of people.  As industry insiders, it’s easy for us to forget the bubble we live in — just because everybody we know uses something doesn’t mean it will ever actually take off an be popular with non-geeks.  But Facebook is clearly on a path to provide a dominant monopolistic standard for social networking data.

Breaking this monopoly would be difficult for Google even without the advantages of a natural monopoly.  People’s natural laziness makes a third social network (after Facebook and Twitter) unlikely to succeed as well.  So on the face of it, Google‘s got a very tough road ahead.  It’s tempting to declare G+ dead on arrival because of these intrinsic forces, but there are other reasons why I think they actually have a decent shot.  But I’ll save that analysis for another story.

It was quite a chore to get Apple to admit that the cause was a hardware problem, and fix it.  But I finally succeeded, so I thought I’d share some of my experiences.  I’ll explain what a Kernel Panic is, how they sometimes can be caused by faulty software but often indicate hardware problems, how they differ from other kinds of crashes, and provide a guide on how to read a Mac OS X kernel panic report.

Dealing with the “Genius” Bar staff

“Genius” is what Apple calls its first tier of technical support.  I find the brand unfortunate and insulting for everybody involved.  There is no intelligence test required to work as a “genius” — just some minimal training on how to follow Apple customer service scripts like an obedient robot.  Knowing Apple, I wouldn’t be surprised if the “Genius” staff are required to follow these scripts verbatim and face not only termination but punitive lawsuits for deviating from the party line.  Keep this in mind when dealing with them.  Also know that they have some discretion in the outcome of your visit, but the discretion exists within guidelines that they cannot control.

Some tips on getting past the “genius” from my limited experience.  Print out your kernel panic reports and bring them in.  The more the better.  Highlight the relevant parts.  I’m not sure if bringing a bad attitude with you helps or not — they want to make their customers happy, but they don’t like their “genius” title challenged with logic.  I also recommend persistence.  Following their stupid advice and showing them that it did no good will help.  I’m not sure if understanding what’s going on will or not.  But if you’d like to understand more about why your Mac is crashing, read on…

Kernel panics and hardware failures vs regular software failures

There are two basic ways your Mac can crash.  First, an application might lock up on you and become unresponsive.  You get the spinning beachball of death, and eventually have to Force Quit your application, losing whatever work you hadn’t saved.  This kind of user mode failure is very common with buggy software.  If the beachball is getting you down, the problem is almost certainly caused by bad software, not by a hardware problem.  In OS 9 and before, this kind of failure could have taken down your entire machine, but since the introduction of the BSD kernel in OS X, the system is designed to allow one application to fail while protecting all the other applications.

Sometimes though your entire Mac will crash hard.  Without warning your system displays a full-screen message saying “You need to restart your computer. Hold down the Power button for several seconds or press the Restart button.” in several languages.  This is OS X’s last ditch attempt to tell you something about what happened before it goes completely teets up.  It’s formally known as a kernel panic.  Sometimes the system is so screwed it can’t even get that error message onto the screen before it dies.

Kernel panics indicate a serious problem, either with the computer’s hardware, or the low-level software in the operating system. In fact there are only three things that can cause a kernel panic:

1. Faulty hardware causes a problem that the OS doesn’t know how to deal with
2. A bug in OS X itself
3. A bug in an OS plugin called a kernel extension or kext

Firstly, if the hardware itself has problems, then kernel panics are a common way they manifest themselves.  Similarly, if the operating system itself has any bugs, they could take down the entire system.  The third option could be caused by third-party software, while the first two are entirely Apple’s responsibility.  So when it comes to dealing with the “Genius” behind the bar, the first two are fairly straightforward.  If you’re seeing this problem a lot, and nobody else is, then it’s probably a hardware problem, and they should replace your hardware. Here’s a thought experiment I tried unsuccessfully with the Apple “geniuses” I had to deal with: Imagine you have a hundred Macs all running the same software, and one of them crashes periodically, but the other 99 don’t.  Would you classify that Mac as having a hardware problem or a software problem?  In my case, the genius insisted that it was a software problem.  In fact he claimed he was certain that if I uninstalled Adobe Flash, the problem would be fixed.  Read on, and you’ll learn how the kernel panic reports themselves show that this explanation is impossible.

Understanding and interpreting Kernel Panic reports

First a bit about what a Kernel Panic is.  Very simply, it’s when something unexpected goes wrong in the operating system kernel.  What’s the kernel?  The kernel is the lowest level of the operating system — the part that’s closest to the hardware.  In modern operating systems, there’s a fairly arbitrary line between what functionality lives in the kernel and what functionality lives in the user space.  The key difference is that when something goes wrong with software in the user space, you get a beachball on the app, but the system survives.  When something goes wrong in the kernel, you get a kernel panic, and the whole system goes bye bye fast.  So it’s critical that any code running in the kernel space be ultra reliable.  You don’t change kernel code quickly or lightly, and you test the hell out of it before you release it.  But code runs faster in the kernel, so most modern operating systems put important things like networking and graphics into the kernel.  The BSD kernel which powers OS X allows the installation of “kernel extensions” or “kexts” which add functionality.  More about these soon.  But suffice to say that when anything goes wrong with any kext, it’s a big deal problem because there’s nothing to fall back on (e.g. can’t display an error dialog if the problem is with the display system), so the system’s reaction is called a panic.  Thus “kernel panic.”

Immediately after a KP, your computer does two things: it stores a bunch of information to help diagnose what caused the problem, and puts up the error screen, if it can.  When you reboot, your computer asks if you want to send the KP report to Apple.  You should do this.  The smarter of the “genius” staff can look these reports up and see that your Mac is actually crashing, but they’ll admit that the contents are too technical for a mere “genius” to understand.  Well I’m going to explain to you what the reports contain and what it means about what’s wrong with your computer.

Here’s a typical crash report from my computer.  In my case, these panics weren’t even accompanied by the “restart your computer message” because as I’ll explain, the problem originated in the graphics system.  My computer just suddenly went black and non-responsive.  I’ve highlighted a few key sections for explanation below.

Interval Since Last Panic Report:  420 sec
Panics Since Last Report:          1
Anonymous UUID:                    8A09F455-1039-4696-8479-xxxxxxxxxxxx
Thu Apr 21 09:00:51 2011
panic(cpu 3 caller 0x9cdc8f): NVRM[0/1:0:0]: Read Error 0x00000100: CFG 0xffffffff 0xffffffff 0xffffffff, BAR0 0xc0000000 0xa734e000 0x0a5480a2, D0, P2/4
Backtrace (CPU 3), Frame : Return Address (4 potential args on stack)
0xbc001728 : 0x21b510 (0x5d9514 0xbc00175c 0x223978 0x0)
0xbc001778 : 0x9cdc8f (0xbe323c 0xc53840 0xbf23cc 0x0)
0xbc001818 : 0xae85d3 (0xe0cfc04 0xe5c9004 0x100 0xb83de000)
0xbc001868 : 0xadf5cc (0xe5c9004 0x100 0xbc001898 0x9bd76c)
0xbc001898 : 0x16c8965 (0xe5c9004 0x100 0x438004ee 0x28)
0xbc0019d8 : 0xb07250 (0xe5c9004 0xe5ca004 0x0 0x0)
0xbc001a18 : 0x9d6e23 (0xe5c9004 0xe5ca004 0x0 0x0)
0xbc001ab8 : 0x9d3502 (0x0 0x9 0x0 0x0)
0xbc001c68 : 0x9d4aa0 (0x0 0x600d600d 0x704a 0xbc001c98)
0xbc001d38 : 0xc89217 (0xbc001d58 0x0 0x98 0x2a358d)
0xbc001df8 : 0xc8ec1d (0xe8e5404 0x0 0x98 0x45e8d022)
0xbc001f18 : 0xc8f0b4 (0xe8e5404 0x124b6204 0x6d39d1c0 0x0)
0xbc001f78 : 0xc8f39f (0xe8e5404 0x124b6204 0x6d39d1c0 0xbc0021e0)
0xbc002028 : 0xca3691 (0xe8e5404 0x1f80d8e8 0xbc00239c 0xbc0021e0)
0xbc002298 : 0xc84d09 (0x6d0b7000 0x1f80d8e8 0xbc00239c 0x0)
0xbc0023f8 : 0xc84f47 (0x6d0c6000 0x1f80d800 0x1 0x0)
0xbc002428 : 0xc87a04 (0x6d0c6000 0x1f80d800 0x0 0x97c6c4fc)
0xbc002468 : 0xca9d40 (0x6d0c6000 0x1f80d800 0x6d09f274 0x140)
0xbc0024f8 : 0xc9b5a9 (0xde94bc0 0x1f80d800 0x0 0x1)
0xbc002558 : 0xc9b810 (0x6d09f000 0x6d09f77c 0x1f80d800 0x0)
0xbc0025a8 : 0xc9bce4 (0x6d09f000 0x6d09f77c 0xbc0028cc 0xbc00286c)
0xbc0028e8 : 0xc98aaf (0x6d09f000 0x6d09f77c 0x1 0x0)
0xbc002908 : 0xc605a1 (0x6d09f000 0x6d09f77c 0x1956a580 0x0)
0xbc002938 : 0xc9a572 (0x6d09f000 0xbc002a7c 0xbc002968 0x5046b1)
0xbc002978 : 0xc648de (0x6d09f000 0xbc002a7c 0x0 0xc000401)
0xbc002ab8 : 0xc9dee6 (0x6d09f000 0x0 0xbc002bcc 0xbc002bc8)
0xbc002b68 : 0xc60c93 (0x6d09f000 0x0 0xbc002bcc 0xbc002bc8)
0xbc002be8 : 0x56a738 (0x6d09f000 0x0 0xbc002e3c 0xbc002c74)
0xbc002c38 : 0x56afd7 (0xcef020 0x6d09f000 0x129bab88 0x1)
0xbc002c88 : 0x56b88b (0x6d09f000 0x10 0xbc002cd0 0x0)
0xbc002da8 : 0x285be0 (0x6d09f000 0x10 0x129bab88 0x1)
0xbc003e58 : 0x21d8be (0x129bab60 0x1ec235a0 0x1fd7e8 0x5f43)
      Backtrace continues...

      Kernel Extensions in backtrace (with dependencies):
         com.apple.GeForce(6.2.6)@0xc55000->0xd0afff
            dependency: com.apple.NVDAResman(6.2.6)@0x967000
            dependency: com.apple.iokit.IONDRVSupport(2.2)@0x95a000
            dependency: com.apple.iokit.IOPCIFamily(2.6)@0x927000
            dependency: com.apple.iokit.IOGraphicsFamily(2.2)@0x938000
         com.apple.nvidia.nv50hal(6.2.6)@0x1592000->0x19a6fff
            dependency: com.apple.NVDAResman(6.2.6)@0x967000
         com.apple.NVDAResman(6.2.6)@0x967000->0xc54fff
            dependency: com.apple.iokit.IOPCIFamily(2.6)@0x927000
            dependency: com.apple.iokit.IONDRVSupport(2.2)@0x95a000
            dependency: com.apple.iokit.IOGraphicsFamily(2.2)@0x938000

BSD process name corresponding to current thread: kernel_task

Mac OS version:
10J869
Kernel version:
Darwin Kernel Version 10.7.0: Sat Jan 29 15:17:16 PST 2011; root:xnu-1504.9.37~1/RELEASE_I386
System model name: MacBookPro6,2 (Mac-F22586C8)
System uptime in nanoseconds: 35829130822125

unloaded kexts:
com.apple.filesystems.msdosfs 1.6.3 (addr 0xbc1e5000, size 0x53248) - last unloaded 12216461868115

loaded kexts:
com.parallels.kext.prl_vnic 6.0 11992.625164
com.parallels.kext.prl_netbridge 6.0 11992.625164
com.parallels.kext.prl_usb_connect 6.0 11992.625164
com.parallels.kext.prl_hid_hook 6.0 11992.625164
com.parallels.kext.prl_hypervisor 6.0 11992.625164
com.apple.filesystems.smbfs 1.6.6 - last loaded 12151022138289
com.apple.driver.AppleHWSensor 1.9.3d0
com.apple.driver.AGPM 100.12.19
com.apple.driver.AppleMikeyHIDDriver 1.2.0
com.apple.driver.AppleHDA 1.9.9f12
com.apple.driver.AppleUpstreamUserClient 3.5.4
com.apple.driver.AppleMCCSControl 1.0.17
com.apple.driver.AppleMikeyDriver 1.9.9f12
com.apple.driver.AudioAUUC 1.54
com.apple.driver.AppleIntelHDGraphics 6.2.6
com.apple.driver.AppleIntelHDGraphicsFB 6.2.6
com.apple.driver.SMCMotionSensor 3.0.0d4
com.apple.kext.AppleSMCLMU 1.5.0d3
com.apple.Dont_Steal_Mac_OS_X 7.0.0
com.apple.iokit.CHUDUtils 201
com.apple.iokit.CHUDProf 216
com.apple.driver.AudioIPCDriver 1.1.6
com.apple.driver.AppleGraphicsControl 2.8.68
com.apple.driver.ACPI_SMC_PlatformPlugin 4.5.0d5
com.apple.GeForce 6.2.6
com.apple.driver.AppleLPC 1.4.12
com.apple.filesystems.autofs 2.1.0
com.apple.driver.AppleUSBTCButtons 200.3.2
com.apple.driver.AppleUSBTCKeyboard 200.3.2
com.apple.driver.AppleIRController 303.8
com.apple.driver.AppleUSBCardReader 2.5.8
com.apple.iokit.SCSITaskUserClient 2.6.5
com.apple.BootCache 31
com.apple.AppleFSCompression.AppleFSCompressionTypeZlib 1.0.0d1
com.apple.iokit.IOAHCIBlockStorage 1.6.3
com.apple.driver.AppleUSBHub 4.1.7
com.apple.driver.AppleFWOHCI 4.7.1
com.apple.driver.AirPortBrcm43224 427.36.9
com.apple.iokit.AppleBCM5701Ethernet 2.3.9b6
com.apple.driver.AppleEFINVRAM 1.4.0
com.apple.driver.AppleSmartBatteryManager 160.0.0
com.apple.driver.AppleUSBEHCI 4.1.8
com.apple.driver.AppleAHCIPort 2.1.5
com.apple.driver.AppleACPIButtons 1.3.5
com.apple.driver.AppleRTC 1.3.1
com.apple.driver.AppleHPET 1.5
com.apple.driver.AppleSMBIOS 1.6
com.apple.driver.AppleACPIEC 1.3.5
com.apple.driver.AppleAPIC 1.4
com.apple.driver.AppleIntelCPUPowerManagementClient 105.13.0
com.apple.security.sandbox 1
com.apple.security.quarantine 0
com.apple.nke.applicationfirewall 2.1.11
com.apple.driver.AppleIntelCPUPowerManagement 105.13.0
com.apple.driver.DspFuncLib 1.9.9f12
com.apple.driver.AppleProfileReadCounterAction 17
com.apple.driver.AppleProfileTimestampAction 10
com.apple.driver.AppleProfileThreadInfoAction 14
com.apple.driver.AppleProfileRegisterStateAction 10
com.apple.driver.AppleProfileKEventAction 10
com.apple.driver.AppleProfileCallstackAction 20
com.apple.driver.AppleSMBusController 1.0.8d0
com.apple.iokit.IOFireWireIP 2.0.3
com.apple.iokit.IOSurface 74.2
com.apple.iokit.IOBluetoothSerialManager 2.4.0f1
com.apple.iokit.IOSerialFamily 10.0.3
com.apple.iokit.CHUDKernLib 208
com.apple.iokit.IOAudioFamily 1.8.0fc1
com.apple.kext.OSvKernDSPLib 1.3
com.apple.driver.AppleHDAController 1.9.9f12
com.apple.iokit.IOHDAFamily 1.9.9f12
com.apple.iokit.AppleProfileFamily 41
com.apple.driver.AppleSMC 3.1.0d3
com.apple.driver.IOPlatformPluginFamily 4.5.0d5
com.apple.driver.AppleSMBusPCI 1.0.8d0
com.apple.nvidia.nv50hal 6.2.6
com.apple.NVDAResman 6.2.6
com.apple.iokit.IONDRVSupport 2.2
com.apple.iokit.IOGraphicsFamily 2.2
com.apple.driver.BroadcomUSBBluetoothHCIController 2.4.0f1
com.apple.driver.AppleUSBBluetoothHCIController 2.4.0f1
com.apple.iokit.IOBluetoothFamily 2.4.0f1
com.apple.driver.AppleUSBMultitouch 206.6
com.apple.iokit.IOUSBHIDDriver 4.1.5
com.apple.iokit.IOSCSIBlockCommandsDevice 2.6.5
com.apple.iokit.IOUSBMassStorageClass 2.6.5
com.apple.driver.AppleUSBMergeNub 4.1.8
com.apple.driver.AppleUSBComposite 3.9.0
com.apple.iokit.IOSCSIMultimediaCommandsDevice 2.6.5
com.apple.iokit.IOBDStorageFamily 1.6
com.apple.iokit.IODVDStorageFamily 1.6
com.apple.iokit.IOCDStorageFamily 1.6
com.apple.driver.XsanFilter 402.1
com.apple.iokit.IOAHCISerialATAPI 1.2.5
com.apple.iokit.IOSCSIArchitectureModelFamily 2.6.5
com.apple.iokit.IOUSBUserClient 4.1.5
com.apple.iokit.IOFireWireFamily 4.2.6
com.apple.iokit.IO80211Family 314.1.1
com.apple.iokit.IONetworkingFamily 1.10
com.apple.iokit.IOUSBFamily 4.1.8
com.apple.iokit.IOAHCIFamily 2.0.4
com.apple.driver.AppleEFIRuntime 1.4.0
com.apple.iokit.IOHIDFamily 1.6.5
com.apple.iokit.IOSMBusFamily 1.1
com.apple.kext.AppleMatch 1.0.0d1
com.apple.security.TMSafetyNet 6
com.apple.driver.DiskImages 289
com.apple.iokit.IOStorageFamily 1.6.2
com.apple.driver.AppleACPIPlatform 1.3.5
com.apple.iokit.IOPCIFamily 2.6
com.apple.iokit.IOACPIFamily 1.3.0

The first line is fairly clear — how long has your system been running since its last crash?  If this is less than an hour, as it was for my computer, then your machine is completely FUBAR.  Less than a day and you’ve still got a seriously unstable computer.  (Hint for any “genius” that might be reading this article: take the number of seconds, divide it by 60 using the Calculator app on your store-issued-iPad, and that will give you the number of minutes.  Divide that new smaller number by 60 again to get an even smaller number which is hours.  If you can figure out how to get to number of days by yourself, it’s time to apply for the “Genius Lead” job.)

The Anonymous UUID is an effectively random code that allows Apple to lookup the crash reports for your computer when you go into the store.  Then there’s the date.  Straightforward.

The line which starts “panic” is the closest thing you’ll find to a concise explanation of what went wrong. In all likelihood this will be a jumble of words and numbers that make no sense, but it’s a great string to Google.  If you’re having a hardware problem, this message will probably stay about the same with each KP.  Googling my error message “NVRM[0/1:0:0]: Read Error 0×00000100” turns up a bunch of people with similar problems — computer going black without warning, often while playing World of Warcraft.

The next section titled “backtrace” is worthless unless you’re actually diving into the source code that caused the problem.  Skip over it.  But the section after it is extremely interesting and relatively easy to interpret.

The section titled “Kernel Extensions in backtrace (with dependencies)” actually tells you what part of the system failed.  Read this one closely and try to make sense of it. In the case of my example, there are three kernel extensions involved with the crash.  They are called “com.apple.GeForce” and “com.apple.nvidia.nv50hal” and “com.apple.NVDAResman”.  The first one is fairly obvious — GeForce is the kind of graphics chip in the macbook.  The second one is also pretty clear — NVidia is the company that makes GeForce, and nv50hal I would guess means “NVidia 5.0 Hardware Abstraction Layer” or something similar.  I’m not sure what NVDAResman is but looking down a bit I see it’s related to “IOGraphicsFamily”.  This paints a really clear picture that the failure is in the graphics system.  Moreover, since every line here starts with “com.apple” we know the failure is entirely in code written by Apple.  There is no third-party software involved in this crash.

For my particular crash, it’s important to know something about the graphics hardware of these MacBooks, since all evidence points to the graphics hardware.  This generation of macbooks have two graphics chips — a faster one from Nvidia, and a more battery-friendly one from Intel.  The nvidia chip which is apparently having problems is always used when the computer has an external monitor plugged in, or when something fancy is happening on the built-in screen.  A nice utility called gfxCardStatus can help you understand this complexity, and will definitely give you a leg up on the “genius.”

The following line starting with “BSD process name” can also be important.  This will sometimes tell you which user-level app originated the call into the kernel which failed.  In my case it was “kernel_task” which provides no additional information.

The next section gives some basic info about the Mac — hardware and OS versions.  What follows is a complete list of kernel extensions (kexts) installed.  This gives you a bit more ammo in dealing with the “genius” who is probably ignoring you at this point anyway.  You can look through this list and see everything that might possibly contribute to a kernel panic.  In my case, the only software modules that aren’t from Apple are some drivers from Parallels for running my Windows virtual machine.  So the only reasons my Mac might kernel panic are because of a hardware problem, a bug in OS X itself, or something going wrong with Parallels.  Understanding this should, in theory, be very helpful when talking to your local neighborhood “genius” but unfortunately they are simple bots that only run scripts authored in Cupertino and are not permitted to listen to logic.

Apple’s Propaganda about Flash

When the “genius” told me my Mac’s problem was that I had Adobe Flash installed, I just laughed at first.  Flash is installed on something like 97% of desktop computers, and very few of them regularly turn themselves off for no reason.   Moreover, the kernel panic report lists every piece of software that could possibly contribute to the kernel panic, and neither the word “flash” nor “adobe” appear anywhere in the list.  But then I realized he wasn’t joking.

Apple’s ongoing arguments with Adobe over Flash are well publicized.  The root of the issue, in very brief summary, is that Apple sees Adobe’s Flash as a strategic threat to their incredibly profitable iPhone platform.  The poor “genius” I’m stuck with has become a pawn in Apple’s PR battle, throwing himself on the grenade of propaganda just to spread FUD about Flash.  I tried reasoning with him, explaining that Adobe’s software doesn’t run in the kernel, and therefore cannot cause a kernel panic.  The job of the kernel is to protect users from badly written software crashing the whole machine. But he would not budge.  I imagined a “genius” script which read as follows:

Mac is crashing…

1. Run hardware diagnostic tests.

2. Address any identified hardware problems.

3. If hardware tests come back clean, tell customer that the problem (whatever it is) is caused by Flash.  Tell them to uninstall it, and see if that helps.

Here I imagine the Dantesque trap of the rare “genius” who actually understands how OS X works.  I’m telling the customer something which is impossible on its face, and he knows it.  He’s arguing with me telling me I’m being stupid.  But I signed a contract with Apple saying I would defame Adobe, and deviation from this contract will bring the wrath of Steve’s legal team on me.  I just have to smile and say things like “yeah, that’s the really strange thing about this particular software problem — it only affects certain computers.  But it’s definitely caused by Flash.”

One might reason that Flash could cause kernel panics because it makes more extensive use of the graphics system than other applications.  But in this case, Flash isn’t the actual problem.  Flash is exposing the underlying problem, as would any software which works the graphics system hard.  Thus lots of people with the same problem as me who play World of Warcraft.  If the “genius” advice ever works, it’s just because Flash is the most graphics intensive software that many people use on their Macs.  The actual problem is still either a bug in OS X, or a hardware problem.

Consider the advice not to use Flash on your Mac in analogy to a car.  (A high-end MacBook actually costs as much as some cars.)  Imagine that your car sometimes just turned its engine off while you were in the middle of driving it – catastrophic failure with no warning or apparent reason.  You go to the dealership and they can’t find anything wrong with it, but ask if you ever listen to electronic music?  Well, yes, sometimes.  That’s the problem!  It’s the electronic music which is causing your car to malfunction.  So stop listening to it, and the problem will be fixed.  Umm, what?  The closest thing to the truth, by analogy, would be that any bass-heavy music (graphics-intensive application) is stressing out some weak connection in the electronics.  But because the car dealership is owned by the local philharmonic, they’re blaming it on that awful music the kids listen to.   Using your misfortune and their incompetence to push an unrelated political agenda.

It’s an interesting glimpse into how Apple is using their retail presence to advance a strategic PR goal.  Evidence that Apple has grown up as a company to the point where their own motives are more important than doing what actually helps customers.  *sigh*  At least I got my MacBook fixed.

Please democratize SSL certificates.  The ability to serve HTTPS:// pages without scaring users is currently controlled by a handful of “trusted authorities” whose business is to make it difficult to secure web communications.  Google, you have the ability to disrupt this oligarchy and empower individuals to make the web safer.

The web is a safer place when information passed between browsers and web servers is encrypted — that is when URLs start with HTTPS instead of HTTP.  The recent introduction of FireSheep demonstrated to the world just how insecure normal (HTTP) web communications are — anybody on your network with a simple browser plugin can impersonate you.  In fact, FireSheep democratized the ability to steal session authentication by bundling it up in a manner that is easily used by the masses.  Google’s own proposed SPDY protocol, whose primary goal is to make the web faster, is willing to slow down in the name of security.  “Although SSL does introduce a latency penalty, we believe that the long-term future of the web depends on a secure network connection.”  We all want a safer web, so please help us achieve that by making it easier to set up HTTPS on our web servers.

There is no technical challenge here.  All modern browsers and servers are capable of safely encrypting the information passed between them.  Encryption protects users against eavesdropping and session hijacking a la firesheep.  Today’s challenge to secure web communications does not lie in the encryption, but the authentication.  The HTTPS protocol begins with the server presenting its security “certificate” which is meant to assure the user they have not reached an imposter web site.  This assurance is provided courtesy of the oligarchy of trusted certificate authorities, for a fee and a hassle.  Alternately, servers can present a “self-signed certificate” which provides equally good encryption, but no assurance that the server is who it claims to be.  But instead of recognizing self-signed certificates as being safer than no security at all, today’s popular browsers do their best to terrify and/or inconvenience users when visiting sites with self-signed certificates.  Certainly there is some value in authenticating the web server, but is that value worth the cost of allowing eaves-dropping and session hi-jacking on the vast majority of web sites?  I think not.

The current standard practice is backwards.  An HTTPS request to a server using a self-signed certificate offers encryption but not authentication.  This is clearly safer than a plain-text HTTP request, which offers neither encryption nor authentication.  But browsers tell users that self-signed certs are worse than unsecured communications.  (Chrome is actually worse than others.)  Deploying SSL on a commercial scale is also complicated by shared IP addresses for multiple sites, which again interferes with authentication, but not encryption.  The certificate verification UI already demonstrates varying levels of trust as shown below.  But self-signed certificates which offer encryption without authentication are incorrectly indicated.  Let’s remove the simple barriers which are preventing encrypted web communications.

The best technical path to fix this mess is immaterial here — many options exist.  Changing browser behavior to make self-signed certs less scary is one path, although it’s not a complete solution because of the legacy of every installed browser.  A new free service that signed anybody’s certificate with a trusted cert would work, provided that company had sufficient clout to get their root cert recognized.  (Google, you can do this.)  Empowering any domain registrar to sign SSL certs also makes sense since they’re the ones ultimately authenticating who owns a domain.  This choice wouldn’t immediately bring certificate prices to zero, but would greatly accelerate the trend we already see of lowering prices.  Perhaps a bloom-filter algorithm similar to what Chrome uses to identify malware sites could differentiate those sites whose identity has actually been verified through stricter measures, where self-signing should not be trusted.  A deeper technical analysis is needed to determine the best tactics, but clearly Google has both the necessary skills and level of influence needed to effect this change.

Additionally, Google uniquely has the motivation to make the web safer.  Google long ago recognized the value of primary demand stimulation — more web use means more web searches which means more advertising revenue for Google.  Open standards do not advance without leadership from selfishly interested parties.  The state of SSL certificates mirrors a political situation that desperately needs legislative intervention — a special interest group (the root certificate authorities) has a strong financial incentive to maintain status quo, even though every individual marginally benefits from the change.  Google is the company that stands to benefit the most from a safer web.  So please Google, act now to bring democracy to the safe exchange of information on the web by enabling anybody to freely secure their web traffic.

Obviously there are huge differences.  Chernobyl was a massive disaster that killed thousands of people, the only accident to ever reach level 7 on the International Nuclear Event Scale (INES).  When I started writing this article, Fukushima was classified as level 4, although that was before the containment building at reactor 3 exploded, and trouble really started in reactor 2.  I had written that it was likely to be re-classified as level 5, and now lots of people are saying they think it might end up as level 6.  I had written that I think it’s extremely unlikely to reach level 7 where thousands of people die from radiation poisoning, but the way things are going, I’m not so confident of that any more.  :(

For a decent explanation of the defense-in-depth strategies of the Fukushima reactors, read this overly-optimistic article.  This article has been widely distributed and republished because its “you’re all over-reacting” message is a nice one to hear and it comes from a seemingly credible source, a scientist at MIT.  But the article has an interesting past, originally including a major technical confusion, mixing up moderators which speed up nuclear reactions with control rods which slow them down.  This mistake was fixed fairly quickly, and then article moved to a new location hosted by MIT, along the way shedding its re-assurances that nobody would get any more radiation than from “a long distance flight”.  Clearly things are worse than that.  Nonetheless, Fukushima was built with many layers of protection, making a Chernobyl-scale disaster much less likely.  But things just keep getting worse there.

Fukushima faces the same problem Chernobyl was trying to fix

As we’ve all probably heard, the Chernobyl reactor exploded while performing an experiment.  The causes of the disaster are many, but most fundamentally the reactor design was unstable.  Relying on cooling water as a nuclear damping material gave the RBMK-style reactors a positive void coefficient meaning that as the water boiled from liquid to gaseous state, the nuclear reaction accelerated.  This is fundamentally unstable since it can create a positive feedback cycle, as it did during their fateful experiment.  The reactor heats up, which boils water, and since steam is less dense than liquid water there is now less nuclear damping material to slow the reaction, so it goes faster.  (Modern reactors don’t do this.)  In fact just 36 seconds after operators started the experiment, somebody hit the “Oh Shit” button (which unfortunately due to even worse design actually exacerbated the problem), and seconds later the reactor core tragically exploded.  Chernobyl’s core didn’t have time to melt — it just exploded.  Then large amounts of radioactive graphite burned in a hot fire which carried toxic ash high into the atmosphere.  Thousands got sick and died.

Despite what the Soviets wanted everybody to think afterwards (and even convinced the IAEA for 7 years), the motivation for the experiment at Chernobyl was wise and well-intentioned.  The operators were not insane, stupid, nor psychotic.  They knew that their reactor relied on the external power grid to run its cooling systems.  Of course they had backup diesel generators on site in case the power grid failed, but they also knew these generators could take up to a full minute to kick in.  That seemed like too long of a gap, so they were trying something creative — using the momentum in the plant’s own steam turbine to power the cooling pumps as the turbine was coasting down, unpowered.  They were thinking to themselves “hey, we’ve got this great power source, why don’t we use it to run the cooling pumps instead of relying on the external grid.”  Great idea.  They’d tried the experiment a couple times before.  It hadn’t worked.  This time it really didn’t work.  But because the reactor was so unstable when the experiment started that a slight decrease in cooling caused it to explode, not because the idea was flawed.

The heart of Fukushima’s problems are the same — the electrical grid around them was taken out by the earthquake.  They shut down their own reactions almost instantly after the quake, and thus were no longer producing their own electricity.  So to power the cooling pumps they needed to switch to backup power.  Unfortunately the backup generators failed, most agree due to the tsunami.

So Fukushima has this ironic problem.  They have an incredibly hot thing.  Even 48 hours after stopping the fission reaction, the core is still producing megawatts of decay heat.  Enough heat to boil 20 tons of water each hour.  They need electricity to run the pumps to cool down this incredibly hot thing.  But they don’t have any electricity.  There’s an electrical power plant (a device to turn heat into electricity) with tons of heat coming off of it, but they don’t have any power to run the cooling pumps, so it overheats.  Ironic, no? This irony was at the core of the experiment that Chernobyl was attempting — use the energy of the offline plant to run the cooling systems.

Safer designs are possible

In principal it seems you should be able to design a reactor that uses this vast quantity of heat (which is power — heat equals power) to run the systems needed to cool the thing off.  Fundamentally this is just an engineering problem.  Shouldn’t we be able to design something that can keep itself cool using its own energy even when disconnected from the grid?  Happily the answer is yes.  But sadly the answer was not yes in the 1970′s when these plants were built.  Not quite at least.

In fact, these old GE Mark I reactors do have emergency core cooling systems designed to help with this, but were never meant to be a complete solution, and clearly didn’t work.  New experimental designs achieve cooling completely passively without any need for active pumping.  But AFAIK these designs have never made it to commercial scale.

A major lesson of Fukushima is clear: extremely unlikely disaster events are highly correlated with each other.  So safety systems should not have external dependencies.   I believe nuclear power has an important place in our path away from fossil fuels towards renewables, but to get there, we need safer designs.

• Hosting any kind of service person – plumber, electrician, cable, etc
• Grocery shopping and cooking
• Gardening
• Dealing with house insurance
• Maintenance such as painting, roofing or windows

The key here is that the amount of effort required to do this for a large house with say 2xN people is less than twice the amount of effort required to do this for a normal house with N people in it.  In some cases it will hardly require any more effort at all for a large house.  But even for something like waiting for the cable guy, the amount of effort required will probably increase slightly for a large house — because the large house will require somewhat more cable services than a small house would.  But generally, the bigger house is more efficient.  My simplified representation was “effort = tasks / people” which is reasonably accurate for a number of useful cases.

There are some ways that economics of scale can work against you.  Specifically with utility prices.  Utilities like water get more expensive the more you use, as a way to discourage people from using more water than they need.  This works against you when you have many people living in a single house which the city classifies as a “single family house” and charges penalizing prices when usage goes above what they consider reasonable for a single family.  Right now, I recognize this as a limitation that I’ll just deal with because the absolute cost is not very high.

Another factor that scales badly is relationships.  That is to say, with lots of people around, there are many relationships to be maintained.  Every additional person you bring into the house forms a relationship with every existing house member.  Each relationship has a reciprocal pair — I have one with you, and you have one with me.  So the number of relationships in a house with N people is N*(N-1).  (This assumes your housemates are sane enough to not pick fights with themselves.)  If any of these relationships sour, then there’s a problem which can make the whole house uncomfortable.  For this reason, it’s valuable to pick housemates who are low-drama.  This table numerically lists the number of opportunities for drama as a function of number of residents in the house:

There’s another limiting factor in increasing the size of a house, which is decreased responsibility of ownership.  When a valuable object is owned by a single person or two people, they typically take very good care of it.  They know that if anything bad happens to it, they need to fix it, or deal with it being broken.  But as the number of owners increases, the sense of ownership and responsibility that comes with it diminishes.  At the extreme end of this are publicly owned goods like subways or parks.  As your house gets bigger, people will care less about taking care of it.  There are aspects of our house where we feel that we are bumping up against this limit practically speaking, and if we took more residents on, we fear the quality of life would degrade.

Here are the slides from the talk.

I start out talking about the need for keeping a web page up to date. I talk about polling as a natural but expensive solution to this. Then I talk about how COMET works, a.k.a. hanging GET or long polling. Then I talk about the difficulties of building a COMET stack from scratch and why you shouldn’t. Then I talk about moving to a higher level of abstraction with hookbox and what a publish/subscribe pattern is. Then I build a demo app using hookbox for a simple web chat. The source code for the web chat example is at https://github.com/leopd/hookbox-demo.

Although I’ve been talking about this kind of ideal since the 1990s, I am not nearly brave enough to try a life-defining social experiment like this without some evidence that it can actually work.  Fortunately, some of our friends are braver than me.  A few years ago we watched two couples both with pregnant wives buy a house together with the intention of raising their kids together.  It has worked out fabulously for them. They have been an inspiration and a model for many of us who have followed.  I put together this map  on the right to demonstrate how the idea has spread.  The green points show houses just like ours — where multiple unrelated / unmarried people have come together to co-own a large supposedly single family house (with a single kitchen) with the intention of raising their kids together.  The blue dots are houses of friends of mine whose that are very similar but don’t meet all those criteria.

I seeded this map with just my friends’ houses around Capitol Hill.  If you know of others and want to add them, feel free to go edit the Google Map yourself.  For security reasons, I haven’t included any identifying information about the houses, and have only located them as accurately as the closest intersection, and I encourage you to do the same.

The point of all this is to show that we might be crazy, but we’re not the only ones.  As another point of support, the map below comes from cohousing.org showing the locations in the greater Seattle area of larger planned cohousing developments.  Click through to find similar communities across the country.

The style we have is a single large house with with lots of people living in it.  Amazingly, this almost 7,000 square foot house is officially zoned as a single family dwelling.  I really like the single family who lived here before us, but I have a hard time envisioning how they used all the space.  This is the densest, most communal style of housing.  We effectively all share a single kitchen.  There is a second kitchen in the house, but  it gets used maybe once a month.  Whether or not you share a kitchen is a critical differentiator in the level of intimacy of a household. People need to eat every day, and so people are always going through the kitchen.  Sharing a kitchen means we’re always seeing each other and interacting.  If we had our own food storage / preparation areas, then we could and likely would spend far less time interacting with each other.

We also considered buildings which in many ways look and act like a single large house, but where each family unit has their own dedicated space, including a small kitchen.  This style allows for much more isolation and privacy within the house.  Not having to interact in order to eat means that you can spend much less time with the other people in your house.  I was originally a proponent of this style.  Partly because I think it makes for a more liquid ownership structure — if you can sell somebody what’s more like a condo unit in a fairly intimate condo building, the transfer is likely going to be much easier.  Now I’m glad I did not get my way because I love the intimacy of our household.  I know of groups who have purchased entire apartment buildings together, with some units dedicated as common areas.  This is an easy way to re-purpose an existing structure towards a co-housing  purpose.  A benefit of this strategy is that it’s easier to find people who will want to join, because of the reduced intimacy.

Going further in this direction there are a variety of ways to build sets of independent, nearby houses which are optimized for use as a community.  The website cohousing.org offers a bunch of pointers to communities of this kind, which are surprisingly common.  Houses with a common walk-way in the middle and a group meeting area with an industrial kitchen for example.  This style marries many of the advantages of owning your own house (privacy) with some of the advantages of living in a close-knit community.  This style works well for professional land developers, because it requires buying a large chunk of land and building lots of houses.  Some of our early plans explored a small-scale option of this kind, which again I’m glad we didn’t do because I don’t think as a group we would have survived the design and construction process.

At the far end of the spectrum there’s the option of literally just buying existing single-family homes near each other.  My previous house was within a few blocks of a great many of my friends.  This is a traditional neighborhood, but done right if you’re actually good friends with your neighbors.  I also know a group of folks who bought a set of houses which are literally adjacent to each other, making it much more like the planned communities above.

When considering the options here, the basic trade-off I see is between intimacy and privacy.  It’s tempting to say that more privacy increases re-sale value, but I think it’s more accurate to say that more privacy makes the investment more liquid.  Intimacy brings all sorts of social benefits, and one of the largest determinants of intimacy is the extent to which you share a kitchen.

Our house actually operates on consensus for most decisions. Operating on consensus is short-hand for everybody has to agree before something happens.  Another way to put this is that everybody has veto power over everything.  It is this fact which most leads to the slowness of decision making that I alluded to.  It can take a long time to reach consensus on issues.  But people are generally happy when they do.  The biggest source of stress is often that things aren’t moving quickly enough.  This leads me to joke sometimes that an issue is “working its way through congress” before it gets decided, which I think contributed to me mis-representing the governance system that we use.

We do have a separate politburo-style committee which is responsible for financial decisions.  For issues like when to refinance it makes sense for only certain members of the household to contribute: those with a direct vested interest in the outcome.  Maintenance and repairs of the house similarly get dealt with in this sub-group, not because other residents don’t have a vested interest, but because it’s our responsibility and we generally figure the other residents would rather not deal with things like hiring a painter.  Even if they did, their incentives would differ slightly.  Sometimes meta-issues around residency like how many people the house should have sometimes get taken up by the politburo, but we do our best to keep these discussions open.

I know of other group houses which operate with similar multi-tiered governance systems.  The hierarchy often seems to follow legal ownership of the house, which makes sense.  Sometimes more power is reserved by the owners.  Clearly there’s a continuum of possibilities here which would get unhealthy on either end.  A strict dictatorship by the owner would probably make all other residents unhappy fairly quickly.  On the other side a house where the owner has no more power than the other residents, and gets out-voted on issues pertaining to physical maintenance could lead to the house falling into dis-repair.  I’ve heard that the Evergreen Land Trust model sometimes has this problem.  ELT is something I don’t know very much about, but deserves its own write-up.

One closing comment about house governance relates to communication.  When decisions need to get made, how will your house communicate the discussion?  We use a combination of an email list and periodic in-person house-meetings which are fairly formal and infrequent.  I know other houses rely fairly heavily on SMS, or chance discussion.  As in most things with co-housing, there are many right answers.  The key is finding a system that works well for everybody you live with, and being open to change if it seems not to be working.

The topics I touch on are:

• Raising kids in a group house
• Choosing your housemates
• How to deal with somebody needing to sell their share of a house
• Hiring a lawyer to write a Tenancy in Commons contract
• How to get a crazy loan
• Living with lots of people
• Governance systems for a house
• Capitalist vs Communist chore systems
• Gamifying chores
• Hiring a housekeeper
• Economies of scale in a group house
• How cooking scales up
• Sharing food in general
• Limitations of accounting
• Letting go of control
• Mis-behaving furniture

And when I say “touch on” I mean it.  Each of those topics are lucky to get a full sentence in my 5-minute talk.  There’s so much more I had considered including, but with an Ignite talk, you’ve got to make tough choices about what gets included.  I could write an entire blog post about each of the topics above, and I just might.  (Leave a comment if there’s something in particular you’d like to hear more about.)  Here’s the list of topics I had included in earlier drafts of this talk, but all got cut before the final version:

• How relationships scale in a big group
• What does privacy mean, and what really matters
• Analogy to college dormitory lifestyle and its limitations
• Personality traits to seek or avoid in co-housing partners
• Social vs. Legal Contracts and what belongs in each
• Balancing preservation of house sanctity vs. owners’ rights in contracts
• Financial ownership models and associated accounting techniques
• How living in a group can minimize interpersonal differences
• Wanderlust in desk accessories
• Analogy between marriage and co-housing
• Personal efficiencies through livability sustaining systems
• Techniques for dealing with clutter
• Architectural features that support group living

And each of these could easily get a 500 word essay as well. Encourage me, and I’ll write them! :)

The simple way to counter buffer bloat is simply to reduce the size of the transmit buffer in each piece of your network gear.  Most linux systems default to a transmit buffer of 1,000 packets, each of which can be 1.5 kilobytes, meaning that 1.5 megabytes of data can get queued up waiting for a chance to go across the network.  Any application that is trying to move a lot of data through a clogged network will fill this buffer.  That’s fine for the buffer-filling application, but any other application will suffer.  So, for example, if you’re watching youtube and your roommate is trying to surf the net, your roommate’s web page requests will suffer very long latency, because their small web pages must get in this megabyte-long line along with your youtube video before they can be delivered.  If your DSL line runs at say 10 mbps, then it’ll take 1.2 seconds for that 1.5 MB buffer to work fit through your pipe.  Since it takes at least 2 round-trips to get a web page that means your roommate’s web page will take at least 2.4 seconds to show up, no matter how small it is!

Gettys quotes Kleinrock that the ideal size of a network buffer is (bandwidth) x (latency).  Say your bandwidth is 10 mbps.  Latency to any web page you’re likely to visit in the US should be less than 100ms, so let’s use that.  This puts your ideal buffer size 125k.  Buffer sizes are usually configured in terms of maximum number of packets.  Typically the maximum packet size (MTU) is 1500 bytes, resulting in the ideal theoretical buffer size of 83 packets for a typical fastish home network line.  Please redo these calculations yourself and experiment with how different numbers affect your system. (Be careful not to set your buffer size to zero as it could lock up the device’s network.)  Remember that linux (which is likely what your wifi router is running) defaults to 1,000 packets!

[Update shortly after posting: a reader suggested I try setting my buffer to be much smaller still.  So I went down to just 2 packets, and noticed that my ping times are much more reliable now when the network has more than one thing.  His caveat which I will echo is that this will mess with your system if your router is trying to do any kind of traffic shaping, i.e. QoS.  But otherwise protocols like TCP will keep everything running fine.]

If your home wifi access points are using DD-WRT as mine are, here’s how you set them to use a more sane buffer size:

1. Log in to your router’s admin web page.

2. Select the Administration tab and the Commands sub-tab

3. Type in the following commands into the box:

ifconfig eth0 txqueuelen 2

ifconfig eth1 txqueuelen 2

4. Click the “Save Startup” button at the bottom.

There — you’re done!  For alternate techniques to configure your dd-wrt router for this kind of thing, see the wiki page on Startup Scripts.

I’m sharing this information because it took me a while to figure out.  This problem is not well documented.  I’m trying this out now on my house’s network now.  In some controlled tests it seems like it might be somewhat better.  But my tests have not been able to replicate the really horrible situations I’ve seen on our network which I suspect come from lots of simultaneous users.  So it’ll be a while before we know for sure if this was a good change.  To be clear, I don’t know if this advice is good or not. It could reduce your network’s maximum effective bandwidth, but hopefully it will do so by reducing the maximum latency, which is often a very good trade-off.  This advice is consistent with the advice Gettys offers in terms of optimizing buffer sizes, and make sense to me.  YMMV.  If you try it out, please leave a comment on whether or not it helps you.

Hello Board of Directors,

I am writing to express my deep concern with your active support of Estrella Creamery.  I have been a long-time supporter of yours going back to the central co-op days.  But the issues with Estrella have me currently avoiding your store.

I see two fundamental values of the co-op at odds here.  The co-op wants to support small and local food producers.  Great.  But the co-op also has a mission to provide healthy products to its shoppers.  Estrella has a very troubling trail of documentation showing that their food has not been reliably safe.  But the co-op is choosing to place more weight in the local/community value than the health value.  I find this extremely troubling.  Having a pregnant wife, I refuse to shop at the co-op until this issue is resolved satisfactorily.  In fact I am actively discouraging my friends from shopping there as well.

I have heard many weak excuses for why Estrella’s food isn’t that dangerous.  For example “nobody has been proven to have gotten sick from our food” is a claim they’ve made.  This does little to bolster their case — tracking food-borne illnesses is difficult even for large national producers where dozens of illnesses help mark the trail.  Additionally it demonstrates naivete in how risks should be assessed.  It is analogous to saying “I never wear a seatbelt but I haven’t been hurt in a car accident.”

I’ve also heard arguments to the effect that people should have the right to choose to eat high quality but dangerous food.  That is a fine argument.  But it goes directly against the co-op’s mission to provide healthy food.  I loved the fact that the co-op avoids stocking food with high-fructose corn syrup and trans-fats.  I thought that by shopping with you I could spend less time worrying about what I was putting into my family’s bodies, even if it cost me a bit more.  Your vocal support of Estrella has destroyed my trust in your store to put my health first.  Clearly healthfulness is just one of several competing priorities for you.  I wish it weren’t.  Health is far more important to me than supporting the artisan lifestyle.

Please recognize that your own stated values are in conflict here.  Whether you realize it or not, your store is at a turning point in defining itself.  You are likely to lose customers either way you decide.  But I strongly encourage you to do so deliberately, with a sense of purpose.  Selfishly, so that I may again have a great store in the neighborhood that I trust, I hope you choose health.

Sincerely,

Leo Dirac

It gives an introduction to South, which is the most popular database migration tool for Django. If you’re using Django on any kind of mature project and you’re not using South, you should really take a look at this. Without a framework like South you’re probably very scared of changing your model schema and making your life a whole lot harder than it needs to be. Without further ado…

When: Tue, Dec 14th 6:00pm – 7:30pm  (Meetings are on 2nd Tuesday of each month)

Where: Web Collective, 1402 3rd Avenue – Suite 925 – Seattle

(3rd and Union, 9th Floor, Vance Building, kitty corner to Benaroya Hall)

Please try to be here *by 5:55pm*, since the doors lock and we have to let you in
after 6.

My talk will be an introduction to South Migrations in Django and how you can use them to change your database schema on the fly. You don’t have to be afraid of changing your models after you’ve started using them.

I’m looking forwards to seeing everybody there!

Incidentally, if you’re consider rails for a new project, my post on whether Django or Rails is more popular has been getting a lot of interest lately.

Here’s the preso.

Here’s how that math works, Kurzweil explains: The design of the brain is in the genome. The human genome has three billion base pairs or six billion bits, which is about 800 million bytes before compression, he says. Eliminating redundancies and applying loss-less compression, that information can be compressed into about 50 million bytes, according to Kurzweil.

About half of that is the brain, which comes down to 25 million bytes, or a million lines of code.

This reasoning is IMHO flawed and overly optimistic.  It’s an interesting idea to compare the complexity of these two systems by comparing their bit representations.  I think the idea has merit, at a very rough level — that is I think you can compare the complexity of a genome to the complexity of a piece of software on a rough order-of-magnitude scale. The biggest flaw in Kurzweil’s argument is that he magically throws in a factor of 16x improvement in his favor by saying the genome can be “compressed.”  Well, software executables can be compressed too, a fact that Kurzweil conveniently ignores.  So I’d follow his reasoning to say that a human brain simulator probably needs about 10 – 100 million lines of code.  (I’m deliberately including 0 significant digits here to indicate the roughness of this approximation.)  This puts a human brain simulator on par with the some of world’s most sophisticated software projects so far, which seems about right, at least to an order of magnitude or so.

Strong reactions

PZ Myers published a wrathful condemnation of Kurzweil’s argument titled “Ray Kurzweil does not understand the brain.”  If you sift through the name-calling you see that Myers assumes a specific tactic in building the brain simulator: starting with the human genome and deriving the brain’s functionality from it.  This strategy will certainly work, once we have solved the protein-folding problem, and more generally have the ability to do quantum chemical simulations of kilogram-sized masses of organic chemicals.  Which is to say it’s theoretically possible (we might be living in a software simulation of our universe for all we know), but completely intractable with current technology.  For comparison, our best quantum chemical simulations if you push them top out at maybe a dozen atoms right now.  So being able to simulate an entire kilogram of organic matter is nowhere in sight.

Tactics to simulation

I agree with Myers that we are nowhere near being able to interpret the genome well enough to understand how it makes a brain.  But we probably don’t need to in order to simulate a brain.  By analogy, consider the Super Nintendo (SNES) Emulator, which is another kind of simulator many of us have experience with.

SNES emulators let you play all the old Nintendo games but on a modern computer instead of original SNES hardware.  Let’s say somebody handed you a box and a stack of cartridges and told you to build a Nintendo simulator.  What would you do?  Well, clearly you could open up the SNES box and reverse engineer the circuit boards to figure out all the wiring.  You’d probably figure out that the CPU was important — a variant on the 65816, which was essentially the 16-bit version of the 6502 some of us grew up with in our Commodore 64s and Atari 800s.  So you could (theoretically) crack open the 65816 CPU chip itself, put it the through an electron microscope and understand every transistor it used to interpret the instructions. In this way you could reliably create an emulator which completely replicated every aspect of the SNES. Such a simulation would replicate all of its bugs, timing quirks and everything, but it would work and be extremely expensive to simulate.

This is analogous to the tactic PZ Myers seems to be assuming Kurzweil would take to simulating a human brain. But Kurzweil would actually start at a much higher level of abstraction.  Simulating every protein in every neuron is like building an SNES emulator by simulating every transistor in the original Nintendo’s hardware. The key to getting those SNES games to work does not lie in replicating the design of the CPU which interprets the instructions.  The key is figuring out how to run those instructions on modern hardware.  By moving up through levels of abstraction, we can simulate the system much more cheaply and easily, although there’s a chance edge-case behavior won’t be captured properly.  (What if our world is a simulation and we bump into the edge-cases?)

Similarly, the key to simulating a human brain anytime soon does not lie in understanding every chemical pathway in human neurons.  Although if we did understand neurons at this level, we would have a great head start at simulating a brain.  Success in simulating a human brain will come by recognizing higher levels of abstraction in neuronal function.  We have known for a very long time that neurons communicate by “firing” electrical signals which are transmitted chemically at synapses.  The details of these behaviors are complex and determined by a great many interdependent chemical systems, but it seems highly likely that we can replicate the key behaviors of human neurons at this level of abstraction without needing to understand everything underneath supporting them.  If we can replicate the firing behavior of neurons in sufficient detail, we don’t care what the proteins underlying them are doing.  The key question here of course is what is “sufficient detail.”  I expect that question is one that researchers who are genuinely interested in reverse-engineering the brain will actually focus their attention on.

Once we can simulate the firing behavior of neurons, simulating a brain becomes much more of an engineering problem than a scientific one.  Still it’s going to be a massive engineering challenge, and gathering the input data will probably require a bunch of new science.  Then the philosophers can debate the meaning of free-will if our brains are Turing-complete.

To get across the chasm, your technology really needs to work well.  Apple seems to have done that with FaceTime.  But there’s more than just having it work — you also need to explain to the public why it’s important.  Here, Apple is just paying for that with traditional advertising.  They’re putting out lots of touching, heartwarming commercials showing the value of video chat.  Apple is spending millions of dollars to explain to people why they’ll like video chat. Primary demand stimulation.  They’re working to overcome people’s biases against  the technology or the idea — that it’s clunky, or the extremely common “I don’t want to see the people I’m talking to” reaction, which is really pretty funny when you think about it.  Of course, there will be some times when you will prefer audio only, but that’s going to be the exception when the technology is good enough.

It’s all playing out in a textbook fashion.  The geeks are all crying that there’s nothing new here, that this technology has been around forever, and they don’t understand why it’s such a big deal.  But it is a big deal, because video chat is finally entering the mainstream.

Some side-effects of this that I pointed out before, but are perhaps more clear now as the story unfolds, are that this will benefit existing established video chat vendors.  Apple is explaining to people that video chat matters.  This will help Skype and Google and Cisco with their products.

Speaking of Cisco, there’s another prediction coming true: Cisco is pushing into consumer video chat.  I had guessed 2012, but barely more than a week after my last post, they announce Cius, a video chat terminal.  Kinda like a Flip phone fused with a linksys router, but running Android and in a pretty nice looking case.

Video chat is in a similar place today to where mp3 players were 10 years ago.  There are lots of video chat solutions out there on the market.  Skype is the most well known.  I helped launch Google’s video chat system across Gmaill, iGoogle and Orkut during my tenure there.  It definitely is one of the best on the market, and it’s still only appealing to early adopters.  I mean “early adopters” in the classic Everett Rogers sense, which is to say folks for whom the extra value of the new technology outweighs the hassles of using it.  This is a step beyond the “innovators” category, who are willing to bend over backwards debugging a brand new product just because they understand that it will be important later.  Video chat has been available to innovators for a great many decades.

With iPhone 4, Apple will push video chat to the early majority category. Apple has a history of sitting on potential technologies until all the bugs are worked out, which is fundamentally what’s needed to appeal to more than just early adopters.  I’m pretty sure FaceTime will be no exception.  In 6 months, video chat won’t be this geeky thing that people put up with out of desperation.  It will begin to be integrated into normal culture.  We’ll start to see television dramas and movies incorporate it as just a way some people communicate, rather than as a way to demonstrate how high-tech somebody is.  People who aren’t geeks will start to use video chat.

Video chat really matters

Those of us who have lived deeply with video chat understand its value.  There is a ton of additional content transmitted in video that helps communication on many levels.  It allows for a more nuanced informational discussion, but more importantly IMHO, it allows distant communication to be much more personal and emotional.  Anybody who has tried professional collaboration with another team that is thousands of miles away knows that this level of communication is at least as important for business uses as it is for social communication.  The first time you meet your collaborators in person, they become more real, more trustworthy, easier to talk to, especially about difficult subjects like problems that might arise in a project *gasp*.  Video chat is certainly not as good as meeting people in person, but it is a huge step above email, IM or phone.  (Getting drunk together I believe represents the highest professionally-accepted level of humanization.)

Human-to-human communication has always been the killer feature of computer technology. Video chat makes synchronous communication fundamentally better, and as such will become a major part of everybody’s life in the developed world in the years to come.

What does this mean for everybody else?

The history of technology innovation tells us that a couple things typically happen when an emerging technology pushes into the majority segments.  First, established players will all get a boost.  Apple will be doing a huge favor to Skype and Google Video Chat by removing the veil of geekiness from their products.  Apple’s huge investment in making this product work well will make all consumers more willing to try alternatives.

Another common side-effect is that the space will get more difficult for new entrants.  This usually happens as the technology standardizes.  There becomes a “normal way of doing things” that people start “to get.”  Before a technology can reach the majority, it will typically bounce around dozens of different modalities as everybody tries to find a way of doing it that resonates with the market.  This uncertainty represents a clear opportunity for start-ups and the subsequent standardization is the closing of that opportunity.  Another reason the space usually gets harder for startups is that economies of scale start to kick in as production levels ramp up to meet the larger demand.  This naturally favors large companies, since it raises the amount of investment needed to compete.

Which Social Graph?

Another reason the space might get harder for newcomers is the natural monopoly of social graphs — consumers are better off if there is a single definitive place to keep track of their contacts rather than having to replicate and maintain a different list for each service.  As such, social graphs are important assets to anybody in this space.  But if Tim O’Reilly gets his way (which I hope he does) and we end up with a loosely-coupled internet OS, this won’t be a problem for startups, as they’ll just be able to draw from an openly available graph, say from a Google or Facebook.

I’m really not sure how this aspect will play out.  My guess is that Apple will rely on the de-centralized social graph which is the contact list built into every iPhone.  It’s a less useful corporate asset than if it were properly cloud-hosted, which will make it harder for them to expand the service to OS X machines.  Perhaps they’ll make something useful out of mobileme here, but I have my doubts.  But given the revenue they get from App Store sales, it’s not clear that the OS X machine is even a major part of their consumer strategy going forwards.  If so, this would likely be a strategic shift for them, as the inclusion of web-cams on essentially every OS X machine for years was probably done in anticipation of making a major push into video chat at some point.

Don’t forget Cisco

In addition to the obvious players like Google and Skype, this is also incredibly important for Cisco too.  Cisco has long been interested in video chat.  Why?  The same underlying reason Intel has been investing in multi-media since the late 1980′s.  Multimedia on PCs needs lots of CPU power, and video chat needs lots of bandwidth.  It’s called “primary demand stimulation.”  To be very clear: Cisco wants everybody to use video chat because video chat uses lots of bandwidth, and when people are using lots of bandwidth, Cisco sells more big routers.

Cisco is in the final phases of buying Tandberg, who is the biggest supplier of video-chat hardware for businesses.  Their aquisition of Flip last year seems strategically odd in isolation, but in this context makes perfect sense.  They are building (buying) expertise in consumer-electronics which can handle high quality video.  Take a Flip camera, add a network (like a linksys wifi box) and you’ve got a video chat terminal.  I predict we’ll see such a toy out of Cisco in about 2012, as video chat fills the early majority segment and edges against the late majority.

I set about answering this question with data.   I quickly found a list of oil spills on wikipedia.  A quick pass through google spreadsheets and a few regexs later, and I’ve got the data in a form that it can be graphed with protovis, a wonderful web-based visualization package.  An initial look shows some interesting trends. (Sorry IE users – modern browser required.)

First, it’s important to emphasize that this is a log-scale graph.  Given the dynamic range of the input, it’s the only reasonable way to visualize what’s there, but if you’re not used to reading log-scale graphs, the data will be deceptive.  In short, being a little higher on the graph means that the spill is a lot larger.  In fact if would be very reasonable to only include the spills near the top of the graph when thinking about “big spills.”  But I wanted to present the entire data set for completeness and analysis.

It’s interesting to note the general downward trend at the bottom of the graph.  I believe this is not a real effect at all, but a result of selective memory.  The smallest spill on this graph was only a couple months ago — the Great Barrier Reef spill in April.  Are we to believe that in the preceeding 100 years of oil exploration there had never been a spill of less than 10 tons of oil, and only a single other spill of less than 100 tons?  Of course not.  I bet spills of this size have happened dozens if not hundreds of times, but 50 or 100 years ago nobody bothered writing them down.  Or if they did write it down, the event has been filtered out of our collective historical memory before making it into wikipedia.  The Lakeview gusher in 1909 is another interesting example of this effect.  This certainly wasn’t the only oil production accident before 1930, but it was clearly an important major accident, and so has been remembered far better than others.

I’ve highlighted a few other spills because of their historical interest.  The Gulf War oil spill (purple dot) of 1991 is exceptional in that it was not an accident, but a deliberate act of war.  As such, it should not be considered in answering the question of whether oil exploration has been getting safer.  The Exxon Valdez spill (light blue dot) in 1989 is large in our memory, but in context we can see that it was not at all a large spill by historic standards.  But clearly the Deepwater Horizon spill (green dot) is huge, ranking as one of the largest spills ever and certainly the largest spill in quite some time.  But aside from this current mess, there does seem to be a real trend towards increased safety in oil exploration.

Again, the log-scale graph makes this somewhat hard to read intuitively.  Because the spills near the top are so much larger than the ones below them, a fair approximation of the sum of all spills can be found by simply considering the points along the top envelope, which is generally decreasing.  Looking just at the last several decades on a linear scale, this trend becomes more clear: since about 1980, serious oil spills have been getting smaller / less frequent. Now we see visually that the majority of spills listed are tiny compared to the few big ones.  I scaled the graph so only the bottom of the uncertainty bar for the gulf war oil spill.  Also note that I’ve kept the middle dots at the geometric average of the low and high estimates, which works visually on the log-scale graph, and makes logical sense given the nature of the problem.

Another factor to consider is that the total amount of oil being produced during this time period has been generally increasing.  I’ve overlaid data from the US Energy Information Administration on global oil production rate, scaled to the average amount produced each hour, to get it to show up on the same scale of this graph.   Another interesting comparison which I haven’t included is the average size of each well, or the number of wells being drilled per unit time.  My understanding is that oil exploration has been getting more difficult over time in that we’re having to drill deeper to get at relatively smaller oil deposits.  Again, this reinforces the idea that we have been getting better and safer — we’re spilling less even though we’re drilling more holes.  Except for the Deepwater Horizon.

Feel free to browse the javascript source code of the graphs for further details, inspiration, double-checking, or whatever.

Having been writing code for 25+ years, the differences between programming languages fade away in my mind.  I know that I, like any good software engineer, can be productive in basically any language.  Certainly within object-oriented languages, (which is where essentially all serious software engineering happens these days) the differences in the language itself are IMHO small compared with other factors in choosing a platform.  Other important factors I consider include:

• Quality and availability of libraries.
• Quality of tools like IDEs, debuggers, automation systems.
• Size and healthy of the active community using the framework.  (i.e. If I run into a problem, how easy is it to google the answer?)
• Ease of hiring people who already know the platform.

All of these considerations are IMHO more important than compiled vs. interpreted or run-time performance or whether the language is statically typed or dynamically typed, or even if it’s open source vs. proprietary technology.  But there is a single common factor which directly feeds into all four of the criteria I list above: How many people are actively using the platform?

The Candidates

The last time I was really writing much code was back in 2007.  At the time Ruby on Rails was the coolest thing since sliced bread.  I played around with it, was amazed by how easy it was to quickly put together simple database-driven websites, and got very frustrated with it when I wanted to color outside of its lines.  A major concern of mine was the level of “magic” that happens behind the scenes — this makes Rails beautiful and elegant when it works, but difficult to debug or extend.  I blogged about my experiences with rails and in particular my conclusion that at the time Rails was not ready for large, complex projects, partly because of a lack of good tools, libraries and sensible error messages, all of which can be fixed by more users.

Around then I started working for Google and stopped writing code.  The next year or so saw a couple big things happen for Ruby on Rails.  Twitter famously had trouble maintaining even 99% uptime (fail whale anybody?), and everybody knew they were running on Ruby on Rails.  Also, Google launched AppEngine, which supported Python, not Ruby, and pointed people like me to an alternative high-level web framework: Django.

So now, in 2010, I return to the fray, and I’m trying to decide between the two frameworks.  There are of course other alternatives, but for the purpose of brevity, I’ll leave out my process of reducing my choices to these two: Rails or Django?

[Update: Commenters point out correctly that PHP has its own MVC frameworks, the most popular of which I believe are CodeIgniter and CakePHP.  While PHP is vastly more popular than either Django or Rails, none of its MVC frameworks come close in terms of popularity, and raw PHP is not comparable in terms of sophistication.]

Measuring community activity

I often use Google Trends to measure relative interest in technologies.  My time is very valuable, so even downloading something and reading its documentation is an investment I’d rather shortcut if I can.  This crude measure of relative search activity can actually be quite telling, and has saved me a bunch of time in choosing packages.

The relative values are hard to take literally since “django” as a single word search query will naturally be higher than a 3-word query like “ruby on rails”, but the search for “rails” by itself will clearly have lots of irrelevant searches.  Likewise, some people searching for “Django” won’t be looking for the web framework, but rather the guitarist or the movie or what-have you.

But it is clear that search activity for Ruby on Rails peaked in 2007-2008 and has been declining since then, while Django has been on a steady upwards trend.  [2012 update: We shouldn't consider this too strongly though since the Django trend is probably due to non-python versions of the term, and honestly both have been pretty flat in recent years.]

Elsewhere I can find evidence that Rails is still a more used platform.  Stackoverflow has twice as many questions about Ruby on Rails vs Django: 13,882 versus 7,496.  To me this indicates pretty clearly that Rails is more active than Django.  Either that or Rails is more confusing and people ask more questions about it, but I doubt that.  [2012 update: now the numbers are 66,845 vs 29,570, continuing the trend.]

The Tiobe Index attempts to objectively measure activity across programming languages.  By its easily manipulated measure, Python has twice the activity of Ruby, independent of the django / rails frameworks, with both languages in decline.

Meanwhile it’s easy to find comparisons on the net between the two.  Everything from content-free videos to people declaring rails dead, and everything in between.

On balance, I find the stackoverflow numbers the most compelling, unbiased indication that Ruby on Rails has more activity than Django / Python.  Even though I’ve been frustrated by it in the past, by my own objective criteria, that seems to make Rails a better choice for building a new web application.

What’s your opinion?

If you have experience with both frameworks, I’d love to hear your experiences.  Please leave a comment.

Simultaneously, technological advances have allowed advertising to progress along a different axis – to become more targeted.  Advertising used to only be broadcast widely through newspapers and television shows.  The best an advertiser could do to ensure their message reached the right kind of people was to select the aggregate demographics of everybody who read a particular magazine.  Now the internet allows ads to be targeted as precisely as you’d like.  Today, Google lets you get your message only in front of people who are about to buy a product like yours.  The ability to connect to people who have expressed an intention to “buy digital camera” is a literal gold-mine, making billionaires out of Larry, Sergey and Eric.  As effective as it is, targeted advertising won’t replace broadcast advertising, because there is still value in abstract brand-building.  Rather, the two will complement each other.

Enter Social Media

Social media has been all the buzz recently.  At its core it’s just a more convenient way for friends to communicate.  The “killer app” for computers has always been helping people communicate, and this is just another chapter in that book.  With this new communications medium comes a new opportunity for organizations to tell their stories.  In fact, I believe that social media will bring another tectonic shift in the entire marketing industry, possibly as important as search-based advertising.  As consumers have gotten more and more sophisticated at filtering out advertising from broadcast media, advertisers have gotten more and more desperate in their attempts to connect with people.  Social media marketing offers a new path – instead of hearing about products and services through ads, people can hear about products and services from their own friends.  Exactly how this will play out through Twitter/Facebook/Foursquare/whatever is not at all clear to me right now, but I fundamentally believe this change is coming, and it will take the entire marketing industry with it.  Klein and her fans are free to unplug from popular culture in order to avoid the onslaught of brand advertising, but they would be foolish to stop talking to their friends just because their friends are happy with things they’ve bought.

This vision is one of the main things that prompted me to jump off the comfy Google cruise liner and start paddling hard in Banyan Branch‘s crowded dinghy.

Is marketing intrinsically evil?

I sometimes feel a need to justify this line of work to those who think that marketing is inherently dirty.  I admit that I’m more of a capitalist than many of my friends, but I certainly recognize that capitalism has its limits.  The vast majority of economic transactions are both consensual and mutually beneficial, and I will argue vigorously that there is nothing wrong with an economic system consisting of these transactions.  The biggest exception to this happens when transactions are not mutually beneficial because one party is not fully informed.  But what we’re doing is helping people share honest opinions and feedback about the things they buy and use.  By lubricating the flow of information between real people, I believe social media will reduce the effectiveness of deceptive marketing.  Moreover, it will help companies connect to their customers and hone their goods to people’s real concerns and desires.  It will help hold companies accountable for their mistakes, and enable companies to better make things that make people happy.

Additionally, I will point out that my employer represents no small amount of “pure good” for the world, including organizations such as The Bill & Melinda Gates Foundation and Vittana, helping them tell their stories.

Taking a chance on a startup

Why did I choose this opportunity out of the sea of possibilities?  I evaluated the landscape as an investor would, since I am investing no small chunk of my life in this effort.  From my entrepreneurial training and experience, I know that smart investors care more about the people than the specific business plan.  The plan will almost certainly change, but the key management will not.  Having known one of the founders of Banyan quite well for a number of years, I am certain that many key elements for success are in place.  The corporate culture and governance will be solid.  I will be working in an environment where I am supported, and where I can learn and grow as a manager and a technologist.

Exactly what will I be doing or building?  I admit I’m not sure yet, but I have some very interesting ideas that I won’t be sharing here anytime soon.  I am sure that my work is very well positioned to be a part of a major shift in an entire industry — a rare opportunity.  Whether or not my work will play a key role in this shift is somewhat out of my hands — these things are always a roll of the dice.  But in another sense, it’s entirely within my control, and this is what I love about working in a small company.  There’s almost nothing but work between me and effective execution of our ideas.  Many people tend to exaggerate the importance of the idea itself, forgetting that it is incredibly important to execute well on whatever ideas you have.  I’ve heard people say that they had the idea for YouTube years before YouTube did.  How quickly we forget the dozens of other companies all working on the same problem in 2006, which almost all fell by the wayside because they didn’t execute as well as YouTube did.  Ideas matter for sure.  But hard work is critical.

If you’d like to jump on this raft and start paddling too, get in touch with me.  I need a few key rock-star developers who are’t scared of chaos and can think creatively about business problems.

I was patiently waiting for the feature to be turned for my account, which wasn’t happening.  I kept getting the same old spreadsheet program.  But then I found the setting to enable it.  So I’m sharing with y’all an explanation of how you too can get these great new features.

Go to http://docs.google.com/ and click the select “Settings” link in the upper-right hand corner, and choose “Documents Settings”.

Then choose the “Editing” tab and check the checkbox next to “New version of Google documents”.

Now any new documents you create will use the new real-time editor.  And anybody you share them with will get the new features without having to set this up for their account.

In Walmart’s case, it’s generally suppliers who get squeezed.  Walmart demands that manufacturers of goods produce them at the lowest possible price so that Walmart can charge the lowest prices in their stores.  They really do try hard to pass the savings on to you.  Another case that is less well known is with so-called “interchange” fees for debit and credit cards, charged by the card networks like Visa and Mastercard.  Back in 2003, Walmart pushed hard on Visa and Mastercard to charge less for debit card transactions since they are both lower risk (because of pin-code use) and cheaper to process (verifying signatures is expensive).  The cynical will point out that with lower fees, Walmart just gets to keep more profit.  Which is true.  But they are genuinely motivated to lower prices for consumers, since that’s their main selling point.  So it’s a win-win – Wal-Mart’s motivations to lower costs are closely aligned with consumer’s desires to pay less.

Apple has similar desires for their network-connected gadgets like iPhones and iPads.  Apples wants people to be able to connect their devices to the network for as little as possible. Apple has clearly negotiated very hard with AT&T to demand low monthly rates on data plans for these devices.  Next month you’ll be able to buy an iPad with a 3G data plan for just $15 / month.  That is basically unheard of in the US.  For people on a limited budget, the iPad is the cheapest way to get online.  Compare this to other data plans available from major U.S. carriers:

The Apple / AT&T rates are the lowest in each of their categories, except Verizon’s smartphone data plan which ties the AT&T iPhone plan.  The iPad rates are extremely low compared to data plans for laptops, and also when when you consider that tethering plans or phone data plans require paying an extra $30/mo – $50/mo for a voice plan.  The unlimited iPad plan is literally half what it costs to get 3G on any other laptop, and it doesn’t come with the 5 GB limit that other plans do.  You might argue that the iPad can’t do as much as a full laptop, which is true.  So you might then argue that iPad won’t tax the network as much as a laptop, which I doubt considering the propensity to consume video on such a device.  So you can’t trade torrents on an iPad, which from an Intellectual Property perspective is just fine with me.

My guess (and this is pure speculation) is that Apple negotiated these rates by offering AT&T a share of the revenues generated through App Store purchases.

Again, the cynical will point out that Apple is just trying to grab the lion’s share of economic surplus for itself, which is true.  But nonetheless, this is a case where Apple’s desires and our desires as consumers line up well.  In a very real way, Apple is fighting on our behalf for lower prices from AT&T.

Apple products are expensive. Apple gets high margins on its hardware, allowing it to recoup large investments in NRE (non-recurring engineering) to design the hardware and its accompanying software.  This is a great place to be from a competitive standpoint, because as a company they don’t need to squabble over the cheapest parts to try to deliver the best prices to consumers.  So long as they can maintain a sufficiently large customer base to support the practice, it is an easy place to defend against competition from.  Certainly a lot easier than being Dell or HP, who struggle with operational efficiency to compete on price, and try to innovate within a very narrow window defined by their platform.

Apple’s success at selling high-end products has secondary benefits for the rest of the ecosystem.  Because the products are expensive, they tend to be purchased by people with more disposable income. So the segment of the computer market which buys Apple products self-selects to be very attractive demographic for many other reasons.  Advertisers love to get their products in front of people who are more-willing-than-most to buy something expensive / unnecessary / fun.

Similarly, app developers know that if they write an app for iPhone / iPad, the people who are able to buy it are much more likely to be willing to pay a couple bucks for something silly than, say, somebody who bought the cheapest smartphone they could afford because they felt they really need that functionality.  I had previously speculated that Apple’s platform play required a very large distribution base to attract developers, which is not quite correct.  The strategy is successful even with a relatively small market, provided that the market is segmented properly.  Which in this case it clearly is.

Preparing for this talk has been a lot of fun.  I’m guessing it will be fun to deliver as well.  (I’m writing this to be posted immediately after my talk, so I can’t know for sure yet!)  The whole process reminds me how much I love my career — the huge impact I can have on making people’s lives better.  This particular talk was a very good reminder to me how much I rely on my training as a scientist to perform this job.

This kind of information and much more is available from a cool web site called Space Weather.  For example, did you know that just a week ago an asteroid got as close to the Earth as the moon is?  It was just a 27 m in size, so not earth destroying.  But these encounters are happening all the time.  Browsing around SpaceWeather a bit more and you’ll find great pictures of the sun like this one.  Or the current interplanetary magnetic field measured in nanoTeslas.

All in all, fun stuff.

I don’t want to give too much away, but from watching the trailer you can tell it’s gonna be good.  The story explores “emoticlips” which are a way to digital encode and transmit emotions, like a podcast.  Drama heats up when a viagra ad shows up, something about blackmail.  And my favorite line asks if you’ve tried Googling “omnipotent self-aware botnets.”

I just saw a fun trailer for the play on YouTube, shared here for your convenience…

You can get your tickets now from Brown Paper.

In the course of learning about people’s personalities for helping them get dates, they also gather deep demographic data.  Recently they analyzed these data with respect to politics.  They show a series of graphs that would make Edward Tufte proud, analyzing people’s preferences along the political plane defined by permissiveness vs government control over economic and social issues.  They look at how these preferences change with age, and how relatively important each axis is.  Their conclusions match exit poll data quite nicely and demonstrate analytically that the Republican party is much more focused in the issues it cares about, while the Democratic party draws in people whose opinions are much more diverse.

It’s a long article, but I heartily encourage you to read it, or at least skim the diagrams and play with the animation on the 6th chart.  Nice analysis.  Very nice presentation.  Nice job, folks.

The Blue Brain Project is the first comprehensive attempt to reverse-engineer the mammalian brain, in order to understand brain function and dysfunction through detailed simulations.

Others have described it as “a step toward the superbrain” or even ”the most interesting project in human history.”

I agree that this project is extremely important.  In my mind, this line of research is the only reliable way to achieve AI – artificial intelligence.  There’s little doubt in my mind that a sufficiently powerful computer could simulate in detail the operation of every neuron in a human brain.  Someday we’ll get there.  If you buy the argument so far, then the only thing standing between us and strong AI is an extremely detailed scan of a brain — down to the sub-neuron level.  In principal this should be possible by freezing a brain, slicing it up into very thin sections, and scanning them.  More sophisticated 3-D imaging techniques might even make this possible without having to cut somebody’s head off, but I’m not necessarily counting on that.  Besides, for the purpose of creating artificial intelligence, we just need to do this once, and it doesn’t even need to be perfect.

In the past, I’ve explored the question of whether or not such an AI would have free will.  This fascinating question about the nature of consciousness is fairly abstract, along with other ones related to the nature of consciousness, and what destructive uploading means to the person whose head gets cut off and sliced up.  (Do you just take a nap when they cut your head off, and then wake up inside the computer?  What happens when they spin up a second process?  Etc. Etc. Etc.)  But all of this philosophizing pales when I realize they’re actually doing it!

IBM is supplying one of their Blue Gene/L “supercomputers” with 8,000 processors — definitely cool hardware.  I’m not sure what supercomputer really means in the era of massively parallel cloud computing, but maybe this is exactly it.  Read more in their FAQ.  Or watch this TED talk:

Thanks to Joe Duck for digging this one up.

Scores of developers have had iPad’s for weeks now. They’ve had to sign non-disclosure agreements, and have the iPad locked in a separate room that random employees couldn’t access. And even that wasn’t enough. The iPads are literally chained to the desk with steel cable and a lock. Apple comes by the office with a suitcase, installs the iPad in a bolted case, chains it to the desk and locks it there. And they they do occasional surprise visits just to make sure it’s still there.

What’s more, Apple has told developers that they are monitoring the location of the device as well.

Apple.  Gotta love them.

Mike says he can type 50 wpm on it.  That’s really quite cool.

I’ll be developing marketing and distribution programs to broaden access to clean water for families living on as little as a few dollars a day.  Every year over 1.6 million people, largely children under the age of 5, die of diseases caused by unclean water and poor sanitation.

I really admire Jeremy for his can-do attitude, working on very important problems that are not at all close to home.  I invite all of you to follow along on his blog Clean Water For All and if you feel so moved to help them out.

UniXKCD command line console

My favorite webcomic, Randall Monroe’s brilliant XKCD, is running a command-line version of itself today.  A few commands you might want to try include:

• find
• wget http://xkcd.com/
• Make me a sandwich
• go west

Google renames itself to Topeka

In honor of Topeka, Kansas renaming itself Google in a bid to get ultra-high-speed broadband installed, Google has renamed itself Topeka today.  Although Google is well known for April Fool’s jokes I believe this is the first time any have been on the homepage.

YouTube’s TEXTp mode

YouTube has the option to render most any of its videos in ASCII by adding the &textp=fool parameter onto the URL.  Looking at bandwidth graphs I can’t tell if they’re actually sending ASCII over the wire, or doing the conversion client-side.  Fun trick though.

Bing’s funny cows

Bing has one of their defining pastoral pictures, this time literally bucolic, but with fake cows.  They’ve supposedly been bread to only make non-dairy creamer.  Glad you’re trying, folks.

Great job everybody!  It’s awesome to see what you can do without leaving your browser.

The big problem I have with the current variations of the CC license is the “who knows where else” part.  Folks are free to use my content, so long as they attribute it to me, without ever letting me know about it.  And that removes a big part of the fun for me — I get a huge kick when somebody tells me they want to publish my photo.  As a matter of politeness, I’ve taken to notifying content creators when I use their CC license, so they can get the joy.  But the license variations can’t require that.

What I’d like to see is a “Notification Required” variation of the Creative Commons License.  It would go alongside the current variations:

• Attribution
• Share-alike
• Non-commercial
• No derivative
• Notification (not yet implemented)

It would be executed similarly to the Attribution requirement — somewhere the content creator needs to explain to re-users how to attribute, and in this case how to notify.  In fact, I’ve updated the CC notice by my photos requesting notification.  I’m asking people to jump through some small hoops to contact me as a form Turing Test.  But if you understand English, they’re really pretty easy.

I know others would like to make similar requests when their content is re-used, so I think this should become a part of the standard array of options for CC licenses.

I’m quite excited about being invited to speak in this forum.  Science and physics in particular have always been very close to my heart.  Although I spend my days working on what I consider very practical problems compared to pure science of physics, I always enjoy musing on the underlying rules that govern our universe.

The conference is in Portland at the Pacific University of Oregon on April 16th and 17th.  If you have reason or occasion to attend, I encourage you to do so.