Embracing Chaos

Ray Kurzweil recently predicted that we’d be able to reverse engineer the human brain by 2020.  He makes an argument that a brain simulator would need about a million lines of code:

Here’s how that math works, Kurzweil explains: The design of the brain is in the genome. The human genome has three billion base pairs or six billion bits, which is about 800 million bytes before compression, he says. Eliminating redundancies and applying loss-less compression, that information can be compressed into about 50 million bytes, according to Kurzweil.

About half of that is the brain, which comes down to 25 million bytes, or a million lines of code.

This reasoning is IMHO flawed and overly optimistic.  It’s an interesting idea to compare the complexity of these two systems by comparing their bit representations.  I think the idea has merit, at a very rough level — that is I think you can compare the complexity of a genome to the complexity of a piece of software on a rough order-of-magnitude scale. The biggest flaw in Kurzweil’s argument is that he magically throws in a factor of 16x improvement in his favor by saying the genome can be “compressed.”  Well, software executables can be compressed too, a fact that Kurzweil conveniently ignores.  So I’d follow his reasoning to say that a human brain simulator probably needs about 10 – 100 million lines of code.  (I’m deliberately including 0 significant digits here to indicate the roughness of this approximation.)  This puts a human brain simulator on par with the some of world’s most sophisticated software projects so far, which seems about right, at least to an order of magnitude or so.

Strong reactions

PZ Myers published a wrathful condemnation of Kurzweil’s argument titled “Ray Kurzweil does not understand the brain.”  If you sift through the name-calling you see that Myers assumes a specific tactic in building the brain simulator: starting with the human genome and deriving the brain’s functionality from it.  This strategy will certainly work, once we have solved the protein-folding problem, and more generally have the ability to do quantum chemical simulations of kilogram-sized masses of organic chemicals.  Which is to say it’s theoretically possible (we might be living in a software simulation of our universe for all we know), but completely intractable with current technology.  For comparison, our best quantum chemical simulations if you push them top out at maybe a dozen atoms right now.  So being able to simulate an entire kilogram of organic matter is nowhere in sight.

Tactics to simulation

I agree with Myers that we are nowhere near being able to interpret the genome well enough to understand how it makes a brain.  But we probably don’t need to in order to simulate a brain.  By analogy, consider the Super Nintendo (SNES) Emulator, which is another kind of simulator many of us have experience with.

SNES emulators let you play all the old Nintendo games but on a modern computer instead of original SNES hardware.  Let’s say somebody handed you a box and a stack of cartridges and told you to build a Nintendo simulator.  What would you do?  Well, clearly you could open up the SNES box and reverse engineer the circuit boards to figure out all the wiring.  You’d probably figure out that the CPU was important — a variant on the 65816, which was essentially the 16-bit version of the 6502 some of us grew up with in our Commodore 64s and Atari 800s.  So you could (theoretically) crack open the 65816 CPU chip itself, put it the through an electron microscope and understand every transistor it used to interpret the instructions. In this way you could reliably create an emulator which completely replicated every aspect of the SNES. Such a simulation would replicate all of its bugs, timing quirks and everything, but it would work and be extremely expensive to simulate.

This is analogous to the tactic PZ Myers seems to be assuming Kurzweil would take to simulating a human brain. But Kurzweil would actually start at a much higher level of abstraction.  Simulating every protein in every neuron is like building an SNES emulator by simulating every transistor in the original Nintendo’s hardware. The key to getting those SNES games to work does not lie in replicating the design of the CPU which interprets the instructions.  The key is figuring out how to run those instructions on modern hardware.  By moving up through levels of abstraction, we can simulate the system much more cheaply and easily, although there’s a chance edge-case behavior won’t be captured properly.  (What if our world is a simulation and we bump into the edge-cases?)

Similarly, the key to simulating a human brain anytime soon does not lie in understanding every chemical pathway in human neurons.  Although if we did understand neurons at this level, we would have a great head start at simulating a brain.  Success in simulating a human brain will come by recognizing higher levels of abstraction in neuronal function.  We have known for a very long time that neurons communicate by “firing” electrical signals which are transmitted chemically at synapses.  The details of these behaviors are complex and determined by a great many interdependent chemical systems, but it seems highly likely that we can replicate the key behaviors of human neurons at this level of abstraction without needing to understand everything underneath supporting them.  If we can replicate the firing behavior of neurons in sufficient detail, we don’t care what the proteins underlying them are doing.  The key question here of course is what is “sufficient detail.”  I expect that question is one that researchers who are genuinely interested in reverse-engineering the brain will actually focus their attention on.

Once we can simulate the firing behavior of neurons, simulating a brain becomes much more of an engineering problem than a scientific one.  Still it’s going to be a massive engineering challenge, and gathering the input data will probably require a bunch of new science.  Then the philosophers can debate the meaning of free-will if our brains are Turing-complete.

It’s fascinating watching a disruptive technology cross the chasm. It’s a rare opportunity in one’s technical career to see this happen to a technology that one has been intimately involved with. That’s exactly what’s happening with video chat technology that I worked on at Google, as Apple pushes the technology into the early majority phase of adoption.  Crossing the Chasm is the name of a classic book on innovation by Geoffrey Moore which describes the process of taking a technology beyond just geeks.  The process is so difficult that Moore refers to it as the Chasm.  Apple is a master of technology strategy, so we can all benefit from watching them do this well.

To get across the chasm, your technology really needs to work well.  Apple seems to have done that with FaceTime.  But there’s more than just having it work — you also need to explain to the public why it’s important.  Here, Apple is just paying for that with traditional advertising.  They’re putting out lots of touching, heartwarming commercials showing the value of video chat.  Apple is spending millions of dollars to explain to people why they’ll like video chat. Primary demand stimulation.  They’re working to overcome people’s biases against  the technology or the idea — that it’s clunky, or the extremely common “I don’t want to see the people I’m talking to” reaction, which is really pretty funny when you think about it.  Of course, there will be some times when you will prefer audio only, but that’s going to be the exception when the technology is good enough.

It’s all playing out in a textbook fashion.  The geeks are all crying that there’s nothing new here, that this technology has been around forever, and they don’t understand why it’s such a big deal.  But it is a big deal, because video chat is finally entering the mainstream.

Some side-effects of this that I pointed out before, but are perhaps more clear now as the story unfolds, are that this will benefit existing established video chat vendors.  Apple is explaining to people that video chat matters.  This will help Skype and Google and Cisco with their products.

Speaking of Cisco, there’s another prediction coming true: Cisco is pushing into consumer video chat.  I had guessed 2012, but barely more than a week after my last post, they announce Cius, a video chat terminal.  Kinda like a Flip phone fused with a linksys router, but running Android and in a pretty nice looking case.

I believe that the iPhone 4 will be remembered as the device that invented video chat.  Just like how the iPod is often seen as the first real mp3 player.  It wasn’t at all of course.  There were dozens of mp3 players before it.  But the iPod set a new quality bar which was so much higher than everything before it, that it redefined the space, and actually made it accessible to the mainstream.

Video chat is in a similar place today to where mp3 players were 10 years ago.  There are lots of video chat solutions out there on the market.  Skype is the most well known.  I helped launch Google’s video chat system across Gmaill, iGoogle and Orkut during my tenure there.  It definitely is one of the best on the market, and it’s still only appealing to early adopters.  I mean “early adopters” in the classic Everett Rogers sense, which is to say folks for whom the extra value of the new technology outweighs the hassles of using it.  This is a step beyond the “innovators” category, who are willing to bend over backwards debugging a brand new product just because they understand that it will be important later.  Video chat has been available to innovators for a great many decades.

With iPhone 4, Apple will push video chat to the early majority category. Apple has a history of sitting on potential technologies until all the bugs are worked out, which is fundamentally what’s needed to appeal to more than just early adopters.  I’m pretty sure FaceTime will be no exception.  In 6 months, video chat won’t be this geeky thing that people put up with out of desperation.  It will begin to be integrated into normal culture.  We’ll start to see television dramas and movies incorporate it as just a way some people communicate, rather than as a way to demonstrate how high-tech somebody is.  People who aren’t geeks will start to use video chat.

Video chat really matters

Those of us who have lived deeply with video chat understand its value.  There is a ton of additional content transmitted in video that helps communication on many levels.  It allows for a more nuanced informational discussion, but more importantly IMHO, it allows distant communication to be much more personal and emotional.  Anybody who has tried professional collaboration with another team that is thousands of miles away knows that this level of communication is at least as important for business uses as it is for social communication.  The first time you meet your collaborators in person, they become more real, more trustworthy, easier to talk to, especially about difficult subjects like problems that might arise in a project *gasp*.  Video chat is certainly not as good as meeting people in person, but it is a huge step above email, IM or phone.  (Getting drunk together I believe represents the highest professionally-accepted level of humanization.)

Human-to-human communication has always been the killer feature of computer technology. Video chat makes synchronous communication fundamentally better, and as such will become a major part of everybody’s life in the developed world in the years to come.

What does this mean for everybody else?

The history of technology innovation tells us that a couple things typically happen when an emerging technology pushes into the majority segments.  First, established players will all get a boost.  Apple will be doing a huge favor to Skype and Google Video Chat by removing the veil of geekiness from their products.  Apple’s huge investment in making this product work well will make all consumers more willing to try alternatives.

Another common side-effect is that the space will get more difficult for new entrants.  This usually happens as the technology standardizes.  There becomes a “normal way of doing things” that people start “to get.”  Before a technology can reach the majority, it will typically bounce around dozens of different modalities as everybody tries to find a way of doing it that resonates with the market.  This uncertainty represents a clear opportunity for start-ups and the subsequent standardization is the closing of that opportunity.  Another reason the space usually gets harder for startups is that economies of scale start to kick in as production levels ramp up to meet the larger demand.  This naturally favors large companies, since it raises the amount of investment needed to compete.

Which Social Graph?

Another reason the space might get harder for newcomers is the natural monopoly of social graphs — consumers are better off if there is a single definitive place to keep track of their contacts rather than having to replicate and maintain a different list for each service.  As such, social graphs are important assets to anybody in this space.  But if Tim O’Reilly gets his way (which I hope he does) and we end up with a loosely-coupled internet OS, this won’t be a problem for startups, as they’ll just be able to draw from an openly available graph, say from a Google or Facebook.

I’m really not sure how this aspect will play out.  My guess is that Apple will rely on the de-centralized social graph which is the contact list built into every iPhone.  It’s a less useful corporate asset than if it were properly cloud-hosted, which will make it harder for them to expand the service to OS X machines.  Perhaps they’ll make something useful out of mobileme here, but I have my doubts.  But given the revenue they get from App Store sales, it’s not clear that the OS X machine is even a major part of their consumer strategy going forwards.  If so, this would likely be a strategic shift for them, as the inclusion of web-cams on essentially every OS X machine for years was probably done in anticipation of making a major push into video chat at some point.

Don’t forget Cisco

In addition to the obvious players like Google and Skype, this is also incredibly important for Cisco too.  Cisco has long been interested in video chat.  Why?  The same underlying reason Intel has been investing in multi-media since the late 1980’s.  Multimedia on PCs needs lots of CPU power, and video chat needs lots of bandwidth.  It’s called “primary demand stimulation.”  To be very clear: Cisco wants everybody to use video chat because video chat uses lots of bandwidth, and when people are using lots of bandwidth, Cisco sells more big routers.

Cisco is in the final phases of buying Tandberg, who is the biggest supplier of video-chat hardware for businesses.  Their aquisition of Flip last year seems strategically odd in isolation, but in this context makes perfect sense.  They are building (buying) expertise in consumer-electronics which can handle high quality video.  Take a Flip camera, add a network (like a linksys wifi box) and you’ve got a video chat terminal.  I predict we’ll see such a toy out of Cisco in about 2012, as video chat fills the early majority segment and edges against the late majority.

Recently one of my friends asked whether or not there was a general trend towards improved safety in oil exploration.  Coming from a mechanical engineering background, he noted that things like bridges and buildings have gotten safer over time through failures.  Every new structure is built with the collective wisdom of the many failures before it.  And with each failure, we learn how to avoid that specific kind of failure.  Are the same principals at play in oil exploration?

I set about answering this question with data.   I quickly found a list of oil spills on wikipedia.  A quick pass through google spreadsheets and a few regexs later, and I’ve got the data in a form that it can be graphed with protovis, a wonderful web-based visualization package.  An initial look shows some interesting trends. (Sorry IE users – modern browser required.)

First, it’s important to emphasize that this is a log-scale graph.  Given the dynamic range of the input, it’s the only reasonable way to visualize what’s there, but if you’re not used to reading log-scale graphs, the data will be deceptive.  In short, being a little higher on the graph means that the spill is a lot larger.  In fact if would be very reasonable to only include the spills near the top of the graph when thinking about “big spills.”  But I wanted to present the entire data set for completeness and analysis.

It’s interesting to note the general downward trend at the bottom of the graph.  I believe this is not a real effect at all, but a result of selective memory.  The smallest spill on this graph was only a couple months ago — the Great Barrier Reef spill in April.  Are we to believe that in the preceeding 100 years of oil exploration there had never been a spill of less than 10 tons of oil, and only a single other spill of less than 100 tons?  Of course not.  I bet spills of this size have happened dozens if not hundreds of times, but 50 or 100 years ago nobody bothered writing them down.  Or if they did write it down, the event has been filtered out of our collective historical memory before making it into wikipedia.  The Lakeview gusher in 1909 is another interesting example of this effect.  This certainly wasn’t the only oil production accident before 1930, but it was clearly an important major accident, and so has been remembered far better than others.

I’ve highlighted a few other spills because of their historical interest.  The Gulf War oil spill (purple dot) of 1991 is exceptional in that it was not an accident, but a deliberate act of war.  As such, it should not be considered in answering the question of whether oil exploration has been getting safer.  The Exxon Valdez spill (light blue dot) in 1989 is large in our memory, but in context we can see that it was not at all a large spill by historic standards.  But clearly the Deepwater Horizon spill (green dot) is huge, ranking as one of the largest spills ever and certainly the largest spill in quite some time.  But aside from this current mess, there does seem to be a real trend towards increased safety in oil exploration.

Again, the log-scale graph makes this somewhat hard to read intuitively.  Because the spills near the top are so much larger than the ones below them, a fair approximation of the sum of all spills can be found by simply considering the points along the top envelope, which is generally decreasing.  Looking just at the last several decades on a linear scale, this trend becomes more clear: since about 1980, serious oil spills have been getting smaller / less frequent. Now we see visually that the majority of spills listed are tiny compared to the few big ones.  I scaled the graph so only the bottom of the uncertainty bar for the gulf war oil spill.  Also note that I’ve kept the middle dots at the geometric average of the low and high estimates, which works visually on the log-scale graph, and makes logical sense given the nature of the problem.

Another factor to consider is that the total amount of oil being produced during this time period has been generally increasing.  I’ve overlaid data from the US Energy Information Administration on global oil production rate, scaled to the average amount produced each hour, to get it to show up on the same scale of this graph.   Another interesting comparison which I haven’t included is the average size of each well, or the number of wells being drilled per unit time.  My understanding is that oil exploration has been getting more difficult over time in that we’re having to drill deeper to get at relatively smaller oil deposits.  Again, this reinforces the idea that we have been getting better and safer — we’re spilling less even though we’re drilling more holes.  Except for the Deepwater Horizon.

Feel free to browse the javascript source code of the graphs for further details, inspiration, double-checking, or whatever.

One of my responsibilities in my new job is to lay the groundwork for development of the company’s technology.  One decision that was pretty easy to make is that we’ll be building tools that you can use from a web browser, that is to say, a web application.  Next comes the choice of what language and framework to build the web application in.

Having been writing code for 25+ years, the differences between programming languages fade away in my mind.  I know that I, like any good software engineer, can be productive in basically any language.  Certainly within object-oriented languages, (which is where essentially all serious software engineering happens these days) the differences in the language itself are IMHO small compared with other factors in choosing a platform.  Other important factors I consider include:

• Quality and availability of libraries.
• Quality of tools like IDEs, debuggers, automation systems.
• Size and healthy of the active community using the framework.  (i.e. If I run into a problem, how easy is it to google the answer?)
• Ease of hiring people who already know the platform.

All of these considerations are IMHO more important than compiled vs. interpreted or run-time performance or whether the language is statically typed or dynamically typed, or even if it’s open source vs. proprietary technology.  But there is a single common factor which directly feeds into all four of the criteria I list above: How many people are actively using the platform?

The Candidates

The last time I was really writing much code was back in 2007.  At the time Ruby on Rails was the coolest thing since sliced bread.  I played around with it, was amazed by how easy it was to quickly put together simple database-driven websites, and got very frustrated with it when I wanted to color outside of its lines.  A major concern of mine was the level of “magic” that happens behind the scenes — this makes Rails beautiful and elegant when it works, but difficult to debug or extend.  I blogged about my experiences with rails and in particular my conclusion that at the time Rails was not ready for large, complex projects, partly because of a lack of good tools, libraries and sensible error messages, all of which can be fixed by more users.

Around then I started working for Google and stopped writing code.  The next year or so saw a couple big things happen for Ruby on Rails.  Twitter famously had trouble maintaining even 99% uptime (fail whale anybody?), and everybody knew they were running on Ruby on Rails.  Also, Google launched AppEngine, which supported Python, not Ruby, and pointed people like me to an alternative high-level web framework: Django.

So now, in 2010, I return to the fray, and I’m trying to decide between the two frameworks.  There are of course other alternatives, but for the purpose of brevity, I’ll leave out my process of reducing my choices to these two: django or rails?

Measuring community activity

I often use Google Trends to measure relative interest in technologies.  My time is very valuable, so even downloading something and reading its documentation is an investment I’d rather shortcut if I can.  This crude measure of relative search activity can actually be quite telling, and has saved me a bunch of time in choosing packages.

The relative values are hard to take literally since “django” as a single word search query will naturally be higher than a 3-word query like “ruby on rails”, but the search for “rails” by itself will clearly have lots of irrelevant searches.  Likewise, some people searching for “Django” won’t be looking for the web framework, but rather the guitarist or the movie or what-have you.

But it is clear that search activity for Ruby on Rails peaked in 2007-2008 and has been declining since then, while Django has been on a steady upwards trend.

Elsewhere I can find evidence that Rails is still a more used platform.  Stackoverflow has twice as many questions about Ruby on Rails vs Django: 13,882 versus 7,496.  To me this indicates pretty clearly that Rails is more active than Django.  Either that or Rails is more confusing and people ask more questions about it, but I doubt that.

The Tiobe Index attempts to objectively measure activity across programming languages.  By its easily manipulated measure, Python has twice the activity of Ruby, independent of the django / rails frameworks, with both languages in decline.

Meanwhile it’s easy to find comparisons on the net between the two.  Everything from content-free videos to people declaring rails dead, and everything in between.

On balance, I find the stackoverflow numbers the most compelling, unbiased indication that Ruby on Rails has more activity than Django / Python.  Even though I’ve been frustrated by it in the past, by my own objective criteria, that seems to make Rails a better choice for building a new web application.

What’s your opinion?

If you have experience with both frameworks, I’d love to hear your experiences.  Please leave a comment.

A number of years ago, a bunch of my friends were reading Naomi Klein’s book No Logo and getting really riled up by it.  The book is certainly written to make you angry, describing how brands and logos have become more and more prominent in our society as the marketing industry has become more sophisticated at delivering their messages.  When I read it, I had a very different reaction.  I found it to be a fascinating history of marketing.  Klein gives examples of how advertising of the past was very simple — think back to classic TV ads which amounted to a person standing in front of a camera saying little more than “Buy this dogfood.  It will feed your dog.”  When television was young, these ads worked.  But as people got used to it, they learned to tune these simple messages out.  What has followed has been a steady co-evolution of new marketing techniques and people learning to understand them and be less swayed by them.  If you’re old enough, you’ll remember that first Diet Pepsi commercial that ran before Top Gun in theaters.  Remember how odd it was to see a commercial in movie theaters?  Or consider the evolution of product placement within movies — how actors used to turn their heads and unnaturally hold their beverage so the entire logo was clearly visible on the side of the bottle.  Now it’s much more common to just see a part of a logo — enough to be recognized and enter the subconscious, thus bypassing the conscious filters which weed out blatant product placement.  Klein presents this history, punctuated with outbursts of “we’re not going to put up with this any more!”

Simultaneously, technological advances have allowed advertising to progress along a different axis – to become more targeted.  Advertising used to only be broadcast widely through newspapers and television shows.  The best an advertiser could do to ensure their message reached the right kind of people was to select the aggregate demographics of everybody who read a particular magazine.  Now the internet allows ads to be targeted as precisely as you’d like.  Today, Google lets you get your message only in front of people who are about to buy a product like yours.  The ability to connect to people who have expressed an intention to “buy digital camera” is a literal gold-mine, making billionaires out of Larry, Sergey and Eric.  As effective as it is, targeted advertising won’t replace broadcast advertising, because there is still value in abstract brand-building.  Rather, the two will complement each other.

Enter Social Media

Social media has been all the buzz recently.  At its core it’s just a more convenient way for friends to communicate.  The “killer app” for computers has always been helping people communicate, and this is just another chapter in that book.  With this new communications medium comes a new opportunity for organizations to tell their stories.  In fact, I believe that social media will bring another tectonic shift in the entire marketing industry, possibly as important as search-based advertising.  As consumers have gotten more and more sophisticated at filtering out advertising from broadcast media, advertisers have gotten more and more desperate in their attempts to connect with people.  Social media marketing offers a new path – instead of hearing about products and services through ads, people can hear about products and services from their own friends.  Exactly how this will play out through Twitter/Facebook/Foursquare/whatever is not at all clear to me right now, but I fundamentally believe this change is coming, and it will take the entire marketing industry with it.  Klein and her fans are free to unplug from popular culture in order to avoid the onslaught of brand advertising, but they would be foolish to stop talking to their friends just because their friends are happy with things they’ve bought.

This vision is one of the main things that prompted me to jump off the comfy Google cruise liner and start paddling hard in Banyan Branch’s crowded dinghy.

Is marketing intrinsically evil?

I sometimes feel a need to justify this line of work to those who think that marketing is inherently dirty.  I admit that I’m more of a capitalist than many of my friends, but I certainly recognize that capitalism has its limits.  The vast majority of economic transactions are both consensual and mutually beneficial, and I will argue vigorously that there is nothing wrong with an economic system consisting of these transactions.  The biggest exception to this happens when transactions are not mutually beneficial because one party is not fully informed.  But what we’re doing is helping people share honest opinions and feedback about the things they buy and use.  By lubricating the flow of information between real people, I believe social media will reduce the effectiveness of deceptive marketing.  Moreover, it will help companies connect to their customers and hone their goods to people’s real concerns and desires.  It will help hold companies accountable for their mistakes, and enable companies to better make things that make people happy.

Additionally, I will point out that my employer represents no small amount of “pure good” for the world, including organizations such as The Bill & Melinda Gates Foundation and Vittana, helping them tell their stories.

Taking a chance on a startup

Why did I choose this opportunity out of the sea of possibilities?  I evaluated the landscape as an investor would, since I am investing no small chunk of my life in this effort.  From my entrepreneurial training and experience, I know that smart investors care more about the people than the specific business plan.  The plan will almost certainly change, but the key management will not.  Having known one of the founders of Banyan quite well for a number of years, I am certain that many key elements for success are in place.  The corporate culture and governance will be solid.  I will be working in an environment where I am supported, and where I can learn and grow as a manager and a technologist.

Exactly what will I be doing or building?  I admit I’m not sure yet, but I have some very interesting ideas that I won’t be sharing here anytime soon.  I am sure that my work is very well positioned to be a part of a major shift in an entire industry — a rare opportunity.  Whether or not my work will play a key role in this shift is somewhat out of my hands — these things are always a roll of the dice.  But in another sense, it’s entirely within my control, and this is what I love about working in a small company.  There’s almost nothing but work between me and effective execution of our ideas.  Many people tend to exaggerate the importance of the idea itself, forgetting that it is incredibly important to execute well on whatever ideas you have.  I’ve heard people say that they had the idea for YouTube years before YouTube did.  How quickly we forget the dozens of other companies all working on the same problem in 2006, which almost all fell by the wayside because they didn’t execute as well as YouTube did.  Ideas matter for sure.  But hard work is critical.

If you’d like to jump on this raft and start paddling too, get in touch with me.  I need a few key rock-star developers who are’t scared of chaos and can think creatively about business problems.

It’s been two and a half years since I started working for Google.  Now, as I walk out the door, I’m going to try to capture some of my thoughts about the experience and share them here.

In the summer of 2007, as I was finishing up my MBA, I was faced with a choice of where to work.  I had this job at Google, and was also considering spinning up a startup on my own.  I had spent the last several years studying entrepreneurship and wanted to put that training into action.  One thing that really pushed me to Google was some advice from a very wise friend who was also looking for a new gig.  He said something like this.

“So many people have adjustable rate mortgages, it’s ridiculous.  They’re going to reset and people won’t be able to afford to pay them any more.  I think a lot of people are going to default on their mortgages and lose their houses.  I bet that will drag the entire economy down.  That’s why I’m looking for a job in a nice big stable company.”

He went to Amazon.  Having tried to start a small venture in a recession previously, I decided that Google would be a great place to hide out.  I also figured (correctly!) that I could learn a lot about how an extremely well-run company works from the inside.

Bottoms-Up Management

I learned how bottoms-up management works.  The idea of giving as much control as possible to the individual contributors who are working on the problems themselves.  Google exemplifies the saying “let 1,000 flowers bloom” with teams seemingly running in every direction at once.  It’s worked very well for them, and I can see a lot of value in this structure.  But I also see its limitations.

Having spent a while working on building infrastructure projects at Google, I experienced first-hand the limitations of bottoms-up management.  Individuals are empowered to make their own decisions, and they optimize for what’s best for themselves, which means their careers and their projects.  It’s easy to optimize towards a local-minimum which is great for your project, but not necessarily ideal for the company as a whole.  Similarly, top-down directives are extremely difficult to enforce.  Executives can make decrees about how things should work, and it just doesn’t happen.  I remember walking out of a review with Larry, Sergey and Eric and hearing one of the Directors respond “well that’s Sergey’s opinion.”

I spent a bunch of time trying to encourage teams to standardize parts of their backend infrastructure on shared components.  Shared technology is rarely the easiest or fastest thing for an individual project, so let me tell you the road was challenging.  I’ll refrain from pointing out the holes in Google’s products that I was trying to plug which are still open.  (To be clear, I’m referring to missing features and UX annoyances, not security holes.)  But I also learned some important lessons in how to design such infrastructure so that it is a win-win-win for all the teams involved.

Alignment with Customers

The best thing about working at Google was the clear alignment with customers.  Because Google is so strong in web search, they can afford to invest heavily in primary demand stimulation.  Like Campbell’s paying for an ad campaign that says “Soup is good food”, it makes financial sense for Google to work very hard just to get people to use the web more.  It goes beyond the motto “focus on the user” — Google really does want whatever the user wants, at least 99% of the time.

This makes huge classes of problem solving so much easier, because there’s no cognitive dissonance trying to remember the difference between a great user experience and a sound business decision.  This particular aspect of the business was an extremely refreshing change coming from Real Networks.  I remember working hard on re-designing the Real Player installer experience and tackling that specific problem head on.  The company had become addicted to the pocket change they could scrape out of their unsuspecting customers by surreptitiously taking over their computers when the Player was installed.  None of that BS at Google.

Looking forwards

Google faces some serious challenges in the years ahead.  For many of the reasons I pointed out in 2007, Facebook continues to be a serious threat to Google’s primary business.  Android faces an uphill battle against Apple’s dominance in the all-important mobile space.  Amazon seems to be squarely in control of the hosted cloud computing space.  And Microsoft seems to be pulling their head out of the sand for office productivity software.  None of these problems will be easy for Google to solve.  But I’m confident the amazing folks I worked with there will put in very strong showings.

As for me, I’m going to work in social media.  I’m very excited about my new role which I’ll write more about soon.

Google recently launched some major improvements to their online document-editing suite.  The spreadsheets are faster and more powerful.  But IMHO the most interesting change is an update to the word-processor (originally known as Writely) which allows for real-time collaboration.  Now the text documents act like the spreadsheets do. You can see where in the document your collaborators are working, and see each keystroke as they type them.  The immediacy of this collaboration removes concerns about synchronizing changes and whether or not your document has “saved” recently or not.

I was patiently waiting for the feature to be turned for my account, which wasn’t happening.  I kept getting the same old spreadsheet program.  But then I found the setting to enable it.  So I’m sharing with y’all an explanation of how you too can get these great new features.

Go to http://docs.google.com/ and click the select “Settings” link in the upper-right hand corner, and choose “Documents Settings”.

Then choose the “Editing” tab and check the checkbox next to “New version of Google documents”.

Now any new documents you create will use the new real-time editor.  And anybody you share them with will get the new features without having to set this up for their account.

Even though Apple products are expensive, there’s a surprising similarity between Apple and Wal-Mart: both companies push hard on other parts of the value chain to deliver lower prices for consumers.

In Walmart’s case, it’s generally suppliers who get squeezed.  Walmart demands that manufacturers of goods produce them at the lowest possible price so that Walmart can charge the lowest prices in their stores.  They really do try hard to pass the savings on to you.  Another case that is less well known is with so-called “interchange” fees for debit and credit cards, charged by the card networks like Visa and Mastercard.  Back in 2003, Walmart pushed hard on Visa and Mastercard to charge less for debit card transactions since they are both lower risk (because of pin-code use) and cheaper to process (verifying signatures is expensive).  The cynical will point out that with lower fees, Walmart just gets to keep more profit.  Which is true.  But they are genuinely motivated to lower prices for consumers, since that’s their main selling point.  So it’s a win-win – Wal-Mart’s motivations to lower costs are closely aligned with consumer’s desires to pay less.

Apple has similar desires for their network-connected gadgets like iPhones and iPads.  Apples wants people to be able to connect their devices to the network for as little as possible. Apple has clearly negotiated very hard with AT&T to demand low monthly rates on data plans for these devices.  Next month you’ll be able to buy an iPad with a 3G data plan for just $15 / month.  That is basically unheard of in the US.  For people on a limited budget, the iPad is the cheapest way to get online.  Compare this to other data plans available from major U.S. carriers:

The Apple / AT&T rates are the lowest in each of their categories, except Verizon’s smartphone data plan which ties the AT&T iPhone plan.  The iPad rates are extremely low compared to data plans for laptops, and also when when you consider that tethering plans or phone data plans require paying an extra $30/mo – $50/mo for a voice plan.  The unlimited iPad plan is literally half what it costs to get 3G on any other laptop, and it doesn’t come with the 5 GB limit that other plans do.  You might argue that the iPad can’t do as much as a full laptop, which is true.  So you might then argue that iPad won’t tax the network as much as a laptop, which I doubt considering the propensity to consume video on such a device.  So you can’t trade torrents on an iPad, which from an Intellectual Property perspective is just fine with me.

My guess (and this is pure speculation) is that Apple negotiated these rates by offering AT&T a share of the revenues generated through App Store purchases.

Again, the cynical will point out that Apple is just trying to grab the lion’s share of economic surplus for itself, which is true.  But nonetheless, this is a case where Apple’s desires and our desires as consumers line up well.  In a very real way, Apple is fighting on our behalf for lower prices from AT&T.

Apple has done a fabulous job in recent years of asserting itself as a major player in the computer industry.  One of their tools for accomplishing this has been a fanatical commitment to high-quality products.  They strive to make every product they offer to be the best in its class, and they’ve largely succeeded at doing this.  (And have used some very clever strategies to maintain this appearance when their products weren’t quite measuring up.)  This has given them an incredibly strong brand.  But it also allows them to position themselves in an enviable place in terms of market positioning.

Apple products are expensive. Apple gets high margins on its hardware, allowing it to recoup large investments in NRE (non-recurring engineering) to design the hardware and its accompanying software.  This is a great place to be from a competitive standpoint, because as a company they don’t need to squabble over the cheapest parts to try to deliver the best prices to consumers.  So long as they can maintain a sufficiently large customer base to support the practice, it is an easy place to defend against competition from.  Certainly a lot easier than being Dell or HP, who struggle with operational efficiency to compete on price, and try to innovate within a very narrow window defined by their platform.

Apple’s success at selling high-end products has secondary benefits for the rest of the ecosystem.  Because the products are expensive, they tend to be purchased by people with more disposable income. So the segment of the computer market which buys Apple products self-selects to be very attractive demographic for many other reasons.  Advertisers love to get their products in front of people who are more-willing-than-most to buy something expensive / unnecessary / fun.

Similarly, app developers know that if they write an app for iPhone / iPad, the people who are able to buy it are much more likely to be willing to pay a couple bucks for something silly than, say, somebody who bought the cheapest smartphone they could afford because they felt they really need that functionality.  I had previously speculated that Apple’s platform play required a very large distribution base to attract developers, which is not quite correct.  The strategy is successful even with a relatively small market, provided that the market is segmented properly.  Which in this case it clearly is.