I’m going to give two examples of natural monopolies in high tech. They are not the perfect examples used in textbooks, but I think they are illustrative, and offer valuable lessons.

Natural Monopoly of Operating Systems

Operating systems are a good example of a natural monopoly. As much as we all value choice as a driver of innovation, the plain truth is that almost everybody is better off if there is a standard operating system upon which higher-level applications can be built. Application developers benefit because they have a single clear platform upon which to build. If there were two or three dominant operating systems, application vendors would need to build a separate version of their application for each one in order to reach consumers, which is considerably more effort. Similarly, the standard benefits consumers because they have a single choice which gives them the benefit of all the applications written on it.

Gates & Allen understood this long before most, which prompted them to drop out of school and pursue Microsoft with vigor. Windows succeeded in creating such a natural monopoly, enabling a rich ecosystem of third-party software vendors (ISVs in MS parlance) to create value for consumers without needing to worry about what chipset underlies the graphics card or network adapter their customers’ computers. In this way, Microsoft enabled the creation of value for PC customers and wealth for ISVs, and the monopoly persists in a form to this day.

But all is not rosy in this world. Other companies want to sell operating systems. People want choice. Once entrenched, the monopolist has a tendency to make choices which benefit the monopolist more than the consumer — Microsoft continues to exhibit this behavior even as their monopoly power fades. In classic natural monopolies like utilities, explicit regulation controls the monopolist’s abuse. With Windows, a combination of limited government intervention and competitive innovation ultimately limited their influence.

Social networks as natural monopolies

Online social networks also exhibit properties of a natural monopoly. A well built social networking service like Facebook creates tremendous economic opportunities. Particularly if the service exposes its valuable social graph data through an API that other services can use. Almost any online service can be made more compelling by incorporating social graph data. The existence of a publicly usable social graph dataset provides an economic boost to the entire tech sector.

This boost tends to create a winner-take-all situation.  When third-party services rely on a social API service, they reinforce consumer’s use of that service.  Third parties’ lives are easier when there is a single standard, because they only need to code to a single API in order to gain the benefits of the social graph.  Here the analogy to operating systems is clear.  The social network provides a platform upon which others can create value.  The value creation process is easier if there is a single standard social network upon which to build. These characteristics make the social networking monopoly natural.

A behavioral characteristic of social networking sites’ users also helps create a monopoly. People enjoy the benefits of having their social network defined online, but they do not enjoy the effort of defining it. Us geeks (everybody reading this and probably most of your friends) are willing to spend hours organizing our friends into circles or searching for people we know to connect with them. Some of us even enjoy it. But for most normal people this very quickly becomes a boring waste of time, especially if they’ve already done this once or twice on different websites.  Most people are not willing to maintain multiple social networks. Once they are invested in one, the barrier to switching is quite high.

Implications for Google+ in competing with Facebook

Facebook’s dominance is rapidly approaching monopoly levels.  They have crossed the tipping point where they are fast on their way to becoming the de-facto standard for social graph data, if they haven’t already.  The nature of social networks as supporting a natural monopoly means that Facebook’s rise will be supported more strongly than it would be otherwise.  When considering Facebook’s dominance, we readers must remember our place in the ecosystem as geeks.  We and our friends, are the innovators and early adopters who are far more willing to try the new thing, because we see intrinsic value in progress, and are far less perturbed by unrefined products.  The fact that recently Facebook’s fastest growing demographic was women over 55 shows that the service has crossed Moore’s chasm and now appeals to the majority of people.  As industry insiders, it’s easy for us to forget the bubble we live in — just because everybody we know uses something doesn’t mean it will ever actually take off an be popular with non-geeks.  But Facebook is clearly on a path to provide a dominant monopolistic standard for social networking data.

Breaking this monopoly would be difficult for Google even without the advantages of a natural monopoly.  People’s natural laziness makes a third social network (after Facebook and Twitter) unlikely to succeed as well.  So on the face of it, Google’s got a very tough road ahead.  It’s tempting to declare G+ dead on arrival because of these intrinsic forces, but there are other reasons why I think they actually have a decent shot.  But I’ll save that analysis for another story.

Please democratize SSL certificates.  The ability to serve HTTPS:// pages without scaring users is currently controlled by a handful of “trusted authorities” whose business is to make it difficult to secure web communications.  Google, you have the ability to disrupt this oligarchy and empower individuals to make the web safer.

The web is a safer place when information passed between browsers and web servers is encrypted — that is when URLs start with HTTPS instead of HTTP.  The recent introduction of FireSheep demonstrated to the world just how insecure normal (HTTP) web communications are — anybody on your network with a simple browser plugin can impersonate you.  In fact, FireSheep democratized the ability to steal session authentication by bundling it up in a manner that is easily used by the masses.  Google’s own proposed SPDY protocol, whose primary goal is to make the web faster, is willing to slow down in the name of security.  “Although SSL does introduce a latency penalty, we believe that the long-term future of the web depends on a secure network connection.”  We all want a safer web, so please help us achieve that by making it easier to set up HTTPS on our web servers.

There is no technical challenge here.  All modern browsers and servers are capable of safely encrypting the information passed between them.  Encryption protects users against eavesdropping and session hijacking a la firesheep.  Today’s challenge to secure web communications does not lie in the encryption, but the authentication.  The HTTPS protocol begins with the server presenting its security “certificate” which is meant to assure the user they have not reached an imposter web site.  This assurance is provided courtesy of the oligarchy of trusted certificate authorities, for a fee and a hassle.  Alternately, servers can present a “self-signed certificate” which provides equally good encryption, but no assurance that the server is who it claims to be.  But instead of recognizing self-signed certificates as being safer than no security at all, today’s popular browsers do their best to terrify and/or inconvenience users when visiting sites with self-signed certificates.  Certainly there is some value in authenticating the web server, but is that value worth the cost of allowing eaves-dropping and session hi-jacking on the vast majority of web sites?  I think not.

The current standard practice is backwards.  An HTTPS request to a server using a self-signed certificate offers encryption but not authentication.  This is clearly safer than a plain-text HTTP request, which offers neither encryption nor authentication.  But browsers tell users that self-signed certs are worse than unsecured communications.  (Chrome is actually worse than others.)  Deploying SSL on a commercial scale is also complicated by shared IP addresses for multiple sites, which again interferes with authentication, but not encryption.  The certificate verification UI already demonstrates varying levels of trust as shown below.  But self-signed certificates which offer encryption without authentication are incorrectly indicated.  Let’s remove the simple barriers which are preventing encrypted web communications.

The best technical path to fix this mess is immaterial here — many options exist.  Changing browser behavior to make self-signed certs less scary is one path, although it’s not a complete solution because of the legacy of every installed browser.  A new free service that signed anybody’s certificate with a trusted cert would work, provided that company had sufficient clout to get their root cert recognized.  (Google, you can do this.)  Empowering any domain registrar to sign SSL certs also makes sense since they’re the ones ultimately authenticating who owns a domain.  This choice wouldn’t immediately bring certificate prices to zero, but would greatly accelerate the trend we already see of lowering prices.  Perhaps a bloom-filter algorithm similar to what Chrome uses to identify malware sites could differentiate those sites whose identity has actually been verified through stricter measures, where self-signing should not be trusted.  A deeper technical analysis is needed to determine the best tactics, but clearly Google has both the necessary skills and level of influence needed to effect this change.

Additionally, Google uniquely has the motivation to make the web safer.  Google long ago recognized the value of primary demand stimulation — more web use means more web searches which means more advertising revenue for Google.  Open standards do not advance without leadership from selfishly interested parties.  The state of SSL certificates mirrors a political situation that desperately needs legislative intervention — a special interest group (the root certificate authorities) has a strong financial incentive to maintain status quo, even though every individual marginally benefits from the change.  Google is the company that stands to benefit the most from a safer web.  So please Google, act now to bring democracy to the safe exchange of information on the web by enabling anybody to freely secure their web traffic.

Video chat is in a similar place today to where mp3 players were 10 years ago.  There are lots of video chat solutions out there on the market.  Skype is the most well known.  I helped launch Google’s video chat system across Gmaill, iGoogle and Orkut during my tenure there.  It definitely is one of the best on the market, and it’s still only appealing to early adopters.  I mean “early adopters” in the classic Everett Rogers sense, which is to say folks for whom the extra value of the new technology outweighs the hassles of using it.  This is a step beyond the “innovators” category, who are willing to bend over backwards debugging a brand new product just because they understand that it will be important later.  Video chat has been available to innovators for a great many decades.

With iPhone 4, Apple will push video chat to the early majority category. Apple has a history of sitting on potential technologies until all the bugs are worked out, which is fundamentally what’s needed to appeal to more than just early adopters.  I’m pretty sure FaceTime will be no exception.  In 6 months, video chat won’t be this geeky thing that people put up with out of desperation.  It will begin to be integrated into normal culture.  We’ll start to see television dramas and movies incorporate it as just a way some people communicate, rather than as a way to demonstrate how high-tech somebody is.  People who aren’t geeks will start to use video chat.

Video chat really matters

Those of us who have lived deeply with video chat understand its value.  There is a ton of additional content transmitted in video that helps communication on many levels.  It allows for a more nuanced informational discussion, but more importantly IMHO, it allows distant communication to be much more personal and emotional.  Anybody who has tried professional collaboration with another team that is thousands of miles away knows that this level of communication is at least as important for business uses as it is for social communication.  The first time you meet your collaborators in person, they become more real, more trustworthy, easier to talk to, especially about difficult subjects like problems that might arise in a project *gasp*.  Video chat is certainly not as good as meeting people in person, but it is a huge step above email, IM or phone.  (Getting drunk together I believe represents the highest professionally-accepted level of humanization.)

Human-to-human communication has always been the killer feature of computer technology. Video chat makes synchronous communication fundamentally better, and as such will become a major part of everybody’s life in the developed world in the years to come.

What does this mean for everybody else?

The history of technology innovation tells us that a couple things typically happen when an emerging technology pushes into the majority segments.  First, established players will all get a boost.  Apple will be doing a huge favor to Skype and Google Video Chat by removing the veil of geekiness from their products.  Apple’s huge investment in making this product work well will make all consumers more willing to try alternatives.

Another common side-effect is that the space will get more difficult for new entrants.  This usually happens as the technology standardizes.  There becomes a “normal way of doing things” that people start “to get.”  Before a technology can reach the majority, it will typically bounce around dozens of different modalities as everybody tries to find a way of doing it that resonates with the market.  This uncertainty represents a clear opportunity for start-ups and the subsequent standardization is the closing of that opportunity.  Another reason the space usually gets harder for startups is that economies of scale start to kick in as production levels ramp up to meet the larger demand.  This naturally favors large companies, since it raises the amount of investment needed to compete.

Which Social Graph?

Another reason the space might get harder for newcomers is the natural monopoly of social graphs — consumers are better off if there is a single definitive place to keep track of their contacts rather than having to replicate and maintain a different list for each service.  As such, social graphs are important assets to anybody in this space.  But if Tim O’Reilly gets his way (which I hope he does) and we end up with a loosely-coupled internet OS, this won’t be a problem for startups, as they’ll just be able to draw from an openly available graph, say from a Google or Facebook.

I’m really not sure how this aspect will play out.  My guess is that Apple will rely on the de-centralized social graph which is the contact list built into every iPhone.  It’s a less useful corporate asset than if it were properly cloud-hosted, which will make it harder for them to expand the service to OS X machines.  Perhaps they’ll make something useful out of mobileme here, but I have my doubts.  But given the revenue they get from App Store sales, it’s not clear that the OS X machine is even a major part of their consumer strategy going forwards.  If so, this would likely be a strategic shift for them, as the inclusion of web-cams on essentially every OS X machine for years was probably done in anticipation of making a major push into video chat at some point.

Don’t forget Cisco

In addition to the obvious players like Google and Skype, this is also incredibly important for Cisco too.  Cisco has long been interested in video chat.  Why?  The same underlying reason Intel has been investing in multi-media since the late 1980’s.  Multimedia on PCs needs lots of CPU power, and video chat needs lots of bandwidth.  It’s called “primary demand stimulation.”  To be very clear: Cisco wants everybody to use video chat because video chat uses lots of bandwidth, and when people are using lots of bandwidth, Cisco sells more big routers.

Cisco is in the final phases of buying Tandberg, who is the biggest supplier of video-chat hardware for businesses.  Their aquisition of Flip last year seems strategically odd in isolation, but in this context makes perfect sense.  They are building (buying) expertise in consumer-electronics which can handle high quality video.  Take a Flip camera, add a network (like a linksys wifi box) and you’ve got a video chat terminal.  I predict we’ll see such a toy out of Cisco in about 2012, as video chat fills the early majority segment and edges against the late majority.

I was patiently waiting for the feature to be turned for my account, which wasn’t happening.  I kept getting the same old spreadsheet program.  But then I found the setting to enable it.  So I’m sharing with y’all an explanation of how you too can get these great new features.

Go to http://docs.google.com/ and click the select “Settings” link in the upper-right hand corner, and choose “Documents Settings”.

Then choose the “Editing” tab and check the checkbox next to “New version of Google documents”.

Now any new documents you create will use the new real-time editor.  And anybody you share them with will get the new features without having to set this up for their account.

Preparing for this talk has been a lot of fun.  I’m guessing it will be fun to deliver as well.  (I’m writing this to be posted immediately after my talk, so I can’t know for sure yet!)  The whole process reminds me how much I love my career — the huge impact I can have on making people’s lives better.  This particular talk was a very good reminder to me how much I rely on my training as a scientist to perform this job.

UniXKCD command line console

My favorite webcomic, Randall Monroe’s brilliant XKCD, is running a command-line version of itself today.  A few commands you might want to try include:

• find
• wget http://xkcd.com/
• Make me a sandwich
• go west

Google renames itself to Topeka

In honor of Topeka, Kansas renaming itself Google in a bid to get ultra-high-speed broadband installed, Google has renamed itself Topeka today.  Although Google is well known for April Fool’s jokes I believe this is the first time any have been on the homepage.

YouTube’s TEXTp mode

YouTube has the option to render most any of its videos in ASCII by adding the &textp=fool parameter onto the URL.  Looking at bandwidth graphs I can’t tell if they’re actually sending ASCII over the wire, or doing the conversion client-side.  Fun trick though.

Bing’s funny cows

Bing has one of their defining pastoral pictures, this time literally bucolic, but with fake cows.  They’ve supposedly been bread to only make non-dairy creamer.  Glad you’re trying, folks.

Great job everybody!  It’s awesome to see what you can do without leaving your browser.

I’m quite excited about being invited to speak in this forum.  Science and physics in particular have always been very close to my heart.  Although I spend my days working on what I consider very practical problems compared to pure science of physics, I always enjoy musing on the underlying rules that govern our universe.

The conference is in Portland at the Pacific University of Oregon on April 16th and 17th.  If you have reason or occasion to attend, I encourage you to do so.

Diversifying from hardware

Since Jobs pulled the company out of the doldrums, Apple has been a manufacturer of high-end hardware.  This is a very nice place to be. Their margins are very high.  If you compare the hardware bits that go into a Mac to those in any PC, Apple’s prices are much higher.  PC manufacturers squabble over low single digit margins because their product is almost completely commoditized. HP, Dell and IBM struggle to differentiate themselves in the market.
Meanwhile Apple can charge a hefty premium for good industrial design and software that is slicker than windows.  iPod followed in this tradition of high margins by setting the bar for usability in portable
media devices and following up with fantastic marketing.

Being a niche retailer of high-end products is a comfortable and stable place to be.  However, having a single line of revenue isn’t good for a large company, so diversifying makes a lot of sense.  Thus Apple’s current push into services.

The iTunes music store was an important pre-cursor to the current push into services.  Even though iTunes moves a massive volume of music, if you work through the accounting confusion, they’re not actually making much money there.  Because the established music industry giants are completely fucked and have no idea what to do, they drive the margins of all big online retailers down to a pittance.  But iTunes did something very important for Apple: it established a billing relationship with customers. Cellular operators have this kind of ongoing billing relationship with their customers which enables them to push high margin, low utility products like ring-tones. Similarly, people are used to spending small quantities of money in iTunes to get music. So iTunes is the perfect precursor for an App Store.  In this way, Apple doesn’t even need to rely on the carrier’s billing relationship
to build a services business.  This will be very important for Apple in coming years as carriers increasingly become just another provider of wireless bandwidth.  In the coming decades, the value will not come from piping bits around.  It will come from the services built on those pipes. 

MobileMe is a very Apple service

MobileMe is a cloud-hosted email, contact and calendaring solution. There’s nothing revolutionary about this.  Google offers all of these services for free.  By charging for these services, Apple is implicitly promising to provide a better solution.  Considering their vertical integration into popular hardware, it’s not hard to imagine that they will succeed at this.  Google will probably remain committed to supporting open standards for working with hardware.  By using proprietary protocols, Apple can provide a higher-quality product and support it better on the few platforms they care about.  It’s a classic story we’ve seen in this industry before.

MobileMe’s launch was a disaster.  This isn’t surprising as Apple is yet to demonstrated great skill in online services.  Personally I believe they’ll figure it out, because it’s important to their long-term objectives, and they’re a smart company, and the skills to do this well are getting easier and easier to find. But from a marketing perspective, MobileMe is nothing new for Apple.  Because MobileMe is a premium service for which there is a very good free alternative, MobileMe still targets Apple’s classic market segment. They’re still targeting people who are willing to pay extra to have something really polished.  They can stick to their classic bag of tricks, like the ads that make fun of people who aren’t as cool as the mac devotees.

App Store is new: a platform play

App Store on the other hand is going to be much harder for them to pull off.  For App Store to succeed, their primary challenge is not to attract paying customers, but developers.  Because independent software developers (ISVs) are the ones who are actually creating value in the App Store.  App is just a distributor taking a cut on that.  So what attracts developers to the App Store?  Customers do.

It’s a real chicken and egg problem.  Nobody’s going to build software unless there are customers to buy it, and it’s hard to get customers without cool apps.  Well right now Apple has the whole farm, but it’s a very small farm.  There are millions of iPhones out there, representing potential customers.  Moreover, the iPhone SDK is very rich and capable when compared to its competitors.  Qualcomm’s BREW, Windows Mobile, Palm OS, Blackberry and Symbian have all faltered for one reason or several.  Google’s Android holds much promise and hope, but at this
point it’s complete vapor-ware.  So for now, Apple has almost all the mindshare of mobile application developers.

But how long will this dominance last?  Software platforms are natural monopolies, meaning the economics tend to be winner-take-all.  The chicken and egg “problem” can easily turn into a virtuous cycle, pushing a winner to the top. Application developers are fickle and will code to whatever platform has the best distribution.  iPhones are very popular, but they are still only used by a small fraction of all mobile subscribers.  Until the distribution of iPhones reaches a critical mass, their dominance as a mobile application platform is very shaky.

The mobile app challenger is HTML

But the laundry list of alternatives shows that the competition is fragmented.  What could unseat Apple?  IMHO it’s not another application platform, open or proprietary.  It’s the web.  Every high-end phone can display web pages, and increasingly they’re using high-quality javascript engines that can run real web applications. Webkit, the super-fast open-source HTML/JS engine behind Safari is showing up in Symbian devices, Blackberries and Windows Mobile phones.

Apple’s challenge is to make the proprieray iPhone SDK compelling to developers.  The alternative is to build a web application that works well on webkit, and works on every high-end phone.  They can attract developers in two basic ways — make the native features of their SDK more compelling, and provide a large market for distribution of the applications.

Charging for distribution of these applications is a gimick that won’t last long.  Soon all the interesting applications will be free, but tied to cloud services that have their own business models independent of the mobile client.  Premium applications will start to seem a lot like premium ringtones pretty fast.  Still, it will help bootstrap this market for Apple so long as there are no serious competitors.

To stay on top, iPhones need distribution quickly

One thing that ties all these points together is that Apple’s continued success with App Store hinges on having wide distribution of iPhones. They are currently throttling the distribution of iPhones for some reason, possibly
because of software glitches.  But the aggressive $199 pricing is clearly aimed at attracting a new larger customer base that will help maintain their dominance in the mobile application space.

Long term they might be happy getting by offering premium versions of applications that are freely available on the web.  But something tells me they’re actually trying to break open the mass market on this one. This is Jobs’ big play.  It’ll be really interesting to see how it works out once Android hits the streets.

Google just launched another way to access the Google Talk network.  It’s a web-based instant messaging chat client optimized for the iPhone browser.  It’s not my primary project or my secondary or tertiary, but I did write a blog post about it and made sure the whole thing got out the door today.

If you have an iPhone, try it out at www.google.com/talk.  Warning: Non-iPhone browsers will be directed away.

Why bigger is better for online advertisers

The reason lies in network effects of the online search + advertising industry.  Imagine you’re an ad buyer which is to say you have a service you want consumers to find online.  Unless you’re a huge company, you have limited energy to expend buying your ads.  So rather than buying and managing separate ads from each Microsoft, Google and Yahoo, you’re likely to just deal with a single ad publisher.  The sensible ad buyer will choose the ad publisher which gives them the most value for limited effort. 

Right now the clear choice for an online advertiser is Google.  Because they have the most search traffic, they are best able to reach customers.  Combined with their adsense network, Google clearly has the largest inventory for an ad buy making them the natural choice for anybody not willing to spend a lot of energy managing their online advertising.  This logic underlies the recent acquisitions of Doubleclick, AvenueA/Razorfish/whomever, and now Yahoo!  Network effects in advertising mean that the largest network will be the most sucessful.  So the mergers will continue as far as the anti-trust regulators allow them to until a handful of bitter enemies remain.

This much might be obvious to some of my readers.  But I felt like sharing this analysis since I’ve read nothing in the common press that explains the basic economic motivation of this deal.

Wrinkles, twists

An irony of the network effect comes from the auction nature of keyword buys.  Advertisers bid for the right to get their message in front of customers.  When more advertisers are competing for the targeted eyeballs of consumers, the prices for advertising go up.  This means that prices will tend to be higher on the larger ad networks.  So bargain seekers can get more advertising for their dollar by seeking out smaller networks.  This appears to contradict the logic that bigger is better for ad networks.  But many advertisers are limited not so much by budget but by the ability to reach highly qualified customers.  If you are selling poodle tattooing services in the pacific northwest, odds are you will not hit max out your advertising budget on any of the ad networks simply because not that many people are searching for your services.

I could probably fill pages with speculation about the culture clash between Microsoft and Yahoo and other reasons why it will or won’t work.  But if you’re interested in that stuff, I’m sure you’ll have no problem finding it in the backwaters of the blogosphere.  I can’t help but drop a couple relevant ideas though.  First, from what I hear, the executive management at Microsoft is so dysfunctional right now, Yahoo will provide fertile new ground for their turf wars.  If the top bosses are adept, they will use the many iterations of re-orgs to sluff off ineffective execs to projects where their overall damage can be minimized.  Second, I think I hope Microsoft has evolved enough humility to understand that they’re better off simply shutting down Yahoo’s services than forcing everything to port over to NT servers.  Right, guys?

Disclaimer

I feel compelled to point out that the opinions expressed here are mine and mine alone.  In no way does this article reflect any official position of my employer.  This is my personal analysis of the economics behind the industry I work in.

I spent my first week in Mountain View at the Googleplex.  My entering class of "Nooglers" were subjected to inane videos and boring HR discussions.  But a couple hours into it we powered up our laptops and within 15 minutes I’d found the internal wiki and started reading project plans for every internal initiative I found interesting and a bunch that I didn’t.  I devoured the information and didn’t unplug myself until wee hours of the morning.

I’ve been spending a lot of time over the last few years analyzing the industry and thinking about what opportunities exist for creating value by solving people’s problems on the net.  Many of those I’ve captured here.  Now I look at the world differently in terms of what problems are still left to be solved because I can see that Google is in the process of solving many of the problems I’d identified.  It’s a little difficult for me to remember what I thought before knowing Google’s plans.  A myriad of half-written blog posts help remind me.  I had been planning on finishing many of them but now I don’t feel so comfortable doing so.  For example, writing about security holes inside Gmail is fun target practice from the outside, but questionably ethical from the inside even though I’d identified it before joining.  The same applies to unexploited business opportunities.

I’ve been asked a lot about the best and strangest things encountered during my week in Mountain View.  The best thing was hands-down the food.  It’s amazing.  Almost every building has their own restaurant with a theme.  My building had a tapas-inspired restaurant featuring many small plates and often fabulous seafood.  The best hamachi sushi I’ve ever had was served there on a real shiso leaf with some light sauce I can only describe through the ecstasy I felt from it.  They served black cod, which I love love love.  (I’ve got a great recipe I need to post to addgarlic.)  Pumpkin bread pudding.  Fresh figs everywhere.  Chilled beet soup.  Even simple things like a ham and cheese sandwiches on fresh bread with arugula were fabulous.  Other cafes have themes like organic hippy foods, dishes prepared with a maximum of 5 ingredients, or everything grown within 150 miles.  It’s all amazing.  As a result I found myself drawn to campus in a predictably Pavlovian manner.

One thing I’ve said for a while that this trip reinforced was the idea that you can’t pay your corporate cafeteria’s chef too much money.  You can get a chef for $50k/yr or $150k/yr.  That extra $100k/yr will do so much more for employee satisfaction than pretty much any other way to spend the money.  Sure you’ll end up spending some more on ingredients or subsidies.  (Or else the chef will leave.)  But it’s worth it.  A couple years ago Real hired a new Chef, Ariel IIRC for their cafeteria and the food got so much better I started bragging to my friends about it.  A little while later a number of things happened at about the same time — Real’s stock dropped, Ariel moved on and life at Real wasn’t as much fun any more.  I won’t try to extract the causality relationships between those events here.

The oddest thing I saw was definitely the automatic toilets.  They’ve got butt-warmers, front and back washing sprays, dryers and more things that I never figured out.  I wonder if they weigh you and keep a high-score list for largest excretion.

Now I’ve got the fire-house turned on full bore and am trying to add value for my team from a position of relative ignorance and keep up with everything going on around me while finishing up a full load of business classes.  But I wanted to take a few minutes to share what’s been going on with you my dear readers.

Most significantly is around intellectual property.  Google’s IP policy for its employees can be effectively summarized as "All your base are belong to us."  It’s a fairly standard employment agreement — anything I do or think of on Google’s time or using Google’s equipment belongs to Google.  The only exception is if I do something entirely on my own that is not related to Google’s current or reasonably foreseeable future business.  I’m not a lawyer, but California and Washington laws both read about the same.  The thing with Google is that essentially nothing in technology is outside of that scope.  Designing juggling balls or running shoes might be.  This was a concern for me in considering the position.  But in the end I couldn’t resist.

I suspect this means I won’t be able to post as much about what the industry needs to be doing.  On the flip-side, hopefully I’ll be in a position to be getting the industry to do these things.  People often ask me what I’ll be working on, and I always answer honestly that I don’t know.  As a Product Manager I’ll be working on products but not writing code — this is similar to a PM role in other companies, but there are very few at Google and their relationship is much more of a peer than in some companies.  As to products, I think everybody has to work on ads as a kind of penance.  But hopefully when I get my feet on the ground I’ll be working on all the things I’ve been posting about here.

Personally this means I’m going to be extremely busy for a while as I finish up a full quarter at school and start up a new job.  I’ll also be a cross-bridge commuter heading into Kirkland most days which I’m really unexcited about.  But the opportunity to work with lots of brilliant people and have a huge impact on the world makes up for it.  I’m pretty excited!

Economically, companies should be valued at the present value of their free cash flows.  That is to say, project forward all the possible ways the company might behave, and take a probability-weighted average (expectation value) of the total dividends the company would pay in each of these scenarios.  Discount these cash flows by an appropriate discount rate and you’ll get a fair market value for the company.  This is called fundamental analysis.

Now anybody who’s tried their hand at such financial calculations will know there’s a lot of judgement calls involved.  Small differences in numbers like discount rate or growth rates have huge effects on the results, and these numbers are hard to judge.  So it’s definitely possible to come up with a believable (by some) model of future cash flows that will value any currently successful company at whatever huge valuation you want.  But that doesn’t make it correct.  Is Facebook worth $300 per user?  It’s not possible for me to click on a $10 CPM ad every day for 100 years, but maybe they can add more users to grow into that?  Maybe?  It sure seems high.  I think there’s something else going on.

For context, think back to March of 2005 when Yahoo bought Flickr.  IMHO that made Google feel bad because Picassa wasn’t doing so well.  I think they saw this as a big missed opportunity to help organize the world’s photos.  I think this was big on their minds when they paid too much for YouTube.  And Google is still very far from monetizing this investment.  But they now control the dominant way that videos are communicated on the net.  This has to help them feel good about getting closer to their corporate mission of organizing the world’s information.  Since it’s not clear right now how they’re going to achieve that goal for photos.

Now consider Facebook.  Left and right, Facebook’s internal applications are surpassing total usage of th best dedicated net applications.  Their invitation app gets many times more usage than evite, and I believe their photos app is actually well beyond flickr in terms of usage too.  I don’t know where they stand for videos right now.  But it’s clear that they are a force to be reckoned with.  As I’ve written before, their application platform is potentially game-changing because it’s very attractive for information service developers and democratizes the process of product development in a novel and powerful way. 

For all these reasons, I think Facebook has the potential to dislodge Google as king of the hill.  No, Facebook isn’t going to become the dominant search engine, or even the dominant deliverer of internet advertising.  But I think Facebook could become the dominant way the humans communicate with each other using computers.  This could be the leverage they need to claim the crown of innovative thought leader on the internet.  If I were running Google, I’d be concerned about this possibility.  If I were running Microsoft, I’d be excited to get a piece of this.  Any piece.  Because even a tiny piece (like <2%) means that Google can’t take control of Facebook.  And yesterday, Microsoft got their foot in that door.  So, the game is on.  It’s gonna be fun.

I haven’t seen anything work differently yet.  But that’s really the way it should be.  The Google Reader implementation of gears where you need to explicitly tell it when you’re online vs offline isn’t an ideal experience.  And the Gears API doesn’t require apps to be written that way either — it’s just easier.

I tried unplugging from the net and nothing interesting happened.  I couldn’t update my calendar.  I couldn’t load my calendar — I was hoping maybe they’re just caching the javascript in gears which would be a nice step.  But it doesn’t appear to actually be doing much yet.

It’s just a matter of time.  I predicted first half of 2008 and I’m sticking to that.

Original Web 1.0

Universal access to massive volumes of data.  Being able to search
through masses of data and find what you want.  Connecting people to
huge databases really well.  Key examples:

• Online telephone books
• Web search
• Huge e-commerce sites

But in all of these applications, the data set is static.  User activity will not change the data for anybody else.

Web 2.0.1: Democratizing use of the data

The users of these data make the data better.  They can collectively
organize the data.  (i.e. tags)  They can help filter good data from
junk.  (i.e. voting)  Or they can help you find the data that are most
interesting to you.  (i.e. collaborative filtering).  In other words,
you can interact with the data.

Any "Web 2.0 company" worth their salt has an API that federates out their raw data.  This enables other sites to use the data in new and novel ways.  But the primary problem with this paradigm is that anything built using these API’s is done from the ground up.  Using the gmail POP interface, it’s possible to build a better UI for gmail.  But to do so you need to first build an entire AJAX mail client — no small feat.  Better would be the ability to add features into the gmail UI itself.  But this is really the standard in web 2.0 API’s today.

Web 2.0.2: Democratizing the feature set

The next big trend will be enabling users to make more compelling ways to interact with
the data.  Users can change not just the data, but how other users see
and use the data.  Sometimes this means API’s with UI hooks.  Or other ways to enable new functionality into an existing site.  This kind of platform enables Independent Software Vendors to improve upon the UI’s that the original sites created.

Facebook is doing this by allowing ISV’s to add new communications features to their site.  Google Maps is doing this through maplets that allow developers to create new ways to interact with mapping data from within the fabulous Maps UI.  Right now these are the only two examples of web 2.0.2 platforms that I’m aware of.

Building this kind of API is very challenging.  There are several very different ways to go about doing it.  Here are a couple of ideas:

• Provide a server-server API that includes content generated from the ISV’s servers into the main experience.  Facebook style.
• Allow developers to author XML files that define new algorithms that are interpreted on the primary host’s servers.  Yahoo pipes is a service in this style, but they’re not doing anything to enhance an existing service so it doesn’t really meet my 2.0.2 criteria.
• Allow developers to author javascript plugins to run on the client machine.  Greasemonkey is essentially doing this.  This strategy has the best shot for a lot of applications in the long term, IMHO.  But it comes with some serious problems right now.

Doing this correctly would allow ISV’s to add new features to Gmail.  Think about it — if I wanted to change the way gmail messages were displayed, or how addressing happened, or whatever it was, this kind of platform could provide hooks for making gmail better in a way that a POP interface never would.  And even though a POP-style interface theoretically could do this, there would never be momentum because having a high-level base to build upon means that there are network effects from the extensions.  (Rails achieves a similar advantage over other web frameworks — just having a standard, any standard, means people will build upon that standard rather than argue over which library to use and extend none of them.)

3rd-party javascript

The big problem with this approach is security.  There is none.  You need to completely trust the ISV before you should allow their code to run in the context of your site.  The kind of editorial review required to do this today would completely kill the democratic goal of such a platform.

The world needs a security sandbox to run third-party javascript code inside.  This way primary site hosts could allow ISV’s to run their code on client machines safely.  Here are a few examples of places this kind of tool could be used.

ISV’s could…

• Add UI features to Gmail
• Create alternate ways to share and discuss images on Flickr
• Define new mathematical formulas to run client-side on a web spreadsheet
• Create new playlist selection / shuffling algorithms for Rhapsody

… and much more.  Even better, individual users (not developers) could pick which UI extensions they wanted to use.  Any site which provides such an API has democratized the feature development process in a very important way.  Not only does it provide a distributed mechanism to figure out which features are best, but it allows users to self-segment as to which features work for them.  Without such a mechanism the entire service must have the same features for everybody, which means product designers must play a political game where they’ll never make everybody happy.  Right now I think really only Facebook has solved this problem.

Building a security sandbox is an area that Microsoft could probably do best and fastest.  They are good at code API’s and layered security models,a nd they have a perfect place to do it with Silverlight and the CLR.  They’re trying to position Silverlight as a faster way to run DHTML, which is something else the world desperately needs right now.  But I just can’t imagine them doing anything this innovative or generally valuable.  Doesn’t sell more Office.  Doesn’t sell more Windows.  They don’t even really have many services that could use third party extensions, and they’ve lost touch with the ISV’s who might build such extensions too.  Google Gears could conceivably add such an extension.  There’s precedent there considering the javascript threading extensions they provide.

This will be a difficult problem to solve, I have no doubt.  But I hope somebody with the resources to leverage a solution takes it on, because I think it would really make the world a better place.

Other alternatives include forwarding list-mail to another list-only account and deleting it, but that is frankly kinda lame.  Or if there was a way to create a filter that runs only on messages left in the inbox after the other rules have run, then I could forward the inbox messages to another account that I read with my phone.  But I don’t think that option is currently available either.

Gmail is written using a classic MVC pattern — Model, View, Controller.  The View reflects changes in the Model, and the Controller tells them what to do.  Clearly the View is our web page, and in this implementation, the Model is stored on Google’s servers.  What this means is that when you do something like archive a message, you’re not going to see the results until a message is sent to Google’s servers and the response comes back.  Now Google has really fast servers, but networks are slow.  By following this design pattern, Gmail is easy to code new features for and maintain relatively bug-free, but it’s never going to be all that fast.

Gmail engineers could code up optimizations for special cases.  For example, when I’m looking at my inbox and I hit archive on a message, they could detect this case, and just remove it from the page and then tell the server afterwards.  This kind of coding will quickly make the Gmail hard to maintain and result in lots of bugs.  Especially when you consider that multiple clients can connect to a single gmail account at once.  Strictly following MVC makes it easy to resolve conflicts in one place — the server.

A Proposed Solution

There is a solution though.  Google Gears can help.  The Gmail UI could be (and for all I know currently is being) re-architected to use Google Gears as the Model in the MVC pattern.  This means any operation in the UI would be committed very quickly to the SQLite database on your hard drive.  Then another thread would synchronize these changes in the background with the master model on Google’s servers.  Everything works — UI operations are reflected to the user extremely quickly.  We don’t lose all the benefits of storing all our data on the server.  And as an added bonus, we can still work on our e-mail when our computers are offline.  Yet another reason why Google Gears is important.

So when is this coming?  Synchronization and it’s evil twin, conflict resolution, are complex software tasks, so it could take a while.  But if I were placing bets, I’d guess we’ll see this in the first half of 2008.

In this environment it’s tempting to design products that assume customers will always be well connected.  It is certainly easier to build compelling services to users that have a good pipe to the net on them at all times.  So this begs the question: If customers will soon always have good broadband net access, why do we need a client-side data store like Google Gears?  For example, somebody working on a subscription music service might conclude that it’s a waste of time building portable mp3-players with local storage since soon enough everyone will have broadband access everywhere, so why not just stream the music off the net?

There are several good reasons why client-side storage is still important and will continue to be important into the future:

• Wireless net access sucks down battery.  Always will.  It’s physics.  Local access to data will always cost less battery.  This won’t change no matter how pervasive broadband is.
• Pervasive net access is expensive. Arguably we’re already in a world where some people have pervasive net access.  Verizon EVDO cards do pretty darned well in this country, for $60/month.  But it will be a long time before most people have it.  Higher speeds will always demand a premium.
• Net access is unreliable.  Especially wireless access, but wired too.  Packets collide.  Transmission patterns have nodes.  Routers flap.  Cables get unplugged.  Laptops wake up and can’t figure out where they are for a while.  Something gets misconfigured.  If your software is designed to gracefully degrade when the network is unreliable, your customers will be happier, because it’s going to happen.  Remember what Outlook/Exchange was like when the entire Outlook UI would freeze while waiting for the Exchange server to respond to any request?  Please don’t do that to your users.

Once web applications are fully embracing it, Google Gears will close most of the functionality gap between native-client applications and web applications.  I believe it’s really important, and I’m really glad that there’s industry consensus around Google Gears and that other offline browser storage projects have deferred to it.  I’d hate to see web app developers trying to choose between several different client-store plugins.

What I’ve been thinking about building in my copious spare time is a web-based feed-reader that assumes you over-subscribe to feeds.  That is, it expects you to "subscribe" to more feeds than you can fully consume.  These days many of the most popular feeds on the web meet this criterion even if that’s all you subscribe too.  I don’t have time to follow any one of TechCrunch, Scobleizer, Engadget, or even Radar in their entirety — I generally don’t even get to skim all their headlines.  But I know people in my social network do, and when they do it would be a small extra effort for them to help me identify the posts that are worth me reading.

This could be done by explicitly recommending articles to friends, or by tagging, or rating, or any of a number of well-understood-yet-often-poorly-implemented mechanisms.  Additionally, I could subscribe to a meta-feed coming out of a single-friend or a set of people in the social network graph that could expand several levels.  And of course there would be meta-feeds covering the aggregate opinions of all users.  The result would be that I could "express mild interest" in a feed by "subscribing to it" and the system would help me figure out which of the voluminous posts were actually worth reading.  Or if other users tagged posts, I could find good posts on a particular topic.  It would encompass a lot of the utility of digg, techmeme and link blogs all at once.  Another step in the process of democratizing information consumption.

I’ve been talking with friends about building this in the context of a facebook app for reasonably obvious reasons.  I’d call it "the outside world" as a reference to the fact that college kids are generally so isolated from external news, and this would be a social way for those few who do read the traditional-news to share good things with their friends.  Facebook’s restrictions on apps processing social networking metadata would make somegood features difficult, but the advantages in marketing and lower barrier to entry probably outweigh that.  Now my idea is out there for the world, so I’m not getting a jump on anybody.  If anybody wants to take this idea and run with it, drop me a line and I’d be happy to help advise.  I might just do it anyway because the Facebook market and the Google Reader market are both healthy and the basics just aren’t that hard.

But it sounds like you’ll have stiff competition.  Quoting from Blogoscoped’s analysis of the video:

Google’s recent big social effort is called Mocha-Mocha (or
Mocka-Mocka?), and will become the infrastructure for all social stuff
across all of their applications. As a part of this, a new
feature called Activity Streams will be introduced or at least
implemented in Reader this quarter. This will be comparable to
Facebook’s News Feed (Minifeed?) feature, and integrate Gmail’s
addressbook and contact list.

Also there will be some other Gmail and Orkut integration, but this might just mean there will be links to Reader.

Hearing that Brad Fitzpatrick has joined Google and because it’s the kind of thing I do, I’ve been putting some thought into how Google could reasonably add social networking features to their services.  I’ve been talking to folks about how Facebook is currently Google’s biggest strategic threat because they’ve done such a good job integrating the social network into new feature development, and in doing so have democratized new feature development in a way the world has never before seen.  This need struck me as a good way to start integrating social networking features into Google. 

Orkut is and only ever will be a toy IMHO.  Let the Brazilians keep playing with it and don’t push it on the rest of us.  Between contacts and knowledge about whom we chat and e-mail with, gmail has vastly more meaningful set of social networking data.  As we’ve learned watching LinkedIn and okcupid and other social networks thrive side by side, it makes sense to have different social networks for different purposes.  Orkut is a toy network and should not be the basis of anything more meaningful.  Sorry, Orkut.

Amazon makes extensive use of so-called A/B testing to try out new UI’s and optimize the user flow.  This works very well for them because their end goal is very well defined: they want people to buy stuff.  They are facing a very hard optimization problem, but their objective function is clear and easy to measure.  So they can try out new UI’s for 1% of users, and if it does well according to this well-defined metric, roll it out to a broader audience.  This is essentially best practice for any modern successful online company.

Google has done a lot to democratize the internet — notably by democratizing search through PageRank which allows anybody to implicitly vote on the relative merit of a web page.  They have also democratized the way some product development choices are made through through their policy of encouraging developers to build whatever they want in 20% of their time.  The result is that everything you can possibly imagine is probably being worked on by at least one googler, and the ideas with merit gain momentum and get built into real services.  But before they get launched to the public they still must be approved by a central authority.  Sure Google does A/B testing like everybody else, which is great for UI tweaks and to verify that new services won’t crash when hit with massive traffic.  But it’s extremely difficult to do A/B testing on major changes to functionality.  For example, it’s hard to imagine testing a change to how g-mail delivers mail through this kind of test.  Moreover, depending on how the test goes, the change is either rolled out to the entire user base or not at all.

Facebook’s platform offers another alternative.  ISV’s have the opportunity to offer major new kinds of functionality to Facebook users in a very democratic way.  Users can try out the new features, and if they like it, they’ll tell their friends about it, and the feature will spread.  Some features which are only appropriate for a certain segment of a user base can naturally find that segment.  This mechanism doesn’t really lower the cost of adding new functionality compared to how Google does it — Google is always launching new features that you’d never know about without reading their dozens of product blogs.  But it democratizes the process of figuring out which of these new features are valuable enough for a mass audience.  To continue with the democracy analog, these decisions are still made by a communist-style central-planning committee in Google’s world, whereas Facebook users can vote with their keyboards on what features are worth using.  This will make the Facebook platform very competitive in the arena of user’s attention.

I plop a few bucks into an adwords account and almost instantly the site shows up in the public search index, with a whopping pagerank of 2.  Not high enough that it was an obvious mistake to omit it from the index, but also not so low that it clearly doesn’t deserve to be in the index because there are other more important pages being indexed.  "Do no evil" they say.  This isn’t clearly a case of evil.  But it certainly seems like a bias.

There were lots of page-search engines along the lines of Alta Vista.  They crawled the web and indexed the contents of each web page.  They would try to figure out which web page best matched your search keywords based entirely on the contents of the pages themselves.  This didn’t work very well since spammers could fill their pages with keywords they liked that didn’t necessarily add any value to you.

Yahoo searched a database that was built manually instead of by crawling.  Yahoo’s staff would catalog thousands of web sites and categorize them according to keywords.  This provided a higher level of quality since a human reviewed every entry, but they were having trouble keeping up with the explosive growth of the net.

Then there was these two punk Stanford kids with their upstart Google.  Google was a lot more like Alta Vista than Yahoo in that they automatically crawled and indexed the entire web.  But they judged which pages were useful not based on what was on the page itself, but on other pages on the net that link to it.

We all know who won.  But I’d like to share a perspective on why that uses a political analogy. Google democratized search.  Yahoo was based on a communist model.  Alta Vista was complete anarchy.  Democracy won because it gives power to the people, and the aggregate opinion of millions of people is almost always better than even a carefully chosen set of experts.

Yahoo’s model was analogous to having a central politburo that makes all decisions.  Provided the politburo is skilled and benevolent, this can be a great solution.  But if the system they control gets too big, it just won’t work.  Alta Vista’s ranking system gave everybody speaking equal say in what happened, which amounted to total anarchy.  Google allowed every web page on the net to cast a vote on which pages were the most important ones.  (Beyond that, the pagerank system iterates so that some votes coming from more important pages count more than other votes — the details of implementation are always key.)  Information democracy is achieved by giving everybody a say in what’s important and aggregating the reults.

There are numerous examples of democratic services completely obsoleteing services based on communist editorial systems.  Wikipedia democratized the encyclopedia and has replaced Encarta.  Youtube democratized internet video clips and replaced iFilm.  In many more cases, the democratic service hasn’t replaced the centrally-controlled services, but provides a strong alternative.  E-Bay democratized shopping.  Blogs have democratized news.  Open-source software has democratized software development.

Clearly democratization isn’t a silver bullet for every problem.  Expertise is much more rare and valuable in some fields than others.  But if your business today is based on having a database that your staff maintains, take note!  Somebody’s probably out there right now figuring out how to build a competing business where anybody in the world can contribute to their database.  And pretty soon they’re gonna be taking pot-shots at your market.  Managing user-generated content is really hard.  Counting votes is really hard.  But if it’s done well, it will dominate any system based on central editorial control.  With the help of computers, groups of people can solve problems far more effectively than individuals can.  This truth will not change.

(I brought this idea up at the Decibel Festival talking about music editorial systems and received a lot of positive feedback about it, so I wanted to post it.  But I must give credit for the vocabulary to my good friend Ramez Naam — I first heard it when he was critiquing a startup’s business model as being communist.  At first I laughed, but later I appreciated his wisdom.)