Embracing Chaos

I spent Saturday hanging out with about a dozen hackers building Tagmindr: Remember the future.  Here’s the site’s self-description:

Put any bookmark in a time capsule and we’ll send it to your future self.

Give us your del.icio.us username and we’ll feed you anything that you’ve tagged as: "tagmindr" and "remind:YYYY-MM-DD". We’ll remind you via RSS, SMS, Email or IM, so long as it’s RSS.

(SMS, Email and IM coming later.)

The use case is that you find a page about a product or service that you’d like to look at sometime later so you tag it into del.icio.us thinking you’ll get back to it, but of course you never will.  With Tagmindr you can set a specific date when it will pop up in your feed reader so you will remember to check it out again.

Brian Dorsey came up with the idea and gathered a bunch of us together at his house with the goal of building a web 2.0 app in 6 hours.  We spent an hour or two setting up our dev environments and talking over the goals of what we were going to do.  Then we did a skills inventory, and divided up into teams to start doing the work.  I worked on the back-end team which was a ton of fun.  The project is written using Django, an MVC-based web application framework for Python, which is conceptually quite similar to Rails.  It’s got a few things that are way cooler than rails and a few things that are definitely lacking.  From noon to 6pm we coded, while others did graphic design, HTML layout, and wrote copy.  I had to leave fairly promptly but at the time it seemed we had slipped just a bit — there were still a few issues rendering the design on the production server, and the back-end code still had a couple of bugs.  It seemed like another hour or two’s work total.

Thanks and props to all the wonderful people I met and got to work with.  Special thanks to Anders for holding my hand through basic Python and Django to a level of minor productivity.  I gotta say that Python is really clean.  Makes me realize how much Ruby can look like incomprehensible Perl.

After 6.5 years of having dreadlocks, I decided I was done.  So while I was at Burningman this year, I had my friends help me cut them off.  I wandered around on Saturday with a pair of scissors, and would ask friends one at a time if they would help me by cutting off one of my dreads.  Some of them went into a bag for future nefarious purposes, and some of them got put into my new dread hat, and others got burned ceremoniously.  In all, 33 people helped cut my hair.  "It takes a village" Mason said.  I did a tiny amount of trimming myself afterwards, but really this is a community haircut.  At some point I’ll need to get a professional to clean it up.  But considering how it was formed, I think it’s not bad.

Now, with a few weeks of introspection and perspective, I can explain my thought process and motivation.  A big part of it was that I was sick of maintaining them.  For a pale guy like me, keeping dreads looking half-way decent is a lot of work — arguably more than I was doing.  I was attracted to the hairstyle initially because of its promise of never having to do anything with your hair again.  That might be true for somebody with thick, curly african hair, but not for me.  I had to go at them with a crochet hook for an hour or two every couple of weeks to keep from looking too fuzzy.

Now that they’re gone, I realize a number of things about them that were really downers: I had a giant lump of matter stuck to my head.  It got in the way when I lay down.  None of my hats really fit anymore.  Helmets have all needed re-adjustment.  I was always pulling it out of my face.  My hair took forever to dry.  All these hassles are gone.  But bigger than that, now people can run their fingers through my hair!  I feel like a purring cat whenever it happens.  I probably have a limit for how long I find that activity enjoyable for, but I haven’t found it yet.

But the most substantial reason was that I knew I was ready for structural changes in my life.  I knew I needed to focus my energy on school rather than my job.  I’d been working at Real Networks for 4.5 years and learned a ton, but doing both has been a killer.  As I squeezed the scissors through that first lock I was thinking to myself, "If I can cut off my dreads, I can make other big changes in my life."  It was an act of empowerment.  I control my destiny.  I actually haven’t exactly quit my job — I’m taking a leave of absence from Real for 6 months while I finish up my MBA at which point I’ll consider all my options.  But I am in control of my time, and time is the most critical resource any of us have.

Next step in taking control of my life: Puppy.  That one’s still pretty scary for me.

My talk about Venture Capital Term Sheets that I mentioned earlier has been accepted for the next Ignite Seattle which is tomorrow night at the CHAC.  If just you want to come hear me talk, I’ll be speaking sometime after 9:45 PM.  But the rest of the night (schedule here) looks fabulous so I plan to show up at the beginning at 6:30 PM.  I’m doing my best to make this talk slower and more sane than my last talk.

I hope to see you tomorrow!

Here’s a shout out to my friends — this weekend I’m making my third attempt to climb Mt. Rainier.  (The first two were successful.)  But we’re in for a challenge since the glaciers are really broken up.  I’m trying to focus this blog on more technology and business issues, so I won’t be talking much about it here.  If you’d like to follow along, check out my ironically named outdoors blog Safety Fourth.  (Look good, act cool, have fun… Safety fourth!)

A number of folks have been courting me for positions as a software development engineer recently.  Many of them sound really fun.  I love writing code, and even though it hasn’t been my professional focus for years, I think I’m still alright at it.  But I’ve had to come to a sober realization that I simply cannot take a job where writing code is my primary function.  I’m just not physically capable of it.  I just about cried during a job interview once when explaining this.

People come in my office and say "that’s a cool keyboard."  My response is generally something like "that’s what it takes."  I don’t use a $300 keyboard because it looks cool.  It’s more like a wheelchair.

I have repetitive strain injury.  It’s not carpal tunnel syndrome.  My wrists basically never bother me.  It’s more my pinkies, more the left than the right.  I can think of 3 causes.  In high-school I took a 3-day bike tour from San Diego to Santa Barbara wearing gloves that weren’t well padded.  For about a week afterwards I had no feeling in my left pinkie.  Then in 1996 a marathon perl coding session of several 100-hour weeks left my pinkies pretty sore because of all the mixed-case variables in our code. 

Also, I suspect cubital tunnel syndrome might have some part of it.  I’m not expert but my understanding is that it’s a result of spending too much time with bent elbows stretching some nerves causing problems with pinkies.  I understand the easiest thing to help is to sleep with straight arms.

The ironic thing is that with a good ergonomic keyboard I can write English text day
in and day out, but more than about 10-15 hours/week of coding and
things start to hurt.  Writing code on a laptop in bed, fun as it is, will get to me after about 10 minutes.  Code involves tons more time on the special keys
than English.  You don’t need a lot of
[]{}()<>*=+-~&^_!#@$%/|\ in e-mail or in specs.  And you don’t
need to bounce around the arrow keys for editing nearly so much
either.  The Kinesis advantage keyboard helps quite a lot, because it moves most of the keys that the pinkies do under the thumbs.  But it’s not quite enough for me, because the shift keys are still the responsibility of the pinkies.  I’ve tried using a foot-pedal for shift, but haven’t been able to
re-train myself.  And I’m not sure I really want to limit myself by
having a job I can only do in front of a super-tricked out
workstation.  Screwing up (gee it sure would be nice to get
something done on this here laptop) means having to use a microphone
and eat grams/day of ibuprofen for a few months which I’ve done and really really
sucks.

So physical disability has sent me down an interesting career path towards middle management.  I have tons of fun in this capacity, but sometimes I miss making things work with my own hands.

Also, if any of you have hints of repetitive strain injury, I strongly encourage you to shell out the cash for a good keyboard.  It’s way cheaper than the alternatives.

I just got back from foo camp, a small unconference-style event held on the Sebastopol campus of O’Reilly.  FOO stands for Friends Of O‘Reilly.  (The name came out of a joke about having a "foo bar" at a conference.  This bar served me too much wine over the course of the weekend.)  Tim O’Reilly likes the conference because it helps him spot upcoming
trends early, which is an important part of O’Reilly’s business both as
a publisher of technology books, and as an organizer of large public
conferences.

The conference is small and invitation only, and pretty much everybody there was happy to be invited to spend a weekend with such a carefully chosen group of thought-leaders.  I was personally quite honored and surprised to be invited back this year because last year it was made quite clear that we shouldn’t expect to be invited back since keeping fresh opinions around is important for the function of the event.

This year I met far more people than last year, and felt much more comfortable striking up a conversation with whomever happened to be standing next to me.  Part of that is of course my own attitude, but based on others’ comments it sounded like a lot of people had similar experiences.  Several people remarked publicly at how uncharacteristically social they felt.

I suspect that a big contributor to this effect was a comment that Tim made on the first day during introductions.  Tim said to not just talk to your existing friends, and that we were all more likely to get invited back if we were more social with strangers.  He said that we were all invited for a reason, and so we all had interesting ideas to share.  This concept helped me break down shyness and intimidation barriers.  Later Tim made an analogy to making new synapses in the global brain.  (I rather like the idea of being a neuron in some huge hive mind.  I’m excited to read Toby Segaran’s upcoming book on collective intelligence, and really enjoyed meeting him this weekend.)

If the causal relationship I’m hypothesizing here is real, I see it as a great example of the economic principal that incentives effect behavior, and more specifically how small incentives can shift social dynamics.  I didn’t get a chance to ask Tim how deliberate or off-hand that comment was, but I’ll encourage him to repeat it next year.

Here’s the scene: It’s 1:30 PM.  In 30 minutes the CEO of your company starts a conference call with analysts to announce quarterly earnings.  PR told you he is going to tell the Wall Street analysts how cool your team’s website is.  It is quite a success — in 18 months it has rocketed from non-existence to the world’s fourth most popular site in a very competitive industry.  Sounds great to get some recognition, right?  Only problem is, today your site’s kinda broken.

The night before a database upgrade got confused half-way through with no possibility to roll back.  One of the two production databases got upgraded to the new schema and the other didn’t.  As you’d spent most of the day diagnosing, the new schema didn’t quite work with your app — some fraction of pages generated from this database came out wrong.  Busted.  Missing.  Scrambled.  Paper white.  Ugh.

After hours of group futzing between you and a couple dozen other folks, you’ve managed to get the problem mitigated.  Your app now appears to be reliably generating correct non-borked pages.  But the site that the world sees is still messed up, because of your content distribution network (CDN) partner.  The CDN caches copies of your site across the world, moving it closer to customers for faster display and reducing the load on your own app servers.  But over the course of the day, the CDN has cached copies of many broken pages.  You can of course clear the individual cache for any broken page you find, causing the CDN to fetch a clean accurate copy from your app servers.  But the site has millions of pages — how are you ever going to find all the pages that need flushing?  With 30 minutes until press time it’s not impossible. 

The only reliable way to clear all the broken pages out of the cache is to wipe clean the whole CDN cache.  Push the big reset button.  This is a fairly big deal because it means millions of cached pages will have to be wiped from the CDN and fetched from the app servers again.  Is there time before the peering eyes of Wall Street come looking?  Clearing the caches takes about 15 minutes.  Filling them back up again — who knows.  The popular stuff will fill in fast, but the long tail will probably take a while.

To make it worse, clearing those caches will mean a big increase in traffic to the app servers.  You’ve hit the button before during code releases.  But always very late at night when traffic is light.  Early afternoon is about as high as traffic gets.  These systems are not the most stable in the world right now — you’re not sure if they’ll survive a cache clear in the middle of the afternoon.  Any web site will slow down with lots of traffic.  But too much traffic and these systems crash.  Break.  Stop working at all.  And often won’t get back up without a lot of help.  Sometimes such crashes will ripple back through dependent systems and it takes hours to figure out what’s happened.  Maybe even take the whole company off-line for a while, and that’s always fun to explain to the execs afterwards.

This is the risk of hitting the big button and clearing the caches.  Best case is the site runs slowly for a while as the caches repopulate.  Worst case, the whole system goes completely south while the analysts are checking it out.  Alternately you could just leave the site in its somewhat-broken but mostly working state for the analysts to look at.

So, what do you do?

A friend from college pointed out to me that engineers get paid for their judgment.  Doing rote calculations doesn’t demand a high salary.  Using your experience and opinion to weigh alternatives does.  Considering the relative merits of trade-offs, especially when the stakes are high — that’s where you really need somebody who is wise and experienced.

I have to digress for a moment to consider what’s really going on here when I say "the stakes are high."  In this industry, a big stupid mistake where you muck with live running machinery that you shouldn’t be means thousands of people don’t get their web page for a while.  Compare this to a friend who makes cheese for a living, and mucked around with live running machinery and got badly hurt.  A mistake on the production web servers potentially could have destroyed millions of dollars of abstract shareholder value.  But nobody was going to get their arm ripped off.  (Warning — these pictures are really gross.)  Anyway…

So what did I do when faced with this dilemma recently?  Me?  I went for it — I hit the button.  And everything was fine.  For a while the site was really slow while the caches refreshed.  Many CPUs were pegged from our app tier back through the databases that the whole company relies on.  But nothing broke.  And when pages finally loaded they looked good.  After about an hour, everything was back to normal.  Most everybody never noticed a thing. 

Just another exciting, adventurous, yet entirely unglamorous day in the life of a software engineer.

First, I’d like to welcome everybody landing here after searching for something on Google.  I hope you find what you’re looking for.  I know Google has been crawling my site nearly since its launch, and I’ve been passively wondering when would this site show up in Google’s search index.  It’s been in Yahoo and MSN for ages, and getting more and more links from high profile sites all the time.  Well I just got the answer: as soon as I gave Google money.  Surprised?  As a birthday present to myself, I bought a few adwords like:

I plop a few bucks into an adwords account and almost instantly the site shows up in the public search index, with a whopping pagerank of 2.  Not high enough that it was an obvious mistake to omit it from the index, but also not so low that it clearly doesn’t deserve to be in the index because there are other more important pages being indexed.  "Do no evil" they say.  This isn’t clearly a case of evil.  But it certainly seems like a bias.

I am both proud and awed by the productivity of the rhapsody.com development team.  Just two months after Rhapsody.com added playlists, a huge new feature has been added: a personal music library for bookmarking your favorite content.  Along with it is a fabulous new AJAX library manager which gives users quick visual access to a large collection of music in their web browser.

What makes this even more impressive is that one of those two intervening months included the end of year holidays.  When I’m doing long-term project scheduling, I generally write off 3 weeks out of December because of vacations and general lack of focus.  So they did all this in about 5 useful weeks.

I attribute this productivity to a team that has fully embraced agile development practices.  We use schedule-driven releases, which have a ton of advantages over feature-driven releases that I won’t detail right now.  (Avoiding feature-creep is huge.)  In 2006 we put out 10 releases with major new features, and almost no crunch time.  At this point the team has a solid understanding of several important things:

• Their feature velocity — How much work can they get done in a month?
• Staggering dependent work — How to break apart a problem into things that can get done early
• Keeping the pipeline full — This one’s my favorite, and requires explanation.  Read on…

I like to draw analogies between software development and a traditional manufacturing factory.  In a well organized team, the bottleneck is going to be the development team.  Every business function suffers from diseconomies of scale as more people are added because of communication overhead.  But the development function, actually writing the code, has this problem way worse than quality assurance, program management, visual design, user experience testing, or product management.  Writing code requires such intensely detailed knowledge that adding people efficiently requires massive amounts of information to be shared.  The bandwidth between human brains isn’t high enough to support this properly yet.  So, in a well proportioned team, the devs are the bottleneck.

As anybody who’s taken intro to operations management will tell you, the key to keeping a factory running at peak capacity is to keep the bottleneck as busy as possible.  That means accumulating a safety stock of work-in-progress inventory in front of the bottleneck.  In software engineering terms, that translates to having a stash of complete product plans, visual designs and functional specs ready for the development team to work on.  In other words, make sure the devs are never waiting  for anybody else to tell them what to build next.  This is an aspect of agile project management I don’t hear discussed much.  But my team has figured it out.  The overall result is a team that is always working hard, rarely stressed, and extremely productive at putting out products everybody is proud of.

Another great aspect of the team is that everybody feels ownership over the product.  Innovation comes from everywhere.  Try bookmarking something in your library.  You’ll need to sign up for a free trial account first, then hit one of the plus buttons next to some music and select "Add to Library."  Normally you might wonder where to go from here to work with your library.  But if you try it, I’m certain it will be obvious to you what to do next.  This simple, subtle, eye-candy user-education  feature didn’t come from product management or creative design.  It was one developer’s idea that the team ran with, and it’s one of my favorite features right now.  This isn’t an agile practice per se, but it sure makes a difference in the overall product quality.

I wish I could take credit for this accomplishment, but my input has been mostly just guidance.  Good job, team.  Keep it up!  (By the way, if you’re a rock-star java developer looking for a better-than-your-current job in Seattle or SF, drop me an e-mail.  We’re hiring.)

So I’m preparing my slides for my Ignite Seattle talk tomorrow night (tonight? Tuesday night) and I go over to my friends’ place to practice with them and I am reminded that the format is not 15 slides for 20 seconds each but rather 20 slides for 15 seconds each!  So now I’m trying to split each of my slides into four thirds and rejigger all the timings.  Fun!

I’d like to take a few moments out of my busy schedule to apologize in advance to anybody expecting a polished coherent lecture from me.  I decided to take an extra class at school this quarter, and midterms are on us in a big way.  That combined with a great wilderness first aid class that took up my entire weekend, I’ve been averaging less than 5 hours of sleep each night for the last week, and it’s starting to limit my critical thinking abilities.

This format is also completely absurd.  But I think that’s part of the idea.  My friend Barry is taking a rather sensible approach of repeating a few slides with minor variations — a sensible cop out if you ask me!

Still, I encourage everybody to come.  It’s starts at 6:30 PM at CHAC which is at 12th and Pinke.  It’s going to be a fun information-rich geeky thought-provoking chaotic time.  For my loopy part, I’m embracing the chaos.  Go team!