Great job everybody!  It’s awesome to see what you can do without leaving your browser.

Broadly, the concept of information democracy is that an increasingly large number of people are able to influence how information is aggregated.  Wikipedia is a clear and simple example of allowing anybody to contribute to what used to be authored by a select few — “The Encyclopedia.”  Google’s Pagerank algorithm democratized web search.  Today’s most successful software is democratizing the feature set by allowing users to vote on how they want to use it.  The general principal is that large numbers of individuals can together make better decisions than any small group.  Applying this principal to culture, we can predict that a cultural democracy will produce “better culture” than what was available before.

Information technology makes it cheaper and easier to both create and to distribute culture.  With the right software, any laptop today has all the power of a professional music or video studio.  Sure the quality won’t be as good without professional inputs (microphones, cameras, etc) but the cheap stuff is good enough for a lot of things.  Obviously the internet makes distribution of this content trivially easy, which is disrupting traditional media businesses.  Easy creation and distribution of cultural content is an important part of creating a cultural democracy, but it is not the critical enabling step.

The key to democratizing culture is in the editorial process.  If everybody is contributing cultural content that is easily distributed, but there’s still a small group deciding which pieces everybody watches, we’re still in a cultural dictatorship.  Enabling the mass public to “vote” on content is the democratizing step.  That enables the collective intelligence of all media consumers to help choose what should become part of mass culture.  So instead of some programming executive trying to guess what will be popular, the question almost becomes moot — whatever is popular becomes popular culture.  Actually making this work is not at all straightforward.  I’ll save a full description of the necessary ingredients for another post, but we can look at a couple examples.

Youtube does this quite well.  It blurs the line between sharing a video clip with your friends and publishing it as a piece of mass culture.  Any video that isn’t marked private is submitted into a kind of massive popularity contest.  Videos that get millions of views are undeniably bits of popular culture.  For music, last.fm does a good job of being inclusive, but hasn’t quite taken off.  When I started building social features into Rhapsody I hoped they could democratize the music editorial process but that hasn’t happened yet.  Like many things in social media there’s a chicken and egg problem with scale which Youtube has clearly gotten past, but music is still struggling with.

Cultural Democracy is “retro”?!

This post is inspired by a recent story by Heather Chaplin that NPR aired describing participatory culture in video games.  The surprising part of the story for me was the assertion that this trend is not modern but in fact “retro.”  The story points out that before analog broadcast media, most culture was participatory — singing, dancing, crafts, etc.  Analog technology created the possibility of cultural hegemonies, and digital technology is breaking them down. A fine point, implying that the 20th century will likely be unique as the only period in human history when popular culture was dictated by an elite group of editors.  Thanks for the interesting tidbit.

]]> http://www.embracingchaos.com/2009/12/participatory-culture-and-the-democratization-of-information.html/feed 1 http://www.embracingchaos.com/2009/01/rhapsody-profil.html http://www.embracingchaos.com/2009/01/rhapsody-profil.html#comments Sun, 04 Jan 2009 19:00:00 +0000 leodirac http://wp.embracingchaos.com/2009/01/rhapsody-profil.html Excuse my newbie exuberance, but OMG Rhapsody.com finally launched profile pages!!!  They’ve been up for a while now, which makes me think they’re for real this time.  A couple of you might remember that this feature was live for something like a week in early 2007.  But it was very slow and didn’t live long.  Sniff.

I worked hard to make this feature possible when I was working at Real.  The fact that I couldn’t get it re-launched was a big motivator for me to move on to greener pastures.  I saw making Rhapsody social as an important evolution of the music catalog’s organizational schema.  It’s also an attempt to bring the product into what Tim O’Reilly would call Web 2.0.  Tim’s canonical essay is long-winded, but I really liked how he summarized it in a recent interview on NPR — basically the product gets better as people use it.  The millions of people who use Rhapsody are an asset that has been almost completely unused, except to take their money.  I saw it as a way to take on one of the product’s biggest shortcomings.

Rhapsody has tons of music.  TONS.  Rhapsody almost certainly has something you want to listen to right now, regardless of who you are or what your current mood or situation is.  It’s a strong statement, but there really is that much music.  The problem is figuring out what you want to listen to.  Rhapsody has a great categorical index of music, so if you know you want to listen to D&B or Emo or Vocal Jazz, no problem.  Or if you know specifically the name of something you want to listen, just search for it.  Other than that, you can take the homepage recommendations, browse the catalog manually, or sift through Playlist Central, a dumping ground for unvetted playlists that is a case study in how not to use user-generated-content (UGC) on a website.

Picking good music is difficult.  This is what DJ’s get paid for.  I originally wanted this feature to be called "DJ Pages."  The idea was to give a voice to the small fraction of Rhapsody users who are fanatical about the product.  People who are serious music buffs love Rhapsody, and if given a voice would and still might add tremendous value to the music catalog.  Right now the editorial voice in Rhapsody is controlled by a politburo of paid editors.  They’re really good, but they’re just a handful of hands.  DJ Pages would democratize the music editorial process so anybody with an opinion can contribute.  The social graph becomes the voting process to select who’s worth paying attention to, just like with pagerank.  What Tim calls Web 2.0, I like to refer to the democratization of information.  Partly because it’s fun to call people Communists when they cling to control of information, but mostly because the analogy is apt and helpful.

The Rhapsody team has made an important step in this direction of openness.  I hope they keep running with it.  If you want to see what’s been playing on my Sonos at home, check out my profile page.  But most importantly, I’d like to express my CONGRATULATIONS to everybody who made this possible again and the first time!!!!11!!1

My girlfriend has a kindle that she very much enjoys.  One of the biggest benefits from it she gets is having a large amount of content in a very small device.  She is a scientist who is very much an information worker.  Having access to a great many research papers in searchable form is very useful for her.  (If only the PDF import worked on multi-column papers!)  She also tends to live out of a backpack, so being able to have several interesting things to read at any give time is very appealing.

So she’s often reading her kindle on the bus.  She’s noted one interesting difference between reading her Kindle and reading a regular book while on the bus.  When she’s reading a normal book, people will ask her what booj she’s reading or will look at the cover and just talk to her about the book itself.  With the kindle the question is always "how do you like the gizmo?"  Which gets old after a while.

Here’s a suggestion to Amazon on how to address this social problzem: offer full-color PDFs of the covers of books that you purchase for the Kindle, so people can print out their own covers.  These could slide into a convenient holder on the Kindle’s attractive leather case.  Long-term it’d be great to have a color e-paper cover for the book, but we’re not holding our breath for that one.

PubSub 101: Push vs Pull

PubSub is short for "publish subscribe" which is a common design pattern describing a way to distribute information to interested parties.  The publisher tells a server about new information, and the server fans the information out to everybody who has registered interest in that topic or channel.  Data consumers find out about the new information very quickly, with relatively little load on the whole system, since the pubsub mechanism provides a means to "push" data to them. 

By contrast, almost all of the web today follows uses a "pull model" where a data consumer only finds out about new information when it gets around to checking if there is something new.  This data distribution model is significantly simpler because the server only needs to keep track of the content, not who is interested in knowing about it.  Modern networks are optimized for this kind of query-based traffic where data consumers (clients, web browsers) initiate connections to servers, such that it’s often impossible for servers to initiate conncetions to clients because of firewalls or NAT.

The downside of the pull model is that the only way a data consumer can find out if thanything is new on the server is to "check back frequently" or "poll" the server for changes.  If you want to know within 15 minutes if anything new has been posted, you have to ask the server at least every 15 minutes "anything new?"  No.  "How about now?"  No.  "Got anything yet?"  No.  Mulitply this by potentially millions of interested data consumers and you end up spending a lot of network bandwidth and server resources doing very little.  Even worse, the problem scales horribly.  If clients want to know about changes within 5 minutes instead of 15, that puts 3 times the load on the server.  Want to know within a few seconds?  Forget it — the servers would crash.  There’s an intrinsic delay in distributing information in this model, and reducing that delay is very expensive.

XMPP as an alternative to polling

XMPP is the protocol used for Instant Messaging by Google Talk and Jabber and a large number of small servers.  In order to deliver instant messages, XMPP systems maintain persistent connections between all machines that allow packets of data to be pushed with very low latency — IM messages are typically delivered within a second of sending them.  So it’s natural to want to use this infrastructure to deliver other data more efficiently than through polling HTTP.

The XMPP PubSub spec known as XEP-0060 describes how to do exactly this at the protocol level.  But for a variety of reasons, this standard has not gained wide adoption.  IMHO the biggest reason is that there isn’t a very pressing need.  The current system is horribly inefficient, but it works.  Moreover, it puts the burden of inefficiency squarely in the hands of the information publishers.  Popular publishers are just expected to shell up for necessary hardware to meet the demands of their readers, and with advertising they can typically recoup the necessary investment.

Another way to state that is that pubsub hasn’t found its niche yet.  IMHO this is partly because the mechanism is so useful it can be applied to almost anything.  Not just breaking news, but everything from e-mail mailing lists to doorbell chimes get used as examples of how XMPP pubsub technology could be applied.  Not wanting to exclude any of these potentially interesting uses, the protocol remains very generic.

Micro-blogging, Atom and Yesterday’s Realization

One place where the current HTTP model breaks down is micro-blogging, which is the generic term for services like Twitter or Facebook’s udpates.  Here, the payload of actual content is very small, so the overhead of checking far outweighs the "useful data" which is delivered.  Also, because the information is social (i.e. "Heading to Broadway for a bite — wanna come?") consumers demand it be delivered quickly.  Nonetheless, current micro-blogging services still rely on polling clients, and their servers suffer as a result.

Yesterday, a group of us including Blaine Cook, Anders Conbere, Evan Prodromou, and XEP-0060 co-author Ralph Meijer were discussing how to apply XMPP PubSub to micro-blogging.  This was likely obvious to many there already, but during the discussion I had a realization.  We aren’t solving this problem from whole cloth.  RSS and Atom feeds already describe all the information we need.  We just need to find a way to substitute XMPP for the assumed transport HTTP.

So we discussed mechanisms for mapping an Atom URL to an XMPP PubSub Node.  (We pretty much ignored RSS because RSS isn’t as cool for reasons I really don’t understand.)  We talked about putting a link-rel tag in the feed to point to the XMPP PubSub node.  This would look something like 

   <link rel="alternate" type="xmpp/pubsub" href="xmpp:twitter.com?;node=users/leopd" />


Even better, the URL for these nodes should be guessable from the URL for the HTTP feed.   The above node would be the normal place to look for a the pubsub version of http://twitter.com/leopd.  Even though it’s not as generic and robust to have a standard mapping like this, I think it’s an important way to speed adoption of a new standard.  The code to do a bit of string manipulation is vastly easier than fetching the URL and looking for a link-rel tag.  And developers are intrinsically lazy (for good reasons!) so making things easier for them means they’ll succeed a lot more.

Ever pragmatic, Blaine pointed out that we should use HTTP for things it is good at, and not re-invent them in XMPP.  I wholeheartedly agree.  Re-transmission is a key example.  What happens if a client is offline when a new post happens, and so never hears about it?  Answer: The clients should fetch the historic archive of the feed over HTTP.  These feeds exist today — no need to improve on them.  If all the posts have sequence numbers on them, then it’s easy to figure out if you’ve missed one.  So all the posts from a user should have sequence numbers.  I don’t think this is standard in Atom feeds today.

The story unfolds…

There’s a lot more to be worked out and standardized here.  And clearly many more people need to voice their opinions before we can reach consensus.  Sadly I can’t be down in Portland today to continue the discussion, so this post will have to take my place as I return to my regular daily commitments.  If you’d like to stay tuned as the story unfolds, you’ll have to poll this site, as I can’t yet give you a PubSub node to subscribe to for updates.  If I could it would probably be something like xmpp:embracingchaos.com?;node=xmpp — try it.  By the time you read this, it might be working!

FeedHub is promising in that it is explicitly trying to solve this exact problem that I’ve identified.  Scoble refers to it as a Custom Techmeme.  But I’m fairly sure FeedHub won’t succeed as it currently exists. Why?  Because it doesn’t incorporate collective intelligence from the social network.  If they write it as a Facebook app, then they’d get here quickly.  But I just don’t believe that enough people will be willing to train this thing for its own sake.  Moreover, there’s no leverage — no network effect.  So even if millions of users have personally trained feedhub for themselves, that doesn’t make it any better for a new user who hasn’t started yet.  It’s just not Web 2.0 enough.

Overall I have to say "Good try."  One thing business school has taught me is that identifying a genuine and solvable need is about the most important thing to creating a business, and they’ve hit on that well.  But the execution seems to fall just a bit short of what’s needed these days.

• Build your app without social networking data
• Start from scratch with your own social network
• Integrate your app with Facebook

The third choice is so simple, it is the obvious best choice for most new information services.  As I see it, this is the fundamental power of the Facebook platform and why they’re going to go very, very far.

What I’ve been thinking about building in my copious spare time is a web-based feed-reader that assumes you over-subscribe to feeds.  That is, it expects you to "subscribe" to more feeds than you can fully consume.  These days many of the most popular feeds on the web meet this criterion even if that’s all you subscribe too.  I don’t have time to follow any one of TechCrunch, Scobleizer, Engadget, or even Radar in their entirety — I generally don’t even get to skim all their headlines.  But I know people in my social network do, and when they do it would be a small extra effort for them to help me identify the posts that are worth me reading.

This could be done by explicitly recommending articles to friends, or by tagging, or rating, or any of a number of well-understood-yet-often-poorly-implemented mechanisms.  Additionally, I could subscribe to a meta-feed coming out of a single-friend or a set of people in the social network graph that could expand several levels.  And of course there would be meta-feeds covering the aggregate opinions of all users.  The result would be that I could "express mild interest" in a feed by "subscribing to it" and the system would help me figure out which of the voluminous posts were actually worth reading.  Or if other users tagged posts, I could find good posts on a particular topic.  It would encompass a lot of the utility of digg, techmeme and link blogs all at once.  Another step in the process of democratizing information consumption.

I’ve been talking with friends about building this in the context of a facebook app for reasonably obvious reasons.  I’d call it "the outside world" as a reference to the fact that college kids are generally so isolated from external news, and this would be a social way for those few who do read the traditional-news to share good things with their friends.  Facebook’s restrictions on apps processing social networking metadata would make somegood features difficult, but the advantages in marketing and lower barrier to entry probably outweigh that.  Now my idea is out there for the world, so I’m not getting a jump on anybody.  If anybody wants to take this idea and run with it, drop me a line and I’d be happy to help advise.  I might just do it anyway because the Facebook market and the Google Reader market are both healthy and the basics just aren’t that hard.

But it sounds like you’ll have stiff competition.  Quoting from Blogoscoped’s analysis of the video:

Google’s recent big social effort is called Mocha-Mocha (or
Mocka-Mocka?), and will become the infrastructure for all social stuff
across all of their applications. As a part of this, a new
feature called Activity Streams will be introduced or at least
implemented in Reader this quarter. This will be comparable to
Facebook’s News Feed (Minifeed?) feature, and integrate Gmail’s
addressbook and contact list.

Also there will be some other Gmail and Orkut integration, but this might just mean there will be links to Reader.

Hearing that Brad Fitzpatrick has joined Google and because it’s the kind of thing I do, I’ve been putting some thought into how Google could reasonably add social networking features to their services.  I’ve been talking to folks about how Facebook is currently Google’s biggest strategic threat because they’ve done such a good job integrating the social network into new feature development, and in doing so have democratized new feature development in a way the world has never before seen.  This need struck me as a good way to start integrating social networking features into Google. 

Orkut is and only ever will be a toy IMHO.  Let the Brazilians keep playing with it and don’t push it on the rest of us.  Between contacts and knowledge about whom we chat and e-mail with, gmail has vastly more meaningful set of social networking data.  As we’ve learned watching LinkedIn and okcupid and other social networks thrive side by side, it makes sense to have different social networks for different purposes.  Orkut is a toy network and should not be the basis of anything more meaningful.  Sorry, Orkut.