$wpsc_version = 169; » XMPP PubSub: a great compliment to Atom/RSS - Embracing Chaos

By BoLOBOOLNE payday loans

XMPP PubSub: a great compliment to Atom/RSS

I spent the day yesterday at XMPP Summit #5 alongside OSCON in Portland.  It was a great chance to catch up with old friends and meet a few new ones.  But my favorite part was the break-out discussion of XMPP PubSub as it relates to micro-blogging.  We discussed what hopefully will emerge as a standard way to associate an existing Atom/RSS feed with an XMPP PubSub Node.  First some background on the relevant technologies.  Feel free to skip ahead if you understand this stuff.

PubSub 101: Push vs Pull

PubSub is short for "publish subscribe" which is a common design pattern describing a way to distribute information to interested parties.  The publisher tells a server about new information, and the server fans the information out to everybody who has registered interest in that topic or channel.  Data consumers find out about the new information very quickly, with relatively little load on the whole system, since the pubsub mechanism provides a means to "push" data to them. 

By contrast, almost all of the web today follows uses a "pull model" where a data consumer only finds out about new information when it gets around to checking if there is something new.  This data distribution model is significantly simpler because the server only needs to keep track of the content, not who is interested in knowing about it.  Modern networks are optimized for this kind of query-based traffic where data consumers (clients, web browsers) initiate connections to servers, such that it’s often impossible for servers to initiate conncetions to clients because of firewalls or NAT.

The downside of the pull model is that the only way a data consumer can find out if thanything is new on the server is to "check back frequently" or "poll" the server for changes.  If you want to know within 15 minutes if anything new has been posted, you have to ask the server at least every 15 minutes "anything new?"  No.  "How about now?"  No.  "Got anything yet?"  No.  Mulitply this by potentially millions of interested data consumers and you end up spending a lot of network bandwidth and server resources doing very little.  Even worse, the problem scales horribly.  If clients want to know about changes within 5 minutes instead of 15, that puts 3 times the load on the server.  Want to know within a few seconds?  Forget it — the servers would crash.  There’s an intrinsic delay in distributing information in this model, and reducing that delay is very expensive.

XMPP as an alternative to polling

XMPP is the protocol used for Instant Messaging by Google Talk and Jabber and a large number of small servers.  In order to deliver instant messages, XMPP systems maintain persistent connections between all machines that allow packets of data to be pushed with very low latency — IM messages are typically delivered within a second of sending them.  So it’s natural to want to use this infrastructure to deliver other data more efficiently than through polling HTTP.

The XMPP PubSub spec known as XEP-0060 describes how to do exactly this at the protocol level.  But for a variety of reasons, this standard has not gained wide adoption.  IMHO the biggest reason is that there isn’t a very pressing need.  The current system is horribly inefficient, but it works.  Moreover, it puts the burden of inefficiency squarely in the hands of the information publishers.  Popular publishers are just expected to shell up for necessary hardware to meet the demands of their readers, and with advertising they can typically recoup the necessary investment.

Another way to state that is that pubsub hasn’t found its niche yet.  IMHO this is partly because the mechanism is so useful it can be applied to almost anything.  Not just breaking news, but everything from e-mail mailing lists to doorbell chimes get used as examples of how XMPP pubsub technology could be applied.  Not wanting to exclude any of these potentially interesting uses, the protocol remains very generic.

Micro-blogging, Atom and Yesterday’s Realization

One place where the current HTTP model breaks down is micro-blogging, which is the generic term for services like Twitter or Facebook’s udpates.  Here, the payload of actual content is very small, so the overhead of checking far outweighs the "useful data" which is delivered.  Also, because the information is social (i.e. "Heading to Broadway for a bite — wanna come?") consumers demand it be delivered quickly.  Nonetheless, current micro-blogging services still rely on polling clients, and their servers suffer as a result.

Yesterday, a group of us including Blaine Cook, Anders Conbere, Evan Prodromou, and XEP-0060 co-author Ralph Meijer were discussing how to apply XMPP PubSub to micro-blogging.  This was likely obvious to many there already, but during the discussion I had a realization.  We aren’t solving this problem from whole cloth.  RSS and Atom feeds already describe all the information we need.  We just need to find a way to substitute XMPP for the assumed transport HTTP.

So we discussed mechanisms for mapping an Atom URL to an XMPP PubSub Node.  (We pretty much ignored RSS because RSS isn’t as cool for reasons I really don’t understand.)  We talked about putting a link-rel tag in the feed to point to the XMPP PubSub node.  This would look something like 

   <link rel="alternate" type="xmpp/pubsub" href="xmpp:twitter.com?;node=users/leopd" />

Even better, the URL for these nodes should be guessable from the URL for the HTTP feed.   The above node would be the normal place to look for a the pubsub version of http://twitter.com/leopd.  Even though it’s not as generic and robust to have a standard mapping like this, I think it’s an important way to speed adoption of a new standard.  The code to do a bit of string manipulation is vastly easier than fetching the URL and looking for a link-rel tag.  And developers are intrinsically lazy (for good reasons!) so making things easier for them means they’ll succeed a lot more.

Ever pragmatic, Blaine pointed out that we should use HTTP for things it is good at, and not re-invent them in XMPP.  I wholeheartedly agree.  Re-transmission is a key example.  What happens if a client is offline when a new post happens, and so never hears about it?  Answer: The clients should fetch the historic archive of the feed over HTTP.  These feeds exist today — no need to improve on them.  If all the posts have sequence numbers on them, then it’s easy to figure out if you’ve missed one.  So all the posts from a user should have sequence numbers.  I don’t think this is standard in Atom feeds today.

The story unfolds…

There’s a lot more to be worked out and standardized here.  And clearly many more people need to voice their opinions before we can reach consensus.  Sadly I can’t be down in Portland today to continue the discussion, so this post will have to take my place as I return to my regular daily commitments.  If you’d like to stay tuned as the story unfolds, you’ll have to poll this site, as I can’t yet give you a PubSub node to subscribe to for updates.  If I could it would probably be something like xmpp:embracingchaos.com?;node=xmpp — try it.  By the time you read this, it might be working!

  1. Ram M. says:

    XMPP/PubSub as a trigger and then the actual content download through http is what I have implemented for a our internal services. The preference would be to keep the messages through xmpp servers to be smaller size, then to send the large size messages.

  2. tuttle says:

    I really dislike this poling everywhere and I very much appreciate these efforts! But to make things easier for the beginning: why not just use XMPP/PubSub as a trigger for RSS/Atom feed updates? So all the content transport is as usual, but the client, if the feed supports it, doesn't poll anymore but waits for the "new feed entry" message on the XMPP connection. I think this is the most modular and most likely successful approach, since it is less effort to implement. In general, although capable of a lot more, XMPP/PubSub can have a success as a "trigger protocol".

  3. Jean-Claude Mason says:

    Great article! Just as a contribution, I found http://www.nardol.org/2009/3/18/pubsub-with-xmpp4r-simple that eases XMPP and PubSub.

  4. But yes, Fritzy, there are so many invalid RSS feeds out there that it is problematic to parse as XML.

  5. RSS may not be as well-crafted as ATOM but it is XML, or should be.
    And it may well be that HTML and RSS are more widely adopted because of their imperfections and not in spite of them.

  6. leodirac says:

    RSS doesn't need to be well-formed XML? Really?

    Woah. I *hate* RSS. Atom rules!

  7. Fritzy says:

    RSS is to Atom as HTML is to XHTML. In short, RSS, while based on XML, isn't XML. You can't reliably parse it as XML, and thus it's not as convenient of a payload in XMPP as Atom.

  8. Thanks for the helpful post. I think we can do the sequence numbers via SHIM headers (XEP-0131). And it would be good to standardize the link rels. Perhaps that belongs in XEP-0240? We also need to standardize the ad-hoc command (XEP-0050) for getting a node of yours to subscribe to another node for lightweight chaining or repeaters. Then I think we're most of the way there. :)

  1. There are no trackbacks for this post yet.