User:dolander on SWiK

Amazon EBS - Elastic Block Store has launched

Thu, 28 Aug 2008 18:07:54 -0700

Today marks the launch of Amazon EBS (Elastic Block Store), the long awaited persistent storage service for EC2. Details can be found on the EC2 detail page, the press release and Jeff Barr's posting over on the AWS evangelists blog. Also the folks at Rightscale have two detailed postings: why Amazon EBS matters and Amazon EBS explained.

With the launch of the Elastic Block Store we complete an important milestone in offering a complete suite of storage solutions as part of the Amazon Infrastructure Services. Back in the days when we made the architectural decision to virtualize the internal Amazon infrastructure one of the first steps we took was a deep analysis of the way that storage was used by the internal Amazon services. We had to make sure that the infrastructure storage solutions we were going to develop would be highly effective for developers by addressing the most common patterns first. That analysis led us to three top patterns:

Key-Value storage. The majority of the Amazon storage patterns were based on primary key access leading to single value or object. This pattern led to the development of Amazon S3.
Simple Structured Data storage. A second large category of storage patterns were satisfied by access to simple query interface into structured datasets. Fast indexing allows high-speed lookups over large dataset. This pattern led to the development of Amazon SimpleDB. A common pattern we see is that secondary keys to objects stored in Amazon S3 are stored in SimpleDB, where lookups result in sets of S3 (primary) keys.
Block storage. The remaining bucket holds a variety of storage patterns ranging special file systems such as ZFS to applications managing their own block storage (e.g. cache servers) to relational databases. This category is served by Amazon EBS which provides the fundamental building block for implementing a variety of storage patterns.

I have written before about the basic features of Amazon EBS:

Amazon EBS will be offered in the form of storage volumes which you can mount into your EC2 instance as a raw block storage device. It basically looks like an unformatted hard disk. Once you have the volume mounted for the first time you can format it with any file system you want or if you have advanced applications such as high-end database engines, you could use it directly.
Developers can create multiple volumes, in size ranging from 1 GB to 1TB. This volume will be created within a specified Availability Zone and will be accessible by your EC2 instances running in that Availability Zone. As to be expected with a volume abstraction only one instance can have the volume mounted at any given time. Volumes can migrate and be reattached to other instances if necessary for failure handling or application migration reasons.
The consistency of data written to this device is similar to that of other local and network-attached devices; it is under control of the developer when and how to force flush data to disk if you want to bypass the traditional lazy-writer functionality in the operating systems file-cache. Because of the session oriented model for access to the volume you do not need to worry about eventual consistency issues.

However Amazon EBS isn't just a massive volume storage array within an Availability Zone, it provides a unique feature that allows for the creation of novel storage management scenarios: the ability to create snapshots and store those snapshots into Amazon S3. These snapshots can then be used as the starting point for creating new volumes within any availability zone.

We see developers use this feature for long term backup purposes, for use in rollback strategies, for (world-wide) volume re-creation purposes. Snapshots also play an important role in building fault-tolerance scenarios when combined with managing applications using Elastic IP addresses and Availability Zones.

Congratulations to the EBS team for delivering a great service that will help a lot of EC2 customers managing their storage efficiently.

M&A - it ain't over till it's over

Thu, 28 Aug 2008 18:07:19 -0700

The economy is clearly slowing down and the IPO market is nonexistent. As I have always said, this is the time to hunker down and tweak your business to get your model right. If you are interested in exiting today, M&A continues to be the only viable path along that front. Having been through a number of acquisitions and potential acquisitions through the years, one point I must remind you of is that any deal isn't over until its over. On the surface, this seems so obvious. And yes, once a term sheet is signed and a price and general terms are agreed to, you are in great shape. But recently, through discussions with other VCs and entrepreneurs, I am hearing about more situations where strategic buyers may significantly change the deal terms after more serious due diligence or even potentially walk away from a deal. This can be especially painful if you have spent a number of months meeting with the strategic and going through due diligence in lieu of running your business. Trust me, this happened to one of my portfolio companies last year and reasons cited can include we had a change of strategic priorities and or look at the economy, there is no way we can value you like we did when we started the deal.

While I can offer you no protection from this happening to you, all I can say is to be prepared and skeptical, be willing to walk away, and make sure that you both do enough diligence and meet with the right decision makers before you sign any term sheet and embark on the extended process. Once the term sheet is signed, run like hell to get the deal closed because the longer a deal lingers the more opportunity there is for it not to happen. Keep the hammer down and always have next steps and a defined timetable. In addition, to the extent that the strategic acquirer has made other aquisitions in the past, I would try to leverage your personal network to reach out to some of the VCs or entrepreneurs involved to get a flavor for how the strategic will run their due diligence process and what doozies or surprises the strategic throw at you. Before you start spending your money from the acquisition, remember there is a lot that can change and that probably will change so keep that in the back of your mind as you go through the process.

Selling to large enterprises costs big dollars no matter how frictionless your sale is

Thu, 28 Aug 2008 18:07:19 -0700

I have written a number of times about frictionless sales and how on-demand companies have a huge opportunity to reduce their sales and marketing costs and subsequently scale their business more efficiently. Here is an excerpt from a prior post:

Frictionless sales means reducing the pain for customers to adopt and use a service/product and consequently reducing the cost of sales and marketing to get a customer and generate revenue. As I mention in an earlier post, "The less friction you have in your sales and delivery model, the easier it is to scale. The easier it is to scale the faster and more efficiently you can grow." The lowest friction sale can be a user clicking on a web page and the content owner getting paid for it. The highest friction sale is spending lots of money on marketing and trade shows and having a large, direct sales force of expensive reps pounding the pavement for months trying to close a large deal with an enterprise customer. Follow that with a 3 month implementation process to get the customer happy. There are various grades of friction between these two extreme points like open source business models, software as a service, and reseller/OEM-type models as other forms of packaging and delivering a product/service. And of course, each of these models requires a different methodology and way of marketing and selling to a customer. Ultimately what you want is sales leverage where every $1 you spend on sales and marketing equals multiples of that in terms of revenue.

The perception that it is much easier to scale definitely holds true if you are selling to consumers, small businesses, and workgroups within large organizations. However, it seems that many public on-demand vendors are feeling the pressure to deliver growth and ultimately need to feed the revenue machine by going after larger customers. And what many companies are learning is that no matter how on-demand your software is, if you are selling to huge enterprises you are going to have to spend huge dollars in sales and marketing. Sales cycles are long no matter how you slice it and even if there is no massive hardware and software installation, many large companies want to have their service customized and integrated, even lightly, with other systems. in other words, many of these high flying on-demand vendors are starting to look more like the old software companies they are trying to replace. As per a Wall Street Journal article today, it seems that many of these public on-demand companies are finding out the hard way that no matter how frictionless your sales process is, the bigger the company you sell to, the more it is going to cost you.

There is nothing to install, so workers can start using online software without the aid of the tech department. That makes it easier for companies that sell online software to get into a business than their on-premises competitors.

Seizing on this, investors bought into online-software companies in a big way. During the first 10 months of 2007, shares of 15 online-software companies tracked by Thomas Weisel Partners increased in value 61%. Since then, however, these companies have lost about a third of their value.

Wall Street has realized that it isn’t enough to simply offer online software—you have to have a sales strategy that can make your offering a corporate standard. It is possible to get individuals, project teams or small businesses to buy online software through word-of-mouth marketing, but it is hard to make money from these groups—at least the kind of money necessary to become a billion-dollar company.

In order to get there, they can’t operate like an Internet start-up, letting their technology spread virally as end users hear about it. They need to sell to the same executives and information-technology professionals who made purchasing decisions before online software was an option. Businesses have a lot riding on the decision to use one product or another. And while having pockets of workers advocate for a particular piece of software is a plus, the execs who sign the big checks still want to see demos, vet the seller and do all the things they have always done when they buy software.

So if you are an on-demand vendor, either stick to your focus of scaling with SMBs and consumers which requires a completely different sales and marketing approach more rooted in traditional online budgets and telesales or be prepared to spend some real dollars if you truly want to go after the big guys.

How Facebook Keeps Memcached Consistent Across Geo-Distributed Data Centers

Thu, 28 Aug 2008 18:07:08 -0700

Last year I wrote a blog post entitled When Databases Lie: Consistency vs. Availability in Distributed Systems where I talked about the kinds of problems Web applications face when trying to keep data consistent across multiple databases spread out across the world.

Jason Sobel, a developer at Facebook has some details on how they've customized MySQL to solve a variation of the problem I posed in a blog post entitled Scaling Out where he writes

A bit of background on our caching model: when a user modifies a data object our infrastructure will write the new value in to a database and delete the old value from memcache (if it was present). The next time a user requests that data object we pull the result from the database and write it to memcache. Subsequent requests will pull the data from memcache until it expires out of the cache or is deleted by another update.

...

Consider the following example:

I update my first name from "Jason" to "Monkey"

We write "Monkey" in to the master database in California and delete my first name from memcache in California and Virginia

Someone goes to my profile in Virginia

We don't find my first name in memcache so we read from the Virginia slave database and get "Jason" because of replication lag

We update Virginia memcache with my first name as "Jason"

Replication catches up and we update the slave database with my first name as "Monkey"

Someone else goes to my profile in Virginia

We find my first name in memcache and return "Jason"

Until I update my first name again or it falls out of cache and we go back to the database, we will show my first name as "Jason" in Virginia and "Monkey" in California. Confusing? You bet. Welcome to the world of distributed systems, where consistency is a really hard problem.
Fortunately, the solution is a lot easier to explain than the problem. We made a small change to MySQL that allows us to tack on extra information in the replication stream that is updating the slave database. We used this feature to append all the data objects that are changing for a given query and then the slave database "sees" these objects and is responsible for deleting the value from cache after it performs the update to the database.

...

The new workflow becomes (changed items in bold):

I update my first name from "Jason" to "Monkey"

We write "Monkey" in to the master database in California and delete my first name from memcache in California but not Virginia

Someone goes to my profile in Virginia

We find my first name in memcache and return "Jason"

Replication catches up and we update the slave database with my first name as "Monkey." We also delete my first name from Virginia memcache because that cache object showed up in the replication stream

Someone else goes to my profile in Virginia

We don't find my first name in memcache so we read from the slave and get "Monkey"

Facebook's solution is clever and at first I couldn't shake the feeling that it is an example of extremely tight coupling for database replication to also be responsible for evicting expired items from your in-memory cache. After some thought, I realized that this is no different from the SqlCacheDependency class in ASP.NET which allows you to create a dependency between objects in your ASP.NET cache and those in your SQL database. When the underlying tables change, the Cache is updated to reflect this change in database state.

In fact, the combination of replication and the SqlCacheDependency class should mean that you get this sort of behavior for free if you are using ASP.NET caching and SQL Server. Unfortunately, it looks like Microsoft's upcoming in-memory distributed object caching product, Velocity, won't support SqlCacheDependency in its initial release according to a comment by one its developers.

Of course, there is a significant performance difference between actively monitoring the database for changes like SqlCacheDependency does and updating the cache when updates made to the database are received as part of the replication stream. I wonder if this pattern will turn out to be generally useful to Web developers (at least those of us who work on geo-distributed services) or whether this will just go down as a clever hack from those kids at Facebook that was cool to share.

Now Playing: Rihanna - Disturbia

Some Thoughts on Amazon's Elastic Block Store

Thu, 28 Aug 2008 18:07:08 -0700

According to Werner Vogels's blog post entitled Amazon EBS - Elastic Block Store has launched, it seems that my friends at Amazon have plugged a gaping hole in their cloud computing platform story. Werner writes

Back in the days when we made the architectural decision to virtualize the internal Amazon infrastructure one of the first steps we took was a deep analysis of the way that storage was used by the internal Amazon services. We had to make sure that the infrastructure storage solutions we were going to develop would be highly effective for developers by addressing the most common patterns first. That analysis led us to three top patterns:

Key-Value storage. The majority of the Amazon storage patterns were based on primary key access leading to single value or object. This pattern led to the development of Amazon S3.
Simple Structured Data storage. A second large category of storage patterns were satisfied by access to simple query interface into structured datasets. Fast indexing allows high-speed lookups over large dataset. This pattern led to the development of Amazon SimpleDB. A common pattern we see is that secondary keys to objects stored in Amazon S3 are stored in SimpleDB, where lookups result in sets of S3 (primary) keys.
Block storage. The remaining bucket holds a variety of storage patterns ranging special file systems such as ZFS to applications managing their own block storage (e.g. cache servers) to relational databases. This category is served by Amazon EBS which provides the fundamental building block for implementing a variety of storage patterns.

What I like about Werner's post is that it shows that Amazon had a clear vision and strategy around providing hosted cloud services and has been steadily executing on it.

S3 handled what I've typically heard described as "blob storage". A typical Web application typically has media files and other resources (images, CSS stylesheets, scripts, video files, etc) that is simply accessed by name/path. However a lot of these resources also have metadata (e.g. a video file on YouTube has metadata about it's rating, who uploaded it, number of views, etc) which need to be stored as well. This need for queryable, schematized storage is where SimpleDB comes in. EC2 provides a virtual server that can be used for computation complete with a local file system instance which isn't persistent if the virtual server goes down for any reason. With SimpleDB and S3 you have the building blocks to build a large class of "Web 2.0" style applications when you throw in the computational capabilities provided by EC2.

However neither S3 nor SimpleDB provides a solution for a developer who simply wants the typical LAMP or WISC developer experience of building a database driven Web application or for applications that may have custom storage needs that don't fit neatly into the buckets of blob storage or schematized storage. Without access to a persistent filesystem, developers on Amazon's cloud computing platform have had to come up with sophisticated solutions involving backing data up manually from EC2 to S3 to get the desired experience.

EBS is the final piece in the puzzle that had prevented Amazon's cloud computing platform from being comparable to traditional hosting solutions. With EBS Amazon is now superior to most traditional hosting solutions from a developer usability perspective as well as cost. Google App Engine now looks like a plaything in comparison. In fact, you could build GAE on top of Amazon's cloud computing platform now that the EBS has solved persistent custom storage problem. It will be interesting to see if higher level cloud computing platforms such as App Engine start getting built on top of Amazon's cloud computing platform. Simply porting GAE wholesale would be an interesting academic exercise and a fun hacking project.

Now Playing: T.I. - Whatever You Like

I Want a Windows App Store

Thu, 28 Aug 2008 18:07:08 -0700

Last week my blog was offline for a day or so because I was the victim of a flood of SQL injection attacks that are still hitting my Web site at the rate of multiple requests a second. I eventually managed to counter the attacks by installing URLScan 3.0 and configuring it to reject HTTP requests that resemble SQL injection attacks. I found out about URLScan in two ways; from a blog post Phil Haack wrote about Dealing with Denial of Service Attacks where it seems he's been caught up in the same wave of attacks that brought down my blog and via an IM from Scott Hanselman who saw my tweet on Twitter about being hacked and pointed me to his blog post on the topic entitled Hacked! And I didn't like it - URLScan is Step Zero.

This reminded me that I similarly found another useful utility, WinDirStat, via a blog post as well. In fact when i think about it, a lot of the software I end up trying out is found via direct or indirect recommendations from people I know. Typically through blog posts, tweets or some other communication via a social networking or social media service. This phenomenon can be clearly observed in closed application ecosystems like the Facebook platform, where statistics have shown that the majority of users install new applications after viewing them on the profiles of their friends.

One of the things I find most interesting about the Facebook platform and now the Apple App Store is that they are revolutionizing how we think about software distribution. Today, finding interesting new desktop/server/Web apps either happens serendipitously via word of mouth or [rarely] is the result of advertising or PR. However finding interesting new applications if you are a user of Facebook or the Apple iPhone isn't a matter of serendipity. There are well understood ways of finding interesting applications that harnesses social and network effects from user ratings to simply finding out what applications your friends are using.

As a user, I sometimes wish I had an equivalent experience as a user of desktop applications and their extensions. I've often thought it would be cool to be able to browse the software likes and dislikes of people such as Omar Shahine, Scott Hanselman and Mike Torres to see what their favorite Windows utilities and mobile applications were. As a developer of a feed reader, although it is plain to see that Windows has a lot of reach since practically everyone runs it sometimes I'm envious of the built in viral distribution features that come with the Facebook platform or the unified software distribution experience that is the iPhone App Store. Sure beats hosting your app on SourceForge and hoping that your users are blogging about the app to spread it via word of mouth or paying for prominence on sites like Download.com.

A lot of the pieces are already there. Microsoft has a Windows Marketplace but for the life of me I'd have never found out about it if I didn't work at Microsoft and someone I know switched teams to start working there. There are also services provided by 3rd parties like Download.com, the Firefox Add-Ons page and Tucows. It would be interesting to see what could be stitched together if you throw in a social graph via something like Facebook Connect, an always-on well integrated desktop experience similar to the Apple App Store and one of the aforementioned sites. I suspect the results would be quite beneficial to app developers and users of Windows applications.

What do you think?

Now Playing: Metallica - The Day That Never Comes

RESTful JSON: Bringing REST and RPC Closer Together

Thu, 28 Aug 2008 18:07:08 -0700

My recent post, Explaining REST To Damien Katz, led to some insightful comments from Dave Winer and Tim Bray about what proponents of building RESTful Web services can learn from remote procedure call (RPC) technologies like SOAP and XML-RPC.

In his post, Dare left something out (and it's important) Dave Winer wrote

Really ought to include it in your thinking, Dare and everyone else. You're missing out on something that works really well. You should at least learn the lessons and add to REST what it needs to catch up with XML-RPC. Seriously.

What's missing in REST, btw, is a standard method of serializing structs, lists and scalar types. The languages we use have a lot more in common than you might think. We're all writing code, again and again, every time we support a new interface that could be written once and then baked into the kernels of our languages, and then our operating systems. Apple actually did this with Mac OS, XML-RPC support is baked in. So did Python. So if you think it's just me saying this, you should take another look.

Dave has a valid point, a lot of the time communication between distributed systems is simply passing around serialized objects and/or collections of objects. In such cases, having a lightweight standard representation for serialized objects which is supported on multiple platforms would be beneficial. Where Dave goes astray is in his assertion that no such technology exists for developers building RESTful Web services. Actually one does, and it is widely deployed on the Web today. JavaScript Object Notation (JSON) which is described in RFC 4627 is a widely deployed and well-defined media type (application/json) for representing serialized structs, lists and scalar values on the Web.

There are libraries for processing JSON on practically every popular platform from "corporate" platforms like Java and the .NET Framework to Web scripting languages like Python and Ruby. In addition, JSON is attractive because it is natively available in modern Web browsers that support JavaScript which means you can use it to build services that can be consumed by Web browsers, other Web services or desktop applications with a single end point.

Tim Bray cautioned REST proponents against blindly rejecting the features typically associated with RPC systems and SOAP/WS-* in his post REST Questions where he wrote

Has REST Been Fortunate in its Enemies? · I have been among the most vocal of those sneering at WS-*, and I’m comfortable with what I’ve said. But that doesn’t prove that the core WS-* ideas are wrong. Here are some of the handicaps WS-* struggled under:

Lousy foundational technologies (XSD and WSDL).

A Microsoft/IBM-driven process that was cripplingly product-linked and political.

Designers undereducated in the realities of the Web as it is.

Unnecessary detours into Architecture Astronautics.

As a result, we should be really careful about drawing lessons from the failure of WS-*. Specifically:

Just because the XSD type system is malformed, you can’t conclude that the notion of schema-driven mapping between program data types and representation payloads is harmful.

Just because WSDL is a crock, you can’t conclude that exposing a machine-readable contract for a service is a necessarily bad idea.

Just because UDDI never took off, you can’t conclude that service registries are dumb.

Just because SOAP has a damaging MustUnderstand facility and grew a lot of ancillary specification hair, you can’t conclude that some sort of re-usable payload wrapper is necessarily a dead-end path.

Just because the WS-* security specifications are overengineered and based on a shaky canonicalization spec, you can’t conclude that message-level security and signing aren’t sometimes real important.

And so on. I personally tend to think that schema-driven mapping is hopeless, contracts are interesting, registries are a fantasy, and payload wrappers are very promising. But I don’t think that the history of WS-* is a very good argument for any of those positions.

In a lot of situations where applications consume XML, the first thing the application does is convert the XML into an object graph representative of the business logic of the application. The SOAP/WS-* way of doing this was to define an XSD schema for the XML payload and then use some object<->XML mapping layer to convert the XML to objects. The problem with this approach was that there is a fundamental impedance mismatch between XSD types and OO types which led to horrible interoperability problems since no two platforms could agree on how to map the various esoteric type system features of XSD into the structs, lists and scalar types that are the building block of all OO type systems. However these problems go away if you use a data format that was explicitly designed for describing serialized data objects like JSON.

Providing a machine readable description of a service's end points is useful, especially on the Web where multiple services may expose the same interface. For example, when you visit my weblog at http://www.25hoursaday.com/weblog/ using Firefox 2 or higher and Internet Explorer 7 or higher the browser immediately lights up with a feed icon which allows you to subscribe to my Atom feed from your Web browser. This happens because I've provided a machine readable description of my feed end points on my blog. The Atom Publishing Protocol, which is one of the most well-designed RESTful protocols out there, has a notion of service documents which enable client applications to discover the capabilities and locations of the various end points of the service.

If you put together the notion of service documents with using JSON as the payload format for a service endpoint, you're close to getting the touted programmer friendliness of RPC technologies like XML-RPC & SOAP/WSDL while still building a RESTful service which works with the Web instead of against it.

The only problem is how to deal with statically typed languages like C# and Java. These languages need the types of the objects that application will consume from a Web service defined up front. So if we could just figure out how to come up with service documents for JSON services that included the notion of a class definition, we could pretty much have our cake and eat it to with regards to getting the ease of use of an RPC system while building a RESTful service.

If this sounds interesting to you, then you should go over and read Joe Gregorio's original post on RESTful JSON and then join the restful-json mailing list. Joe's proposal is on the right path although I think he is letting his background as an editor of the Atom Publishing Protocol specification skew his thinking with regards to what developers would find most useful from a Json Publishing Protocol (JsonPub).

Now Playing: G-Unit - Beg For Mercy

Best Practices for Web Sites Seeking to Prevent Service Degradation due to 3rd Party Widgets

Thu, 28 Aug 2008 18:07:08 -0700

I've been reading about the Ning vs. WidgetLaboratory drama on TechCrunch. The meat of the conflict seems to be that widgets from WidgetLaboratory were so degrading the user experience of Ning that they had to be cut off. The relevant excerpts from the most recent TechCrunch story on the war of words are below

For those of you not closely following the drama between social network platform Ning and a popular widget provider called WidgetLaboratory, you can read the background here . On Friday Ning unceremoniously shut down their access to Ning, making all those widgets vanish.
...
In an email to WL on August 2 (more than three weeks ago), CEO Gina Bianchini wrote “Our only goal is to have you build your products in such a way that doesn’t slow down the networks running your products or takedown the Ning Platform with what you’re doing. Both of those would result in us needing to shutdown WidgetLaboratory products and that’s has never been our first choice of options. Hopefully, you know this after 8 months of working with us.”

Ignoring the he said, she said nature of the communication between both companies, there is a legitimate concern that 3rd party widgets included on the pages of a Web site can degrade the performance to the extent that the site becomes unusably slow. In fact, TechCrunch has had similar problems with 3rd party widgets as Mike Arrington has mentioned on his personal blog which led to him excluding the widgets from his site.

Typically, widgets are embedded in a site by including references to Javascript hosted on a 3rd party site in the page's HTML. This means rendering the page is dependent on how quickly the script files can be downloaded from the 3rd party site AND how long it takes for the script to execute especially since it may also fetch data from one or more servers as well. Thus a slow server or a badly written script can make every page that embeds the widget unbearably slow to render. Given that the ability to embed widgets is a key feature of social networking sites, it is important for such sites to figure out how to isolate their user experience from badly written widgets or widgets hosted on slow Web servers.

Below are some best practices that have emerged on how social networking sites can immunize themselves from the kinds of problems Ning has had with WidgetLaboratory

Host the Scripts Yourself: If you have a popular site, it is quite likely that you have more resources to handle lots of page views than the typical widget developer. Thus it makes sense to take away the dependency on externally hosted scripts by hosting the widgets yourself. Microsoft encourages developers to submit their gadgets to Windows Live Gallery if they want to build gadgets for my.live.com or Windows Live Spaces. For it's AJAX homepage service, Google does not require developers to submit gadgets to them for hosting but instead caches gadget data for hours at a time which means they are effectively hosting the gadgets themselves for the majority of the accesses by their users.
Keep External Dependencies off of Pages that Need to Render Quickly: In many cases, it isn't feasible to host all of the data and content related to widgets that are being shown on your site. In that case, you should ensure that the key scenarios on your Web site are insulated from the problems caused by slow or broken 3rd party widgets. For example, on Facebook viewing someone's profile is a key part of the user experience that is important to make sure happens as quickly and as smoothly as possible. For this reason, Facebook caches all 3rd party content that shows up on a user's profile and requires applications to call Profile.SetFBML to add content to the profile instead of providing a way to directly embed widgets on a user's profile.
Make It Clear Who Is to Blame if Things go Awry: One of the issues raised by Ning in their conflict with WidgetLaboratory is that user pages wouldn't render correctly or would show degraded performance due to WidgetLaboratory's widgets but Ning would get the support calls. This kind of user confusion is avoided if the user experience makes it clear when the failure of a page to render correctly is the fault of the external widget and when it is part of the hosting site. For example, Facebook Canvas Pages for applications make it clear that the user is using a 3rd party application and not part of the core Facebook experience. I've seen lots of user complain about the slowness of Scrabulous and Scrabble but never seen anyone who thought that Facebook was to blame and not the application developers.

Following some of these practices would have saved Ning and its users some of their current grief.

Now Playing: Ice Cube - Get Money, Spend Money, No Money

Fixing the HTTP Polling Problem: Some Thoughts on FriendFeed's Simple Update Protocol (SUP)

Thu, 28 Aug 2008 18:07:07 -0700

Paul Buchheit of FriendFeed has written up a proposal for a new protocol that Web sites can implement to reduce the load on their services from social network aggregators like FriendFeed and SocialThing. He unveils his proposal in his post Simple Update Protocol: Fetch updates from feeds faster which is excerpted below

When you add a web site like Flickr or Google Reader to FriendFeed, FriendFeed's servers constantly download your feed from the service to get your updates as quickly as possible. FriendFeed's user base has grown quite a bit since launch, and our servers now download millions of feeds from over 43 services every hour.

One of the limitations of this approach is that it is difficult to get updates from services quickly without FriendFeed's crawler overloading other sites' servers with update checks. Gary Burd and I have thought quite a bit about ways we could augment existing feed formats like Atom and RSS to make fetching updates faster and more efficient. Our proposal, which we have named Simple Update Protocol , or SUP, is below.
...
Sites wishing to produce a SUP feed must do two things:

Add a special <link> tag to their SUP enabled Atom or RSS feeds. This <link> tag includes the feed's SUP-ID and the URL of the appropriate SUP feed.

Generate a SUP feed which lists the SUP-IDs of all recently updated feeds.

Feed consumers can add SUP support by:

Storing the SUP-IDs of the Atom/RSS feeds they consume.

Watching for those SUP-IDs in their associated SUP feeds.

By using SUP-IDs instead of feed urls, we avoid having to expose the feed url, avoid URL canonicalization issues, and produce a more compact update feed (because SUP-IDs can be a database id or some other short token assigned by the service). Because it is still possible to miss updates due to server errors or other malfunctions, SUP does not completely eliminate the need for polling.

Although there's a healthy conversation about SUP going on in FriendFeed in response to one of my tweets, I thought it would be worth sharing some thoughts on this with a broader audience.

The problem statement that FriendFeed's SUP addresses is the following issue raised in my previous post When REST Doesn't Scale, XMPP to the Rescue?

On July 21st, FriendFeed had 45,000 users who had associated their Flickr profiles with their FriendFeed account. FriendFeed polls Flickr about once every 20 – 30 minutes to see if the user has uploaded new pictures. However only about 6,000 of those users logged into Flickr that day, let alone uploaded pictures. Thus there were literally millions of HTTP requests made by FriendFeed that were totally unnecessary.

FriendFeed's proposal is similar to the Six Apart Update Stream and the Twitter XMPP Firehose in that it is a data stream containing information about all of the updates users are making on a particular service. It differs in a key way in that it doesn't actually contain the data from the user updates but instead identifiers which can be used to determine the users that changed so their feeds can be polled.

This approach aims at protecting feeds that use security through obscurity such as the Google Reader's Shared Items feed and Netflix's Personalized Feeds. The user shares their "secret" feed URL with FriendFeed who then obtains the SUP ID of the user's feed when the feed is first polled. Then whenever that SUP ID is seen in the SUP feed by FriendFeed, they know to go re-poll the user's "secret" feed URL.

For services that are getting a ton of traffic from social network aggregators or Web-based feed readers it does make sense to provide some sort of update stream or fire hose to reduce the amount of polling that gets done. In addition, it also makes sense that if more and more services are going to provide such update streams then it should be standardized so that social network aggregators and similar services do not end up having to target multiple update protocols.

I believe that at the end we will see a continuum of options in this space. The vast majority of services will be OK with the load generated by social networking aggregators and Web-based feed readers when polling their feeds. These services won't see the point of building additional features to handle this load. Some services will run the numbers like Twitter & Six Apart have done and will provide update streams in an attempt to reduce the impact of polling. For these services, SUP seems like a somewhat elegant solution and it would be good to standardize on something, anything at all is better than each site having its own custom solution. For a smaller set of services, this still won't be enough since they don't provide feeds (e.g. Blockbuster's use of Facebook Beacon) and you'll need an explicit server to server publish-subscribe mechanism. XMPP or perhaps something an HTTP based publish-subscribe mechanism like what Joshua Schachter proposed a few weeks ago will be the best fit for those scenarios.

Now Playing: Jodeci - I'm Still Waiting

Raising Venture Capital: How Much Money Matters

Thu, 28 Aug 2008 18:07:02 -0700

After watching a bazillion venture pitches, I've come to the conclusion that every VC Pitch should end the same way -- with the ask. If you want to crescendo into it, feel free to summarize why it is your technology is life changing, but finish with the ask -- "we are looking to raise six million dollars." Don't beat around the bush. Come right out and ask for the money. After all, that's what you're there for.

There are a number of reasons VCs want to hear what you're raising. And it isn't just the obvious one. Yes, it is helpful to know how much money a company is hoping you will invest. But there are other more valuable pieces of information that come out of the ask.

First of all, the amount of money you are raising is a good general indicator of how much you think the company is worth. I was in a pitch once learning about pretty interesting but pretty early stage technology. From where I sat, it seemed to me that the company could use single digit millions to take the technology to the next step. Yet, when we got to the slide that stated how much the company was raising, I learned that they were hoping to raise more than $50M. By my assessment, $50M would buy the vast majority of the company. Clearly the company felt differently -- they were hoping to sell closer to 20% of the company. It certainly refocused the conversation on what the company felt was the justification for such a high valuation and led to a very interesting discussion of the underlying economics of the company's business.

The thing I find most interesting about how much money a company is raising is not the actual number itself, but rather the conversation about how the company arrived at that number. What is interesting to me is what the company plans on doing with that money? What are the milestones the company can reach with that much money? Could they do it for less? What would they do if they had more money?

For me, the right question isn't "how much money do you want to raise?" The right question is "how much money should you raise?" Ask some entrepreneurs and they will tell you, the right amount of money to raise is as much as they possibly can (some recent monster financings suggest that strategy). That makes no sense to me. The right amount of money to bring into the company is enough to reach sufficient milestones to raise more money at a higher price at a future date (or, in some rare cases, enough to get to cash flow positive). If all goes well, the money I invest will be used to drive all sorts of risk out of the business, enabling the Company to raise the next round at a much higher valuation.

Figuring out the right amount to raise is more art than science but can have a big impact on the Company. If you raise too little money, you may run out before you have proven the business sufficiently to raise additional capital. In other words, raising too little money can be fatal. On the other hand, if you raise too much money early on, you could well be selling off too much of the company for too little capital. Companies should leverage early stage venture money to drive up the value of the company (by proving out as much of the business as quickly as possible), so that the next time the company fundraises, they will be able to bring in larger amounts of money while suffering smaller amounts of dilution.

Unfortunately, the perfect amount of money to raise is not always obvious. So the question isn't whether a company is raising the "right" amount of money. The question is, "why is the company raising the amount of money it is raising?" A great deal can be learned about a company from their answer to that question. So when you go out to raise money, be prepared to not only answer how much you are hoping to raise, but also why?

Building Modern Web Apps? Better Have A Deep Competency in Web 2.0, Open APIs, Widgets, Social Apps, and Much More

Thu, 28 Aug 2008 18:06:52 -0700

The Web has an interesting property that those building Web applications and online businesses usually encounter soon after they first launch: It has its own unique and unforgiving rules for success and failure. Appreciating them requires a certain level of understanding of the intrinsic nature of the Web and how it works. Actually leveraging those rules requires an even deeper and more profound understanding of the Web. The challenge these days? The Web competency bar is climbing fast.

To drive the right decisions in what they do product designers, marketing teams, software architects, developers, strategy officers, and other key roles in today's generation of online businesses need to have a solid handle on an extensive array of Web topics. This ranges from appreciating why plain old HTTP is so good at underpinning the Web to more sophisticated topics like modern application architecture, the latest in online user experiences, next generation computing models (grid/cloud/utility/SaaS/PaaS), cost-effective scalability, user identity, network effects, Jakob's Law, analytics, operations, user community, as well as the many compelling new distribution models that are nearly mandatory in the first release of most products.

This extensive set of competencies is what's required nowadays to deliver a credible online product to a receptive user base and it has dramatic implications for both uptake and overall cost/time-to-market. Worse, this body of knowledge has become extensive enough that many Web startups frequently fall far short of what they need to know in order to be successful with these far flung practice areas.

Does this complex body of knowledge mean the era of the two-to-five person Web startup is coming to a close? Not at all, at least not yet. The productivity level of the latest tools and techniques remains almost astonishing though the level of knowledge required of these teams is creeping up and up. And as we'll see, new models for product distribution are pushing the capability envelope of the typical Internet startup team to the point we may very well see the day soon that they won't have all the skills necessary to deliver a fully-scoped modern Web application. It is also one reason why fewer and fewer Web startups have the goods to be all around hits out of the gate.

Certainly, varying depths in subject matter are required depending one's exact role in a Web business, but Web-oriented products are fundamentally shaped the vagaries of the network itself. Tim O'Reilly himself still has the best quote on the subject: "Winners and losers will be designated by who figures out how to use the network." And as we'll see, the Web is driving the evolution of a major new generation of online distribution models.

Why Adopting New Distribution Models Is Crucial

As an example of this, I've been tracking some of the latest discussions around the hot topic du jour in the Web world: Social networking applications. Specifically, it's been interesting to watch the surprisingly low level of industry attention around the titanic competition brewing between social networking application formats from Web giants Facebook and Google. Why is this? Some might say it's because these applications still have largely unproven business models. Others, like Nick O'Neill at the Social Times recently observed (rightly in my opinion) that the struggle may have to do with a deficit in understanding why these new types of Web applications are so important. Nick notes that these widget and social networking style models for packaging and distributing Web apps often "have more eyeballs looking at their products than television channels have" and the challenge is that too many people just "don’t know what any of this means", despite the major players divvying up the online pie for themselves. With the size of these next generation distribution audiences, ignorance has an extremely painful price: failure to produce results and growth, poor engagement with the marketplace, and loss of market share.

An excellent summary of the truly massive, but largely underappreciated scale of these new Web application models was last week's TechCrunch piece on the progress of Google's OpenSocial, an increasingly successful model for creating portable social networking applications that will run on any OpenSocial-compliant site. Erick Schonfeld reported that OpenSocial now has a total reach of an astonishing 350 million users and it will soon be 500 million. There are over 4,500 OpenSocial apps today, a healthy number for the application format but a small drop in the bucket compared to the number of Web sites in the world. But the key is that these applications are integrated much deeper into the social fabric of an engaged audience, interjecting themselves into the daily personal and work habits of the "captive" users of social sites and even have access to the personal habits and data of users of these sites. Facebook's story is impressive as well with over 37,000 applications that have been installed over 700 million times.

And social networking applications are just one of many news ways that applications have to be packaged and distributed, yet far too many organizations persist in a very 1990s view of Web experiences, namely that Web sites themselves are the center of online product design. Many even think that some of these other new distribution models are interesting but not part of their core online product. Unfortunately, that's very much a parochial view in the present era. Federated applications, atomized content and functionality, 3rd party product ecosystems through open APIs, and much more are required to establish a strong and resilient network effect which fends off competitors that are themselves bringing these potent new competencies to bear.

In fact, one of the things we emphasize over and over again in our conference workshops and in Web 2.0 University is that having a Web site is usually the least interesting things about new products. Worse, it makes the customer have to find you amongst tens of millions of other sites. Instead, these new models tend to focus on going to the customer, instead of making them come to you which is a much harder proposition. This can instantly give you the ability to reach millions of potential people with dramatically lower effort and cost, as long as you have something interesting to offer.

Unfortunately, the number of capable practitioners of these new distribution models remains relatively small compared to the large body of experts in traditional Web product development. Demand is also low for these new skills as most organizations have been painfully slow to appreciate how much online product development has changed. A quick search of the job aggregator SimplyHired tells the tale: Nearly a thousand Web designer positions are available while only 36 OpenSocial and 40 open API positions are open, for example. This despite the the latter skills being able to project a product across the Web into hundreds of social sites or create an API that allows the product to be incorporated into countless other products for far less cost per customer than traditional methods.

The lesson here is that these new models still have a lot of fertile, unclaimed territory and many otherwise fierce competitors have not yet become fully aware of these new opportunities. Get your piece of the pie while there's still time.

I also find that the Web development industry has been slow to change, particularly outside the valley, and there is depressingly scarce information on how to deliver well on things like widgets, open APIs, social networking applications, and even syndication. To help with this, I've put together a short primer and some good references for those that want to get started.

Because the good news is that there remains tremendous opportunity for growth and success -- for both startups and traditional businesses -- if they will actively begin incorporating these new product delivery models into their own online capabilities.

Overview of Online Product Delivery Models

Web sites. This the classic model for Web presence. During the early Web, creating a Web site was just about the only option for engaging with those online (e-mail being the other.) Most early Web sites were used for publishing and not for user participation or peer production. These days, Web sites are still important, though by no means mandatory, and have their content syndicated via RSS and ATOM (pushing the content to where it's wanted), provide an access point to obtain widgets, and maintain user identity, and create communities of users. Upshot: They've evolved a lot but Web sites are only part of an extensive set of capabilities that must be brought to bear in the Web 2.0 era.
Syndication. It took ten years for the Web community to figure out a workable syndication model. Now RSS and ATOM are now the expected models used to distribute content off a single site and across the Web. Countless aggregation services now exist that make a site's information embedded in their services as well as a way to offer users a method for pulling information from a site and experienced in a means of their choosing, from Google Reader and Newsgator to the innovative Yahoo! Pipes. Most sites still heavily underutilize syndication even for notifications and pushing out frequently changed information to draw attention to it much less the strategic ecosystem and integration opportunities it affords.
Web 2.0 applications. You might argue that Web 2.0 itself is not a product distribution model but a set of design patterns and business models and that would be a true statement. However, in this context we're referring to the fact that Web 2.0 apps package up the 3rd major type of networked value: user participation. Before then, Web sites and syndication primarily had only centrally produced content or functionality that they could expose over the network and offer to the marketplace. In other words, user participation its purest form -- sometimes known as peer production -- ultimately results in products like Mechanical Turk and Predictify that provide direct networked access to user participation, but there are many fine gradations to this. The bottom line, Web 2.0 applications plug the user into the network like never before and are a critical rung in the distribution ladder since it offers access to the largest set of content and information by harnessing collective intelligence.
Open APIs and Web services. This is one of the most important long-term decisions most online businesses can make. Offering an open API lets anyone take the online components of a business, from its data and functional capabilities to the users themselves, and makes them open and accessible over the Web to be incorporated into other products and services, sometimes in the form of mashups and sometimes in the form of entire online products. Amazon, one of the first Web companies in existence and is hence far downrange in terms of the experience curve, has been using this distribution model with notable success recently. So have hundreds of others. The real challenge has been how foreign this model is to the original Web model and thus to the various management and development competencies in most organizations. It's much more an a way to OEM a product and leverage the customers and investments of hundreds of other partners. However, overall, it affords the potential for much larger business outcomes than could ever be created with point Web presence. It's now considered a significant oversight not to have an open API available for the typical online product.
Web widgets. Selecting parts of a Web site and it's data and packaging it up to make it run inside a portable, user distributable widget has been growing more and more popular over the last few years. For example, WidgetBox currently distributes 74,000 different kinds of Web widgets from its partners to over 1.2 million other sites. Widgets lets users distribute a Web site to other places on the Web at no extra cost and it also creates an ecosystem effect, where other Web sites users become the users of the new site. The YouTube badge is a notoriously well-known example of this that also helped drive the extraordinarily fast growth of the site. Like APIs, widgets are now considered a mandatory must-have for new and existing online products. But unlike APIs where it's up to the API users, figuring out users want out of your site's widgets is still an art form.
Social networking applications. Sometimes viewed as an extension of the Web widget model, social networking applications are applications designed to run inside of popular social networking environments and usually have capabilities that tap into and make use of the social graph information resident in a user's social network account. This is an amazingly fast moving field as you can see from a recent post on the latest happenings on the OpenSocial blog, to the extent it's hard even for well-funded companies to keep up. However, despite skepticism that large businesses can be built exclusively through a social networking application, it's become ever more essential for a site to make its capabilities accessible usable in these environments. Not only will users help distribute online products in these formats to their contacts but it also increases the overall usage of the your application including participation and its consequently growth of a site's network effect. While not yet considered mandatory for online products, the ease with which these social network applications can be created and the large numbers of users they make available makes it a smart distribution option for most Web businesses. Like widgets, however, figuring out what users will find engaging in a social networking application featuring your online product takes some research and experimentation. However, the results can be very rewarding and some social networking applications have millions of daily users. See the Plaxo Pulse story on Mashable for the details of how OpenSocial drove a 5x improvement in traffic in only 3 weeks.
Semantic Web and Web 3.0. The Semantic Web, one of the original visions for the World Wide Web, has taken a while to arrive but it's beginning to look like it may hit critical mass in the next 12-24 months. Combined with Web 3.0, which takes the architectures of participation at the core of Web 2.0 and drives it through a lens of Semantic Web capabilities. The benefits can be profound and can greatly increase the value and leverage of information on the Web. While this is very much not prime time yet, unlike #1-#6 above, it likely will be and smart organizations can get ahead of the learning curve and get an early market lead using these techniques. For now, however, I recommend that most organizations focus on executing well on the first six items before tackling this and waiting for the technologies to finish emerging and maturing.

The list above should provide good guidance for starting move into the potent new models for distribution on the Web. I'm seeing, however, that because of the major shifts in strategy and product design emphasis these techniques demand, most organizations take an inordinately long amount of time to become effective with them. The lesson here: Start small now and build core competency. Small investments now can pay off later in terms of valuable experience made from early experiments and pilots. When done right,
these new distribution models can become the dominant channels that the world uses to interact with your business, like they already have with Amazon and Twitter.

I'll be talking about these and other strategic online product design topics in my upcoming Building Next Generation Web Apps Workshop at the inaugural Web 2.0 Expo 2008 NYC next month. I'll have more details about this deep-dive session in an upcoming post.

Web 2.0 Continues As Most Used New Internet Term

Mon, 18 Aug 2008 11:10:56 -0700

While it's no longer quite so fashionable to label your Internet startup a "Web 2.0" company these days, the popularity of the term remains extraordinarily high and is presently used today both far and wide in traditional media and social media. The Google Trends graph in the figure to the right tells the overall story; global search interest in Web 2.0 is more popular than "social media" and "social networking" combined and by a significant margin. About the only other strategic technology concept that has anywhere near the same volume of world-wide interest is service-oriented architecture (SOA), which as it turns out is also surprisingly closely related to Web 2.0. Granted, Google Trends is not a scientific, "bet-the-business" kind of source, but it's a pretty darn good barometer.

Even for someone who spends much time with Web 2.0 concepts, I was surprised at this and I carried out a little cross checking from other sources and they all show the same disparity: Web 2.0 is still far and away one of the most popular terms to describe the intrinsic nature of many new online applications and businesses. This apparently highlights large scale demand for a broad enough term that rightly captures the innovations, new trends, and technologies that have emerged in the online space in the last few years. Web 2.0 has fit this bill better than any other single meme including the read/write Web, Social Computing, the Social Web, and the New Internet, to name just a few alternatives (and conceptually incomplete) terms that have been suggested.

The only real problem with this is that term itself has sometimes devolved into a vague buzzword that is often substituted as a simple synonym for social software or rich user experience techniques such as Ajax. Part of this is that the early investigation into Web 2.0 trends attempted to use it as a placeholder until the real underlying patterns were actually identified. This work resulted in the famous Web 2.0 meme-map that began to put meat on the bones and ultimately resulted in the excellent Web 2.0 Principles and Best Practices by my good friend John Musser. However, the lack of early specifics, though a brilliant move that allowed the right concepts to emerge from research into what was happening online, rather prescribing it blindly, also left a lasting impression of a vague, somewhat shapeless term for "newness" in the online world to the extent that even Tim Berners-Lee himself was left doubting.

However, it does appear that we are now left with both a very popular term that also has an increasingly large body of serious work that puts tremendous substance behind it. Academics such as Amy Shuen and her excellent Web 2.0: A Strategy Guide have produced enormous formal texts based on intensive research. A quick search of Google Scholar shows that over 14,000 references can be found. So too does the popular Web 2.0 Expo conference series continue and it has been expanding in recent years to the East Coast, Europe, and Asia. While the hype itself has largely dissipated and Gartner's 2008 Hype Cycle report says it's entering the trough of disillusionment, it also notes that "it will emerge within two years to have transformational impact, as companies steadily gain more experience and success with both the technologies and the cultural implications." I could not agree more.

Web 2.0: The Concepts Spread to Other Fields

I've previously covered what Web 2.0 means exactly and the virtual ink spilled on this often surprisingly complex subject is itself vast. The Wikipedia definition of "Web 2.0" remains one of the most popular entries on the site and the number of offshoots of the term has been a saga in itself, from the early days of Advertising 2.0, Law 2.0, Library 2.0 to the newer, (generally) widely accepted terms Enterprise 2.0 and Government 2.0.

For simplicity's sake, however, this is what we normally use to provide the most straightforward definitions of all things Web 2.0:

Web 2.0 - The continuously changing, participatory Web with a focus on building collective intelligence on myriad devices and primarily servicing The Long Tail.
Web 2.0 in the Enterprise - Web 2.0 as applied to business and not consumer activities.
Enterprise 2.0 - The social, collaborative network with emergent behavior and structure.

At this point there are some that like to invoke Buzzword Bingo at such seemingly gratuitously coining of new terms, but I personally find this a crucially important point: The global network of the Web itself, which is shaped continually by the endless participation of hundreds of millions of users around the clock, is no more than a reflection of those that shape it (which are then shaped themselves by it.) That the principles of Web 2.0 cross all disciplines, types of business, types of government, languages, as well as types of people and culture has fostered an interesting phenomenon. Namely, each of these topical areas are in the various stages of translating how Web 2.0 transforms and improves what they do, from architectures of participation and harnessing collective intelligence to radical decentralization (with cloud computing being the most interesting new example) and open service ecosystems.

This "localization" of Web 2.0 into specific verticals appears to be a natural competitive response by those trying to incorporate the latest best practices and proven technique into their work. In fact, I find that non-technologists and those whose professions are not spent in the world of software or in Internet businesses have a hard time incorporating, indeed translating, the Web 2.0 body of knowledge to their line of work. So one by one, we can thank a largely self-appointed group of experts have taken the trouble to map the 2.0 works into the many aspects of the world that are steadily being remade by the increasingly pervasive presence of the Web.

And Web 2.0 isn't standing still, we certainly haven't figured out all the ways that we can leverage the network yet. As we start thinking beyond Web 2.0 we begin considering where sensor-gathered information of every description, location-awareness (the iPhone will drive this like few other devices today), and the glimmerings of semantic capability, we can see that eventually Web 2.0 will -- like Web 1.0 -- evolve into something else in its own right.

It took us almost 10 years to figure out how to begin to use the Web properly and it may take another 10 years from now before most of us are incorporating the lessons of web 2.0 deeply into how we run their businesses. The result will be a transformed business and competitive landscape with products and services created and delivered in ways very unlike today (see my Web 2.0 predictions for 2008 for some details on this). It's also clear that the long-term implications will go well beyond that, similar to the way that the telephone, television, and especially the printing press changed how information was created, who could access it, and how it was owned and distributed. The parallels stop there since the deepest implications of 2.0 is a tremendous shift of control from the center of our networks to the edge.

What other 2.0 memes are you tracking? Please put in comments below.

Explaining REST to Damien Katz

Sun, 17 Aug 2008 05:11:51 -0700

Damien Katz recently caused a stir on a bunch of the blogs I read with his post entitled REST, I just don't get it where he wrote

As the guy who created CouchDB, I should be a big cheerleader for RESTful architectures. But the truth is, I just don't get it.

For CouchDB, REST makes absolutely insanely perfect sense. Read a document, edit, put the document back. Beautiful. But for most applications, enterprise or not, I don't see what the big win is.

I know what is wrong with SOAP, and it has everything to do with unnecessary complexity and solving the same problems twice. But what is the big advantage of making all your calls into GET PUT and DELETE? If POST can handle everything you need, then what's the problem?

I guess what I mean to say is just because SOAP is a disaster, doesn't somehow make REST the answer. Simpler is better, and REST is generally simpler than SOAP. But there is nothing wrong with a plain old POST as an RPC call. If its easy to make all your calls conform to the RESTful verb architecture, then that's good, I guess.

His post made the rounds on the expected social news sites like programming.reddit and Hacker News, where I was amused to note that my blog is now being used as an example of silly REST dogma by REST skeptics in such discussions. From reading the Damien's post and the various comments in response, it seems clear that there are several misconceptions as to what constitutes REST and what its benefits are from a practical perspective.

Background: The Origins of REST vs. SOAP

The Representational State Transfer (REST) architectural style was first described in Chapter 5 of Roy Fielding's Ph.D dissertation published in 2000. It describes the architecture of the Web from the perspective of one of the authors of the HTTP 1.1 specification which was published the year before in 1999. Around the same time Don Box, Dave Winer and a bunch of folks at Microsoft came up with the Simple Object Access Protocol (SOAP) which they intended to be the standard protocol for building distributed applications on the Web.

Over the following years SOAP was embraced by pretty much every major enterprise software vendor and was being pushed hard by the W3C as the way to build distributed applications on the Web. However a lot of these vendors weren't really interested in building software for the Web but instead were more interested in porting all of their technologies and scenarios from enterprise integration technologies like CORBA to using buzzword compliant XML. This led to a bunch of additional specifications like XSD (type system), WSDL (IDL) and UDDI (naming/trading service). Developers initially embraced these technologies enthusiastically which led to the enterprise software vendors pumping out dozens of WS-* specifications. During this period not many thought or talked much about REST since no one talks about boring Ph.D dissertations.

In 2002, a canary in the coal mine emerged in the form of Mark Baker. On mailing lists frequented by Web services types such as xml-dev and xml-dist-app, Mark began to persistently point out that SOAP was built on a bad foundation because it fundamentally ignored the architecture of the Web as defined by Roy Fielding's thesis. At first a lot of people labeled mark as a kook or malcontent for questioning the trend of the moment.

By 2005, the industry had enough experience with SOAP to start seeing real problems using at as a way to build distributed applications on the Web. By that year many developers had started hearing stories like Nelson Minar's presentation on the problems Google had seen with SOAP based Web services and sought a simpler alternative. This is when the seeds of Mark Baker's evangelism of Roy's dissertation eventually bore fruit with the Web developer community.

However a Ph.D dissertation is hard to digest. So the message of REST started getting repackaged into simpler, bite-sized chunks but the meat of the message started getting lost along the way. Which led to several misconceptions about what REST actually is being propagated across the Web developer community.

Misconceptions About the REST Architectural Style

With that out of the way I can address the straw man argument presented in Damien's post. Damien states that building a RESTful Web Service is about using the HTTP PUT and DELETE methods instead of using HTTP POST when designing a Web API. In fact, he goes further to describe it as "the RESTful verb architecture" implying that choice of HTTP verbs that your service supports is the essence of REST.

This is incorrect.

Q: What is the Essence of REST? A: The Uniform Interface

REST explains how the Web works by defining the set of constraints on the various components in the current Web architecture. These constraints include

interaction is based on the client-server architectural style. User agents like Web browsers, RSS readers, Twitter clients, etc are examples of Web clients which talk to various Web servers without having a tight coupling to the internal implementation of the server.
communication between the client and server is stateless. The benefits of HTTP being a primarily stateless protocol is that statelessness increases scalability and reliability of services at the cost of introducing some complexity on the part of the client.
the Web architecture supports caching by requiring that requests that are cacheable or non-cacheable are labeled as such (i.e. via HTTP method and various caching related headers).
there is a uniform interface between components which allows them to communicate in a standard way but may not be optimized for specific application scenarios. There are four interface constraints: identification of resources; manipulation of resources through representations; self-descriptive messages; and, hypermedia as the engine of application state.
there can be multiple layers between client and server which act as intermediaries (e.g. proxies, gateways, load balancers, etc) without this being obvious to the requesting client or accepting server.

When you read the above list, the first thing you will note is that it describes the architecture of the World Wide Web. It doesn't describe the architecture of "a typical enterprise" or the internals of a cloud computing application.

Building a RESTful Web Service simply means paying attention to these characteristics of the Web. As you read them, some practical guidelines start becoming obvious. For example, if you want to take advantage of all the caching infrastructure that is built into the Web infrastructure, then you should use HTTP GET for services that retrieve data. This is just one of the many things Web Services built on SOAP got wrong.

The uniform interface constraints describe how a service built for the Web can be a good participant in the Web architecture. These constraints are described briefly as follows

Identification of resources: A resource is any information item that can be named and represented (e.g. a document, a stock price at a given point in time, the current weather in Las Vegas, etc). Resources in your service should be identified using URIs.
Manipulation of resources via representations: A representation is the physical representation of a resource and should correspond to a valid media type. Using standard media types as the data formats behind your service increases the reach of your service by making it accessible to a wide range of potential clients. Interaction with the resource should be based on retrieval and manipulation of the representation of the resource identified by its URI.
Self-descriptive messages: Following the principles of statelessness in your service's interactions, using standard media types and correctly indicating the cacheability of messages via HTTP method usage and control headers ensures that messages are self descriptive. Self descriptive messages make it possible for messages to be processed by intermediaries between the client and server without impacting either.
Hypermedia as the engine of application state: Application state should be expressed using URIs and hyperlinks to transition between states. This is probably the most controversial and least understood of the architectural constraints set forth in Roy Fielding's dissertation. In fact, Fielding's dissertation contains an explicit arguments against using HTTP cookies for representing application state to hammer this point home yet it is often ignored.

Benefits of Conforming to REST and the Uniform Interface to Web Developers

At this point, the benefits of building RESTful services for the Web should be self evident. The Web has a particular architecture and it makes sense that if you are deploying a service or API on the Web then it should take advantage of this architecture instead of fighting against it. There are millions of deployed clients, servers and intermediaries that support REST and it makes sense to be compatible with their expectations.

This doesn't mean you have to use DELETE and PUT when POST might suffice. It does mean understanding the difference between using POST versus using PUT to other participants in the Web architecture. Specifically, that PUT is idempotent while POST is not so a client of your service can assume that performing the same PUT two or three times in a row has the same effect as doing it once but cannot assume that for POST. Of course, it is up to you as a Web service developer to decide if you want your service to provide a more explicit contract with clients or not. What is important to note is that there is a practical reason for making the distinction between which HTTP verbs you should support.

There are other practical things to be mindful of as well to ensure that your service is being a good participant in the Web ecosystem. These include using GET instead of POST when retrieving a resource and properly utilizing the caching related headers as needed (If-Modified-Since/Last-Modified, If-None-Match/ETag, Cache-Control), learning to utilize HTTP status codes correctly (i.e. errors shouldn't return HTTP 200 OK), keeping your design stateless to enable it to scale more cheaply and so on. The increased costs, scalability concerns and complexity that developers face when they ignore these principles is captured in blog posts and articles all over the Web such as Session State is Evil and Cache SOAP services on the client side. You don't have to look hard to find them. What most developers don't realize is that the problems they are facing are because they aren't keeping RESTful principles in mind.

Don't fight the Web, embrace it.