Alex Bosworth's Weblog

I'm Leaving SourceLabs

Tue, 29 May 2007 13:35:55 -0700

Just a heads up to readers of my blog.

You may know that SourceLabs hase employed me for the past two and a half or so years. But today is my last day at SourceLabs: my plans are to travel, take some time to think and adventure, and I plan on continuing to work on little projects and prototypes – that’s just what I like to do.

It’s been a crazy time here at SourceLabs, in most ways I couldn’t imagine a better job: design and build a website and then run and grow it. SWiK has grown from a small prototype I wrote 2 years ago to something that is serving 125+ pages on open source every minute to about a million unique visitors a month.

When SourceLabs started it was an idea with no name, and I started talking to Byron (SourceLabs CEO) about his ideas for a public community oriented website about open source software. After a while SourceLabs picked up a name, an office, we stopped meeting in coffee shops and out of our houses and got an office in Seattle. SourceLabs has come a long way since then, and I recommend highly the startup experience.

I’m going to try posting more often on my blog, however if you want to keep receiving new posts you might want to double check which url you are subscribed to and make sure it’s http://feeds.feedburner.com/AlexBosworth – I’m probably going to switch over to the domain I snapped up recently: alexbosworth.net (nothing there atm). I need to figure out blog engine stuff before I post over there, but if you subscribe to the feedburner feed you won’t even notice.

Anyways stay tuned I will probably continue to churn out little projects and such you may be interested in, or points on development, tech, and other fun stuff.

Digg CommentSpy

Mon, 30 Apr 2007 09:51:05 -0700

Over the weekend I made another script based on the Digg API: Digg CommentSpy

This one tracks live comments posted to Digg, ala LiveMarks. Blue posts are replies to the top thread, yellow are replies to another comment.

I’m now a huge fan of JSON feed based APIs. These scripts have 0 server code, all the work is done in Javascript. This means that scaling the mashup is all on the remote service, not on my puny server.

If you’re using my Digg javascript function, commentspy uses a slightly revised version that allows for ‘min timestamp’ – I use this so that I don’t ask for already seen comments.

You Need Friends on Digg

Thu, 26 Apr 2007 18:21:35 -0700

I’ve written another of my little javascript web apps, this time based on Digg’s new expansive API.

The API is very well designed and I’ve only found a couple problems with it: the license is very restrictive and getting back some info requires many multiple calls to digg, but I guess it points to the problems with web apis as just being poor proxies for direct database calls.

The Digg application is a friends browser. – It basically asks the question: “Who’s digging you”.

I wrote this script because i’ve noticed that the front page of Digg is guarded by social networks of people who digg each other’s stories. If you want to get your story noticed, you want to have digg friends.

Enter any top digger name, you will see a high ratio of repeat diggers – the same is true of my submissions as well (Thanks Pawfoots!).

As I wrote it, I was interested really in just browsing around the networks, so I thought that was fun enough for an app. It’s kind of like an earlier script I wrote: delimages.

Anyways my app started with a single function basically that gives you back a digg response object with what you want. If you write a Digg script, you can use my little Digg function:


var Digg = 
{
    apikey : 'http://sandbox.sourcelabs.com/'
};

Digg.request = function (path, count, cbkName, offset)
{
    var url = 'http://services.digg.com/' + path;

    if (!offset) offset = 0; 

    var params = $H(
    {
        'count'    :  count, 
        'type'     : 'javascript', 
        'appkey'   :  Digg.apikey, 
        'callback' :  cbkName,
        'offset'   :  offset
    });

    url += '?' + params.toQueryString();

    var head = document.getElementsByTagName('head')[0];
    var request = document.createElement('script');
        request.type = 'text/javascript';
        request.src = url;

    head.appendChild(request);
}

All of these JSON Apis are really hurting my Javascript OOP sensibilities. XMLHttpRequest is cool with you making an anonymous function on the fly as the callback, but with JSON the best approach probably is having a callback name passed in the object return, which isn’t the cleanest.

I tried to make do in my script as best I could though.

Clay Shirky: Warcraft pwns Second Life

Tue, 30 Jan 2007 13:20:26 -0800

I read a lot of Clay Shirky’s stuff and he has some great thoughts. In his newest essay he puts forward the premise: Second Life Sucks, World of Warcraft Rules.

He puts it a little better than that though:

Games have at least three advantages other virtual worlds don’t. First, many games, and most social games, involve an entrance into what theorists call the magic circle, an environment whose characteristics include simplified and knowable rules. The magic circle saves the game from having to live up to expectations carried over from the real world.

Second, games are intentionally difficult. If all you knew about golf was that you had to get this ball in that hole, your first thought would be to hop in your cart and drive it over there. But no, you have to knock the ball in, with special sticks. This is just about the stupidest possible way to complete the task, and also the only thing that makes golf interesting. Games create an environment conducive to the acceptance of artificial difficulties.

I’m not sure what he means by a magic circle, but as I put in my previous post on Warcraft and Second Life: Warcraft is very controlled but also very entertaining.

Valleywag and the social software cognoscenti on Many 2 Many apparently have it in for Second Life, probably in response to the gushing and false press reports about Second Life’s successes.

I’m not very interested in Second Life, but I do think that Warcraft is escaping a lot of criticism in these posts. Sure Warcraft is popular, and lots of people like it, but is it good? Eric Rice raises this point by pointing at MySpace: it’s a social software success, but there’s a lot to hate about it.

My main criticism of Warcraft is pretty obvious: it’s a big time waste. Ultimately wasting time is not the best experience. It’s entertaining, but it’s basically like TV: you sit there for a while and it’s not so much a hobby as something to keep you occupied.

As I posted earlier, Warcraft is a controlled experience with carefully planned areas, kind of like a theme park or Disneyland whereas Second Life is an uncontrolled anarchy. But why can’t we have a mix of both?

Take Lego for example, it’s a toy I played with a lot as a kid. With Lego, you get a box full of bricks and it’s a fun experience just to follow the instructions and put them together like they are on the box. But Lego’s genius is that you don’t always do that. A lot of times you just build stuff with Lego, to make your own designs.

We need more game design like that: not to the pointless extreme of Second Life, but a game wherein you can play the game by itself and have fun, or invent diversions from the game to be creative and just build stuff.

What are Machine Tags?

Wed, 24 Jan 2007 15:51:26 -0800

straup from Flickr recently posted about a new tagging feature in Flickr called Machine Tags:

We are rolling out a new feature called “machine tags” that allows users to be more precise in how they tag, and how they search, their photos.

- When a user tags an event with an upcoming ID (for example : “upcoming:event=81334”) we display a link back to the upcoming.org site.

We’ve thought about this idea in SWiK, and I’ve seen something similar on another Yahoo property: del.icio.us with system:filetype:jpg tags that I use to power delimages.

I wonder if it would be too crazy to formalize machine tagging and give it domain namespaces?

For example, if you tag a bookmark on del.icio.us “upcoming.org:event=foocamp”, del.icio.us could run off to upcoming.org and grab some info about foocamp.

Another advantage to machine tags is giving you search access and combination opportunities to locked up system metadata. GMail has a good example of this: if you want to find emails from Alex in gmail, you can search using “From:alex”. There’s a laundry list of these Machine Tags on gmail.

Google of course has the luxury of making every word in your gmail a tag that describes it, but they also add these system tags to add power above that.

Adding domain specific tags could help in gmail too. Making a label in gmail called “upcoming.org:event=foocamp”, tagging emails with that label, and then having Google Calendar remind me on the date of Foocamp all the emails I tagged as related to the event.

Social Software: Second Life or World of Warcraft?

Tue, 19 Dec 2006 19:05:27 -0800

There are an unprecedented number of new online ventures seeking to create user based software, but among these hundreds of offerings there is a split in the core design: Should users run the show?

The design of Second Life highlights this model. The creators of Second Life, Linden Labs, seek to be nothing more than platform developers who rent servers to people who want to play on their platform. This means that all content in the second life world is created by users, in a largely unregulated way. Because there is no rules or purpose to using Second Life, Second Life has become a chaotic and risqué collection of people interacting in what is essentially a giant avatar based chat room.

The other side of this model is World of Warcraft. In WoW, everything in the world is created by Blizzard developers and artists. The experience is scripted and polished, to a perfection that has been the signature of Blizzard products since the first Warcraft was released in 1994. These is not an ounce of content in WoW that wasn’t created by Blizzard, user interface mods aside. Because Warcraft sticks so closely to focusing on a great game experience, Warcraft is an experience much like Disneyland – thoughtfully crafted rides, fun for the whole family, carefully tended and managed to help avoid the chaos people usually bring along with them.

Other services pretty much fall into one of these two models.

Control Chaos

World of Warcraft	Second Life
Amazon	eBay
eBay	craigslist
Google Video	YouTube
Facebook	MySpace
Digg	del.icio.us

This model might suggest that the two models are different segments, but looking closely at the development of all of these successful sites suggests reverse entropy: embracing chaos is not a sustainable way to run a site: the future model for mature social software seems to be Disneyland / World of Warcraft.

As usual, Clay Shirky wrote a great paper on this and way before I even thought about it, in A Group Is Its Own Worst Enemy

The problem is that in unmanaged communities, there’s a power vacuum and eventually someone ends up directing it, typically either spammers or high schoolers with the time and inclination to take over, it’s often a combination of the two. Clay is not the first person to write about this issue.

For a while chaos works quite well. Digg has rode the wave of “user controlled” to the top, all the while introducing algorithmic limits to individual power as well as managing things editorially for quality control. Today’s Digg, with bans on top users to moderate the community as well as algorithmic limitations on vote efficacy is very different from the early Digg versions with a simple set number of votes required to promote a story and a small but loyal following that kept things in line.

del.icio.us manages their chaos as well through putting limits on user interactivity. Joshua Schachter has been quoted on many occasions that he does not want to build a community, but rather a social tool – because of the management problems community brings.

A good example of the promise of chaos and the problems it brings can be seen in a recent patch to World of Warcraft the other day. Although Warcraft is mostly static with a window dressing war that doesn’t change, for holidays and other occasions Blizzard will introduce a seasonal patch with goodies for the time of year.

For Christmas (Winterveil) this year Blizzard introduced reindeer, fruitcake, Father “Winterveil”, and throwable snowballs. Snowball fights abounded. But there was a problem. Players hit with snowballs move back. In Warcraft, only members of the opposing side are supposed to be able to attack each other, but suddenly there was a new game called “snowball your team mates off of the cliff”. Pretty soon the forums exploded with people complaining about their fun being ruined by mean kids with snowballs. Quickly, Blizzard issued a hotfix to end the snowball fights: no more snowball fights with your friends.

This small change highlights the process that all of these social sites go through. Whether it’s Google YouTube turning to censoring comments or it’s del.icio.us blocking my private bookmarking service, control over the community is taken away from the users and put in the hands of the host.

This is not the wrong decision. The impulse is to say Warcraft should have let the snowball fights go on. But World of Warcraft is a great game, a fun game and safe for kids to play, and it’s because of a thousand decisions like this: it’s a constant battle to keep it that way. Disneyland might not be a very interesting place, but I liked it when I was a kid and they have to cope with an enormous scale of visitors, offering a consistent and satisfying experience to each paying customer. The alternative is Second Life, with 1/3 of its player economy devoted to pornography, and an explosive item cloning issue, it’s not something I’d buy someone for Christmas.

There’s a lot to be said for a controlled experience. In SWiK we see spam every single day. People are trying to take control over our pages, abusing our open system. The temptation will always be to close it, to require registration to edit, to put up captchas and email verification and other lockouts. And it’s not a bad impulse. The bad impulse is to stop innovating in the face of chaos. I’m glad Blizzard tried the snowballs, and I’m glad they took them out quickly. The key is the focus on the experience. For experiments and new services, chaos works well and is a lot more interesting, but as things mature people want something that just works rather than something that has promise and flexibility.

Session Security

Mon, 11 Dec 2006 18:50:27 -0800

Brian Ellin wrote an interesting blog post about session security today, but for some reason it’s gone, so I’m going to similarly post on session security.

It’s a tendency of most web developers nowadays who are building interactive websites to put a global login that triggers a long running session cookie, so users don’t need to constantly verify their password to do things associated with their accounts.

This however can be a big time security issue.

Why? A session cookie is something that is sent in HTTP headers, and the HTTP request can come from anywhere and your browser will happily send along the session cookies too.

Ok enough of this talk. Want to see an actual factual exploit?

Go to this page – if you have logged in to Netflix recently, the movie “Hackers” will be added to the end of your Netflix queue.

How was this done? Not by a cookie monster virus I assure you.

It’s only 1 line of HTML.
<iframe src='http://www.netflix.com/AddToQueue?movieid=567905' style='display:none;'></iframe>

This is an example of HTTP method misuse, it’s 1 line because Netflix doesn’t pay attention to the HTTP standard of not making stateful changes through GET requests.

But even if they had used a POST instead of a GET to add a blog to your homepage, session security would still apply. Through Javascript triggering of forms, it’s also easy to create a form like <form action='http://netflix.com'>, make it hidden and then trigger the submit() method on the form with the target an invisible iframe.

How to avoid this?

Include in submission forms a random hidden value that is checked against the session variable. Browser security does include measures to avoid snooping on the contents of remote iframes, or javascript interference with the contents of said frames.

del.icio.us uses this method – on the post pages, a random number is generated and appended to the post. Initially I thought that Google Homepage was vulnerable to this attack, but they use a similar type of defense – setting a random variable on the server and checking for it on the client before taking action.

Issues with Ajax

With Ajax requests and stateless web frameworks like PHP, session security checking can be complicated by race conditions.

Imagine if your Ajax requests were required to be scoped by a random session variable. To be safe, like del.icio.us does, you randomized the variable every page load.

But now you’ve rewritten your web todo list application to use Ajax. What used to be a simple application flow of adding a new task, reloading the page and editing some more, is now a more complicated one: tasks can be edited and added from various endpoints, and simultaneously.

If you’ve kept your application’s logic for session security the same, you might check on each todo list edit for the random session variable, and then send back the new variable in the Ajax response, updating the token on the client side when it receives the response.

But what if the user edits two entries in their todo list before before the first one has had a chance to return a response from the server? Well what happens is that your server receives the second todo list request, sees that its token is not matching, and discards it – an error.

This case would result in obvious errors, but there are a thousand ways it can be less obvious, so be careful.

A good resource for reading more about problems with asynchronous requests and sessions can be found in Marc’s article on Trouble with Ajax and Sessions

Del.icio.us Tools

Mon, 04 Dec 2006 18:11:21 -0800

Even though I’ve moved off of del.icio.us and onto my encrypted web bookmarks manager boz, I still have hundreds of bookmarks saved on del.icio.us, and I use the site for other things like a handful of projects that use the site’s api to do some interesting stuff.

I’m adding a new one to the list today, and updating an old one to start Delicious Tools.

Delicious Tools includes:

A dead link checker – lots of your urls are out of date or broken: check that your bookmarks are still valid.
Google sync – synchronize del.icio.us with your Google search results.

(thx osde-info for the screenshot and braving Flickr’s policies against them)

Other del.icio.us projects:

LiveMarks – streams social bookmarks from various services and does some frequency analysis as well to highlight new and interesting stuff.
Delimages – hotlinks images from del.icio.us image streams.

Notes

Fixing my database

Over my Thanksgiving vacation I stopped checking on my projects for a week or so and that’s when everything decided to go to hell. The MySQL database dedicated to the del.icio.us projects ran out of space, and when I tried to fix it I wiped it :/ I’m pretty lax with these projects and I don’t do sensible things like use a VCS or make comprehensive backups, so it took a bit of work just to get things back to normal. There was also some code rewriting required, and I revisited a lot of queries in various services to bring them back online, such as in LiveMarks which now has a different popularity query and for delimages which now has a simpler tag query. Things should be back to normal though, so if you use any of these services, send me mail if you don’t like the changes or something is broken.

Making the dead link checker

The dead link checker turned out to be a little bit annoying, given that a lot of servers don’t respond properly to HTTP HEAD requests.

The checker works by making a bunch of prototype-driven Ajax calls to a little JSON outputting PHP script that makes a socket connection to the server to do a HEAD request for the URL.

This was fairly straightforward, however there were some tricky bits in that there are a range of weird responses to HEAD requests:

Server just hangs on Connection: close
Server decides to put the entire page HTML in the HEAD response (i’m looking at you IIS 6.0)
Server always returns a wrong response code for the page (Wikipedia and Google Video)

I didn’t do that much about special-casing completely wrong codes, so you’ll have to verify some yourself.

Updates to Kibbutz

I already had a Google sync project called Kibbutz which drives the XML translation of del.icio.us bookmarks to Google Co-op search results.

One thing you might have noticed however if you used it – it didn’t index all of your posts. Del.icio.us has a purposely limited API that requires login credentials to access your full list of public bookmarks, and I didn’t feel like writing an importer. Well I finally wrote one, and now you should be able to get all of your bookmarks indexed.

Now that I’ve written the importer script, it should be straightforward to write some more tools for working with your bookmarks – these will go under delicious-tools, watch that space.

Google: In your business, taking your money.

Mon, 13 Nov 2006 16:48:45 -0800

Jacob Nielsen’s Alertbox published an article at the beginning of this year in which he put forth an argument that said:

Search engines extract too much of the Web’s value, leaving too little for the websites that actually create the content. Liberation from search dependency is a strategic imperative for both websites and software vendors.

This provocative thesis is backed by a compelling analysis of Google’s business model.

Basically, if you are a company that produces widgets at $5 a hogshead, it’s worth it to you to pay $4.99 in commissions to Google for each sale they deliver. And because you are in competition for who gets the advertising with other companies that produce widgets – if you pay less than $4.99, you won’t get an ad at all.

Google thus profits from a situation where online business turn over all their profits to the gatekeeper of the web – so that they can make the penny of profit over their competitors.

This is an interesting argument. Basically he advises you to develop more cost effective channels of marketing, where you can the same amount of click through activity for a lower cost. Despite this being easier said than done, this is good advice.

But what about the impact on the consumer. “When companies compete, you win” is the mantra of the capitalist. The idea being that competition drives prices down and quality up.

Unfortunately, this idea doesn’t work very well with Google’s marketplace. The reason is: Google has no pricing or quality information in their ads. If you search for widgets: you don’t know that although the top result charges $9 for crappy widgets, the bottom result charges $3 for great widgets and good service.

Let’s back up a minute to you and your widget company. You are happily producing widgets at $5 a hogshead, your competitors realize this and pay $4.99 to wipe you off the map, so you pay $4.99 to stay in the running.

Here’s where it gets bad for the consumer. (In other words, there’s a reason if you know what you are doing on the net, you’ll rarely buy via a ‘Google’ Adwords ad).

Your competitors think for a while and realize that if they raised their prices and charged $9 for widgets, they could pay Google $8.99 for ads and wipe you off the map again. Now if you want traffic, you are going to have to pay the gatekeeper $8.99 a sale and raise your price to keep up with it.

Uh oh, now Google is causing raised prices across the board, bad news for the consumer. This is a natural trend for advertising, which is always competitive – the reason for $12+ ticket prices at the movie theater are not so as much bigger budget flicks, as much as huge movie advertising costs. (“Disney, Warner Bros., Sony, 20th Century Fox, Universal, and Paramount—spent, on average, $34.8 million to advertise a movie and earned, on average, just $20.6 million per title.”)

This is not to say the prices of advertisement on Google will trend ever skyward. Although Google has a monopoly on web search, as Jacob points out: they don’t control all channels. Brick and mortar prices can have memorable prices or offer ‘lowest cost’ deals – Very strong brands like Amazon and Newegg can create stores where they gain a reputation for generally low prices and go there and do at least a price check, if they don’t go there first.

And that’s how I do my purchasing through trusted brands or recommended sellers – not through Google Adwords, a system that rewards and promotes vendors for not offering a quality product, but rather for giving up profits to Google and writing misleading advertisements.

Another alternative is Craigslist or Ebay. Both of these have models for real markets, where people compare prices and quality of goods / sellers, rather than just catchy advertisements.

Announcing SWiK-Source

Wed, 20 Sep 2006 16:44:31 -0700

I’m happy to announce that SourceLabs has decided to release SWiK-Source – the code that drives swik.net, under an open source license (GPL v2).

Although we didn’t want to make any promises, open sourcing the code to the wiki has been something we intended to do from the start of the project. Personally, in designing SWiK as an Ajax powered wiki, I don’t think there’s any reason it can’t be used for various purposes beyond driving swik.net, and in fact for the past 6 months internally at SourceLabs we’ve repurposed SWiK-Source to run as our internal wiki to help organize our internal projects. People write weekly status reports in the blog pages, describe design policies in wiki pages, and use tags to avoid a disorganized wiki.

It’s been the single biggest demand by far since we started the project, that we go beyond licensing the wiki pages under an open license, but that the entire engine driving the site be open as well. Designing and building a web service however isn’t exactly the same as building a software product: we didn’t design for environments that aren’t our own production servers.

That will not change, with the release of SWiK Source – it’s still designed to be run on production servers. We’ve abstracted to a configuration file everything that might be specific to a wiki install, such as the desired wiki name and the paths to libraries, but the code is still written for a server environment. While the code is open, setting it up and installing it is not for the faint of heart, and we haven’t tested it in settings too far beyond the servers we run.

The documentation for the project will live entirely on SWiK and be collaborative. If you want to help with the SWiK-Source distribution, that’s what we need: better docs on using SWiK-Source in different settings and for different purposes, or fixing any problems with the current docs. The installer script could use some love too.

All that being said, I’m excited to finally be able to offer SWiK-Source as free software, and at the very least feel free to poke around and see how it all works.

Boz, Videobookmarks upgrades

Fri, 08 Sep 2006 19:20:16 -0700

First, an important update: if you are not subscribed to my feedburner URL that I switched to a long time ago, please switch your subscription to http://feeds.feedburner.com/AlexBosworth. I am moving my blog to SWiK: increased growth of SWiK traffic over the past few months has meant that we have needed to switch our setup to a more load-balanced setup across more webservers. Rather than struggle with MovableType, which has been having a series of problems, I’m just going to blog solely on this blog, which I was only cross posting to before. If you are subscribed to the RSS feed, you shouldn’t even notice the difference though.

Boz news

It’s been a month since I released boz – an aes encrypted, (open source) private bookmarking service. Today I rolled out the latest update, adding bookmarks export, rss support, and n-way tagging, as well as a tag search box.

RSS support was a tricky problem from the encryption standpoint. I use Firefox RSS integration with my del.icio.us bookmarks, and I really wanted the same functionality in boz. There are a handful of bookmarks that are sensitive and certainly not ‘social’, that I still want to have in my firefox livebookmarks. But Firefox can’t decrypt an encrypted RSS feed. The solution is to attach labels to bookmarks that should be included in an RSS feed. The labels can be generic if I don’t want to give away the purpose of the link, but in any case the real title and link of the bookmark is protected by the encryption.

So far the features upgraded since the release of boz have been:

Encrypted tagging
- tag cloud
- tag autocompletion
- tag search
Starred bookmarks – ala gmail
RSS feed – add cleartext labels to bookmarks to have them appear in an rss feed. The link address and title remain encrypted.
export: pull out an encrypted stream of bookmarks

I’m biased, but I have to take back what I said earlier about private and public bookmarks not mixing. I’ve found that they do, because I’ve found myself only bookmarking encrypted bookmarks on boz and not really anything more on del.icio.us. I like to bookmark stupid stuff as well as internal intranet urls, and I am too lazy to figure out if it should be posted to del.icio.us or boz, so sorry if you are one of my network buddies on del.icio.us but this is part of the reason for the rash of updates to boz this month.

boz was originally written to replace a previous project that del.icio.us put the kibosh on called priv.at, and I wrote an importer for people who wanted to migrate bookmarks they saved there that were wiped by del.icio.us, but some have asked for a migration tool to encrypt their existing normal del.icio.us bookmarks in boz. I’m not sure if this is a big demand, but if people want to migrate their bookmarks to boz, let me know, it wouldn’t be too hard to adjust my existing migration script.

VideoBookmarks

I’ve updated VideoBookmarks, the application I built on wikiality to display the most recently bookmarked videos. The most recent update deals with a problem that I’ve had to deal with with delimages—people bookmark the same thing over and over.

VideoBookmarks now tries to only show the same video once.

Ajax: A Throbber For All Seasons

Tue, 05 Sep 2006 15:25:50 -0700

If you are using Ajax, and by Ajax I meant XMLHttpRequest, and you are using it intensively, you may have noticed you want to give people an indication that requests are being sent to the server.

The browser does this by making something called the ‘throbber’ move. This takes various shapes, from stars flying across a giant N, to a windows flag waving, to small dots rotating. In FireFox, or other tabbed browsers, the throbber also acts as a helpful indicator on a tab to indicate whether a tab has finished loading.

As a heavy internet user, I have become attuned to the signal that a webpage is still loading, and in an application that uses XMLHttpRequest, it can be confusing as to what is going on.

So I’ve developed a technique to fake the browser throbbing, by launching a dumb parallel request to the XHR request. The parallel request merely creates an iframe to a page that will load for a long time, triggering the throbber. When I want to stop the throbber, I just destroy the iframe.

I’ve rigged up a demo page on wikiality, so you can see the technique in action and the source code behind it.

PS: Btw to all the word police and wordinistas who wrote in to say that wikiality is the exclusive province of Steven Colbert: If Steven Colbert wants me not to use it, he can ask me himself, and if he doesn’t come over to my office to demand it back, well then he’s a coward and he doesn’t deserve it anyways.

Wikiality

Tue, 05 Sep 2006 01:55:06 -0700

I’ve been working on a wiki project as an experiment for the next generation of SWiK.

The concept is that the web isn’t just a lot of static webpages anymore, it’s Ajax widgetry, Flash videos, fancy stylesheets, etc. Wikis however are dragging behind, stuck in static webpage land, there either isn’t a model for extending the wiki, or if there is it’s too complicated to really be ‘wiki’.

The wikiality concept tries to play a balance between these worlds. It’s a very experimental concept, but I’ve started using it personally and I think it’s started to be interesting. The way it works is via two types of pages: wiki pages and wiki templates.

The templating language is straight JavaScript. Templates are wiki pages that contain an eponymous Javascript class. When templates are embedded in a wiki page, the page then incorporates any referenced templates.

I like building applications in wiki form because:

It’s faster to develop, about two times as fast as a regular application
More customization. If I don’t like how something works in a widget, I can clone the widget, change it subtley, and now I have it working both ways. You can also build your own interface from top to bottom, there’s no constraints of the standard application ui.

I’ve written some applications in wikiality:

Bozpages 2.0 – This is the successor to Bozpages (simple rss feed pages).
- bozpage web2.0
- bozpage tech news
Video Bookmarks – Like delimages shows you the most recently bookmarked image feeds from del.icio.us, Video Bookmarks shows the most recently bookmarked videos from YouTube and Google Video.

I’m still thinking about how the concept works and it’s very experimental and in flux, but I thought I’d share just the basic idea: I think wikis and widgets can work together.

PS. I’ve been looking more into how Digg works, and how the user ecology is evolving, I posted this on my other blog, but I think it might be interesting to people who just read this one, It’s called “The Prisoner’s Dilemma In Digg Story Promotion”. It talks about a theory I’m working on that there might be a weakness in Digg’s story promotion scheme wherein you can get your stories front-paged by entering into a cooperative game with other people trying for the same thing.

How To Provide A Web API

Mon, 21 Aug 2006 17:25:08 -0700

In a world where people are making interdependent webservices, API design and maintenance is pretty important. Unfortunately despite rising use and availability of APIs, there are significant problems with the way even big API vendors are deploying and maintaining their APIs.

What are a few simple rules for providing a web API?

Keep it clean and simple
Stick to standards
Make it about data
Keep it working
Design for updates

Keeping it clean and simple is subjective and a matter of audience. For most developers a simple API is REST/HTTP based, with XML delivery of a known or simple schema, RSS being a good general choice. For JavaScript developers or plugin writers JSON feeds might be preferrable. For enterprise development scenarios, SOAP over HTTP might be better, but generally it’s best to stick with just REST + XML/RSS.

Simple also means don’t be too abstract. Flickr for example chooses in its API to require the use of its internal ids for all API calls. This means for example that every call to find information about a user requires a call first to find the internal id of the user. Del.icio.us on the other hand just requires visible names, in fact internal ids are hidden everywhere.

Sticking to standards is a matter of developing APIs that can plug in to accepted norms. Not only does this make development easier, it makes tooling and other peripheral services work better, and generally standards are written for a good reason. Using REST? Don’t use GET requests to update state, such as in del.icio.us’s urls to delete or add urls like https://api.del.icio.us/v1/posts/add?. Using RSS? Try to stick to mainstream semantically appropriate elements rather than new namespaces, provide well formed XML, don’t stick data in XML attributes, etc.

Make it about data. Leave application design to the application developer. Google’s new Ajax search results API is a good example of an API that isn’t about data, which makes less flexible to build upon. Instead of providing JSON feeds for plugin developers, Google has chosen to build out their own little search results box, with controls and results that cannot be styled, instead of leaving the interface and logic up to the Javascript developer or plugin writer. A better design would have been a simple JSON feed of Google service search results, and a reference object to build an embedded results box.

Keep it working. An application developer working with remote web services should design with the consideration that the remote service can malfunction or die, but that doesnt’ mean that service providers shouldn’t prioritize keeping reliable service high on their list. On SWiK and other development projects I’ve done, every one of the APIs we use (del.icio.us, sourceforge, google, etc) have gone down or had problems, and I learned the hard way not to depend on any of them.

Make a clean upgrade path. There’s no permanent APIs: add a version number and keep developers informed. Flickr calls don’t have a version number but they should. In del.icio.us even browser bookmarklets have versions. Salesforce.com, whose bottom line depends on web service APIs, uses versioned WSDLs. Each new rev of Salesforce.com’s APIs are given a unique WSDL and the backend from that point is kept stable once the WSDL has been issued. This has come into practice because just like native APIs, customers started to build code against buggy behavior, and when the server logic was updated to fix bugs, their code broke. Now if there’s broken code it stays in place, and developers migrate to new services at regular and scheduled intervals.

Recently del.icio.us updated their post API to use a secure encrypted URL, so as not to betray passwords or bookmarks in cleartext if they were posted using the API. A good move, especially as developers are using GET requests to post bookmarks, which may be prompting some routers to cache sensitive user data. Yahoo was nice enough to provide clear documentation and plenty of warning about the change though. After a few months of warning, the old insecure URL was turned off, and legacy requests are redirected to the secure URL, all in all a very good update.

On the other hand, del.icio.us recently updated their rss feed of recent bookmarks: http://del.icio.us/rss to be a bit more ‘digg-esque’. Instead of showing the steady stream of users adding bookmarks to their accounts, it now aggregates the popular urls, showing you something that is currently being bookmarked quite a bit. Well guess what? LiveMarks uses those front page rss feeds to aggregate del.icio.us bookmarking activity. The RSS feed change, which was completely unannounced, significantly impacted that application, and I wish they had given some warning that the URL was going to change, instead of silently changing it. (If you are hosting a mirror for LiveMarks and I haven’t contacted you, please change the aggregation url to http://del.icio.us/rss/recent instead of http://del.icio.us/rss/).

As a sidenote, a personal annoyance is the reluctance of service providers to provide APIs against what they consider to be their most important property: public user data.

Even though many times the data is made public in various forms, such as through RSS feeds or HTML pages, data like my favorites on Flickr or my older bookmarks on del.icio.us require authorization to access, which means as a developer the interface and code needs to be more complicated to use these APIs. The new Google Ajax Search API for example requires a separate API key I must apply for, I can’t use the Google search API key I use normally.

If I want to build an application for del.icio.us for example to offer a cool visualization of all your bookmarks, and you in context of other people who are close to your bookmarking activities, it’s essentially impossible without everyone volunteering their username and password in the clear to me, data about your bookmarks in del.icio.us is behind a firewall unless you sit on the RSS feed and store the aggregation. It’s the same with last.fm, who don’t even offer an API to recover your listening data (which is why I built a last.fm proxy). It’s up to del.icio.us or the service provider as to what they want to offer, but from the developer perspective, a lot of gratuitous authorization and api keys are essentially just another barrier to building the application I am interested in building.

Encrypted Tagging

Mon, 07 Aug 2006 18:04:13 -0700

They said it couldn’t be done! OK – I said it would be a pain in my last post on releasing boz: encrypted web bookmarks.

To me, encrypted tagging seemed like a complicated problem, how can you keep track of tags if the database can’t be trusted to know what the tags are? If you encrypt the same tag word twice it results in different words, so you can’t search the encrypted text for a tag.

Well that actually seemed like fun to work around, so this weekend I rolled out an implementation of tagging into boz that allows for encrypted tags.

This is actually the third release for boz, the second being the ‘stars’ feature. I like the Gmail concept of starring something to make sure you don’t forget it, and I have missed this feature from del.icio.us. There are a variety of important internal addresses to SourceLabs and to SWiK that I’d rather not put on my del.icio.us bookmarks.

The stars feature lets me keep a short list of bookmarks that I hit more frequently, or ones that I need to do something with in the near future.

Tags is just a feature I’ve copied from del.icio.us, and really it’s essential for a bookmarks manager, and a much better model than folders.

The way that it’s implemented to keep things secure is that the client receives a tag dictionary from the server of all the tags that have been used in the past. When new bookmarks are posted, the client consults the dictionary to see if they should flag it with an existing tag from the dictionary, if not, they post an encrypted request to create a new tag.

Now that I’ve started playing around with encryption, I am starting to think it’s a good idea for a lot more applications. Maybe ‘private’ isn’t the new ‘social’, but in migrating to pure web apps off of desktop apps, encrypting content does away with one of the major reservations in leaving the desktop – giving your security over to some third party server.

Private Bookmarks Web Service

Wed, 02 Aug 2006 15:18:35 -0700

It seems to be a trend: if you’re using web applications, forget about privacy. Forget about owning your own data.

A while ago, I had an issue with this aspect of del.icio.us. Once upon a time, there was no way to save private bookmarks: it’s still pretty obtuse. Well, I hacked up something to get around the limitation and create bookmarks anonymously or semi-privately if not privately.

It was just meant to be something to bookmark for myself and maybe something to blog about, but people took to it, and over 11k private bookmarks were saved using it in the space of 8 months. However a short time ago, Yahoo/del.icio.us blocked the service: not only could I not post new private bookmarks for myself, all the ones I and everyone else had saved were wiped out.

Well ok, lesson learned. But I still want to bookmark privately, and I don’t like the way del.icio.us does it: public and private bookmarks are not chocolate and peanut butter, they should be separate. And private bookmarks should be really private, I don’t even want to trust the servers with them.

So I’ve coded up an open source solution: a web service that lets you post bookmarks that even the server doesn’t know about.

How it works is that just before you post a bookmark, your browser encrypts the data and sends the bookmark information encrypted with your private key to the server. To browse your bookmarks, the server sends them back encrypted and your browser then decrypts them.

A special bookmarklet can be used on remote web pages to post, or you can post directly via the interface.

Because of the heavy use of browser encryption and decryption, the entire application is written in JavaScript/Ajax.

Also, if you have used my previous private bookmarks solution, please email me at alex.bosworth+projects at gmail – I’ve set aside your username and I’ll import your bookmarks that were blanked into the new service.

For those interested, I’ll go into some details on how the application was developed:

Developing an encrypted bookmarks web application

I’ve been working on this project in my spare time since a few months ago when Yahoo/del.icio.us gave me the final word that priv.at was blocked for good. There are several challenges to an encrypted bookmarks service that needed to be overcome:

Paucity of quality open source web browser encryption and decryption libraries
Developing a bookmarklet that doesn’t betray the url to the server
Storing an encryption key on the browser beyond a single page load, without persisting it to the server
Ensuring that encrypted data doesn’t become corrupted
Keeping a javascript application fast
Dealing with pages with content that all has to be decrypted
Search and tagging without the database knowing what it’s searching for

The first problem of finding a suitable library was just a matter of going through various libraries, looking at the code and running them through unit tests. Most javascript encryption libraries however were designed as proof of concepts, or coded in very ugly ways, or not for any kind of performance and are therefore very difficult to adapt. This just took methodical testing to find one I liked. I then extended the string object with a .encrypt and a .decrypt method, this allows for encryption to be a simple component of the application.

The problem of developing a bookmarklet is that traditionally you encode the url in a get parameter, and then the server echoes what you asked it to ‘get’ when you hit the post screen. But that implies that the server knows what urls you are interested in. I wanted to avoid that, so my bookmarklet uses the only part of a url that is not passed to the server: the hash.

Storing an encryption key on the browser was another issue that I didn’t really anticipate. It is quite annoying to have to type in your encryption key every time you want to see your bookmarks or post a new one, you get used to being just logged in and having the server remember that you authenticated and it can send you privileged information. But that doesn’t work in this case, you must never tell the server what your private key is, but somehow have JavaScript remember it from page load to page load, which is not something that JavaScript seems to have been designed for. Luckily, the dojo toolkit provides a JavaScript to Flash bridge that allows for permanent storage on the browser, something normally of limited use, but perfect for my purposes.

This was my first practical use of the dojo javascript toolkit, and I have had a mixed experience. I have found on the one hand it’s fairly elegant as an API, provides the functionality I need, and is generally very powerful. On the other hand, it doesn’t always work quite like it should and it creates problems for Safari and Opera: I haven’t even tried IE yet. I decided the tradeoff of having to type your key in over and over was worth losing the minority browsers temporarily, and I’ll look at fixing that at a later stage.

Another problem that I ran into during development of the project was the fact that strange corruptions were taking place in some posts of bookmarks. I would post a bookmark, and it would sometimes return from the server garbled. I could post something 5 times in a row, and 4 could return fine and the fifth would be corrupt. This made it one of the more frustrating issues to pin down. One issue that was obvious is that I had forgotten that encrypting the strings would make them too large for MySQL’s maximum varchar space of 255 characters, which is usually ok for a title and a url.

Another issue is that the encryption library I use doesn’t encode to hex, so it makes data transmission and application design a little more complicated. My normal style of writing JavaScript is to keep everything in the document. Building web applications, you might have various stages of data representation: a database schema, an object schema, a javascript object schema, and finally a document schema. A bookmark is one thing in the database, another as a server object, another as a javascript object, and another as an html node. Because of this, my practice is to generally avoid JavaScript variables and store everything right in the html. I also try to avoid generating html in JavaScript, I prefer to keep things simple and leave all the html generation to PHP.

Except that I discovered that storing the encrypted bookmarks in the document would corrupt them. This meant that PHP had to become a generator of JSON instead of HTML, and JavaScript would then take over the job of generating the pages. The data transmission issue was solved by tracking down the appropriate escape functions in JavaScript and storing the bookmarks in the database escaped for JavaScript.

Finally, all this JavaScript made the application slow, encryption is a processor intensive business and Firefox’s JavaScript engine is sluggish at best, so I’ve limited the number of bookmarks on a page to 15 and tuned the JavaScript to avoid excessive DOM manipulation, which is the biggest CPU killer out there.

Oh yes, lest I forget: tagging. I decided to leave that out for the time being. The server can’t search for a tag, because the encrypted text is different even for the same word encrypted with the same key. Even the same word encrypted twice in a row is not the same. This of course means that I can’t prevent users from posting the same bookmark twice.

If I wanted to implement tagging/search, I would need to either use a different type of encryption that gave back the same result for the same input, or I would need to burn CPU on maintaining a dictionary on the browser side. The original priv.at del.icio.us bookmarks didn’t have tagging, I can still look through dates and page quickly through my bookmarks, so I have left that feature out as being too CPU intensive for a first pass at creating a quick bookmarking application.

Of course I have also published the project source as GPL v2 for those interested.