bigtable on SWiK

the-cassandra-project - Google Code

Tue, 22 Jul 2008 21:51:13 -0700

High Scalability

Tue, 22 Jul 2008 06:52:00 -0700

hypertable - Google Code

Sat, 19 Jul 2008 13:46:02 -0700

Hypertable: An Open Source, High Performance, Scalable Database

Sat, 19 Jul 2008 13:46:02 -0700

the-cassandra-project - Google Code

Tue, 15 Jul 2008 23:50:33 -0700

Facebook's BigTable.

Dare Obasanjo aka Carnage4Life - Project Cassandra: Facebook's Open Source Alternative to Google BigTable

Tue, 15 Jul 2008 17:50:28 -0700

Powerset Blog : First Release of Hbase in Hadoop

Mon, 07 Jul 2008 20:52:43 -0700

Hypertable: An Open Source, High Performance, Scalable Database

Sun, 06 Jul 2008 20:48:17 -0700

Hypertable: An Open Source, High Performance, Scalable Database

Sun, 06 Jul 2008 06:48:57 -0700

the-cassandra-project - Google Code

Thu, 03 Jul 2008 17:47:47 -0700

the-cassandra-project - Google Code

Thu, 03 Jul 2008 16:47:51 -0700

Cassandra is a distributed storage system for managing structured data while providing reliability at a massive scale.

[from bushwald] Is Google's BigTable too private?

Fri, 27 Jun 2008 15:55:24 -0700

"What if Facebook started on Google App Engine? They would be stuck on Google."

Hypertable: An Open Source, High Performance, Scalable Database

Wed, 25 Jun 2008 11:56:29 -0700

Danga home - DJabberd - Gearman - memcached - MogileFS - Perlbal

Wed, 18 Jun 2008 05:22:24 -0700

Perlbal - our Perl-based reverse proxy load balancer and web server.

Wed, 18 Jun 2008 05:22:23 -0700

MogileFS - our open source distributed filesystem. Distributed (meta) file system. Spray files across cheap disks on your network. Pay less for storage. No proprietary on-disk file formats.

Wed, 18 Jun 2008 05:22:23 -0700

Gearman - Low-latency remote function execution. Run code remotely, in parallel, load balanced, in other languages, etc.

Wed, 18 Jun 2008 05:22:21 -0700

How Todd Hoff learned to stop worrying and use lots of disk space to scale

Wed, 21 May 2008 08:15:14 -0700

Todd Hoff, who apparently learned a hell of a lot during a short stint at Yahoo followed by some startups has an extremely well-written and edutaining article about how scaling to a million or more users requires jettisoning more or less everything we know and love about relational modeling.

Even though he uses bigtable (Google’s distributed hash storage system) as his example, in reality this approach works well with relational datastores like MySQL and Oracle too, you just have to think about your data differently and use the databases differently. So I’m including this article in the MySQL and Oracle categories because I think it would be of interest.

Here’s a taste of how it reads:

How do you structure your database using a distributed hash table like BigTable? The answer isn’t what you might expect. If you were thinking of translating relational models directly to BigTable then think again. The best way to implement joins with BigTable is: don’t. You–pause for dramatic effect–duplicate data instead of normalize it. *shudder*

Flickr anticipated this design in their architecture when they chose to duplicate comments in both the commentor and the commentee user shards rather than create a separate comment relation. I don’t know how that decision was made, but it must have gone against every fiber in their relational bones…

But Flickr?s reasoning was genius. To scale you need to partition. User data must spread across the shards. So where do comments belong in a scalable architecture?

The answer is, in case you aren’t following yet, you store it everywhere you might need it and worry about keeping your multiple copies in sync later, if at all.

BigTable data ethics are more Mardi Gras than dinner with the in-laws. Data just wants to have fun. BigTable won?t stop you from hurting yourself. And to get the best results you may have to engage in some conventionally risky behaviors. But if those are the glass bead necklaces you have to give for a peak at scalability, why not take a walk on the wild side?

So anyway, this is awesome stuff and thanks Todd. For your reading and learning enjoyment: Todd Hoff’s “How I learned to stop worrying and use lots of disk space to scale”.

[from yurinotigor] How I Learned to Stop Worrying and Love Using a Lot of Disk Space to Scale | High Scalability

Wed, 21 May 2008 06:18:45 -0700

Powerset Blog First Release of Hbase in Hadoop

Fri, 16 May 2008 08:18:57 -0700

Powerset developed Hbase, I had no idea.

HowHypertableWorks - hypertable - Google Code

Mon, 21 Apr 2008 21:23:07 -0700

Top 10 Reasons to Avoid the SimpleDB Hype — ryanpark.org

Mon, 21 Apr 2008 21:23:06 -0700

good (if contrarian) coverage of the tradeoffs

Hypertable looks really interesting from everything I’ve read but …

Sun, 20 Apr 2008 13:17:24 -0700

… I can’t get over this nagging question, “Why does Zvents need a distributed, sparse matrix, versioned datastore?” I don’t track the calendaring space as much as I used to, and sometimes innovation is self justifying, but I’d feel a lot better about the project if I knew that answer.

Rumor: Google To Launch BigTable As Web Service

Sat, 05 Apr 2008 19:49:07 -0700

Interesting rumor, and totally plausible. Just as Amazon has thought “why not make some money off of all of this great infrastructure we’ve built”, it looks like Google is going to do the same thing: Source: Google To Launch BigTable As Web Service

Google may be releasing BigTable, its internal database system, as a web service to compete with Amazon SimpleDB, according to a source with knowledge of the launch.

This will be one more non-traditional database among the many interesting choices that exist today.

Source: Google To Launch BigTable As Web Service

Fri, 04 Apr 2008 17:47:43 -0700

Google may be releasing BigTable, its internal database system, as a web service to compete with Amazon SimpleDB, according to a source with knowledge of the launch. There are also rumors that press is being pre-briefed on the product, although we haven’t been contacted by Google.

BigTable is a highly scalable database system used internally by Google to support over 60 of its products and projects. A source says Google has plans to announce next week that it will make BigTable available to outside developers as a service. Amazon provides a similar service through SimpleDB, a cloud database solution announced in December.

Google started development on BigTable in early 2004 and began using it actively in February 2005. The non-relational, proprietary system was designed internally to fulfill Google’s peculiar need for access to massive amounts of data at very high speeds (millions of read/writes per second). BigTable is based on the Google File System (GFS) and designed for distribution across thousands of commodity servers that collectively store petabytes of data. Services that rely on it include Google Search, Google Earth and Maps, Google Finance, Google Print, Orkut, YouTube, and Blogger.

The decision to open up BigTable would seem to mark Google’s challenge to Amazon Web Services (AWS) suite, which also includes the Elastic Compute Cloud (EC2) for cloud processing power and Simple Storage Service (S3) for cloud storage. The Amazon triumvirate of SimpleDB, S3, and EC2 is meant solve the scalability needs of web developers with a utility-like model. Customers pay for just the storage, computations, and bandwidth they need, and none they don’t. While Google has yet to announce the pricing for BigTable, we presume it will share the same model as AWS.

If Google does indeed announce public access to BigTable next week, expect the company to follow up with cloud storage and processing solutions as well, since there are substantial synergies between the three.

For more information about BigTable, see a paper (PDF) that was written about it in 2006. You can also watch a talk about it given at the University of Washington in October 2005.

Crunch Network: MobileCrunch Mobile Gadgets and Applications, Delivered Daily.

Zvents releases open-source cluster database - LinuxWorld

Mon, 11 Feb 2008 06:43:44 -0800

Event search firm Zvents is releasing a massively parallel database server, based on a published Google design, as an open source project. The new software, Hypertable, is designed to scale to 1000 nodes,

Random Write Performance in Solid-State Drives

Sat, 02 Feb 2008 19:42:06 -0800

hypertable - Google Code

Tue, 22 Jan 2008 08:30:33 -0800

Open source Bigtable clone 'Hypertable' posts performance numbers (Skrentablog)

Thu, 17 Jan 2008 03:36:43 -0800

Hbase/HbaseArchitecture - Lucene-hadoop Wiki

Mon, 31 Dec 2007 11:01:14 -0800

hbase