9 on SWiK

Venti and Linux

Thu, 28 Aug 2008 18:11:08 -0700

I've been playing around with the plan9port's version of Venti in a bit more depth lately and thinking about Linux-specific optimization opportunities. I've got more thoughts than I'm prepared to write down right now, but I figured I should start somewhere.

First, there seem to be a number of shortcomings for large systems. We've actually just installed a new 36TB storage system here at work with 3 file server nodes connected to it with 64GB, 64GB, and 12GB of DRAM respectively. One of the first things I ran into is Venti seems to behave poorly if you try to give it too big of a mem, icmem or bcmem (i was going for 4GB icmem and 2GB bcmem and mem) -- I'm fairly certain something is failing silently, but haven't had a chance to track it down. Another potential shortcoming on these 16-way, 16-way, and 4-way systems is that venti appears to be single threaded at its core so most of the processing power of these systems is idle while performance suffers.

Most of my current experimentation revolves around looking at using venti to backup and back block devices. vbackup does a reasonable job of this, but the fact that it scans the entire volume is a little problematic (my 80 gb home directory takes 45 minutes to go through even if I haven't modified anything). I'm currently looking at using lvm2/device-mapper to take snapshorts to make backups more coherent and to track changes since the last venti so I only have to have venti operate on changed blocks. This should allow much tighter granularity on dumps than nightly.

Of course, there is little need to keep 5-minute, 15 minute, or even hour level granularity snapshorts forever. I've started to think through a hierarchical venti approach which used a local 'transient' venti which would use a recycled arena for the transient snapshots and then using venti/copy to send coarser granularity snapshots to the next level of the hierarchy (which could be another cache server or be the central venti server).

Besides using device mapper for helping to take snapshots -- it could also be used to create caches and or cow devices on top of a venti score -- assuming I can get something to look like a disk or a disk image that actually serves a vbackup score as a disk.

More later...there's a bunch of stuff I need to play with to understand how the various pieces I'm thinking of interact.

Plan 9 /g/rid information

Wed, 13 Aug 2008 15:05:09 -0700

Plan 9 Authentication in Linux paper available

Sun, 10 Aug 2008 11:12:21 -0700

Ashwin's paper on implementing Plan 9 authentication and capability device for Linux is now available for free from the ACM Archives along with the rest of the special issue on the Linux kernel that I helped co-edit:

http://portal.acm.org/toc.cfm?id=1400097

User:goundoulf

Wed, 12 Sep 2007 05:42:25 -0700

Member of the OpenBox4 project (alternative firmware for the neuf box)

Sand 9 Gets $8M for Nano MEMs

Mon, 14 Jul 2008 11:47:49 -0700

Sand 9, a Boston University spinoff, has received $8 million in a Series A round from Flybridge Capital Partners and General Catalyst Partners. An early-stage investment in a fabless chip company is notable, simply because there are fewer of them than ever. While the firm isn’t making a true semiconductor, it is using the chip manufacturing process to make its nano-mechanical resonator, a component that can both filter and stabilize multiple frequencies on one piece of tiny machinery.

The Sand 9 resonator can replace several components used in making a wireless devices from a cell phone to a headset. Rather than requiring a separate device for each radio found in a communications device, each Sand 9 component can work with multiple frequencies and radios. That means fewer parts and a smaller potential form factor. It sounds like a smart bet, but the real test will be if it works.

Matt Crowley, VP of business development with Sand 9, was cagey about when the firm would have devices to sample, but did say the company should be able to get to product on the $8 million round and the $2 million seed round it raised when it formed in 2007. He added that the Series A round should last through the next two years, and will primarily be used to add engineering staff and get the product out the door.

That’s serious capital efficiency for a fabless company. Crowley said the key is using older semiconductor fabrication tools and plants to build the resonator. So while it’s hard to find a Series A for a processor startup these days, entrepreneurs turning to MEMs or older process technologies are still raking in investment dollars.

installing broadcom on fedora core 9

Thu, 26 Jun 2008 21:55:57 -0700

Service Oriented File Systems

Wed, 16 Apr 2008 12:23:14 -0700

Its fun to abstract and think about web2.0 concepts in terms of Plan 9 concepts. Abstract away the gorp that makes up their implementations and think about some of the fundamental concepts. When interfaces are files, mashups are binds.

The larger picture here is about keeping this simple, language independent, and client independent. I'd like an infrastructure that I could build simple web services out of that can be viewed with a browser, or with a rich-client (like acme plug-ins). I'd like the front-end of these services to be defined in the same way the Octopus approach defines normal GUI's (except from a web-browser, the widgets are implemented by Java script). I'd like the back-end of these to be akin to big table and the chubby lock service, with a back-back-end that is simply Venti versus a database or XML.

I'll admit I'm probably naive about all of this, not having really worked on client-facing applications for some time, but I think it would be a fun thought exercise to go through and perhaps a nice refresher to actually try and implement. While part of the goal is to keep things language independent, I'll probably do things under Inferno to keep them portable for me and open up the potential for interaction with the Octopus crowd.

Of course, this all falls under the copious spare time which I have a definite deficit of -- however, I think dedicating one day a week is doable and I should be able to combine this with my constant desire to get back and look at collaborative facilities (aka warren) under Plan 9 and Inferno.

Blue Gene Project Pages

Fri, 22 Jun 2007 12:08:15 -0700

I've setup an more official website to put publications and presentations relating to the DOE sponsored Plan 9 on Blue Gene work. You can get to it here and I'll be updating it over the next few days.

Torus and tree networks working on Plan 9 BG/L port

Mon, 18 Jun 2007 12:55:01 -0700

We finally worked out the remaining infrastructure issues and are now able to cpu(1) to both I/O nodes (over the Ethernet) and cpu nodes (over the tree network). The cpu nodes are also able to talk to each other over the torus network.

We will be attempting a live demo during the USENIX poster session, so drop by and play with Plan 9 running on a Blue Gene.

Service Oriented Synthetic File Systems

Fri, 30 Nov 2007 10:34:17 -0800

I was doing some thinking about how larger synthetic file systems are structured and implemented under Plan 9 (or using 9P in general).

My toy-use-case for thinking about it was developing a synthetic-file-server based service similar to SourceForge -- where you have many different projects, each with different component services (bug-tracking, version-control, blogs, wiki, forums, etc.)

You could do it by running lots of different servers (per project) and binding them into a name space and then exporting, but that seems kinda awful (particularly if you have dozens of projects you are using the same tool to track). Much better to have a single server running which is able to accommodate multiple projects, each with multiple file services underneath.

Obsession with qid-spaces ends up being distracting, with the way fids work, we shouldn't need to segment the space so rigidly, it seems to me that qids only really need to be unique per directory, outside of that you can track which "module" of the hierarchy you are in by saving some state in the server-side fid tracking structure as you traverse the hierarchy.

dot-dot makes things a little more complicated, but the solution used inside the kernel can be used within file servers as well (namely, keep the whole path in the service-side fid tracking structure).

What's really nice is you can develop plug in file server modules which allow you to compose much more complex single-server-services out of many different file system components -- without worrying about how the qid space is split up and managed.

GSoC 2008 Ideas

Tue, 26 Feb 2008 06:47:50 -0800

My Plan 9/Inferno summer of code ideas (otherwise known as projects I wish I had more time for) - projects should take no longer than 5 weeks to complete (including ramp up, debug, and packaging) -- conservatively projects should be completable in 3 weeks by experienced P9/Inferno developers and 2 weeks by folks unfamiliar with p9/inferno development environments.

mount.9P helper program for v9fs including packaging for debian and fedora -

hueristic for determining transport type

DNS support for resolving hostname <->ip address
ssh-based tunneling support (will require server work as well)
man pages
debian packaging
rpm packaging
(stretch) authentication support (w/p9p)
(stretch) authentication support (w/o p9p)
(stretch) integration with idmap solution

since this is relatively straightforward it will really need to be "super" version including support for ssh tunneling, authentication, etc.
idmap solution for v9fs - maps local uids/gids/error-codes to strings

userspace daemon to provide mapping
hooks in v9fs to use mapping
(stretch) synthetic file server approach to update mappings

OLPC Inferno Environment

fontfs - on-demand ttf solution
metafs - abstract metadata from underlying file system
(stretch) Integrate OLPC translation solution for Inferno

(stretch) OLPC oriented GUI toolkit, window manager, and toolbar
(stretch) Inferno approach to collaboration using file systems

Source Control Management based on Venti backend

single branch version tracking with associated log file

add new files/directories
commit changes
checkout specific version #

(stretch) support multiple branches
(stretch) support repository sync (push/pull)
(stretch) support three-way merge

wrapper for p9p vbackup to make it more user friendly

wrapper which tracks venti scores based on volume being backed up
GUI admin tool

which sets up venti in partition(s) or with files

which assists in configuring backup intervals and volumes

(stretch) time-traveler like GUI for navigating/searching backups
(stretch) support for pruning and/or merging snapshots

GSoCFS for managing future community involvement with GSoC

synethtic file system and web interface
posting project ideas
voting for projects
registering student interest in projects
project milestone tracking
project blogs and wikis
post-summer project success metrics (subjective and objective)
(stretch) syndication points for community monitoring
(stretch) potential integration with SCM
(stretch) potential integration with some form of chat
(stretch) potential integration with name space sharing

(more to come...)

Installing The Native Linux Flash Player 9 On Ubuntu | HowtoForge - Linux Howtos and Tutorials

Mon, 11 Feb 2008 03:36:25 -0800

Chapter 9 - Improving XML Performance

Fri, 01 Feb 2008 11:13:38 -0800

Dovecot 0.99.14 IMAP Server on RedHat Linux 9

Fri, 01 Feb 2008 06:27:12 -0800

SWF9 runtime target, coming together

Thu, 10 Jan 2008 13:27:01 -0800

We’ve got the basic core of OpenLaszlo compiling and running in SWF9. Thanks go to the good folks at g.ho.st (Global Hosted Operating System) for their support in this development. Here is a simple Laszlo app compiled to swf9 format and running in Flash 9. If you don’t have a Flash 9 player installed, you won’t be able to see it.

Simple SWF9 Demo

This is the source code for the app above:

 <canvas width=\"800\" height=\"600\"> 
	
  <view id=\"bar\" x=\"200\" y=\"200\" >
    </view><view id=\"foo\" bgcolor=\"0xcccccc\" x=\"-100\" y=\"-100\"  height=\"200\" width=\"200\"
          onclick=\"this.parent.animate('rotation', 90, 1000, true)\" >
	
    <text fontsize=\"18\" fontstyle=\"italic\">This is some text in a text view</text>
	
    <view bgcolor=\"blue\" width=\"40\" height=\"40\"
          x=\"59\" y=\"59\"
          onclick=\"this.animate('x', 10, 1000);
                   this.animate('y', 10, 1000)\"/>
	
    <view bgcolor=\"red\" width=\"40\" height=\"40\"
          x=\"101\" y=\"59\"
          onclick=\"this.animate('x', 150, 1000);
                   this.animate('y', 10, 1000)\"/>
	
    <view bgcolor=\"green\" width=\"40\" height=\"40\"
          x=\"59\" y=\"101\"
          onclick=\"this.animate('x', 10, 1000);
                   this.animate('y', 150, 1000)\"/>
	
    <view bgcolor=\"yellow\" width=\"40\" height=\"40\"
          x=\"101\" y=\"101\"
          onclick=\"this.animate('x', 150, 1000);
                   this.animate('y', 150, 1000)\"/>
  </view>
	
</canvas>

A number of things have to be working to support this application:

Kernel Sprite implementation
LzEvent and LzDelegates
LzNode is able to instantiate nodes with children, init methods, and setters
LzView can create and manipulate kernel Sprites
Idle events can be regsitered, so that animators can operate
Kernel mouse events are forwarded to the LFC and dispatched to registered listeners
Integration between the Laszlo compiler and the Flex mxmlc compiler

All of these are implemented, although the kernel sprite support needs to be fleshed out and optimized.

Things major that still need to be brought up in swf9 are

constraints: we’re not allowed to store the dependency functions on function in swf9, so we have to make the compiler put them someplace else
XML data loading and parsing
node replicators
media playback
drawview
keyboard handling
text and inputtext views
browser/player communication
debugger (although the Flash fdb debugger can be used for now)
Selection manager, font manager
embedded assets,fonts

We’re got the development going on in a branch named “devildog”, and in about a week or so should have things in place to allow people to start helping out if they want to get certain modules or features completed sooner, or fix bugs or optimize for the Flash 9 platform. There are a lot of new and improved APIs provided by the Flash 9 runtime; better media loading, data loading, and network APIs, as well as much more rational imaging and event model. We can probably take a lot of advantage of these by updating and optimzing the swf9 kernel and the runtime to use this where possible.

An quick overview of the approach to compiling to swf9 is outlined below.

The Flash 9 runtime contains a new virtual machine which has efficient support for JS2 style classes. If type declarations for variables and methods are provided, and use of some dynamic Javascript features is avoided, the application can run faster.

The Laszlo compiler emits AS3-compliant javascript class files, which are compiled by the Flex AS3 compiler, to produce an executable swf9 binary application file.

We currently compile the LFC library as a separate .swc AS3 library file, which is linked to the user application when the application is compiled.

Our plan is to have the LZX tag compiler phase emit real ‘native’ JS class declarations for user-defined classes (and all views, in fact). The LFC is already defined as JS2 style classes, using our own Class.lzs class system, designed by Tucker. We are converting these to be actual JS2 classes, which means no longer using our class initializer protocol. Stuff that is in now in class and instance initialize methods must be coded in some other manner.

Classes which are declared ‘dynamic’ in Flash 9 are slower to execute, since they must look up methods are variables by name at runtime, so we have been avoiding declaring LFC classes this way unless absolutely necessary, and would like to go back again and make another pass when things are all working to see which classes can be optimized to use only static lookups. We probably need to leave LzNode dynamic, given that we allow setAttribute at runtime on arbitrary properties, but there is a lot of room for optimization in the support classes in the system.

Finally! Bring native XML data into your relational database using DB2 Express-C

Thu, 25 Oct 2007 11:06:03 -0700

More than 100 Web 2.0 Online Generators

Mon, 08 Oct 2007 10:27:25 -0700

John Resig - RSS to JSON Convertor

Sun, 16 Sep 2007 14:58:05 -0700

Perl

Search Radar Adds Suggestions to Search Results

Wed, 29 Aug 2007 08:09:50 -0700

The 40 coolest free applications around - Seopher.com

Mon, 27 Aug 2007 09:06:00 -0700

Web 3.0 and the rise of Hybrid-Browsers | Frostfirebuzz.com

Mon, 20 Aug 2007 18:39:37 -0700

liquuid's post :: Netscape 9 beta 1

Thu, 09 Aug 2007 07:24:27 -0700

Celebs, Movies, Music / Published

Wed, 01 Aug 2007 02:27:06 -0700

Plan 9 from User Space

Tue, 19 Jun 2007 16:18:52 -0700

If you don't know what it is, it doesn't matter. If you know what it is, it's awesome.

Nguyen Do Dang Khoa

Fri, 04 May 2007 02:47:23 -0700

Trac nghiem, bai tap Hoa hoc 9

CPU working on Blue Gene

Fri, 20 Apr 2007 18:58:27 -0700

It took quite a bit of fussing about to gateway between the IBM internal Plan 9 cluster and the Blue Gene VLAN, but with the help of Forsyth's new Ethernet driver for BG/l, Inferno as an intermediary on the front-end node, and a few other bits and pieces... we were able to cpu(1) into Blue Gene I/O nodes. The CRN-Tree network is pushing packets back and forth and the Torus is being debugged. Overall its been a very productive 80-hour week.

aNieto2K | Phatfusion, recopilación de utilidades MooTools

Tue, 17 Apr 2007 01:49:51 -0700

Slimbox, the ultimate lightweight Lightbox clone · digitalia.be

Tue, 17 Apr 2007 01:49:51 -0700

More Plan 9 on Blue Gene

Thu, 12 Apr 2007 08:50:30 -0700

For folks who can't read the memory dump in my last post I've got a rdbfs(4) like interface now that gives me access to the BG/l machine state and memory and includes a more readable console interface:

criswell% con -r /usr/ericvh/bgd/0/con0

Plan 9 bgl
cpu0: 0x5202

BG/l Personality
Block: R000-N60_32_4
Memory Size: 536870912 bytes
clockHz: 700000000
Torus Addr: -1 -1 -1
Tree Hops to Top: 255
EMAC h/w Address: 0:d:60:e9:10:9f
Assigned IP Addr: ac186439

Kernel Status Cpus 0 & 1: 3 3
512M memory: 160M kernel data, 352M user, 977M swap
boot...
--rw-rw-r-- c 0 24 Apr 11 2007 /dev/bintime
--rw-rw---- c 0 0 Apr 11 2007 /dev/cons
---w--w---- c 0 0 Apr 11 2007 /dev/consctl
--r--r--r-- c 0 72 Apr 11 2007 /dev/cputime
--r--r--r-- c 0 0 Apr 11 2007 /dev/drivers
--rw-rw-r-- c 0 48 Apr 11 2007 /dev/hostdomain
--rw-rw-r-- c 0 0 Apr 11 2007 /dev/hostowner
--r--r----- c 0 0 Apr 11 2007 /dev/kmesg
-lr--r----- c 0 0 Apr 11 2007 /dev/kprint
--rw-rw-rw- c 0 0 Apr 11 2007 /dev/null
--r--r--r-- c 0 0 Apr 11 2007 /dev/osversion
--r--r--r-- c 0 12 Apr 11 2007 /dev/pgrpid
--r--r--r-- c 0 12 Apr 11 2007 /dev/pid
--r--r--r-- c 0 12 Apr 11 2007 /dev/ppid
--r--r--r-- c 0 0 Apr 11 2007 /dev/random
--rw-rw-r-- c 0 0 Apr 11 2007 /dev/reboot
--rw-rw-r-- c 0 0 Apr 11 2007 /dev/swap
--rw-rw-r-- c 0 0 Apr 11 2007 /dev/sysname
--rw-rw-rw- c 0 0 Apr 11 2007 /dev/sysstat
--rw-rw-r-- c 0 78 Apr 11 2007 /dev/time
--rw-rw-rw- c 0 0 Apr 11 2007 /dev/user
--r--r--r-- c 0 0 Apr 11 2007 /dev/zero
/boot/bind
/boot/boot
/boot/echo
/boot/ls
/boot/ps
/boot/rc
/boot/rcmain
/boot/sleep
Hello Squidboy
sleep 10
hi hi hi hi hihihihihihihihihihihhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh
%

(Picture photoshopping credit: Andrey)

Virtual Systems

Mon, 19 Mar 2007 12:42:15 -0700

Bit the bullet and got Plan 9 running under Xen on my "victim" laptop. It can now run Plan9 natively, under Xen, or under Qemu. I still am toying with the idea of a bridged network mode where I can mount an Inferno devip from a loopback cable to my windows laptop in case I can't find hard-lines to plug into at Watson. This also means my victim laptop is squared away to be a testbed for x86/Libra -- I might even get to a basic "shot-in-the-dark" (ie. no Xen devices, just devshm) configuration sometime during the trip.