v9fs on SWiK

Plan 9 Authentication in Linux paper available

Sun, 10 Aug 2008 11:12:21 -0700

Ashwin's paper on implementing Plan 9 authentication and capability device for Linux is now available for free from the ACM Archives along with the rest of the special issue on the Linux kernel that I helped co-edit:

http://portal.acm.org/toc.cfm?id=1400097

v9fs - SWiK

Sat, 09 Aug 2008 22:07:30 -0700

v9fs - SWiK

Sat, 09 Aug 2008 22:03:45 -0700

Slides from KVM Developer Forum on 9p/virtio

Thu, 19 Jun 2008 17:44:56 -0700

9p/virtio Slides Available

Thu, 19 Jun 2008 17:20:20 -0700

The slides I presented at KVM forum on using 9p over virtio to provide a paravirtualized file service are available on the KVM wiki

Intro to using 9P under Linux

Thu, 29 May 2008 16:44:23 -0700

[9fans] mount 9P on Linux and FreeBSD via FUSE

Thu, 29 May 2008 16:44:23 -0700

Linux Kernel Documentation :: filesystems : 9p.txt

Thu, 29 May 2008 16:44:23 -0700

Grave Robbers From Outer Space - a collection of articles about Plan 9, Inferno and the technology transfer of their technologies to other platforms such as Linux.

Thu, 29 May 2008 16:44:23 -0700

Intro to using 9P under Linux

Thu, 29 May 2008 16:44:22 -0700

Grave Robbers From Outer Space - a collection of articles about Plan 9, Inferno and the technology transfer of their technologies to other platforms such as Linux.

Sun, 18 May 2008 07:07:31 -0700

XCPU - clustering

Sun, 18 May 2008 07:07:31 -0700

Intro to using 9P under Linux

Sun, 18 May 2008 07:07:30 -0700

xcpu quick start

Fri, 25 Apr 2008 19:13:36 -0700

xcpu quick start for impatient people like me
(aka xcpu quickstart) (More information about xcpu is available from http://www.xcpu.org) Make Sure You Have The Pre-Reqs (on Ubuntu 8.04) sudo apt-get install libelfg0-dev sudo apt-get install libssl-dev Acquire the Source

svn co svn://xcpu.org/sxcpu/trunk sxcpu
cd sxcpu
make
sudo make install

To try things out on the same host (single system, no client server)  % su % ssh-keygen -t rsa -f admin_key % mkdir /etc/xcpu % cp admin_key* /etc/xcpu % xcpufs & % xgroupset 127.0.0.1 root 0 % xuserset 127.0.0.1 root 0 root /root/.ssh/id_rsa.pub % xrx 127.0.0.1 /bin/date  To setup a pre-existing cluster with xcpu  On the front end machine (as root):  % ssh-keygen -t rsa -f admin_key % mkdir /etc/xcpu % cp admin_key* /etc/xcpu (copy admin_key to each back end node - below example with ssh) % ssh root@each_back_end_server mkdir /etc/xcpu % scp admin_key root@each_back_end_server:/etc/xcpu  To setup just the compute node (back-end server)  % xcpufs &  Then back on front-end node:  % sudo xgroupset    % sudo xuserset     ~/.ssh/id_rsa.pub
% xrx  /bin/date

trans_fd_simple

Wed, 16 Apr 2008 13:21:41 -0700

Did some really basic performance checks on trans_fd_simple compared to the older code and comparing single lock with reader/writer lock protection on the socket. Turns out my thinking with reader/writer lock was flawed and there are race cases (hence only the single threaded result). Long and short of it appears to be sort of what we figured, that the simple approach does better on single threaded workload due to less overhead, but does much worse on multi-threaded due to added latency and what not. I ran against npfs as well just to check if earlier thinking about whether or not multi-threaded npfs added much -- it appears to add nothing but overhead in this case.

This particular case was from a qemu hosted linux talking to the simulation host server -- which isn't particularly good. I'm going to replicate this test on the blade regression cluster, but wanted to do a quick check before moving on to scatter/gather implementations.

Service Oriented File Systems

Wed, 16 Apr 2008 12:23:14 -0700

Its fun to abstract and think about web2.0 concepts in terms of Plan 9 concepts. Abstract away the gorp that makes up their implementations and think about some of the fundamental concepts. When interfaces are files, mashups are binds.

The larger picture here is about keeping this simple, language independent, and client independent. I'd like an infrastructure that I could build simple web services out of that can be viewed with a browser, or with a rich-client (like acme plug-ins). I'd like the front-end of these services to be defined in the same way the Octopus approach defines normal GUI's (except from a web-browser, the widgets are implemented by Java script). I'd like the back-end of these to be akin to big table and the chubby lock service, with a back-back-end that is simply Venti versus a database or XML.

I'll admit I'm probably naive about all of this, not having really worked on client-facing applications for some time, but I think it would be a fun thought exercise to go through and perhaps a nice refresher to actually try and implement. While part of the goal is to keep things language independent, I'll probably do things under Inferno to keep them portable for me and open up the potential for interaction with the Octopus crowd.

Of course, this all falls under the copious spare time which I have a definite deficit of -- however, I think dedicating one day a week is doable and I should be able to combine this with my constant desire to get back and look at collaborative facilities (aka warren) under Plan 9 and Inferno.

v9fs updates

Mon, 14 Apr 2008 07:53:57 -0700

With the 2.6.25 merge window rapidly approaching, it looks like I'm going to have to hold off on merging the reorganization until 2.6.26. At the moment, I'm still fiddling with the transport API and specifically the rewrite of the file descriptor based transport which is used for sockets, named pipes, etc. With a 2.6.26 target, I should be able to get the async and scatter-gather rewrites complete as well as merge the in-kernel server code from lucho.

As part of the exercise I've created a "simple" version of the code which is synchronous and single threaded (to match spfs). My motivation was to scale back and figure out what the core pieces were in the fd transport, as well as to have a useful performance baseline to measure the overhead (and advantages) of a more asynchronous model. A nice side-effect is that I also have a really simple and straightforward transport module which can be used as an example for those wanted to write their own transports. I'm not quite certain whether I want to merge the simple transport mainline though -- although I may merge it into trans_fd in order to maintain it as a baseline case -- it really amounts to a different request member function which could easily be selected via an option.

In parallel I've setup a few blades in order to measure performance, and perhaps more importantly performance regression. When complete, I hope to publish historical performance regression results back to kernel 2.6.14. At the moment I'm trying to do all collection of results and graphs inside Google docs, which should make an easy place to keep things up to date and simultaneously publish the data. I need to look around to see if anyone has hooks for the Google Data API so I can have my regression framework update the results automatically instead of me having to go in and manually enter the data.

GSoC 2008 Ideas

Tue, 26 Feb 2008 06:47:50 -0800

My Plan 9/Inferno summer of code ideas (otherwise known as projects I wish I had more time for) - projects should take no longer than 5 weeks to complete (including ramp up, debug, and packaging) -- conservatively projects should be completable in 3 weeks by experienced P9/Inferno developers and 2 weeks by folks unfamiliar with p9/inferno development environments.

mount.9P helper program for v9fs including packaging for debian and fedora -

hueristic for determining transport type

DNS support for resolving hostname <->ip address
ssh-based tunneling support (will require server work as well)
man pages
debian packaging
rpm packaging
(stretch) authentication support (w/p9p)
(stretch) authentication support (w/o p9p)
(stretch) integration with idmap solution

since this is relatively straightforward it will really need to be "super" version including support for ssh tunneling, authentication, etc.
idmap solution for v9fs - maps local uids/gids/error-codes to strings

userspace daemon to provide mapping
hooks in v9fs to use mapping
(stretch) synthetic file server approach to update mappings

OLPC Inferno Environment

fontfs - on-demand ttf solution
metafs - abstract metadata from underlying file system
(stretch) Integrate OLPC translation solution for Inferno

(stretch) OLPC oriented GUI toolkit, window manager, and toolbar
(stretch) Inferno approach to collaboration using file systems

Source Control Management based on Venti backend

single branch version tracking with associated log file

add new files/directories
commit changes
checkout specific version #

(stretch) support multiple branches
(stretch) support repository sync (push/pull)
(stretch) support three-way merge

wrapper for p9p vbackup to make it more user friendly

wrapper which tracks venti scores based on volume being backed up
GUI admin tool

which sets up venti in partition(s) or with files

which assists in configuring backup intervals and volumes

(stretch) time-traveler like GUI for navigating/searching backups
(stretch) support for pruning and/or merging snapshots

GSoCFS for managing future community involvement with GSoC

synethtic file system and web interface
posting project ideas
voting for projects
registering student interest in projects
project milestone tracking
project blogs and wikis
post-summer project success metrics (subjective and objective)
(stretch) syndication points for community monitoring
(stretch) potential integration with SCM
(stretch) potential integration with some form of chat
(stretch) potential integration with name space sharing

(more to come...)

Design and Roadmap Information

Wed, 13 Feb 2008 10:52:57 -0800

inferno-olpc circa 1/28/08

Thu, 31 Jan 2008 13:36:00 -0800

This week's update to inferno-olpc includes OLPC-optimized fonts and a customized logon with two extensions:

it binds the Inferno activity "writable" area to important areas of the Inferno name space which the user may want to write to (/usr, /services, /keydb, /tmp). This will be necessary for the next OLPC update which will start enforcing security (including only allowing writing to the activity writable area of the file system. Also bundled is a day-one scenario which will setup template directories to bind from and copying relevant files.
when logon can't find a user it will now prompt to create the user by copying the template /usr/inferno directory to /usr/username. This can probably be generalized back into the standard Inferno base.

I also have a "customized" toolbar which just limits the size of labels to 15 characters so you don't get the awful line wrap. This is a stopgap, a better toolbar metaphor needs to be developed for the OLPC version of Inferno. I also augmented wmsetup with a larger set of applications.

I had a larger agenda that included social-network style "profiles" and integrated security/password setup. But there were too many corner cases to test and "bullet-proof" so I decided to hold that off for later and maybe just incorporate those functions into separate applications.

I'm still a little uncertain of how I want to handle authentication and security. On one side, there could be a central "schoolserver" to setup authentication against. However, as most of the current users don't have such a server, having a pure peer-to-peer mode is probably preferable. Given that every laptop will be an auth-server then, do we setup a laptop wide authentication setup, or do we do a per user authentication setup. The later would seem to make more sense to me as the user-provided password wouldn't have to be stored anywhere and could be use to setup keyfs and the user's own authentication secrets. Other users "registering" with that user for the purpose of collaboration could then provide their own secrets in his keydb, etc. I'm not sure if that makes sense or if there is a better model I should be considering.

The file system security is another pain. Inferno users won't necessarily equal underlying system users (a common problem). This will be made worse with the new OLPC security enforcement which may have different uid's for each "instance" of Inferno (but the same "gid"). I think the right thing to do is a new version of devfs which always keeps group permissions open in the underlying fs and uses extended attributes or some other mechanism to maintain Inferno uid's and gid's for the files.

freefontfs - a styxserver inferno-lab

Thu, 31 Jan 2008 13:36:00 -0800

I've decided doing the dynamic free font file system would be a more gentle introduction to using the new Inferno styxservers modules than a full-blown filterfs like what we plan for metafs.

The most appropriate example would appear to be mntgen.b (which allows for dynamic walk based creation in /n). It uses both the styxservers facilities and the navigator and has a dynamic nature similar to what I want for freefontfs.

Fonts are comprised of two types of file, the base file (..font) and subfonts (..). We only want to generate (and cache) files as they are accessed. The base available 's will be derrived from crawling a directory for ttf files or perhaps from a configuration file.

Following mntgen, it looks like it would be much more simple to keep it as a single level hierarchy (much like /n) -- this may be possible as all the necessary information is encoded in the filename. We just need to parse the single walk correctly.

One question is whether we should prepopulate the cache with a base font size for every ttf available. This seems like a reasonable idea, but probably wouldn't even be necessary in many situations -- perhaps we can leave it as a config option.

metafs

Thu, 31 Jan 2008 13:35:59 -0800

Since I wrote about it in my other post, I might as well explain metafs. Basically in order to accommodate the OLPC file system security model I need to maintain uid, gid, and mode bits for Inferno's view of the underlying file system separately. This is actually a common need within hosted Inferno when you don't necessarily want to keep Inferno user's sync'd with system users. In any case, this amounts to an overlay file system, which stores away the additional metadata.

While I am tempted to keep a central database of the metadata, the most pragmatic solution would seem to either use extended attributes or a hidden file in each directory. Since extended attributes are somewhat distasteful and not available anywhere, I'm going to start with the per-directory hidden file.

In the future this same mechanism might be extendable to implement semantic file system extensions (or extended attributes for that matter) -- allowing for tagged organization and searching of the file system as well as alternative views such as a "journal" style temporal view (which might also be useful for backup purposes). However, these extended metadata scenarios may beg a re-evaluation of the distributed versus centralized storage model.

In any case, the basic plan is squirel away the extra meta data, indexed by the string filename/directory-name in a hidden file. If the directory isn't writable by eve then no extended metadata will be written and the directory will default to the underlying uname/gname/mode -- however, if eve isn't able to access the files, the overlay won't be able to access the data -- so it's likely a moot point.

Inferno Enabled

Wed, 23 Jan 2008 07:39:55 -0800

I've resurrected the Inferno Enabled sticker logo.

PNG Version:
SVG Versions are available in white and black.

Infernal OLPC repository

Mon, 21 Jan 2008 15:10:37 -0800

I've reset up my Inferno git repository on http://git.9grid.us/git based on a mirror of Forsyth's SVN repository. Included now is my working OLPC modifications repository which contains both my core-Inferno patches as well as new mk rules for packaging. I'm going to move on to better OLPC applications support now, but there probably won't be an update on that until after next weekend.

Infernal OLPC Packages Available

Mon, 14 Jan 2008 09:30:00 -0800

Finished working on the details on how to launch Inferno within the OLPC infrastructure and packaged an initial release which is now available via the OLPC wiki.

I need to take the sandbox I built it from and work out a merge into the mainline inferno-os repository. Basically this will amount to moving some bits into olpc specific directories and writing a distribution script which builds the bundle.

Next step after that will be to really go to work on customizing the environment for the OLPC, starting with a custom look and feel for the window manager and a few example applications. That will include supporting scaled true-type fonts and perhaps SVG support. I also need to work out some day-one scripts to help setup the right name space for the OLPC's restricted security model.

OLPC - Hosted Inferno

Mon, 31 Dec 2007 16:10:29 -0800

Got my daughter an OLPC for Christmas, but have spent a lot of time playing with it myself. First order of biz was to load Inferno on it ;) I can run it from the shell quite happily, but am now trying to come up with a Sugar launcher and package for wider deployment. I may start with a simple sugar app that toggles between loading Inferno versus loading sugar. This should be much easier as it'll avoid interacting with the Python gorp. Another alternative would be to just hack the terminal application (which would then act as emu's console), but I'd like to try to find a way to tie it in better.

(UPDATE: launcher is working, now working on packaging)

Service Oriented Synthetic File Systems

Fri, 30 Nov 2007 10:34:17 -0800

I was doing some thinking about how larger synthetic file systems are structured and implemented under Plan 9 (or using 9P in general).

My toy-use-case for thinking about it was developing a synthetic-file-server based service similar to SourceForge -- where you have many different projects, each with different component services (bug-tracking, version-control, blogs, wiki, forums, etc.)

You could do it by running lots of different servers (per project) and binding them into a name space and then exporting, but that seems kinda awful (particularly if you have dozens of projects you are using the same tool to track). Much better to have a single server running which is able to accommodate multiple projects, each with multiple file services underneath.

Obsession with qid-spaces ends up being distracting, with the way fids work, we shouldn't need to segment the space so rigidly, it seems to me that qids only really need to be unique per directory, outside of that you can track which "module" of the hierarchy you are in by saving some state in the server-side fid tracking structure as you traverse the hierarchy.

dot-dot makes things a little more complicated, but the solution used inside the kernel can be used within file servers as well (namely, keep the whole path in the service-side fid tracking structure).

What's really nice is you can develop plug in file server modules which allow you to compose much more complex single-server-services out of many different file system components -- without worrying about how the qid space is split up and managed.

Applying mount namespaces (in Linux)

Wed, 03 Oct 2007 13:12:43 -0700

Decent developer works article describing how to use private name spaces in Linux including the new shared subtree stuff and examples about how to create per-login or per-user (where's per-process?) name spaces.

"Applying mount namespaces" IBM Developerworks

Per-process would be fairly easy to do given the techniques described in the paper -- we just need to work through an example, although it may be easier to do inside the source code of a particular shell.

v9fs

Tue, 11 Sep 2007 16:04:45 -0700

9p-linux - Google Code

Fri, 31 Aug 2007 14:16:12 -0700