(RSS Feed|Atom Feed)

Google Summer of Code 2009 Guidelines (2009/04/23)

If you are not a participant in the 2009 Google Summer of Code program, you can safely skip to the next section; there’s not much interesting stuff for you here.

Otherwise, welcome, and congratulations on making it into the Plan 9 program for the 2009 GSoC. We received over 50 applications and were only able to accept 7 of them. If you have not done so yet, take a moment to send your mentor(s) an email, introducing yourself. These are the people who will be guiding you through the next few months, so it’s good to have an open communications channel.

Communication

We are expecting students to embrace all means of communication during this period, and as such, we’ve created several means by which you can get in touch with us:

Progress Reports

Progress reports must be entered into your blog weekly. Your blog will be formatted in markdown, so these blog posts should be well-formatted text, and perfect for emailing to the plan9-gsoc list as well. After these reports, you and your mentor must convene to discuss them and any outstanding issues. These reports should detail where you are with your project, where you expect to be by the next week, and any blocking issues you are facing (especially if they may place a delay on your project).

This weekly status update is the minimal requirement -- you may of course contact your mentor or blog or email more than once per week, should you feel the need. The mentors and the community at large are there to help you through your project for the summer, so don’t be ashamed to ask questions!

The day to update your blog is Monday between 12:00 and 18:00UTC.

Community Bonding Period (20 April - 23 May)

During the community bonding period, we will be expecting you either to be completing tasks assigned by your mentor to familiarize yourself with the environment in which you will be working, or doing preliminary work on your project (if you are already very familiar with Plan 9). This preliminary work would include design and architectural decisions. During this period, you should be discussing any questions you have about Plan 9 or your project with your mentor. Additionally, you should be actively discussing the architecture of your project with your mentor.

During this time, you will also be set up with access to a version control repository and blog. Unless your project requires otherwise, we encourage you to perform your developmental tasks on an actual Plan 9 system or virtual machine. Examples of exempt requirements are development in Inferno, or working on components for other operating systems (e.g. Glendix, vx32/9vx for Windows). IMPORTANT: All source code must be kept in this public repository, and all development for your project should be committed to the repository as it is available. No excuses.

Interim Period (24 May - 6 July)

Weekly status updates should be sent to your mentor(s). As specified above, these reports should contain your progress for the week, and where you expect to be by the next week. If you have any questions, pose them here. If you are falling behind schedule, let your mentor know what issues you are experiencing, and try to work out a way to catch up.

Midterm Period (6 July to 13 July)

Mentors and students should work together to determine specifics on their mid-term evaluations. We expect development to continue through this phase, and a weekly report will be expected here, too.

Interim Period (13 July - 10 August)

This period of development follows the 24 May - 6 July Interim Period schedule as far as expected work and communication.

Pencils Down Period (10 August - 17 August)

Remove bugs, clean code, discuss the project with your mentor. You should not plan to be using this time for your project, though it is not expressly forbidden. At the end of this period, you’ll be asked to take a survey about the experience, to provide the mentors feedback with how you felt about the project, any problems you had, and to provide suggestions for smoother sailing in the years to come.

Questions

If you have questions about your project, don’t hesitate to get in touch with your mentor(s). If you are unable to get in touch with your mentor(s) over an extended period of time (four or five days; no longer than a week), or are unable to reach them in an urgent situation, get in touch with Anthony Sorace, the project administrator (anothy_x on IRC, anothy@gmail.com via email) or any other mentor who is immediately available.

Working With Plan 9

Plan 9 is not like any other modern operating system you have used. Learning how to use a different operating system is not too different from learning a new skill or language, and the same amount of perseverence and patience is required. The best thing to do before diving into Plan 9 is to accept that you will be learning new things (and not just re-learning things you already knew).

Installing Plan 9

Our hardware support is somewhat lacking. We run on many modern machines, but we do not have drivers for some of the ‘‘newest and coolest’’ devices. The best way to run Plan 9 when learning is by installing it in a virtual machine. The installation process itself is not complicated, but does take quite some time. For this reason, we are providing a new, preinstalled qemu disk image. While Plan 9 does come pre-installed on this machine, there are still plenty of useful administration tasks to learn, all of which will help familiarize you with the system.

The disk image is available at http://206.71.190.158/~dho/plan9-qemu.qcow.bz2. To boot Plan 9 in qemu, simply run: qemu -hda plan9-qemu.qcow -boot c -std-vga.

To become familiar with the Plan 9 operating system, you should install the qemu image and begin becoming acquainted with the system. A good first project to undertake is to configure your qemu VM to be a standalone CPU server. Instructions for configuring Plan 9 as a CPU server are available at http://www.plan9.bell-labs.com/wiki/plan9/configuring_a_standalone_cpu_server/index.html.

Plan 9 Resources

Working With Inferno

Inferno is an operating system designed to function well for implementing distributed systems. Mechiel Lukkien has created a wonderful introductory guide for Inferno at http://www.xs4all.nl/~mechiel/inferno/getting-started.html.

2009 GSoC Update: Accepted Projects (2009/04/20)

Google has officially released the list of accepted candidates per organization. We were lucky enough to receive all 7 slots we asked for. It was a difficult choice; we had many good proposals, several for the same project, and it wasn't terribly easy for us to pick from them. (Maybe we will get more in the future!)

The list of accepted projects for 2009's Google Summer of Code program (in no particular order):

Congratulations again to everyone who was accepted. You should contact your mentor as soon as possible to get acquainted. I will be sending a mailing out in the next couple of days detailing some of our expectations for this year's program, as well as containing some first step-style exercises to get you acquainted with Plan 9.

To those of you who were not accepted into this year's program: we're sorry that you didn't make it, and believe me, some of the picking and choosing was very, very difficult. We'd still love to have you as active members of our project. We're still around on 9fans, here at cat-v.org, and on #plan9 on Freenode.

Thanks again to all applicants and mentors!

--dho

2009 GSoC Update (2009/03/25)

Plan 9 in the 2009 GSoC

Firstly, I'd like to thank Anthony Sorace for investing the time and energy required to get Plan 9 into the Summer of Code program for 2009. His application was a success and we'll be working hard this year to make the 2009 Summer of Code program a huge success for Plan 9.

If (by some stretch) you are a capable Plan 9 developer who is not yet aware of that, we can always use more mentors. Get in touch with Anthony via the plan9-gsoc mailing list.

If you're a student, we've got a plethora of great projects for you -- check out http://gsoc.cat-v.org/ideas/ for these. As usual, we're always willing to hear your ideas, make suggestions, and help you with your proposal. Outside of the plan9-gsoc mailing list, many of us are also on IRC: #plan9-gsoc and #plan9 on irc.freenode.net.

I wish to extend a warm welcome to all interested participants, a huge thanks to everyone who is helping out with the program as an administrator or mentor, and hope that we can get a bunch of cool stuff done this summer.

--dho

Good News (2007/11/10)

Oi

I haven't had much time for hacking on o9fs. The university takes most of my time. But that's not what the post is all about as one can presume... The good news is that there are two people testing and reporting bugs with o9fs which is something every project needs since many times it is not possible for the author to find every possible problem because of, besides from other things, his/her own limited test environment. So, thanks to Matthias Bauer and Christian Kellermann, the project is again quite active.

iru

Final Update (2007/09/25)

I don't know if anyone is still reading this blog, but I wanted to write about the outcome of my project, how GSoC went for me and my future plans related to the project.

In case anyone reading this didn't know, I was working on Inferno authentication, in particular the SPKI infrastructure for Inferno. I completed the project successfully, and produced the following:

- An implementation of Inferno authentication for Plan 9 and p9p
- A SPKI verifier which can produce Inferno certificates
- A SPKI version of keyfs which stores keys and certificates securely,
  and allows these to be queried
- A command which creates SPKI certificates to form part of a chain of
  delegation of authority
- I also adapted a program written by my mentor Charles to create a
  module which performs SPKI reduction.

I'd like to say how GSoC was from the perspective of someone who was never involved in open source development before. In general, I felt that GSoC went well for me. I started off slowly because I found it very difficult to become familiar with Plan 9 and Inferno so quickly since I'd never heard of them until earlier this year. I think this was the main thing I'd change - if I did this again I'd try my best to prepare more. I also should have asked my mentor more questions - at first I was afraid to ask many things in case the questions were stupid. On the other hand, after a few weeks I started working a lot more quickly and I really learned a huge amount during the project, not just about Plan 9/Inferno but also about development and managing a project in general. Overall, I think I produced more useful code than I had expected at the start.

In future, I intend to continue to work on improving or fixing the code I have written this summer, if necessary. However, I also want to become actively involved in the development of Plan 9 and Inferno, and contribute as much as possible to these projects. This includes both work related to my project and also totally different work, since I have a lot of ideas. I guess I'll post to the relevant mailing lists when I want to work on something.

Manual pages (2007/08/31)

manual pages

yesterday i wrote the missing manual pages for the rabin and vac library, and also added some documentation to the venti library. it can also be found in the mercurial repository of course.

Say Hello to Angled 0.1 (2007/08/22)

I feel I'm in the seventh heaven. After a few sleepless nights struggling with Mozilla's XPCOM, I finally got the 9P Firefox plugin to work.

The plugin is called Angled (an anagram of Glenda, the Plan 9 bunny) and is in a pretty simplistic state right now: you can read any files served by 9P right in your browser window. Let's take a step by step look.

First I startup Inferno to start a 9P server. ${INFERNO}/usr/anant/home is symlinked to my actual home directory, /Users/anant:

[theghost anant]$ emu
; runas nobody {listen -A tcp!localhost!1564 {export /usr/anant/home &}}

Let's see what files are actually there:

[theghost web9]$ pwd
/Users/anant/Plan9/web9
[theghost web9]$ ls
README TODO js9p php9p

Alright, I open my browser window and type 'ninep://localhost!1564/Plan9/web9/README' into the address bar. I could also say 'tcp!localhost!1564′, but TCP is the only protocol available for Angled, so it would be redundant. Now, for the goodies: Screenshots!

Read Text files over 9P

Cool! But wait, Angled also displays binary files right in the browser. There's a catch though, it will only work for binary files that can be viewed directly in the browser window. Certain types of files (.doc for example) do trigger a download request, but then become corrupted for some reason.

[theghost content]$ pwd
/Users/anant/Plan9/web9/js9p/angled/content
[theghost content]$ ls
angled.png firefoxOverlay.xul glenda-error.png overlay.js

Let's say I want to view angled.png. Here's what I get:

Angled shows Images too

Okay, but what if you type in a URL that points to an invalid file? Check this out:

Errors in Angled

Sweet! I'm yet to figure out how to transmit the exact error message to that page, so you'll have to make do with that generic image for now.

Okay, now onto the bad parts. Angled doesn't support authentication yet (although the base JS implementation is capable of generating and parsing auth messages). Next, you won't get directory listings (you'll get a bunch of binary gibberish which is actually Rstat messages for the directory's contents). Also, I'm doing the 9P connection and transactions in a blocking thread, so the UI freezes while all that is done. I couldn't feel the difference since I was testing on my local 9P server, but connecting to remote 9P servers won't be a pleasant experience. The solution to this is to create a custom nsIChannel implementation, which is a lot of work… I'll do it when I get to it ;)

Enjoy!

Ventisrv fileformat (2007/08/20)

Ventisrv file format

A few minutes ago I added a document to the ventivac hg repository (doc/ventisrv-fileformat.ms, a troff -ms file) about the file format used by ventisrv. The format is not very complex, the description should be enough to recover data in case of problems (such as disk failures, or bugs--though note that ventisrv will only ever append data to the index/data file, never overwrite it). I also made the file available as ventisrv-fileformat.ps (original postscript) and ventisrv-fileformat.pdf (converted to pdf).

Now getting back to finishing the last bits!

The end (2007/08/20)

  1. The QEMU-on-Plan 9 strategy paper has morphed, growing a lot more discussion of QEMU internals. The latest version is available here.

  2. Changes to libdynld are complete. The tarball is here. Some additional commentary is available here.

FP Final Update (2007/08/20)

I have done a snapshot of my code in a tar.gz file, I don't know how to upload this snapshot so I have copied this to my home dir in gsoc.cat-v.org. 

Finally I have not success with the planning I'd done. Actually the plugin loads and runs libinit and emuinit, but fails while running disinit. I have translated the thread management to pthread, and the problem happens about the exception and error handler, I have problems when calling waserror(), I've seen in the IE plugin that it changes the exception handler, I think that I might do the same but I'm not sure about it.

The error that I had with the free calls is solved, I was using the __clone proc to create threads, now I use pthread_create and it seems to solve the problem, now I have core dumps of the errors and I can use it to solve the problems.

About the other goals of the project the status is:

Plan9 Kernel Booting on OLPC (2007/08/19)

Finally we have Plan9 kernel up and running on OLPC ! Here is the screen shot



Implementing a Protocol in Mozilla (2007/08/18)

Creating a Firefox extension is nothing short of an adventure. I was able to get started pretty quickly, thanks to this web-based quick-start wizard, all the boilerplate code was generated in literally no time.

Now, onto the actual functionality of the extension. I have to implement a protocol handler for the 9P protocol, which essentially means you type in "ninep://sources.cs.bell-labs.com/" and start reading files right off the browser window. (ninep:// because a URL can't start with a number)

This page provides some useful insights and code snippets on the subject of adding a new protocol handler. I was able to get as far as displaying a Glenda image whenever you type in a URL beginning with 'ninep'.

The way this works is you create an XPCOM component that implements a standard interface. Specifically, the newChannel() method is where all the action is. It receives a URL and you do something and return an nsIChannel. Mozilla provides standard nsIChannel implementations for popular protocols such as http, ftp and even the ubiquitous file://.

The intuitive thing to do here would be to do all my 9P processing in the newChannel() implementation and return a stream in a standard channel. However, that's not going to work, since newChannel() would then block and the UI would actually freeze until the 9P transaction completes. Sub-optimal.

The "proper" way to do this would be to create my own implementation of nsIChannel. That way I just create a new nsIChannel in newChannel() and be on my way. nsIChannel would then take care of firing callbacks as and when data arrives. There's somewhere I can start with, and that's the Mozilla implementation of the finger protocol. It's written in C++, however, and I need to figure out how I can map the same to JavaScript (via XPConnect).

Fixing OLPC Crash Problem (2007/08/16)

Today I fixed OLPC crash problem. Problem is with lowraminit() function in memory.c Plan9 reads BIOS data area to figure out free memory below 640 K. But since OLPC doesn't have a PC BIOS, I have modified the OLPC loader to write to BIOS data area that 639 KB is free.

Also I have updated x86amd table in devarch.c I have added Geode-LX processor entry to this table. So that Plan9 will be able to detect Geode processor correctly.

To block or not to block (2007/08/16)

There’s one last hurdle before I finish my GSoC project. I’ve already written the JavaScript that produces binary messages for every possible 9P transaction, and all that needs to be done is to actually send those messages to the 9P server.

In a few of my previous posts, I mention that there were two apparent ways to do this:

a) Send the message wrapped in an XMLHttpRequest to a HTTP server that forwards the message to the actual 9P server.

b) Use Mozilla’s XPCOM components to access Sockets directly in JavaScript.

Well, it turns out that (a) is probably not a solution at all. HTTP is far from what one would call a protocol that supports streaming. A 9P server (correctly) doesn’t return an EOF until you have completed your whole session. So the first time I send an XMLHttpRequest to, say a PHP script, the script blocks forever, since PHP would never know when the first R-message has been actually sent through. I can always peek at the first 4 bytes and find out the length of the R-message, but what then? I can’t close the socket since that would terminate the 9P session, but I have to return from the script for the HTTP response to be actually sent.

PHP doesn’t support threading (proper) so I can’t do the_ select()_ mojo either. How about storing the socket FD in the session variable? Well, this is probably the closest to a good solution but that would limit every client to exactly one 9P session.

Although Mozilla’s XPCOM is one hell of a beast, I think it might be good to just build a firefox extension to access 9P resources. Not exactly sure of how I’m going to make it work, but tentatively, I’m thinking of parsing URIs beginning with 9p:// or something like that. Let’s see how this goes.

Automated boot script for Plan9 Kernel (2007/08/16)

I have created a small forth boot script to automate booting of plan9 kernel on OLPC. You need to create /boot directory on USB thumb drive and copy olpc.fth to /boot.

Once open firmware sees disk:/boot/olpc.fth file it will execute this script as part of its automated boot sequence.

speaksfor (2007/08/15)

Today I completed the speaksfor program. This is an Inferno command which will be used to create a SPKI certificate which forms part of a SPKI "speaksfor" chain of delegation of authority. It works like this: the command can be invoked as

speaksfor S I [T] [V]

where S is the Subject, I is the Issuer, T is an optional tag and V is the optional validity of the certificate. Then a certificate is created which states that Subject S may now speak on behalf of Issuer I regarding the things in the tag T (by default the Subject may speak for I regarding everything). In other words, I delegates part of its authority to S.

In practical terms, currently S is the name of a public key stored in keyfs which I spoke about last time, and I is a public/private key pair read from a file. The Issuer I then signs the certificate with its private key. Although it isn't essential for my project, I'd like to extend speaksfor so that it can also produce name certificates, which would be used to verify that a user is a member of a group.

As for what I have left to do, there are several things. Right now I'm working on a command that does SPKI reduction. I'll write about that next time as I only just started this. I also have a couple of minor bugs to fix and man pages to write.

mset (2007/08/15)

Just checked in mset, nothing to do really with gsoc, just a quick little mandelbrot set viewer I wrote while learning limbo earlier this year. At the most, it might be useful as yet another tk/graphics example, but it's fun to play around. Once I get some spare time I'd like to add some color cycling for fun, some text elements to display the cursor position on the complex plane and the generation of the julia set for a given point on the plane.

Update (2007/08/14)

Long time, no update. Sorry about that.

  1. The QEMU-on-Plan 9 strategy paper has morphed, growing a lot more discussion of QEMU internals. The latest version, which is still a draft (and therefore contains some FIXME notes, sorry) is available here.

  2. Changes to libdynld are complete. The tarball is here. Some additional commentary on these changes is forthcoming.

  3. With all of this in place, and QEMU's 386 target micro-ops library compiling, my dyngen can emit the right kind of code for the dynamic translator. Some comparison is available here.

OLPC plan9 kernel Changes (2007/08/14)

Following files are added to /sys/src/9/pc/ which add EGA emulation on top of OLPC linear frame buffer.

font_sun12x22.h
lfbgeometry.h
lfbega.c
olpc-egainit.c
olpc-egalib.c
olpc-ega.h

Following files are modified :

l.s - Here we need to make a new 4 MB page table entry for mapping OLPC's linear framebuffer. Also instead of calling main, olpc_display_init is called which prints Plan9 on screen using EGA emulation code. After that HLT is called so that machine won't reboot.

mkfile - Added entries for new EGA code.

Fiddling with Binary in JavaScript (2007/08/13)

Since ordinary JavaScript cannot directly communicate with a 9P server (over TCP), we decided to go in for a 2-tier approach: A script on the client generates a 9P message which is sent to a server via an XMLHttpRequest. The server then forwards the message to the actual 9P server. Messages from the 9P server to the client are sent in a similar fashion.

All 9P messages are just binary sequences, which means I need some way of representing a 9P message in JavaScript. A character is always 1 byte, so representing characters is not a problem. For representing integers in binary I use the following snippet of code:

while(num) {
str[str.length] = String.fromCharCode(num % 256);
num = Math.floor(num / 256);
}

where 'num' is the number to be encoded, and str is returned as: str + join("").

This means I can now encode any 9P message as a simple sequence of JavaScript strings. The final string can then be sent along with the payload of an XMLHttpRequest. I'm wondering whether it will be a good idea to encode the string in Base64 first, although an XMLHttpRequest should have no trouble with the string representation either.

FP update (2007/08/13)

Process management is all right. I have used pthreads in order to execute inferno's main in a thread while the main thread still runs as a plugin. Emuinit loads, but Firefox crashes when the ksetenv function is called, now I have comment the calls to ksetenv. A proc is created to execute disinit, I'm trying to trace if this disinit is running ok. Kproc works and everything seems to be ok but ksetenvs.

Then I will work in the console, I want to let emu paint the console in the firefox window (using X11), I hope to achieve it tomorrow.

FIREFOX PLUGIN UPDATE (2007/08/12)

Inferno Plugin update

I have had many problems with the memory management due to the fact that the plugin is loaded as a shared library. This implies that when the inferno code calls to malloc/free/realloc it calls the malloc/free/realloc functions from libc instead the malloc/free/realloc functions defined in alloc.c. This happens because the plugin is a shared library, and when the inferno code calls malloc, it finds the libc malloc before the inferno's malloc redefinition, idem with free and realloc.

In order to solve this issue I renamed malloc, free and realloc from alloc.c to imalloc, ifree and irealloc, writing the preprocessor directives in fns.h:

#define malloc(s) imalloc(s)
#define free(l) ifree(l)
#define realloc(v,s) irealloc(v,s)

But this result in another problem, there are calls to ifree where its argument is not a memory reference reserved by imalloc, and when it tries to free this memory an "not in pools" error ocurrs. In order to find where there is ifree calls to free memory reserved by libc malloc instead inferno imalloc I have used the ElectricFence library (thanks Chris), putting it in the LD_PRELOAD path so any call to malloc/free will call the malloc/free of EF, this way I have found (after many problems) the ifree call culprit of the problem. This call was inside the kstrdup function, the above mentioned function is that:

void
kstrdup(char **p, char *s)
{
    int n;
    char *t, *prev;
    n = strlen(s)+1;
    t = kmalloc(n);

    if(t == nil)
        panic("kstrdup: no memory");

    memmove(t, s, n);   
    prev = *p;
    *p = t;

    free(prev);
}

You can notice that at the end of the kstrdup function it calls to free(prev) (that finally is ifree(prev)), where 'prev' contains the last content of the reference that now references to the string copy 's'. The problems comes when the above mentioned pointer has not been initialized and does not contain memory reserved by imalloc but libc malloc. This happens with the string 'eve' which is initialized in main() (main.c) by calling strdup (libc), internally strdup calls libc malloc, then eve's memory has been reserved by glib malloc not inferno's imalloc, nevertheless, in libinit() eve's content is changed by calling kstrdup(eve,"newstring"), and when it tries to free the previous content of eve it try to free it calling ifree (when it was reserved by libc's malloc not inferno malloc).

To solve this issue I have wrote a strdup redefinition named istrdup that does the same that standard strdup but calling imalloc instead libc's malloc.

In addition, inferno code calls to sbrk to get for memory from the host OS to allow that inferno manages this memory. To be able to reserve memory acting as a plugin it is necessary to to call to NPN_MemAlloc() (Mozilla NPAPI) so I have added a redefinition of sbrk named isbrk that gives memory to inferno using Mozilla's API. In order to let the plugin use istrdup/isbrk instead of strdup/sbrk of libc I have added other directives to fns.h:

#define strdup(s) istrdup(s) 
#define sbrk(s) isbrk(s)

With this the problem of the memory management seems to be solved (for the present).

Process Management

Mozilla Firefox API for Plugins does not offer functions to create and manage process or threads, so it is possible to use directly the same calls for the process creation in Linux. Nowadays I have tried to create processes with kproc and it seems that everything is alright.

Nowadays Status

The problem that I have now is with the calls to ksetenv, when ksetenv is called in emuinit Firefox crash, and the only thing that says is "Firefox: Unknown error 2988458393", I have traced the calls and it seems that ksetenv (env.c) calls to namec() (chan.c), the flow enters the switch(amode) statement and go to the branch 'Acreate' where it calls to walk() and gives the error.

If I comment the ksetenv call, then emuinit runs without problems, but emuinit is called from libinit by:

executeonnewstack(tos, emuinit, imod);

But after this statement nothing statement is executed, so the 'for' iteration of the end of emuinit will keep the control of the main plugin process, and then the plugin does not respond neither offer the NPP functins that Mozilla needs. Then the plugin does not respond to the browser requests and finally crash. The plugin must offer many NPP functions in order to comunicate with Firefox, one of these function is NPP_Init where the plugin initialices and where I have added the call to ipluginmain() (firefox-os.c) that is the entry point to inferno from the plugin.

Maybe a possible solution to this is to create a new independent process where execute ipluginmain while the main process remains running and receiving browser requests...

Planning:

I have this aims:

Memory initialization semms to be solved, process initialization seems to work but it could cause problems, emuinit does not load right due to the ksetenv problem, when I solve this I hope to have an initialized and running inferno.

The next steps are:

Console: Paint the console (emu running) -> On August 14 Hello Worlds: execute a simple "hello world" within the console -> On August 15 Text Input by keyboard to feed the console -> On August 17 Paint graphics and handle mouse events -> On August 22 Clean the code -> On August 25

Please any comments will be welcome.

Thanks.

cocytus update (2007/08/11)

Past few days have been fairly unproductive due to having a few orientation days for school. This weekend I plan to devote to gsoc, once I see how much I can get down over the weekend, I'll have a good idea of what needs to be done/if completing the project is possible/ etc. on Monday. I'll update over the weekend and then give a detailed post on Monday describing what is done, what needs to be done, and whether or not I think it's possible to complete.

Booting plan9 on OLPC (2007/08/10)

Finally Plan9 kernel is up and running on OLPC !!

Here is the complete procedure to boot plan9 on OLPC:

  1. You will need a USB thumb drive and OLPC laptop (of course).

  2. Following 3 files are needs, which should be stored on USB drive.

    a. plan9.ini : plan9 configuration file
    b. 9pcf : plan9 kernel
    c. loader.elf : Bootloader which loads plan9 kernerl (9pcf) on OLPC

  3. When you start the OLPC laptop, you need to interrupt the boot sequence to go to open firmware (OFW) command prompt. Here is the procedure:

    a. Hold down the Game key when machine starts.
    b. When asked (on screen) release the game key, after releasing it, OFW will start probing for hardware.
    c. After finishing the probing, OFW will ask you to press Esc key within 3 seconds. Once you press Esc then you will land up to OFW command prompt.

  4. To check files on attached thumb drive execute following command,
    dir disk:\

  5. Execute following command to boot loader.elf instead of standard linux kernel.
    setenv boot-device disk:\loader.elf

  6. Execute following command to keep USB system alive after OFW transfers control to plan9 boot loader.
    ' noop to go-hook

  7. Execute following command to map OLPC's linear frame buffer to virtual address 0xfd000000.
    h# 910 config-l@ dup 100.0000 -1 mmu-map

  8. Finally, we are ready to boot plan9. Here is the final command,
    boot

And you should see Plan9 printed on screen !!
Currently it only prints plan9, as there is no proper display driver.

Swapping Endian-ness in JavaScript (2007/08/09)

My mentor came up with two interesting ways of swapping endian-ness on JavaScript. The first one he proposed was based on what was "usually done in Plan 9″, something along the lines of:

b = "\1\1\1\2";
n = (b.charCodeAt(0) & 0xff) << 24;
n += (b.charCodeAt(1) & 0xff) << 16;
n += (b.charCodeAt(2) & 0xff) << 8;
n += (b.charCodeAt(3) & 0xff);

…which gives us n = 16843010.

Maht then had a look at the series of JavaScript lectures by Douglas Crockford at Yahoo!. The first of the series tells us that bit shifting is not faster than simple multiplication. Maht gave me this code snippet doing the same conversion, but using multiplication instead of bit shifts this time:

b = "\1\1\1\2";
n = b.charCodeAt(0) * 16777216;
n += b.charCodeAt(1) * 65536;
n += b.charCodeAt(2) * 256;
n += b.charCodeAt(3) * 1;

n, is of course 16843010; the real question is how much longer (or, shorter) did this take.

Venkman is probably one of the more mature "old-school" JavaScript debuggers out there. FireBug, the relatively modern sibling to Venkman certainly has a few nifty features, but profiling is not one of its strong points. After failing to profile the script in FireBug, I used the trusty old Venkman - and it came up with some interesting results:

Venkman Profile Report
Created ………. Thu Aug 09 2007 20:18:54 GMT+0530 (IST)
User Agent ……. Mozilla/5.0 (Macintosh; U; Intel Mac OS X; en-US; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6
Debugger Version . Venkman 0.9.87 [Mozilla rv:1.8.1.6/2]
..
Function Name: multi (Lines 1 - 6)
Total Calls: 1 (max recurse 0)
Total Time: 0.21 (min/max/avg 0.21/0.21/0.21)
Time (ex. calls): 0.21 (min/max/avg 0.21/0.21/0.21)
..
Function Name: shift (Lines 8 - 13)
Total Calls: 1 (max recurse 0)
Total Time: 0.02 (min/max/avg 0.02/0.02/0.02)
Time (ex. calls): 0.02 (min/max/avg 0.02/0.02/0.02)

As you can tell from the function names, multi uses simple multiplication, while shift uses bit-shifting. Turns out that bit-shifting does indeed take a lot lesser time. A Firefox quirk?

Inferno SPKI (2007/08/08)

This is my first post on here, and I guess it is about time I posted some news on my project. My project involves extending the SPKI infrastructure of Inferno by implementing various file servers and commands. Using SPKI for authentication improves scalability and provides a way to define group membership using the "speaks-for" relationship with SPKI certificates.

The first part of my project was not actually related to SPKI, and involved adding support for Inferno authentication to the factotum of Plan 9 and p9p. I completed a basic implementation of this, although it would benefit from further testing.

I then wrote a SPKI verifier. The verifier is implemented as a file server using file2chan, and accepts a SPKI certificate sequence as input. If the sequence is verified correctly, the verifier then serves a file containing an Inferno certificate which can be used to access services.

Right now I am working on a SPKI version of keyfs for Inferno. This keyfs stores SPKI keys and certificates securely in an encrypted keyfile. It is implemented as a Styx file server which serves three directories within /mnt/keys: pk/, sk/, and cred/, which contain public keys, private keys, and all credentials including certificates respectively. Users can add new keys and certificates and name them by writing an S-expression to the "new" file which is provided.

So, next I will probably be adding more features to the keyfs, and hopefully also beginning work on some other related things. My code can be found at http://code.google.com/p/inferno-spki/.

I am enjoying the project a lot and learning so much, even though I have found it difficult. I will give another update very soon.

Plan9 kernel image format (2007/08/07)

Today I decided to leave plan9 ELF booting effort. Plan9 linker's ELF output is not correct. This is creating lot of problems.

So now I am going to write custom boot loader for plan9's kernel.

First thing i need to do is to understand plan9 kernel's format.

Here is the information that i got from a.out.h:

struct  Exec
{
    long    magic;          /* magic number */
    long    text;           /* size of text segment */
    long    data;           /* size of initialized data */
    long    bss;            /* size of uninitialized data */
    long    syms;           /* size of symbol table */
    long    entry;          /* entry point */
    long    spsz;           /* size of pc/sp offset table */
    long    pcsz;           /* size of pc/line number table */
};

I wrote a simple C program to read plan9 kernel image and print above information from it.

Kernel file image size = text + data + syms + spsz + pcsz

Also another important thing that I found out is that, this header is in big endian format, so on x86 architecture we need to swap data to read it correctly.

cocytus update (2007/08/07)

Have implemented some of the data structures...not 100% it has everything needed, but will know as soon as the styx operations are implemented which is the next step. Will begin tomorrow with the fundamental, i.e. read/open/walk. This should give a good feeling for whether or not everything is working as it should and if the data structures are sufficient. Write will then be implemented once I am sure the basics are working properly. A semi-rough timeline would be to get the read/open/walk operations done by Wed-Thurs, then have write working by the end of the week.

Mmu setup problem for kernel startup (2007/08/07)

By default plan9 kernel's text section starts at 0xF0100020 which is around virtual address 3841 MB. 9load loads kernel at physical address 0x100020, and then kernel sets up correct memory mappings using paging.

For OLPC we are using ELF format. OLPC's open firmware determines start address of ELF's txt section from ELF header. If we compile kernel in ELF format and ask linker to link its txt section at 0xF0100020 address, then OFW gives page fault, since this address is not present physically and its not mapped in current page tables.

I am linking ELF kernel at physical address 0x100020. So its loaded correctly by OFW loader. But once control is transferred to plan9 kernel, kernel code expects all address above 0xF0100020. So this is the main problem which needs to be solved.

cocytus update (2007/08/06)

Have the styxserver running, Basically used vacfs as the template for sending and receiving styx messages. Right now is basically just a skeleton for the file server. Am working now on implementing the various operations which will need to be performed on the venti side, i.e. this will be the mapping of the styx messages to the venti operations. Also the data structures invovled are being implemented as well, this step is being done before the operations. The vac and venti libraries contain some basic structures which will be used, but I will need to implement some things ot keep track fo the modified/unmodified states of the blocks.

Rabin fingerprints (2007/08/06)

rabin fingerprinting

I have been a bit quiet, time to bring some news! Quite a bit has happened since the previous post. The most important change is the new rabin fingerprinting code which can now be used by vacput, vacget and vacfs to split blocks not at fixed byte offsets, but at content boundaries. This code for rabin fingerprinting has actually been in the hg repository for some time, but I recently changed a few of the parameters making the block boundaries much better and hadn't yet explained it. So that's what is post is about.

the algorithm

The rabin fingerprinting algorithm calculates a rolling checksum over data (a file to store in venti). The window of the data to look at is configurable, but typically a few dozen bytes long. The rabin module will read through a file, and let the window "slide" over the data, recalculating the fingerprint each time when advancing a byte. When the fingerprint assumes a special value, the rabin module considers the corresponding window position to be a boundary. Al data preceding this window position is taken to be a "block" of the file.

Calculating a checksum relative to a previous one is not very expensive cpu-wise. The operations involved are:

  1. Multiplying the previous checksum by a prime (the prime is a parameter of the algorithm).
  2. Adding the value of the byte that has just come into the sliding window.
  3. Subtracting the value of the byte that has just slid out of the window times the nth power of the prime (where n is the width of the sliding window; the 256 possible values are precalculated during module initialisation).
  4. Taking the modulo of the value by the average desired block size (for now, the average desired block size is required to be a power of two).
  5. Check whether the new checksum has the special value that makes it a boundary.

This means the algorithm has three parameters: prime, width and modulo. The properties that make this algorithm useful for our purpose is that: 1. they are cheap to calculate; 2. they find the same boundaries, no matter where in the file they occur. Thus, when a byte has been prepended to or inserted into a file, this has no influence on how later blocks are formed, as opposed to the case where all block boundaries are at fixed file offsets.

This implementation also allows you to specify a minimum and maximum block size . If a block boundary occurs before the minimum block size is encountered, it is ignored. If no block boundary occurs before the maximum block size, the window is treated as a block boundary anyway and a new block emitted.

At first I had set the width of the sliding window to three, but after reading through lots of data (my homedir, ~7GB in size) with different values for the parameters, it was clear that a window width of three is too small: too few unique boundary patterns existed, and the ones that did exist were found too often. It has now been set to 31. The tests showed that which prime number to choose did not matter (for the values I tested). The modulo (average block size) is reasonably accurate, but the minimum and maximum block sizes, and the fact that end of files result in short blocks, skew the actual average block size a bit. The special value at which to find boundaries had at first been set to zero: not good, patterns of all zero bytes will be a block boundary this way, causing lots of boundaries on "null" data. Better is to use the modulo-1, which is the value currently being used.

not there yet

Okay, so we can split a file in blocks in another way. Unfortunately, this cannot be integrated into vacput/vacget/vacfs without changing the data structures of venti archives. A hash tree stored in a venti archive needs all its blocks to be of a fixed length, typically 8KB. Vacget/vacfs use this assumption when finding the right block for fulfilling a read for a given offset in a file; they use it to determine which score in a pointer block to follow to get to the data block.

Now consider what happens when a (random) file offset in a vac+rabin fingerprinting archive should be read. When starting at the top pointer block of a hash tree, there is no way of knowing below which score the requested offset resides. Previously it could be calculated using the depth of the tree and the width of the tree branches and leaf size.

To solve this, I have changed the format of the pointer blocks. Each score is now accompanied by the size of the subtree the score references. The size referenced by the top score can be found, as usual, in the Entry data structure (which also contains the top score).

When walking from the top down to some file offset, the sizes of the subtrees allow vacget/vacfs to determine which ranges of the file reside below each score in the pointer block.

more (data structure) changes

The pointer block is not the only thing that has changed. To make sure the two types of data can never be misinterpreted, the new pointer blocks are stored with different "type" numbers in venti: the usual values added to 16; so 19 for depth 0, 20 for depth 1, and so on.

There is a new root block version, version 3. The maximum data block and pointer block size can be left intact (though I an currently writing them with "0" filled in).

The "flags" field in the Entry structure has a new bit set (1<<7) to indicate the entry is has variable-size blocks. In theory venti archives can be made with the two types of entries (variable sized blocks or not) mixed.

more ideas

A few more ideas have crossed my mind:

  1. Currently, the new pointer blocks always hold a fixed number of scores (either to more pointer blocks or to data). When a new block is inserted in an existing hash tree (e.g. when writing a new version of an already existing archive) is split in the new archive, it now changes all pointer blocks to right of it in the hash tree. Since we now have variable block sizes anyway, we could as well grow the current pointer block by another score (unless of course it would grow too large). This would mean fewer pointer blocks need to be written when storing a new version of a file. I am not sure how much it really helps though, and whether the added code complexity is warranted.

  2. Rather simple, but currently the size of a subtree is always stored in 8 bytes. This grows the size of one pointer from 20 bytes (the score size) to 28 bytes. The lowest-level pointer blocks, which point to blocks of at most 32KB, will be the most numerous. Thus, perhaps a variable length encoding of the size could be used. But it probably isn't worth the trouble.

  3. Should the rabin parameters be configurable, and parametrized into the root block of an archive? Probably not, well-choosen defaults should help. At least changing the prime doesn't make a difference, the sliding window width should be okay too. Only the average block size may be worth parametrizing, perhaps as "data block size" in the root block. This information is only useful for writing venti archives, so in this case, for writing an archive relative to a previous version.

conclusions

It seems the current rabin fingerprinting code works and can be used to write and read venti archives. I still want to test how fast fingerprinting is (both with and without JIT), and whether not using power of two's as modulo has a noticeable influence on performance.

I am probably forgetting to explain something, please bug me about it on irc or send me an e-mail.

Faulty Page Fault Handler (2007/08/05)

I'm still puzzled by the “hw tlb loading disabled w/o sw loading available” warning displayed by the simulator. I verified that software TLB management is enabled, and that the correct interrupt handler is invoked on a page fault. Having run out of ideas, I moved on to the next issue. The page fault handler (which eventually causes mmuput() to be called) behaves correctly the first time it's called, that is, a vacant TLB entry is used to map the page, but when the next fault occurs, mmuput() panics as it finds that the virtual page it's trying to map already exists in the TLB, which means one of two things: either the checking used in mmuput() itself is buggy, or that there's inconsistency between the hardware TLB & the STLB cached by the kernel, which is unlikely. My biggest problem now is that I can't even use print() for debugging. mmuput() is extremely sensitive to any modification I make (may be because it's running in an interrupt -- there might be some restrictions that I'm not aware of). For example, this is what I get when I modify the condition used in checking if an entry is duplicated:



back (2007/08/02)

Working on o9fs has teached me many more things that I can count, but surely one of the most precious are the real need for simple code. Having no internet for almost two weeks, man pages and source code have been my best friends on the learning process. It could have been a real pain if I wasn't hacking on clean and simple 9P and, even being a unix-like os, the OpenBSD kernel. Even if some people don't like to hear this, good code is documentation.

On the technical part the communication framework is almost totally finished and it's pretty usable. The filesystem can mount, resolve names (lookup was such a pain to get 'right'), open, and close files. Next step is working on Tread related stuff such as read (duh!) and readdir. I spent a great deal of time thinking about how directories should be implemented when the simplest way was obvious: a list of fids.

Before going on the code work, I'm gonna put togheter my notes on the design and implementation of the project for those who wish to have a better undestanding of it.

Update on the cocytus file server (2007/08/02)

I've been somewhat of a recluse this summer, so for those of you who don't know, this project aim is to implement a read/write filesystem for Inferno with a Venti backend. The next few days I will be dissecting vacput (A nice tool written by Oksel) into a lib as it does somethings that we need in the fs, and it is deemed quicker then writing this same functionality from scratch. Once this is complete we will integrate this lib into a styx server (the fs) and make the appropriate mappings between styx calls and the lib. The end product should be a fs which mounts a venti root, caches modified blocks in memory until a time that it will sync these modified blocks back to the venti server. You can think our fs as an overlay for the traditional venti block pointer heirarchy, only it can point to either locally cached modified blocks or pure venti blocks.

Post-mid-term changes (2007/08/02)

First of all, I want to thank all the students and mentors who have worked hard to make this GSoC successful.

The mid-term evaluations are over, and there are both good and bad news and some important changes on how we are going to work from now on.

The good news: some projects have made amazing progress and are well underway to archive their main goals, congratulations to everyone who got this far.

The bad news: many projects are way behind schedule, and what is worse, due to lack of open communication it is impossible to estimate what their exact status is and what changes need to be made to bring them back on track.

During the first half of GSoC everyone got a free hand to organize their projects in any way they wanted, no requirements were made and there was no control of how projects were going.

It was probably naive to assume things would work out fine this way, but it probably was a lesson we needed to learn.

Starting next week every project should at least follow this basic requirements:

If for any reason someone can't comply with one of this requirements, we will be very understanding, but we need to know why.

Lets hope this improves how things work in the second half of the summer and ensures that all projects are successful.

Best wishes to everyone

the Plan 9 GSoC admin.

9load page table setup (2007/08/02)

Today i read the code in 9load's l.s file. I wanted to get information about the way 9load sets up paging for plan9 kernel.

I figured out that 9load turns on paging. It identity maps first 16 MB of memory to 2 GB and above it.

I find this quite odd, normally bootloader never uses paging, when kernel gets control it sets up the page tables.

Now my next task is to check the startup code in kernel, and figure out how it uses this mapping, and if possible remove dependency from 9loads memory mapping.

Cell Memory Management (2007/07/29)

Several bugs have been identified and fixed in the MMU code. Although the first user process still dies prematurely. I hope that we're close to finding the remaining issues. The simulator output has been very useful, although it's really a tedious task having to single-step through long runs of instructions to find where things go wrong.

Most changes have been done in l.s, an assembly file containing processor initialization code & other essential functions, and mmu.c, which has the MMU code specific to the Cell BE. This is a brief summary of the issues that were solved so far:

Being working at a very early stage in the kernel, fixing all of these issues hasn't shown any perceivable improvement yet (other than the internal state of the processor being more like what we expect), simply because any little remaining issue is capable of screwing everything up!

By enabling Memory XLATE_FAULTS debug control in the simulator, I get a message saying “hw tlb loading disabled w/o sw loading available” right after switching to the user process. This is interesting because software management of TLB is enabled in the very beginning, long before the the user process is created, and I don't see that message during that. Anyway, at least I have a clue about where to look next!

init (2007/07/26)

hello, 9.

this is my first post and i'm happy to announce my initial commit to the o9fs repo. the code is not as clean as i want it to be.

this initial commit brings a new virtual filesystem which issues a Tversion message and fake a mount point. after finishing the communication framework i can get back to the real point (i.e. 9P and vfs). if anybody is wiling to give it a try (i doubt it) send me an email at iru.muzgo!gmail asking for instructions on how to install it.

Mapping 9P to REST (2007/07/26)

Now that the PHP bindings to libixp are somewhat usable, I've moved on to the JavaScript portion of my project. The first (and easier!) part of it is to map 9P to a RESTful scheme, so traditional web developers can use 9P without having to learn anything new.

The PHP bindings to libixp was an important pre-requisite to achieve this: presenting a REST interface to an existing 9P serve would require a "bridge" at the middle, to convert REST requests to 9P requests and vice-versa with responses. This "bridge" may be present at any location that is mutually accessible by the client wanting RESTful access and the server providing the 9P service. This bridge can be easily coded in PHP using the new libixp bindings.

To those not very familiar with REST, it is simply a way of accessing resources using plain-old HTTP. It's become quite popular with web developers these days, as a much simpler alternative to SOAP. A lot of web services these days are RESTful, including those offered by Amazon and Yahoo.

From the client's perspective, accessing a 9P resource boils down to 4 things: reading, creating, modifying and deleting. These operations map neatly onto the GET, PUT, POST and DELETE HTTP requests, respectively. And thus, we have our REST URI scheme. This scheme would be enough if my bridge exposes only a single 9P serve as a REST service. As an example, suppose I start a bridge at http://plan9.kix.in/rest/ that exposes only the tcp!sources.cs.bell-labs.com!564 9P serve, to read the file named 'lsr' I would perform a GET request at http://plan9.kix.in/rest/lsr.

However, the plot thickens when I want to design a bridge that allows access to any 9P service (which is definitely better). Now we need to encode the information represented in Plan 9 as: tcp!sources.cs.bell-labs.com!564 into a HTTP URI. The intuitive thing to do would be something like: [ROOT]/[PROTOCOL]/[9P-URI]/[PORT]/[FILE-PATH].

Hmm. That leads to really long URI's like http://plan9.kix.in/rest/tcp/sources.cs.bell-labs.com/564/lsr. Besides that, there are several things that need to be worked out. What happens when you do a GET on a file that is actually a directory? What about parameters to GET that you usually pass to a read() function: Suppose you want to read the first 1024 bytes of a file only?

Comments and Suggestions welcome :)

P.S.: Thanks to some pointers by Kris, the PHP9P client example shown in the previous post now handles binary files correctly.

B4 Laptop Arrived (2007/07/18)

Today I got the B4 laptop. Its very fast compared to previous B1 machine. OLPC is making impressive progress.

I also completed reading about paging and plan9 compiler and assembler papers.

Things to do next (2007/07/12)

I found out that plan9 kernel startup point is in l.s file.
9load enables paging and maps first 16 MB of RAM identity mapped.
OFW doesn't support this. Also plan9 assembly is somewhat different than GNU or Intel assembly. Looking at this issue I think I need to read following things first before proceeding further:

  1. Paging - Page Tables Setup etc.
  2. How to use Plan9 C compiler - Rob Pike
  3. A Manual for Plan9 assembler - Rob Pike

Open Firmware Client Interface Information (2007/07/12)

While handling control to Operating System; OFW passes address of client interface handler in register EAX. So in Plan9 kernel we need to copy this value from EAX and store it in a variable. And we need to do this before somebody else overwrites EAX register.

Rules for making OFW client interface call from Operating System:

  1. CS nad DS selectors should point to flat 4GB segments.
  2. EAX should contain pointer to argument array.
  3. On return from OFW, EAX contains return value.

Baby dyngen (2007/07/11)

Substantive (if demo quality) progress has been made on the dynamic code translator, dyngen. Some demo code is available here (requires upstream dynld code, mirrored here). Commentary about the demo, dyngen, and next steps has been shunted to another file so as not to clog the blog page.

Some updates have been made to the strategy paper to talk about a new translation buffer strategy (thanks to Eckhardt) and say a bit more about dynld, including discussion of an extension to make the immediate value folding dance easier for dyngen.

First OLPC Program (2007/07/10)

To check whether a program compiled using Plan9's compiler works on OLPC I wrote following program. This program fills OLPC's screen with Blue color.

void
_main(void)
{
    int i=0;
    unsigned short *fb = (unsigned short *) 0xfd000000;
    for(i=0; i<1200*900;i++)
    {
        *fb = 0x0015U
        fb++;
    }
}

Here linear frame buffer start address is 0xfd000000.
OLPC Screen Resolution is 1200x900.
Color Depth is 16 bits.
0x0015U is value for Blue color.

I compiled this program using following commands,

8c color.c
8l -H5 -T0x100000 color.8

I got 8.out which was in ELF format. I wanted OLPC's open firmware to load this executable at 1MB so I gave -T0x100000 option to 8l.

To execute this program, copy it to USB disk. Attached this USB disk to OLPC and boot the laptop. You need to press 'x' key in game pad to interrupt the booting sequence. After this you need to press Esc key to enter to open firmware boot prompt. Once you are at open firmware boot prompt execute following command to check whether USB disk is detected properly and appropriate FS driver is loaded.

dir disk:\

This command should print out contents of USB disk's root directory.

Now you need to tell open firmware (OFW) that you want to load 8.out from USB disk and execute it. Execute following command,

setenv boot-device disk:\8.out

OFW by default resets USB system just before handling over control to new kernel. If you need to turn off that behaviour then you should execute following command,

' noop to go-hook

Now we need to tell OFW to map linear frame buffer video memory to 0xfd000000 address. Execute following command,

h# 910 config-l@ dup 100.0000 -1 mmu-map

After this you can boot 8.out by executing command,

boot

Some other useful OFW commands,

reset-all = To reset all hardware and reboot
power-off = Shutdown
print-env = Prints all the Environment
dev / = Sets to root of device tree, after this you can use ls command to print the tree.
.properties = Prints properties of the selected node.

Plan9 Tips (2007/07/10)

I want to documents some tips here, which are quite useful.

Copying files from Plan9 to Linux using floppy:

Create a 1.44 MB floppy image in linux, format it with FAT FS. Use it as 1st floppy disk in qemu.
In Plan9, use following commands to access it:
a:
cd /n/a:

Editing plan9.ini configuration file:

Type following command in Plan9,
9fat:
cd /n/9fat

Command to reboot / halt the system: reboot fshalt

For PC 386:
8c - is the C compiler
8a - is the assembler
8l - is the linker

One typically needs to add u.h and libc.h files as headers for C programs.

Important linker options for 8l are:
-H5 = Output is generated with ELF header
-l = Suppress default loading of startup files.
-T0x100000 = Loads Text section at 1MB

Default Entry symbol is _main

Default Plan9 kernel configuration for PC is pcf

Procedure for Compiling kernel:
cd /sys/src/9/pc
mk 'CONF=pcf'

To compile kernel in ELF format I modified /sys/src/9/pc/mkfile and added -H5 option to $(LD) command.

Herald the PHP 9P Client (2007/07/09)

I've given a few final touches to the 9P client for PHP. All the basic stuff should work (if they don't please let me know!), and I've written a script that shows some of the basic functionality that the client offers. Two actually, this one for the CLI SAPI and this one for the Apache2 SAPI. All they do is read files and show directory listings, but the scripts are a good place to see what the API is like, since there's no official documentation yet. I couldn't locate a proper PHP 5 server to actually host the second example, which would have been cool… but now we'll just have to make do with these screenshots I took off my browser (with the script running on my local Apache):


Startup Screen

Once you've put in the address, you'll get a directory listing of the root:


Viewer

While the CLI SAPI demo script also handles binary files quite well, this one will just send gibberish to your screen if you click on a binary file (text files work fine though). I'm still trying to figure out what's the best way of determining the MIME type of a file so I can send the appropriate HTTP header before transmitting the data itself,  so your browser would know how to interpret it. As for the server-side of things, the code is still in a state of flux and I'm not decided on what kind of API to offer. I'll probably take a break from this part of the project and move on the JS bindings, and then come back to tie this up in the end. That's not to say that the server-side code is not usable, just that it's not very developer-friendly; feel free to play around with it if you're feeling adventurous :)

Hello (2007/07/08)

This my first post !

Hello World !!!

Some more on Object Oriented PHP Extensions (2007/07/06)

Hopefully this is the last of the series on Object Oriented PHP Extensions. We left the last post with a question of how to create objects of a certain class in a method of another. This is especially relevant to the libixp extension, since classes like IxpStat, IxpQid and IxpCFid don't have any constructors of their own, but are rather returned by methods of the IxpClient class. For example, an open() on IxpClient returns a IxpCFid, and the IxpQid is a property of the IxpStat and IxpCFid classes.

How do we get this to work? The helpful folks at #php.pecl on EFNet pointed me to a few examples in the existing PHP codebase that do this. First, you need to initialize an object - which essentially involves:

Once you've done this, most often you'd need to set up certain properties on the class object. In libixp's case, we need to populate IxpStat's, IxpCFid's or IxpQid's properties. You can create a method to populate each classes properties. I use a set of PROPSET* macros to update properties correctly. Check out the code to clear things up - objectinstantiate()_ is the generic method for allocating memory for a object, and the PHP*initialize() set of methods set properties for particular classes. These functions are used in almost all of IxpClient's methods.

Before signing off, another quick word about using properties in your classes. If your class doesn't use any properties, you can get away without doing any memory allocation at all, see the FliteTTS extension for an example of this. However, if you do plan on using properties, you need a bunch of methods for allocating memory to the HashTable that will hold the properties, as well as the structures that your class will use. You can create a generic method createobjectex() that does this, and then create two more methods createobjectnew() and createobjectclone() that will act as handlers for new object creation as well cloning. These handlers are set during the class initialization (most probably located or called in PHPMINIT) by setting the _createobject_ member on the class entry struct and the cloneobj_ member on the classes' standard object handlers.

Now what (2007/07/03)

Once I have a plugin structured program I have to load the VM with it. In order to do it I've written a 'firefox' configuration file, it's a copy of 'emu' configuration file but I've modified it introducing the flags and path's (include's) that it needs to compile inferno as a plugin. First I modify the mkfile and write a new variable "PLUG", then I put this new variable in the compilation objectives (OBJ variable), I put the 'firefox' config file in the CONFLIST, and I redefine the PLUG variable in 'firefox' so that PLUG point to the plugin structure code files. Then I have a shared object lib of 5 MB where the emu VM and the plugin structured program live together.

Now Mozilla can use this lib, and the plugin loads, on the other hand, emu is not running, it's only loaded but not running.

I could try to run emu writing a mainplugin procedure which is called from the plugin init cycle. This mainplugin would call libinit procedure in order to let run the VM. But, it will never success, because my emu is not living under Linux but under Mozilla, and it must adapt to the Mozilla API, it's like Mozilla is the host OS.

But first let me talk about the inferno init model...

Lets talk about plugins (2007/06/27)

Ok, let's talk about plugins,

Firefox plugins are written using the NPAPI (Gecko Plugin API: http://developer.mozilla.org/en/docs/GeckoPluginAPIReference), Mozilla provides an SDK and API in order to let programmers who want to build plugins build it without needing of the entire mozilla source code tree, you can get it here (http://developer.mozilla.org/en/docs/GeckoSDK). there are a lot of documentation right there, however most of these docs are incomplete. You can find examples and tests of using the npapi (http://developer.mozilla.org/en/docs/Plugins:SamplesandTestCases).

If I have to be honest, I couldn't even compile these examples without errors, I don't know if I'm a little bit clumsy... Then, what did I do? Well, the npapi is in C++ with O.O. and I want C pure, then I get the SDK and I write my own NPAPI interface, I write a C version of the needed files, based on the Gecko Plugin API Reference.

I wrote two files, "npngate.c" and "nppgate.c", these files come from the Plugin Side Plugin API, and from the Browser Side Plugin API respectively.

When you have a Plugin you need two things, first your plugin has to provide a minimal set of functions in order to let Firefox talk with him, these functions are defined in the file "nppgate.c", and in the other hand your plugin has to define the interface to communicate with netscape functions, in the file "npngate.c" there are wrapper function for the Browser Side API, and there is the function that fills the Plugin and Browser Side functions table. These two files include the npapi.h (from the NPAPI SDK) where you can find all the functions that must be implemented.

If your plugin code includes these two files Mozilla will recognise it as a valid plugin.

Getting Inferno-Synth Up and Running (2007/06/27)

OMG Update!

Yes, ladies and gentlemen, an update! I will skip the pleasantries and announce, for the second time, I will be posting some updates on the development of inferno-synth. Perhaps this post is a little premature given I don't even have inferno on this machine yet. However, I'll be into the mix soon enough.

Cheers, D

First time (2007/06/27)

Hi people!

I'm here, at last. Well, I'm Bnext and I'm building the Inferno Plugin for Mozilla Firefox (IPMF), from now on I hope to update the blog entries daily.

I've uploaded my first code delivery ;)

Ok, that's all, will go to the code :P

Well hello (2007/06/22)

Apologies for the rather pronounced radio silence up until now, but I have been working with Eckhardt to produce a strategy paper. It identifies what seem to be key hurdles for getting the QEMU code generator working on Plan 9 (under kencc) and proposes at least one and usually more solution per problem.

I would deeply appreciate feedback from any and all interested parties. My email address is nwfilardo@gmail.com.

[Update: now using the absolute path for the paper. Thanks to Oksel for pointing out the need.]

Ventisrv compression (2007/06/20)

Compression

Work done since my previous post: Compression in ventisrv.

As you might know now, ventisrv stores blocks from 1 byte to 56kb in size on disk. These blocks are immutable: cannot be (re)moved or modified. The typical block size is 8kb for normal data blocks, and usually smaller for directory entries and pointer blocks (but that's outside the scope of this post). Now venti from Plan 9, and the newer venti from Plan 9 From User Space try to compress a block before writing it to disk, and only if compression makes it smaller, they store it compressed. This saves disk space for many blocks.

Ventisrv at first did not have the ability to compress blocks, but always stored them 'raw'. With the changes of this week, the option -c enables compression. It isn't on by default because the compression speed limits write throughput: Compression is expensive (more so in Limbo then in C). Also, if you enable this, make sure you have the just-in-time compiled enabled as well.

Some details

Ventisrv handles compression a bit differently from the other venti's. First, ventisrv uses the deflate/inflate compression algorithms (the one used by gzip), whereas the venti's use whack, an algorithm that doesn't have much documentation but seems to at least be favourable when compression smaller blocks. It isn't available for inferno though, and deflate/inflate is: easy choice.

The second difference is that the other venti's write a header and the compressed payload for each block, whereas ventisrv gathers multiple venti blocks, compresses the payload, and writes the the headers first, and then the compressed payload (representing data for multiple blocks). The idea is that compression more data in one go makes the compression history buffer larger, allowing for better compression. A quick test on the data in the venti I have been using for backups for a few months show this really does increase the compression ratio: Gzip on the entire datafile resulted in a compressed data file of ~68% of the original, compressing up to 64KB of blocks resulted in ~70%, and compressing each block separately in ~79%. The downside to compressing multiple blocks in one go is that they all have to be decompressed (at least up to the one found) to read a single block, and all the compressed data has to be read from disk to do that. Thus, a maximum of ~64KB raw/uncompressed data to compress into one block has been chosen. This ensures we don't have to (de)compress or read from disk too much data for each block.

The file format has been "extended" with a new header for compressed blocks. This leaves data files written by previous ventisrv's valid. Note that this code is pretty fresh, may have bugs and hasn't been tested a lot, so use with care (and report bugs back to me please).

There is always room for improvement.

But well, these are mostly details and the benefit is not obvious.

Other news

Another small thing has changed in ventisrv: Support for specifying read-only connections. Specifying -r addr makes ventisrv listen on addr and disallow write and sync messages from clients that connected to that address. With -w addr, writable connections can be listened for.

I've also started on some data fingerprinting code, but that is not working yet, so nothing to show. That's it for now!

Fixed RSS feeds (2007/06/19)

RSS feeds were broken for a while, they should be working properly again, sorry for the inconvenience.

Vac tools in inferno (2007/06/15)

news

good news! the vacget, vacput and vacfs code of the ventivac project is now in the inferno-os subversion repository. many thanks to charles forstyh. this makes it very easy to test that code, so please try it out. the patch to /appl/lib/venti.b (mentioned here) that was needed before, has been included in inferno-os svn as well, so no more patching is required.

ventisrv and vcache are not in the inferno-os svn, so keep checking out the ventivac hg repository (it will also contain updates to the vac tools).

ventisrv testing

for testing, i've set up a ventisrv on gsoc.cat-v.org. note that it is for testing, so don't store your very important data in it, yet. for example, to write a file to it, try this:

; vacput -a net!gsoc.cat-v.org!venti welcome.txt
vac:a39bc0605cc80c7dee50f02c2ef6c502e740c376

now list the contents of the venti archive vac:a39bc0605cc80c7dee50f02c2ef6c502e740c376:

; vacget -t -a net!gsoc.cat-v.org!venti a39bc0605cc80c7dee50f02c2ef6c502e740c376
./welcome.txt

and retrieve the contents of the archive:

; rm welcome.txt
; vacget -v -a net!gsoc.cat-v.org!venti a39bc0605cc80c7dee50f02c2ef6c502e740c376
./welcome.txt
; ls -l welcome.txt
--rw-rw---- U 0 mjl mjl 10 Jun 15 12:59 welcome.txt

and now using vacfs:

; mkdir /n/vac
; mount {vacfs -a net!gsoc.cat-v.org!venti} /n/vac
; ls -l /n/vac/a39bc0605cc80c7dee50f02c2ef6c502e740c376
--rw-rw---- M 4 mjl mjl 10 Jun 15 12:57 /n/vac/a39bc0605cc80c7dee50f02c2ef6c502e740c376/welcome.txt

that's how to use it, enjoy!

First general IRC meeting (2007/06/14)

I hope that all your projects are going well. Given that all projects should be underway by now, and that almost all participants got accounts at http://gsoc.cat-v.org now it is a good time to get together in irc and see what everyone is up to and discuss any issues anyone might have.

Two irc meetings per week are planned, so don't worry too much if you can't attend one of them. The first one will be tomorrow at 22:00 UTC. The location as usual is #plan9-gsoc in irc.freenode.org (and don't forget that you probably can hold of someone there any time of the day or night if you have any questions or problems.)

More on Object Oriented PHP extensions (2007/06/13)

In the last post, I mentioned that the libixp sources may have to be modified to work around callbacks, but Kris quickly pointed out that there's a special aux element meant specifically for developers to put whatever they like into. The aux element is automatically passed to every callback function via the Ixp9Req struct:

struct Ixp9Req {
Ixp9Conn *conn;
Fid *fid;
Fid *newfid;
Ixp9Req *oldreq;
Fcall ifcall;
Fcall ofcall;
void *aux;
};

The aux element is created during the call to ixp_listen:

IxpConn *ixp_listen(IxpServer *s, int fd, void *aux, void (*read)(IxpConn *c), void (*close)(IxpConn *c));

Now how do we wrap this functionality in PHP land? The aux functionality must be passed on the 9P developers using PHP, while still being able to extract the zval that we need to invoke the appropriate callback. Thanks to the marshaller method, we can mangle struct values as much as we like before passing the values into their PHP counterparts. We simply create a struct to wrap both our callback values and the aux element from the user:

typedef struct _IxpAux {
IxpCallback *cb;
zval *aux;
int type;
} IxpAux;

In the marshaller, we simply extract the aux value out from the structure, convert it to a regular PHP-type and pass it along to the PHP-level function using Zend's calluserfunctionex()_ method. That problem solved, onto the next! Creating objects of a class in a method of another class - the IxpCFid class has no constructor, but an object is returned by calls to an IxpClient object. More on that later!

Ventisrv bugfix (2007/06/12)

a short note again. i just fixed a bug in ventisrv that caused it to misread the data file and spew error messages. that has been fixed now in the mercurial repository. there are also a few vacfs changes in there.

so hg pull and try again please!

Ventisrv and more (2007/06/12)

some news on the ventivac front. this post is mostly about how to use ventisrv.b, the (still work in progress) venti server.

most of the code has been cleaned up a bit. not vacfs.b yet though, that one is next. the manual pages have been cleared up a bit. the biggest difference since my previous post has been that ventisrv is now closer to being a useful venti server. it should not be considered stable yet, and the file format might change as well, but i think it's worth a try now!

so, where to start?

the ventisrv manual page (snapshot version) should offer a decent introduction. if anything is unclear after reading it or questions got raised by it, please let me know so i can clarify the manual page.

in short, if you want to run a venti server with a data file of 8 gigabytes and an expected mean blocksize of 8 kilobytes, executing the following commands starts a fresh ventisrv:

echo -n > data
echo -n > index # these first two only the first time ;)
ventisrv -v 8g 8k

data is used to store the data blocks, index keeps the index. ventisrv calculates how much memory it needs based on the arguments passed at startup. specify the verbose option to get it printed.

now vacput your current working directory to the venti:

vacput .

and use the resulting score to read the files in the vac archive again:

vacget -t yourscore

it should be as easy as that. if something fails along the way, please let me know so i can update this post with better descriptions, or fix the bug. please read the README in the ventivac hg repository, it has the instructions to patch inferno's appl/lib/venti.b to make ventisrv work.

a few quick tests with randtest from plan9ports (src/cmd/venti/randtest.c) showed write performance (of new blocks) of 15MB/s and a sequential read performance of over 18MB/s. but the machine was quite fast and i'm not sure it represents an average machine ventisrv might run on. oh, if you want to do a few performance tests yourself, do not specify -D, it kills performance.

what's next?

well, there is no authentication for connections. that does not really bother me, the scores are a nice and simple capability system. the only real worry i have is someone filling up a public venti server. thus, an address listen for read-only connections may do the trick.

compression is missing as well. the easiest would be per-block compression. a "smarter" thing might be to compress larger blocks. i don't have well developed intuition for data compression methods, but i have a hunch that compressing larger blocks (say 128kb) gives the compressor more history to work with and better compression ratios. writing would still be append-only (no problems there), but reading would need special care: for a given file offset, accounting needs to be done to find the location of the 128kb block in the compressed file. if this is a viable option, it may even be done as a sys->file2chan program.

other open issues are listed in the bugs section of the manual page referenced above. solutions (and more problems) are welcome. thanks for listening!

My laptop at last (2007/06/07)

Well, at last I have my laptop out from the technical service, so I don't have to share computer anymore. That means bbye lunix for the summer... And also that I don't have to do any more qemu "magic" to work with two synergy running machines.

Mercurial (2007/06/06)

just a short note. i put the source code so far in mercurial. it can be found at:

http://gsoc.cat-v.org/hg/ventivac/

so, a hg clone http://gsoc.cat-v.org/hg/ventivac/ my-ventivac should suffice to get a copy. see the README on how to compile and install.

enjoy!

Starting (2007/06/06)

Hi all. Here is a first progress report. To start with, here is some code in action:

http://ueber.net/vac/bc32a6a6f09395131bec9f34d75e7cfda4b1ffc7/

This is an http interface to a venti server. The files are served by vacfs.b, which is a vac file server a lot like vacfs from Plan 9. The difference is that this one can serve arbitrary vac root scores: The root directory appears empty. Walking to a score tries to open that score as a vac root score and serves the contents of that vac. The current (work in progress, with at least one severe bug, be warned!) vacfs.b can be found in the vac archive referenced above. As a sidenote, the http interface is a scgi program that reads directories and prints out html.

There are some more interesting bits in the archive. But note that these programs are not final, they may be removed/replaced/redesigned later on. And when I put this in the mercurial repository, I will also make a proper directory hierarchy and mkfile. Here is only a summary of the somewhat working programs in the archive. They also have manual pages which have near-complete usage information. The limbo source files contain lists of work to do, and some of the manual pages have lists as well.

Vacfs, as described above, it can connect to a venti server and serve any score present at that server. A specific score can also be specified at startup which will be openend and it's contents served in the root directory.

Vacget, lists of retrieves the contents of a venti archive. -t just lists them, -x retrieves them.
Vacput, write a file tree to venti. The resulting vac root score is printed to standard out. The tree is always written entirely, i.e. it cannot write only changes relative to a previous archive.

Vcache, a venti cache and/or proxy. It can serve as memory-cache, keeping venti blocks in memory. Requests for non-present blocks are sent to the authorative venti server. A proxy can also be specified: Before reading data from the venti server, the proxy (a venti server, typically one nearby) is queried for the data. Data received from the authorative venti server is written to the proxy server. The proxy can also be used as write-through server: all writes (that normally go only to the authorative server) go to the proxy simultaneously, and the operation succeeds only when both servers respond okay.

Vread and Vwrite, read a single score or write a single data block from/to a venti. Mostly for debugging.
Vparse, parses vac root scores, metablocks, entries, etc.
Ventry, parses and prints a given entry in a file with many entries.

If you want to play around with some of this code, you also need these changes to inferno's appl/lib/venti.b:

http://ueber.net/vac/78d72a3daf13c934f4b161e0268feffefc232f7d/venti.b.diff

That's it for now. Next on the todo list is setting up the hg repository, and working on the venti server.

Updated information (2007/06/05)

Hello there!

I've been updating the project pages. Take a look at them... What is missing? Any comments or questions you can find me on irc.freenode.net.

Cheers!

On Ken (2007/06/05)

The Plan 9 compilers present an interesting challenge in porting. Because Plan 9 was, and is, a research operating system, the architects were allowed to throw away convention, and implement things in the best manner they were able. For the compilers, this has manifested in a number of ways. The most visable to the users is that ANSI C has been both extended and restricted, in ways which Plan 9 programmers generally find agreeable. But, for the compiler architect and porter, there are obvious, larger changes, which present obstacles to porting.

To begin with, the system does not use the standard stack semantics of any given architecture. This used to read that, on Plan 9, the stack always grows away from 0, as I was misled to believe. Actually, on all architectures on Plan 9, the stack grows towards the lower address range, i.e. 0 Arguments and local variables are accessed at known offsets from the stack pointer, and the base register remains free for the compiler to use. While there is much to appreciate about this convention, in order to interoperate with code generated by Linux and BSD native compilers, the compiler must follow the same stack and calling conventions. This may seem like a simple problem to solve, but there are exacerbating factors. For one thing, the compiler loses a register from its normal complement.

So, how do we deal with this? Well, to begin with, it helps to have a working compiler. The idea is to initially link together only code generated by 8c, and load it into memory with a special purpose loader, which preloads a function to map calls to it, using the Plan 9 convention, into system calls, using the BSD convention. Once the compiler this works, we begin transforming the calling convention, while trying to maintain a working compiler. Initially, this means freeing EBP (the aforementioned base pointer register) for use in stack frame generation. From there, work progresses on achieving compatible calling conventions, while mantaining a working compiler all th way.

But, that's not the only area where the compilers diverge from convention. The C compiler, itself, handles, in one program, the preprocessing, lexing, machine code generation, and initial optimization, though it is clearly divided between machine independant and dependant arenas. The objects that it outputs are basically compressed, binary representations of assembly language. The loader processes these objects into machine code, subjecting code from the compilers and assemblers alike to global optimization in the process. This means that the compilers and loaders are intrinsically linked. The two must be used together. The main problem with this is that the design of the loader is not sympathetic to the cause of loading standard object files.

There are several ways to deal with this problem, and it's arguable which way is best. The first option would be to modify 8l to actually link ELF object files. This is not a particularly pleasant prospect, and is likely to yield few benefits. The second is to have 8l output a relocatable ELF object to be linked by the system linker. This would slow things down, and complicate them a bit, but would not pe particularly difficult. The third, which is likely to be the first implemented, is to only link ELF libraries dynamically. The loaders and compilers already been modified to be useful in dynamic linking, but only when used with Plan 9 a.out objects. Modifiying them to generate dynamic ELF executables and shared objects is likely to be fairly painless and achievable.

These are certainly not the only compatibility issues, but they present the general flavor of the problem. It should be fun...

Here's to a sane compiler on Lunix, Kris Maglione

Getting Started (2007/06/05)

Hello everyone! I am very excited to be working with plan9/inferno on such a unique project. There have been a number of ideas floating around with respect to the final goal of my project. I hope to hash out in the coming days what some of the discussed aspirations for the project are, and more importantly, what is most feasible keeping in mind the time that will be required to develop these ideas. In the future, the hope is that we will be able to tackle several different ideas simply because the modular aspects of this project lend themselves to connecting with one another in meaningful ways, using the power of the Limbo programming language, and the architecture of the Inferno OS.

New website code and rss feeds (2007/06/04)

Kris has been heavily reworking and cleaning up the code that runs this site (now can be found in the hg repo under werc) including implementing rss feeds for all blogs!

There have also been some minor improvements, like automatic blog setup for user and project blog directories and other minor fixes.

Dave Eckhardt proposed that we have a table of projects that tracks the last commit and last blog post for each project, a prototype of this can be found in /projects/table.

Enjoy and please report any issues or suggestions in #plan9.

Hello (2007/06/03)

For anyone interested, there's a blogpost script in /home/kris/bin which makes it easy to post to these blogs. The -u option sets the user (default `{whoami}), and the -t option sets the title (defaults to the first line of your post. If you give a filename (or - for stdin), that is used, otherwise, $EDITOR (default vi) is run.

Salutations,
Kris Maglione

Starting up (2007/06/02)

Hello world! I'll be posting my GSoC advances here. Stay connected...

More information about gsoc resources (2007/05/31)

I have created a page that documents the resources and services provided by gsoc.cat-v.org, please take a look and let me know if there is anything else you would like to see.

Mercurial web interface (2007/05/30)

I have setup the web interface for the mercurial repos, feel free to start pushing the repos for your projects under /gsoc/hg/.

Making progress (2007/05/30)

Have made some progress with the gsoc.cat-v.org site and setup, a handful of brave souls have got accounts already, and i hope that most of the work left is to add some real content (I hope others will help out with that ;))

While fixing a bug in the blog code (all six lines of it), I found a neat trick: Given a list of paths, how to list the full paths of all files in those dirs ordered by file name:

; l = ( /foo /bar/baz/bax /) 
; ls $l^'/./' | sort -t. +1

Enough for today, when I wake up I should announce the site to the gsoc list and try to setup an irc meeting for tomorrow so we can get things going. Thanks to everyone that helped debug and test things.

Object Oriented PHP Extensions (2007/05/30)

Before I begin, if you're using Gentoo Linux, don't forget to check out the plan9port ebuild I recently made.

The Summer of Code has officially started - and I will begin my blogging with a quick post on creating object-oriented extensions to PHP, which is what the first phase of my project is all about :)

There isn't much documentation on building OO extensions apart from the README series in the PHP source. If you're looking to build a conventional procedural extension, however, I've found that the 3-part tutorial series on Zend by Sara Golemon is most useful.

I've chosen the libixp (pronounced as lib9p, get it?) library to wrap over. libixp is a small and clean C library that helps you write 9P servers and clients and should serve as a great base for the PHP extension. Incidentally, libixp's current maintainer is Kris Maglione; one of my fellow SoCers! Anyway, the first thing I did was to run the ext_skel script for my new extension but quickly found out that you can't have PHP extensions that begin with a number (I wanted to name the extension 9P). Rinse and repeat with an extension name of 'ixp' instead. Now what?

Marcus Börger had presented a talk on "Implementing PHP 5 OOP extensions" at php|tropics, which, though a bit dated, served well as a guide. The bundled util extension is a good example of some OO code in action. However, as time progresses, I realized that creating a class and setting its properties in itself was getting very monotonous. Surely, this wasn't code that was meant to be typed in by humans; after all, why code when the computer can do it for you?

Yep, Code generation to the rescue! PEAR has a neat package called CodeGen_PECL that takes an XML description of your extension and generates not only the extension code, but also skeletal documentation to go along with it! No more worrying about naming conventions and other such menial stuff - you get straight to coding. Indeed, this is how programming should be. I will recommend this extension to anyone trying to build an extension to PHP, especially if you're creating an OO extension (given the lack of documentation and Zend's clunky object system for PHP5).

Now that I had a neat little skeleton of my extension (CodeGen_PECL indents and comments the code for you too) all I had to do was to begin actually implementing the methods. The client portion of it wasn't such a big deal, but when I started to code the classes that would help you create a 9P server, I hit a roadblock. libixp expects me to pass a bunch of function pointers, one each for every 9P operation (read, write, clunk, attach etc… more details in the 9P specification). I now have to think of some way to map that to the PHP way. Tricky, but the answer lay quite close to me.

I've been working with the PHP-GTK team for quite some time. In fact, it was my first experience with open-source projects in general. PHP-GTK is an OO extension that wraps over the Gtk+ toolkit. Gtk+ has a ton of functions that expect function pointers, their whole signal-callback system is built that way. PHP-GTK maps this behavior to the PHP way by creating a "marshaller" function for every type of function pointer, that expects the same arguments as the function pointer is given. While this marshaller function is passed to the C-level Gtk+ methods, PHP-GTK maintains a separate callback structure that stores the Zend object for the corresponding PHP-level function. Gtk+ (thankfully) provides an extra gpointer for the developer to pass around their own data to callbacks, and PHP-GTK uses this to pass around the callback structure. When the marshaller function is called, it simply reads the callback structure and invokes the PHP-level function with all the parameters that it received (after appropriate conversion of course).

Cool, I could do the same thing with libixp! Except libixp doesn't allow me pass custom data to callbacks:

typedef struct Ixp9Srv {
void (*attach)(Ixp9Req *r);
void (*clunk)(Ixp9Req *r);
void (*create)(Ixp9Req *r);
void (*flush)(Ixp9Req *r);
void (*open)(Ixp9Req *r);
void (*read)(Ixp9Req *r);
void (*remove)(Ixp9Req *r);
void (*stat)(Ixp9Req *r);
void (*walk)(Ixp9Req *r);
void (*write)(Ixp9Req *r);
void (*freefid)(Fid *f);
} Ixp9Srv;

Hmm. Luckily, I'm bundling the libixp sources with the extension; it's a -with extension, not a -enable one ;)
I guess I can hack up the libixp sources to allow for passing this extra data around to the callbacks. Let's see how this goes!

Until next time, Happy Hacking!

Hi there (2007/05/28)

Hullo! ^_^

Site is up (2007/05/27)

The first version of the Plan 9 GSoC site is up!

Things are still being worked on, and most people don't have their accounts setup yet. The mercurial repos and 9P access will be next.

Update: Per-project blogs webpages were also broken, I fixed them now, all blogs should be working properly now.