Say Hello to Angled 0.1 (2007/08/22)
I feel I’m in the seventh heaven. After a few sleepless nights struggling with Mozilla’s XPCOM, I finally got the 9P Firefox plugin to work.
The plugin is called Angled (an anagram of Glenda, the Plan 9 bunny) and is in a pretty simplistic state right now: you can read any files served by 9P right in your browser window. Let’s take a step by step look.
First I startup Inferno to start a 9P server. ${INFERNO}/usr/anant/home is symlinked to my actual home directory, /Users/anant:
[theghost anant]$ emu
; runas nobody {listen -A tcp!localhost!1564 {export /usr/anant/home &}}
Let’s see what files are actually there:
[theghost web9]$ pwd
/Users/anant/Plan9/web9
[theghost web9]$ ls
README TODO js9p php9p
Alright, I open my browser window and type ‘ninep://localhost!1564/Plan9/web9/README’ into the address bar. I could also say ‘tcp!localhost!1564′, but TCP is the only protocol available for Angled, so it would be redundant. Now, for the goodies: Screenshots!
Cool! But wait, Angled also displays binary files right in the browser. There’s a catch though, it will only work for binary files that can be viewed directly in the browser window. Certain types of files (.doc for example) do trigger a download request, but then become corrupted for some reason.
[theghost content]$ pwd
/Users/anant/Plan9/web9/js9p/angled/content
[theghost content]$ ls
angled.png firefoxOverlay.xul glenda-error.png overlay.js
Let’s say I want to view angled.png. Here’s what I get:
Okay, but what if you type in a URL that points to an invalid file? Check this out:
Sweet! I’m yet to figure out how to transmit the exact error message to that page, so you’ll have to make do with that generic image for now.
Okay, now onto the bad parts. Angled doesn’t support authentication yet (although the base JS implementation is capable of generating and parsing auth messages). Next, you won’t get directory listings (you’ll get a bunch of binary gibberish which is actually Rstat messages for the directory’s contents). Also, I’m doing the 9P connection and transactions in a blocking thread, so the UI freezes while all that is done. I couldn’t feel the difference since I was testing on my local 9P server, but connecting to remote 9P servers won’t be a pleasant experience. The solution to this is to create a custom nsIChannel implementation, which is a lot of work… I’ll do it when I get to it 
Enjoy!
Implementing a Protocol in Mozilla (2007/08/18)
Creating a Firefox extension is nothing short of an adventure. I was able to get started pretty quickly, thanks to this web-based quick-start wizard, all the boilerplate code was generated in literally no time.
Now, onto the actual functionality of the extension. I have to implement a protocol handler for the 9P protocol, which essentially means you type in “ninep://sources.cs.bell-labs.com/” and start reading files right off the browser window. (ninep:// because a URL can’t start with a number)
This page provides some useful insights and code snippets on the subject of adding a new protocol handler. I was able to get as far as displaying a Glenda image whenever you type in a URL beginning with ‘ninep’.
The way this works is you create an XPCOM component that implements a standard interface. Specifically, the newChannel() method is where all the action is. It receives a URL and you do something and return an nsIChannel. Mozilla provides standard nsIChannel implementations for popular protocols such as http, ftp and even the ubiquitous file://.
The intuitive thing to do here would be to do all my 9P processing in the newChannel() implementation and return a stream in a standard channel. However, that’s not going to work, since newChannel() would then block and the UI would actually freeze until the 9P transaction completes. Sub-optimal.
The “proper” way to do this would be to create my own implementation of nsIChannel. That way I just create a new nsIChannel in newChannel() and be on my way. nsIChannel would then take care of firing callbacks as and when data arrives. There’s somewhere I can start with, and that’s the Mozilla implementation of the finger protocol. It’s written in C++, however, and I need to figure out how I can map the same to JavaScript (via XPConnect).
To block or not to block (2007/08/16)
There’s one last hurdle before I finish my GSoC project. I’ve already written the JavaScript that produces binary messages for every possible 9P transaction, and all that needs to be done is to actually send those messages to the 9P server.
In a few of my previous posts, I mention that there were two apparent ways to do this:
a) Send the message wrapped in an XMLHttpRequest to a HTTP server that forwards the message to the actual 9P server.
b) Use Mozilla’s XPCOM components to access Sockets directly in JavaScript.
Well, it turns out that (a) is probably not a solution at all. HTTP is far from what one would call a protocol that supports streaming. A 9P server (correctly) doesn’t return an EOF until you have completed your whole session. So the first time I send an XMLHttpRequest to, say a PHP script, the script blocks forever, since PHP would never know when the first R-message has been actually sent through. I can always peek at the first 4 bytes and find out the length of the R-message, but what then? I can’t close the socket since that would terminate the 9P session, but I have to return from the script for the HTTP response to be actually sent.
PHP doesn’t support threading (proper) so I can’t do the select() mojo either. How about storing the socket FD in the session variable? Well, this is probably the closest to a good solution but that would limit every client to exactly one 9P session.
Although Mozilla’s XPCOM is one hell of a beast, I think it might be good to just build a firefox extension to access 9P resources. Not exactly sure of how I’m going to make it work, but tentatively, I’m thinking of parsing URIs beginning with 9p:// or something like that. Let’s see how this goes.
Fiddling with Binary in JavaScript (2007/08/13)
Since ordinary JavaScript cannot directly communicate with a 9P server (over TCP), we decided to go in for a 2-tier approach: A script on the client generates a 9P message which is sent to a server via an XMLHttpRequest. The server then forwards the message to the actual 9P server. Messages from the 9P server to the client are sent in a similar fashion.
All 9P messages are just binary sequences, which means I need some way of representing a 9P message in JavaScript. A character is always 1 byte, so representing characters is not a problem. For representing integers in binary I use the following snippet of code:
while(num) {
str[str.length] = String.fromCharCode(num % 256);
num = Math.floor(num / 256);
}
where ‘num’ is the number to be encoded, and str is returned as: str + join(“”).
This means I can now encode any 9P message as a simple sequence of JavaScript strings. The final string can then be sent along with the payload of an XMLHttpRequest. I’m wondering whether it will be a good idea to encode the string in Base64 first, although an XMLHttpRequest should have no trouble with the string representation either.
Swapping Endian-ness in JavaScript (2007/08/09)
My mentor came up with two interesting ways of swapping endian-ness on JavaScript. The first one he proposed was based on what was “usually done in Plan 9″, something along the lines of:
b = "\1\1\1\2";
n = (b.charCodeAt(0) & 0xff) << 24;
n += (b.charCodeAt(1) & 0xff) << 16;
n += (b.charCodeAt(2) & 0xff) << 8;
n += (b.charCodeAt(3) & 0xff);
…which gives us n = 16843010.
Maht then had a look at the series of JavaScript lectures by Douglas Crockford at Yahoo!. The first of the series tells us that bit shifting is not faster than simple multiplication. Maht gave me this code snippet doing the same conversion, but using multiplication instead of bit shifts this time:
b = "\1\1\1\2";
n = b.charCodeAt(0) * 16777216;
n += b.charCodeAt(1) * 65536;
n += b.charCodeAt(2) * 256;
n += b.charCodeAt(3) * 1;
n, is of course 16843010; the real question is how much longer (or, shorter) did this take.
Venkman is probably one of the more mature “old-school” JavaScript debuggers out there. FireBug, the relatively modern sibling to Venkman certainly has a few nifty features, but profiling is not one of its strong points. After failing to profile the script in FireBug, I used the trusty old Venkman - and it came up with some interesting results:
Venkman Profile Report
Created ………. Thu Aug 09 2007 20:18:54 GMT+0530 (IST)
User Agent ……. Mozilla/5.0 (Macintosh; U; Intel Mac OS X; en-US; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6
Debugger Version . Venkman 0.9.87 [Mozilla rv:1.8.1.6/2]
..
Function Name: multi (Lines 1 - 6)
Total Calls: 1 (max recurse 0)
Total Time: 0.21 (min/max/avg 0.21/0.21/0.21)
Time (ex. calls): 0.21 (min/max/avg 0.21/0.21/0.21)
..
Function Name: shift (Lines 8 - 13)
Total Calls: 1 (max recurse 0)
Total Time: 0.02 (min/max/avg 0.02/0.02/0.02)
Time (ex. calls): 0.02 (min/max/avg 0.02/0.02/0.02)
As you can tell from the function names, multi uses simple multiplication, while shift uses bit-shifting. Turns out that bit-shifting does indeed take a lot lesser time. A Firefox quirk?
Mapping 9P to REST (2007/07/26)
Now that the PHP bindings to libixp are somewhat usable, I’ve moved on to the JavaScript portion of my project. The first (and easier!) part of it is to map 9P to a RESTful scheme, so traditional web developers can use 9P without having to learn anything new.
The PHP bindings to libixp was an important pre-requisite to achieve this: presenting a REST interface to an existing 9P serve would require a “bridge” at the middle, to convert REST requests to 9P requests and vice-versa with responses. This “bridge” may be present at any location that is mutually accessible by the client wanting RESTful access and the server providing the 9P service. This bridge can be easily coded in PHP using the new libixp bindings.
To those not very familiar with REST, it is simply a way of accessing resources using plain-old HTTP. It’s become quite popular with web developers these days, as a much simpler alternative to SOAP. A lot of web services these days are RESTful, including those offered by Amazon and Yahoo.
From the client’s perspective, accessing a 9P resource boils down to 4 things: reading, creating, modifying and deleting. These operations map neatly onto the GET, PUT, POST and DELETE HTTP requests, respectively. And thus, we have our REST URI scheme. This scheme would be enough if my bridge exposes only a single 9P serve as a REST service. As an example, suppose I start a bridge at http://plan9.kix.in/rest/ that exposes only the tcp!sources.cs.bell-labs.com!564 9P serve, to read the file named ‘lsr’ I would perform a GET request at http://plan9.kix.in/rest/lsr.
However, the plot thickens when I want to design a bridge that allows access to any 9P service (which is definitely better). Now we need to encode the information represented in Plan 9 as: tcp!sources.cs.bell-labs.com!564 into a HTTP URI. The intuitive thing to do would be something like: [ROOT]/[PROTOCOL]/[9P-URI]/[PORT]/[FILE-PATH].
Hmm. That leads to really long URI’s like http://plan9.kix.in/rest/tcp/sources.cs.bell-labs.com/564/lsr. Besides that, there are several things that need to be worked out. What happens when you do a GET on a file that is actually a directory? What about parameters to GET that you usually pass to a read() function: Suppose you want to read the first 1024 bytes of a file only?
Comments and Suggestions welcome 
P.S.: Thanks to some pointers by Kris, the PHP9P client example shown in the previous post now handles binary files correctly.
Herald the PHP 9P Client (2007/07/09)
I’ve given a few final touches to the 9P client for PHP. All the basic stuff should work (if they don’t please let me know!), and I’ve written a script that shows some of the basic functionality that the client offers. Two actually, this one for the CLI SAPI and this one for the Apache2 SAPI. All they do is read files and show directory listings, but the scripts are a good place to see what the API is like, since there’s no official documentation yet. I couldn’t locate a proper PHP 5 server to actually host the second example, which would have been cool… but now we’ll just have to make do with these screenshots I took off my browser (with the script running on my local Apache):
Once you’ve put in the address, you’ll get a directory listing of the root:
While the CLI SAPI demo script also handles binary files quite well, this one will just send gibberish to your screen if you click on a binary file (text files work fine though). I’m still trying to figure out what’s the best way of determining the MIME type of a file so I can send the appropriate HTTP header before transmitting the data itself, so your browser would know how to interpret it. As for the server-side of things, the code is still in a state of flux and I’m not decided on what kind of API to offer. I’ll probably take a break from this part of the project and move on the JS bindings, and then come back to tie this up in the end. That’s not to say that the server-side code is not usable, just that it’s not very developer-friendly; feel free to play around with it if you’re feeling adventurous
Some more on Object Oriented PHP Extensions (2007/07/06)
Hopefully this is the last of the series on Object Oriented PHP Extensions. We left the last post with a question of how to create objects of a certain class in a method of another. This is especially relevant to the libixp extension, since classes like IxpStat, IxpQid and IxpCFid don’t have any constructors of their own, but are rather returned by methods of the IxpClient class. For example, an open() on IxpClient returns a IxpCFid, and the IxpQid is a property of the IxpStat and IxpCFid classes.
How do we get this to work? The helpful folks at #php.pecl on EFNet pointed me to a few examples in the existing PHP codebase that do this. First, you need to initialize an object - which essentially involves:
- Calling ALLOC_ZVAL on the zval.
- Setting the type to OBJECT using Z_TYPE_P.
- Calling object_init_ex(), which in turn will call the corresponding create_object method for the class entry.
- Setting the object’s refcount to 1.
Once you’ve done this, most often you’d need to set up certain properties on the class object. In libixp’s case, we need to populate IxpStat’s, IxpCFid’s or IxpQid’s properties. You can create a method to populate each classes properties. I use a set of PROP_SET macros to update properties correctly. Check out the code to clear things up - object_instantiate() is the generic method for allocating memory for a object, and the PHP_initialize() set of methods set properties for particular classes. These functions are used in almost all of IxpClient’s methods.
Before signing off, another quick word about using properties in your classes. If your class doesn’t use any properties, you can get away without doing any memory allocation at all, see the FliteTTS extension for an example of this. However, if you do plan on using properties, you need a bunch of methods for allocating memory to the HashTable that will hold the properties, as well as the structures that your class will use. You can create a generic method create_object_ex() that does this, and then create two more methods create_object_new() and create_object_clone() that will act as handlers for new object creation as well cloning. These handlers are set during the class initialization (most probably located or called in PHP_MINIT) by setting the create_object member on the class entry struct and the clone_obj member on the classes' standard object handlers.
More on Object Oriented PHP extensions (2007/06/13)
In the last post, I mentioned that the libixp sources may have to be modified to work around callbacks, but Kris quickly pointed out that there’s a special aux element meant specifically for developers to put whatever they like into. The aux element is automatically passed to every callback function via the Ixp9Req struct:
struct Ixp9Req {
Ixp9Conn *conn;
Fid *fid;
Fid *newfid;
Ixp9Req *oldreq;
Fcall ifcall;
Fcall ofcall;
void *aux;
};
The aux element is created during the call to ixp_listen:
IxpConn *ixp_listen(IxpServer *s, int fd, void *aux, void (*read)(IxpConn *c), void (*close)(IxpConn *c));
Now how do we wrap this functionality in PHP land? The aux functionality must be passed on the 9P developers using PHP, while still being able to extract the zval that we need to invoke the appropriate callback. Thanks to the marshaller method, we can mangle struct values as much as we like before passing the values into their PHP counterparts. We simply create a struct to wrap both our callback values and the aux element from the user:
typedef struct _IxpAux {
IxpCallback *cb;
zval *aux;
int type;
} IxpAux;
In the marshaller, we simply extract the aux value out from the structure, convert it to a regular PHP-type and pass it along to the PHP-level function using Zend’s call_user_function_ex() method. That problem solved, onto the next! Creating objects of a class in a method of another class - the IxpCFid class has no constructor, but an object is returned by calls to an IxpClient object. More on that later!
Object Oriented PHP Extensions (2007/05/30)
Before I begin, if you’re using Gentoo Linux, don’t forget to check out the plan9port ebuild I recently made.
The Summer of Code has officially started - and I will begin my blogging with a quick post on creating object-oriented extensions to PHP, which is what the first phase of my project is all about
There isn’t much documentation on building OO extensions apart from the README series in the PHP source. If you’re looking to build a conventional procedural extension, however, I’ve found that the 3-part tutorial series on Zend by Sara Golemon is most useful.
I’ve chosen the libixp (pronounced as lib9p, get it?) library to wrap over. libixp is a small and clean C library that helps you write 9P servers and clients and should serve as a great base for the PHP extension. Incidentally, libixp’s current maintainer is Kris Maglione; one of my fellow SoCers! Anyway, the first thing I did was to run the ext_skel script for my new extension but quickly found out that you can’t have PHP extensions that begin with a number (I wanted to name the extension 9P). Rinse and repeat with an extension name of ‘ixp’ instead. Now what?
Marcus Börger had presented a talk on “Implementing PHP 5 OOP extensions” at php|tropics, which, though a bit dated, served well as a guide. The bundled util extension is a good example of some OO code in action. However, as time progresses, I realized that creating a class and setting its properties in itself was getting very monotonous. Surely, this wasn’t code that was meant to be typed in by humans; after all, why code when the computer can do it for you?
Yep, Code generation to the rescue! PEAR has a neat package called CodeGen_PECL that takes an XML description of your extension and generates not only the extension code, but also skeletal documentation to go along with it! No more worrying about naming conventions and other such menial stuff - you get straight to coding. Indeed, this is how programming should be. I will recommend this extension to anyone trying to build an extension to PHP, especially if you’re creating an OO extension (given the lack of documentation and Zend’s clunky object system for PHP5).
Now that I had a neat little skeleton of my extension (CodeGen_PECL indents and comments the code for you too) all I had to do was to begin actually implementing the methods. The client portion of it wasn’t such a big deal, but when I started to code the classes that would help you create a 9P server, I hit a roadblock. libixp expects me to pass a bunch of function pointers, one each for every 9P operation (read, write, clunk, attach etc… more details in the 9P specification). I now have to think of some way to map that to the PHP way. Tricky, but the answer lay quite close to me.
I’ve been working with the PHP-GTK team for quite some time. In fact, it was my first experience with open-source projects in general. PHP-GTK is an OO extension that wraps over the Gtk+ toolkit. Gtk+ has a ton of functions that expect function pointers, their whole signal-callback system is built that way. PHP-GTK maps this behavior to the PHP way by creating a “marshaller” function for every type of function pointer, that expects the same arguments as the function pointer is given. While this marshaller function is passed to the C-level Gtk+ methods, PHP-GTK maintains a separate callback structure that stores the Zend object for the corresponding PHP-level function. Gtk+ (thankfully) provides an extra gpointer for the developer to pass around their own data to callbacks, and PHP-GTK uses this to pass around the callback structure. When the marshaller function is called, it simply reads the callback structure and invokes the PHP-level function with all the parameters that it received (after appropriate conversion of course).
Cool, I could do the same thing with libixp! Except libixp doesn’t allow me pass custom data to callbacks:
typedef struct Ixp9Srv {
void (*attach)(Ixp9Req *r);
void (*clunk)(Ixp9Req *r);
void (*create)(Ixp9Req *r);
void (*flush)(Ixp9Req *r);
void (*open)(Ixp9Req *r);
void (*read)(Ixp9Req *r);
void (*remove)(Ixp9Req *r);
void (*stat)(Ixp9Req *r);
void (*walk)(Ixp9Req *r);
void (*write)(Ixp9Req *r);
void (*freefid)(Fid *f);
} Ixp9Srv;
Hmm. Luckily, I’m bundling the libixp sources with the extension; it’s a -with extension, not a -enable one
I guess I can hack up the libixp sources to allow for passing this extra data around to the callbacks. Let’s see how this goes!
Until next time, Happy Hacking!