User:marc on SWiK

PHP QuÃ©bec talks

Wed, 12 Mar 2008 10:43:31 -0700

This year, as part of my annual trip to Canada and the USA, I’ve been asked to give two talks at the annual PHP Québec conference in Montréal. I haven’t been back to that city since 1993 when I graduated from University, and it will be interesting to see how it goes. (Although I suspect that while Beijing basks in nearly 20C (nearly 70F) weather every day and even Seattle and New York were closer to 10C (50F), Montréal is still hanging below freezing most days and has over a metre of snow on the ground).

I will be giving talks on internationalisation (commonly just called i18n) and giving your database servers a break with memcached. If you’re anywhere in the neighbourhood, come on by for some good fun. I’ll be getting back to regular programming content this weekend.

The slides for my presentations are here:

memcached (FIXED)
i18n

[Sorry for the delay in fixing the memcached link – I have had a severe flu for the last few days. It should be okay now.]

Fifteen years in the making - Nethack Ascension Report

Tue, 08 Jan 2008 22:16:32 -0800

This is a bit of a geeky post, but then this is a geeky computer blog, so … so be it. Regular readers of this blog will know that I’m a fan of Nethack, and have been playing it on and off for over fifteen years (mostly off, but recently I’ve rediscovered it again). Well, finally, after all this time, I can say the following:


Goodbye marcw the Demigod...

You went to your reward with 5941448 points,
The Book of the Dead (worth 10000 zorkmids and 25000 points)
Vorpal Blade (worth 4000 zorkmids and 10000 points)
The Heart of Ahriman (worth 2500 zorkmids and 6250 points)
The Bell of Opening (worth 5000 zorkmids and 12500 points)
The Candelabrum of Invocation (worth 5000 zorkmids and 12500 points)
       8 emeralds (worth 20000 zorkmids),
       2 diamonds (worth 8000 zorkmids),
       2 rubies (worth 7000 zorkmids),
       1 amulet of ESP (worth 150 zorkmids),
       1 amulet of unchanging (worth 150 zorkmids),
and 2497 pieces of gold, after 93897 moves.
You were level 22 with a maximum of 95 hit points when you ascended.

 No  Points     Name                                                   Hp [max]
  1    5941448  marcw-Bar-Hum-Mal-Neu ascended to demigod-hood.        95  [95]
  2    2623722  marcw-Val-Hum-Fem-Neu died on the Plane of Fire.
                Dissolved in molten lava (with the Amulet).           129 [287]
  3    1330849  marcw-Val-Hum-Fem-Neu choked on her food in Gehennom
                on level 33.  Choked on a disenchanter corpse.        248 [248]

Interestingly, I almost always play Valkyries, but decided to try Barbarians for a couple of games. The first game, I made it all the way down to level 24 without finding a single altar (except for a non-aligned one in the mines with a nasty priest next to it) before an Arch-Lich and Titan finished me off. The second game was this one. Most of the rest of the time, I do embarrassing things like choke on things or eat something I shouldn’t have. Need to be more careful, I suppose.

Well, that’s all there is to this post, but it’s exciting news for me. I’m still playing, and now trying other classes (read: dying a lot).

End of an Era

Wed, 18 Jul 2007 18:35:00 -0700

For the last 17 years or so, I’ve been a huge fan of the various BSD-inspired operating systems, starting with SunOS 4.1.x, and then moving on towards the various free flavours available for the PC, such as Bill Jolitz’s 386bsd, then FreeBSD, NetBSD, and OpenBSD. For a while, even I was a regular contributer to the NetBSD community, and enjoyed playing with them all.

When I started installing and running my own servers for mail and web application purposes about 8 years ago, there was an abortive few-month attempt to use Microsoft Windows Server, but since then it’s all been FreeBSD, with the latest lanfear.com server being FreeBSD 4.9-RELEASE (and with an uptime of 2 years, which would have been nearly 3 had my ISP not hacked and rebooted my machine one day).

To this date, various SYSV-inspired features, such as initd and their directory structure leave me with a bad taste in my mouth. I have repeatedly stuck with types of linux such as SuSE (before it sucked), and Ubuntu, that still gave me /etc/rc and familiar directory structures. Mac OS X still gives me warm tingly feelings to this day.

So, it is with some sadness that I recently decided to move from my own dedicated server to a virtual server hosting solution. I’m simply never in the USA any more, and I don’t want to have to worry about my computer going down. A virtual server comes with a guarantee that all hardware problems are the ISPs, and is a bit cheaper to boot. I usually hover around a 0.00 load average, so serious computing power isn’t a necessity for me.

However, the cheapest package with the best bandwidth means my server will, henceforth, be running Ubuntu Server. It is reasonably familiar to me … I can still add things to /etc/rc.local, and the rest of /etc isn’t too alien, and the apt-get scheme seems to work reasonably well. My needs are less these days, as I slowly admit defeat in the email world and let people like Google do it for me, so as long as I can run web apps and a few other fun things, I’m happy. All of my sites and addresses have already been moved to the new server.

The old FreeBSD 1U Dell server will be shut down by the old ISP on Thursday, and put in a box for a friend to go pick up sometime after that. I’ll miss it.

Things I've learned about CoreImage (and Quartz, and OpenGL) in two weeks

Sun, 08 Jul 2007 19:57:11 -0700

I recently spent two weeks converting JustLooking, my Mac OS X Image Viewing program, from NSImage to CoreImage and friends. This experience was overall much easier than I expected, and I have learned a bunch of things, some of which might have been handy to have known in advance.

The good news is that it mostly lives up to the hype. The bad news is that it’s not without tricks and traps of its own. Here are some notes and comments.

Once set up, properly configured, coded, and tuned, CoreImage rocks. Whereas before 10fps NSImage transitions between medium images dragged on an Intel Mac, 30fps transitions on 1GHz G4s are smooth like butter.
Proper configuration is a must. I first just converted my various NSImage calls to their CoreImage counterparts (AppKit makes it easy to put CoreImage into your program), and saw maybe a 10-15% performance improvement.
Using CoreImage pretty much requires you to start getting comfy with OpenGL and its usage in Mac Applications. Instead of NSView classes, you need to use NSOpenGLView. Going to do drawing in the view along with your CoreImage? You’re going to need to learn how to do this in OpenGL (or, if you’re like me, find some samples that do the same thing you’re doing and template heavily).
This one took me the longest time to figure out: In Interface Builder, if your window has a subclassed NSView called MyNSView, simply changing MyNSView’s superclass to be NSOpenGLView instead of NSView and changing your code files likewise is not enough. My views were drawing all sorts of weird crap all over the place and it was driving me crazy. The part I was missing? You have to go back to IB
- Delete your NSView from the window.
- Drag on a new NSOpenGLView
- And finally mark its subclas as being MyNSView (which now inherits from NSOpenGLView instead of NSView).
NSOpenGLView is quirky. The weirdest of them all: It often doesn’t call your drawRect: when you first display it or tell it to change its contents. Thus, it seems as though a lot of people (myself included) have developed code that sets up a single-fire timer for some short period of time (0.1 seconds or so) to cause a redraw, which is respected.
I still cannot get an NSOpenGLView to be semi-transparent. It’s trivial to do this with an NSView—just call [[NSColor colorWithDeviceRed: 0 green: 0 blue: 0 alpha: 0.7] set]. In some situations, overriding - (BOOL)isOpaque might be required so you can return NO. Doing this in the NSOpenGLView, however, is a monumental effort, and something I have to figure out. I found a few examples on the Internet, but was never able to make one work. (If you have done this before, please please please send me mail or add a comment).
Any given CIImage instance you have is probably not an actual image. Instead, it’s a collection of instructions that the CoreImage code will later use to figure out the best way to render your final image to the screen, using all sorts of cool and clever optimisations. Applying resizing, scaling, and rotating transformations are all done at once, as none are actually computed until rendering time.
The filters you can use with CoreImage images are just awesome. From the banal like regular NSAffineTransforms, to impressive scaling filters, to all sorts of colour and distorting filters, you might be forgiven for thinking you could write you own little Photoshop clone in a few hours. The CILanczosScaleTransform filter, in particular, provides excellent results (sadly, it isn’t quite fast enough to use in real time).
The transitions available in CoreImage are all quite cool, and reasonably easy to use. Most simply require the setting of a few params and you’re good to go—one or two might require a mask image. The one thing the docs do not explain very well, however, is how to use the transitions. Oh sure, there’s a full page in the Programmer’s Guide, but it blithely leaves out a few of the key steps, leaving you scratching your head. Which brings me to my next point:
If you’re going to use a CoreImage transition, download and learn to love the CITransitionSelectorSample2 sample. Until you start playing with CoreImage (or have read this article), you might not have noticed that it uses an NSOpenGLView, that it fires a timer to make sure that NSOpenGLView always paints correctly, and more. And It shows very clearly how to use all of pre-canned transitions in Tiger.
The CoreImage classes are not without quirks, possibly bugs (although I’m such a newbie at this that I’m still willing to believe they were faults of my own). In my program I simultaneously do things like rotations, translations (so that images remain centred in the window), and scales to draw images on the screen. On a few occasions, I found that CoreImage would draw the images correctly for a few frames in a transition, and then suddenly start drawing them in unexpected places. Only by eliminating one of the transforms was I able to eliminate these problems.
The CIImage method imageWithContentsOfURL: is extremely useful for quickly loading in images, but won’t let you get at things like EXIF or other image meta-data. You’re far better off using CGImageSourceRef APIs to load in CGImages, and then pass these to the CIImage method imageWithCGImage:. These will even let you load in all the frames in an animated GIF (something which NSImage simply does not).
For thoses cases where you do want to use an NSView, such as printing, you’re still more than welcome to use CIImage objects. JustLooking does its printing in a regular NSView, and just uses AppKit extensions to the CIImage class to do the drawing. Extremely convenient!
This isn’t so much a CoreImage observation as much as an artifact of design changes I made in order to best use CoreImage: Drawing text in an NSOpenGLView is a monstrous pain in the butt. There are various classes and tutorials on how to do this, which is quite disheartening. I avoided tackling this issue and got around this by just using other NSViews.

In summary, I’m extremely glad I took the CoreImage leap. The system is extremely cool, and has pushed NSImage to a minor utility role in my programming world. If you do decide to go for it, take heart—it’s really not so bad, and internet searches will give you all the help you could want.

Website Design Gone Horribly, Horribly Wrong

Thu, 07 Jun 2007 19:27:35 -0700

There are lots of ways in which a website can be annoying. Favourite methods include: rotating and blinking animated GIFs (or worse, Flash), popup advertising windows, unexpected background music files, or just plain all around atrociously ugly page design. (I’ve been quite guilty of this in the past!)

But until you’ve lived in China, or at least spent some time browsing around websites here on the mainland, there’s probably one way to annoy the living bejeezus out of people that you’ve never thought of.

To demonstrate, simply visit any Chinese website, such as the Bank of China or something else such as Chinaren. Don’t worry if you can’t see the characters, they’re not important for this experiment. (Windows XP users can add them by going to Control Panel /International and installing the Asian Font Pack, while Vista and Mac users will have all these fonts installed already).

Once you have one of these pages up in your browser window, click on a link or two. Click on some more links on those pages. Try to get back to where you came from. Within minutes, you’ll have at least a dozen browser windows littering your desktop, or at best, for those Firefox users with the correct settings, dozens of tabs.

You could be forgiven for thinking that this was specific to a few sites with particularly bad design. And you’d be totally wrong. This is completely endemic here in local website design, and is how the locals think that the “Internets” should work. Indeed, there is almost no concept of forward or back button usage any more, and it is not uncommon to see users with well over twenty browser windows littering their desktop at any given time. While Windows users can at least expect the Task Bar to group similar windows, Mac users just end up using the mouse to move the windows out of the way until needed later, or until they just close the browser application completely.

Ultimately, the problem becomes such that, if you want to fix the site design to not do things this way, you will confuse your user. When they click to go to a new page, and they then subsequently finish visiting it, they will close the browser window and proceed to go looking through their other browser windows until they find the one from whence (they hope) they came.

The only thing I can say? At least blatent ripoffs of other sites on the internet don’t seem to have felt compelled to introduce this behaviour into their clones. For everybody else, it’s going to take a while to change this design.

When MySQL Attacks!!!

Fri, 20 Apr 2007 23:34:13 -0700

The Setup

Imagine, if you will, the following scenario:

You design a whole new database schema for your cool new scalable web-application. You’re using MySQL and the InnoDB datbase engine for everything, because your schema is so cool it uses all sorts of foreign keys and transactions and the like.
You quickly set up MySQL and get your application going with your new schema on your development staging machine.
You get MySQL up and running on your live server, play around with it for a bit to make sure it’s working, and then set up a my.cnf file with all sorts of caching and security goodies in it.
You do a backup from your dev machine, restore it to the live server, and ta-daa!!! Your web application is up and running on your live server.

What you might not have noticed, especially if you – like me – have a few thousands rows of data, is that MySQL might have screwed you along the way and not really told you all that clearly.

The Problem

After a few days of operation on my live server, I started to notice a few weird things—foreign keys weren’t being enforced properly, and there were some values in the database that probably shouldn’t have been possible. I furrowed my brows and put it on my list of stuff to investigate.

Well, yesterday, I added a new table to the database, and it went something like this:


mysql> CREATE TABLE Fudgecicles
>(
>  id INTEGER AUTO_INCREMENT PRIMARY KEY,
>  value VARCHAR(255) NOT NULL,
>)
>ENGINE = InnoDB;

Query OK, 0 rows affected, 1 warning (0.10 sec)

Where did that warning come from?


mysql> SHOW WARNINGS;

It is here that MySQL tells me:


+---------+------+-----------------------------------------------------+
| Level   | Code | Message                                             |
+---------+------+-----------------------------------------------------+
| Warning | 1266 | Using storage engine MyISAM for table 'Fudgecicles' |
+---------+------+-----------------------------------------------------+

Augh! No! Bad! Bad, MySQL, Bad! Why on earth would it do that? I didn’t misspell InnoDB or even use “incorrect” casing in the name. There’s nothing wrong with the schema I specified and I’ve done this hundreds of times before.

Well, after some research, I then tried the following:


mysql> SHOW ENGINES;

And MySQL helpfully gave me the following:


+------------+---------+----------------------------------------------------------------+
| Engine     | Support | Comment                                                        |
+------------+---------+----------------------------------------------------------------+
| MyISAM     | DEFAULT | Default engine as of MySQL 3.23 with great performance         |
| MEMORY     | YES     | Hash based, stored in memory, useful for temporary tables      |
| InnoDB     | DISABLED| Supports transactions, row-level locking, and foreign keys     |
| BerkeleyDB | NO      | Supports transactions and page-level locking                   |
| BLACKHOLE  | NO      | /dev/null storage engine (anything you write to it disappears) |
| EXAMPLE    | NO      | Example storage engine                                         |
| ARCHIVE    | YES     | Archive storage engine                                         |
| CSV        | NO      | CSV storage engine                                             |
| ndbcluster | NO      | Clustered, fault-tolerant, memory-based tables                 |
| FEDERATED  | NO      | Federated MySQL storage engine                                 |
| MRG_MYISAM | YES     | Collection of identical MyISAM tables                          |
| ISAM       | NO      | Obsolete storage engine                                        |
+------------+---------+----------------------------------------------------------------+
12 rows in set (0.00 sec)

The InnoDB database engine had been disabled somewhere along the way and I hadn’t even noticed. It is enabled, by default, on the standard Linux and Mac OS X MySQL binaries that I’ve been downloading. So something changed along the way that made this all stop.

Far worse, I then began to worry about my existing tables, all of which we supposed to be InnoDB. Upon executing the following for each of them:


mysql> SHOW CREATE TABLE Fudgecicles;

I would see something like:


| Table       | Create Table
| Fudgecicles | CREATE TABLE `Fudgecicles` (
  `id` int(11) NOT NULL auto_increment,
  `value` varchar(255) NOT NULL,
  PRIMARY KEY  (`id`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8

Sure enough, every single one of my tables was MyISAM, and not the InnoDB it was supposed to be.

The Reason

The problem turns out to be that InnoDB is somewhat finicky about my.cnf settings, and will frequently refuse to operate if settings change in this file in such a way that makes any existing data incompatible with the way it would write new data.

In my case, it turns out that changing the settings for the InnoDB binary data and log file sizes were somehow incompatible with the existing binary data and log files (ibdata1 and ib_logfileX). Upon starting the server, InnoDB finds this inconsistent state and simply refuses to start up. This is the first problem.

The second, and far more serious problem, is that MySQL just switches the database persistence engine on you and only provides a little warning. If you’re loading in thousands – if not tens of thousands – of rows, those warnings are easily lost in the scroll-a-thon that ensues. This is bad behaviour. MySQL should simply refuse to create your table if your selected persistence engine is not available.

The Solution

Fixing this problem involved three parts. The first, and easiest, is to get the InnoDB engine back. You have two choices:

You can either revert to the original settings you had for the my.cnf file to remove the inconsistent state that is freaking out poor ole’ InnoDB.
You can delete the idbdata1 and ib_logfileX files with the inconsistent sizes, thus removing in a different way the same inconsistent state, and allowing InnoDB to startup with your new (and presumably better) configuration values in my.cnf.

I chose the second method.

WARNING: Using this second method can result in data-loss if you’re not careful. Some database engines store things in the binary data and log files before writing them to the actual table files, and if they’re deleted, you might lose those changes. I only selected this path because all my tables were MyISAM, I had a full backup, and spent a good 10 minutes after deleting the files verifying that all data were correctly restored.

You then stop and restart the MySQL database engine, and InnoDB will be back. You can verify that all is well in InnoDB land by executing:


mysql> SHOW InnoDB STATUS;

And you will receive a gloriously long and detailed set of information on how things are going.

Part two of the process involved converting my tables back to InnoDB.


mysql>  ALTER TABLE FishSticks ENGINE = InnoDB;

This proves to be tedious and time consuming, as you cannot simply go from table A to table Z doing the conversion – because of the foreign keys and references, they have to be done in the right order. When one of the ALTER TABLE statements fails, you can find out what happened by re-executing the SHOW InnoDB STATUS command – it will tell you why it wouldn’t convert the table to InnoDB.

But, eventually, I got them all done. It was then that I noticed that none of the foreign keys were set up properly any more.

So, the last step of the process is to re-establish the FOREIGN KEYs. I did this by, for each foreign key in each table I had, executing the following commands:


mysql> ALTER TABLE FishSticks DROP KEY [dead foreign key name];
mysql> ALTER TABLE FishSticks ADD FOREIGN KEY (keyname) REFERENCES Table (fieldname);

The good news is that saving my database and getting back to all sorts of InnoDB goodness only took about an hour in total, for about 30 tables or so.

However, if MySQL had simply reported an error a few days earlier instead of blithely just switching tables types behind the scenes, I might have avoided this whole mess in the first place. Oh well, at least I learned a few neat little commands I can play around with now! Lesson learned!

Here’s to hoping that this article helps some other folks solve the same problem a bit quicker!

Setting up, Configuring, and Using Kannel to send/receive SMS messages

Fri, 13 Apr 2007 22:25:13 -0700

I recently had the oppportunity (necessity) to set up a web application that interacted with many of the users through SMS messages in addition to the more traditional HTML interface. While there are a number of possible software solutions for GSM modems on Windows, on Unix-like platforms the most commonly used one is Kannel. It also has the advantage of being open source and thus very, very free.

However, setting, configuring, and using Kannel tends to be a bit tricky. I’m writing this article (almost a HOWTO) in an attempt to help out anybody who’s undertaking the process themselves and might be able to get some tips and tricks from this. I expect this to not be a terribly popular article, but if I ever need to set this stuff up myself in the future, then I’ll have it written down somewhere at least!

Most of the instructions here will work on any Unix platform, such as Linux, FreeBSD, or Mac OS X. It’s worth noting that I got nearly everything working with OS X, only to be thwarted at the very end because Mac’s no longer have serial ports to use GSM modems. You could, however, easily use some of the more advaned HTTP based SMS services on a Mac server of some sort.

The Scenario

The web site written was a discounting service oriented around cell phones. Users would send the site a message, and then receive a discount at local venues. They could go to the site on the intarwebs to see their total savings and learn about more venues.

The usage pattern was always:

User sends us a message.
We might send them a reply after processing it.
We might also periodically need to be able to send the user a message for other purporses, such as party invitations, etc (based on their preferences).

Downloading and Compiling

Kannel is trivially easy to download and compile. You visit the Kannel.org website and download the latest and greatest gateway-1.X.Y.tar.gz file.

From there:


# mkdir src
# cd src
# tar xfz ../downloads/gateway-1.4.1.tar.gz
# cd gateway-1.4.1
# configure --prefix=/usr/local/kannel

I chose to install to /usr/local/kannel just because I’m that kind of guy who likes to keep everything reasonably separated and organised. You’re free to put it anywhere.

Compile and install.


# make
# sudo make install
password: **************

Configuration Files

You can be forgiven for thinking Kannel is trivially easy thus far—it really is that easy to download, compile, and install. Unfortunately, here is where things get tricky.

You now need to set up a configuration file. This file has a zillion options to support all of the possible and powerful ways in which Kannel can be used. I will be showing strictly how I set it up for the GSM modem we had in the office (It’s a Siemens GSM modem connected to a serial port, and works quite well).

The basic smskannel.conf (in the gw/ directory) has much of the information we want, but we’ll need to add a few things for our GSM modems and to interact with our web server correctly.

Configuration is divided into a few key groups, each representing the key parts of the kannel system, including the server that handles sending and receving the actual SMSes (bearerbox) and the system that handles the final dispatching to your scripts (smsbox).

The core Group

The first part of the file is the “core” group, and the default is pretty close to what we want:


group = core
admin-port = 13000
smsbox-port = 13001
admin-password = bar
#log-file = "/tmp/kannel.log"
#log-level = 0
box-deny-ip = "*.*.*.*"
box-allow-ip = "127.0.0.1"

You’ll want to change the password of course, but everything else is nearly standard. We are assuming that all communication to the kannel server will come from the same physical computer (127.0.0.1). You can set a log file if you are going to be running kannel as a service on your server, or you can just redirect stdout to some file.

Be aware that kannel has various log levels, ranging from 0, which displays information that is only of interest when you’re in the development and debugging phases, to 4, which only displays critical errors and problems. I tend to develop at level 0 and run live servers at level 1. Disk space is cheap.

The smsc Group

Kannel supports a pretty insane number of ways of sending and receiving SMSes, ranging from SMS services over HTTP, to a fake SMS centre for testing/development purposes, to GSM modems, which is what I have used and is the smsc module. These modems use AT-style modem commands and typically hook up over the serial port. To get this going, I set up the smsc group in the smskannel.conf file:


group = smsc
smsc = at
modemtype = auto
device=/dev/ttyS0
my-number = 123123123123
connect-allow-ip = 127.0.0.1
log-level = 0

The my-number field contains the number of your GSM modem’s SIM chip. Again, I only allow connections from my local server, and the Ubuntu Linux serial port is on /dev/ttyS0.

The smsbox Group

The smsbox group helps configure the part of the system that dispatches SMSes received by the core SMS or receives SMSes before they’re sent out. I honestly don’t fully understand what this group really does, but it’s necessary, and pretty trivial to set up.


group = smsbox
bearerbox-host = 127.0.0.1
sendsms-port = 13013
global-sender = 123123123123
log-level = 0

The global-sender field is the outgoing-number of your GSM modem, which for me is the same as the my-number field above.

The Sendsms Group

This group is what allows your web applications to send SMS messages using Kannel. They do this via simple HTTP requests, and configuration here basically requires a user name and password:


group = sendsms-user
username = kanneluser
password = df89asj89I23hvcxSDasdf3298jvkjc839
concatenation= true
max-messages = 10

Since the password is semi-plain and unprotected here, I tend to use one that is complicated and nearly impossible to remember, but quite different from any other passwords that I actualy use for login accounts and the like.

Getting the Messages to your application

The sms-service group configures how Kannel gets messages to your web application. You are allowed to specify a number of these groups, each of which can “catch” incoming messages based on various criteria. My application had all messages go to one processing script, so I just set up one group that caught all incoming messages.


group = sms-service
keyword =
keyword-regex = .*
catch-all = yes
max-messages = 0
get-url = "http://localhost/sms?phone=%p&text=%a"

This particular configuration has Kannel set up to use an HTTP GET request to send the message to my application. The param phone contains the phone number of the sender and the text parameter contains their entire message.

NOTE: The max-messages value was particularly tricky and critical for me: When I first set up Kannel and tested sending messages, I would always get back '<Empty reply from service provider>'. Setting max-messages to 0 tells Kannel to never send a reply directly from the incoming message (you can, of course, initiate your own response later, of course).

Finally, Setting up the modems

Kannel and smsc tends to be pretty good at figuring out everything about your modem by yourself, but you can help them out by including modems.conf in your smskannel.conf file as I did:


include = "/usr/local/kannel/modems.conf"

Running the Server

The hard part is done; all we have to do now is copy over the config files and start the service up:



# cd /usr/local/kannel
# cp ~/src/gateway-1.4.1/smskannel.conf .
# cp ~/src/gateway-1.4.1/gw/modems.conf .
# sbin/bearerbox -v 0 smskannel.conf &
# sbin/smsbox -v 0 smskannel.conf &

I tend to run the last two commands in two separate shell windows when developing/debugging so that I can see the output from the two programs clearly and use the information to help me figure out what’s going on (level 0 really tells you a lot).

Receiving Messages

Kannel will simply call the URL you told it to in the sms-service group and you can process this with whatever HTTP server environment you want. We’re using LAMP right now, but, again, any will do. The incoming phone number and message are in GET parameters. You can, if you want, configure the sms-service to send them as POST messages as well.

Sending Messages through Kannel

The final part our puzzle is to send outgoing SMS messages through Kannel, and has only one little twist. It is also done via an HTTP interface. It requires you to be a little careful about the character set you use. I found I had the most success by using the UCS-2 character set. In PHP5, you can easily use the iconv function to do this for you.

Since I send both English and Chinese messages, my PHP scripts and langugage string files are all UTF-8. Here is the code I use to send messages:



 function sendSmsMessage($in_phoneNumber, $in_msg)
 {
   $url = '/cgi-bin/sendsms?username=' . CONFIG_KANNEL_USER_NAME
          . '&password=' . CONFIG_KANNEL_PASSWORD
          . '&charset=UCS-2&coding=2'
          . "&to={$in_phoneNumber}"
          . '&text=' . urlencode(iconv('utf-8', 'ucs-2', $in_msg));

   $results = file('http://'
                   . CONFIG_KANNEL_HOST . ':'
                   . CONFIG_KANNEL_PORT . $url);
 }

To make this work, of course, you need to have allow_url_fopen set to On.

That’s It

That’s pretty much it. This has been a pretty dry article, but it does contain everything you need to get Kannel up and running and operational. The manual actually does contain everything you could possibly want to know, so keep digging in there if you’re stuck. Finally, there are mailing lists at kannel.org which tend to be quite helpful as well.

Good luck!

Why I love computers

Mon, 19 Mar 2007 11:51:59 -0700

I’m just beginning a short little contract here in China for a month or so to pay some bills and help a friend out. Part of doing work in the computer industry in China is that 1000$ USD for a computer is an outrageous sum of money. Those of us with Apples are either reckless spendthrifts or simply lucky beyond belief (Mac OS X itself is simply not understood … is that a program that runs on top of Windows?). So, this morning, when one of my coworker’s computer broke, I groaned. Not only was it likely to be a huge hassle, but the likelihood of having spare parts or another working computer were quite remote.

In short, he had no network. The cable lights simply did not come on. We switched cables. We unplugged and re-plugged in everything in the office. We uninstalled and re-installed drivers. We fiddled and diddled with Windows ad nauseum. We stole other departments’ switches. No joy.

Finally, in an act of desperation, I opened the computer up, tapped the network port aluminum housing a few times, looked sagely at at the current state of affordable PC hardware (it is pretty cool looking inside those things), and then tried again …. it works fine now.

What the—???

Weird Cocoa Errors 101

Thu, 08 Mar 2007 20:52:18 -0800

The other day, I was working on JustLooking, changing the appearance and the like of a couple of dialogs. As I ran the programs and went to show the dialogs (“panels” in the local terminology), nothing would show and I’d get an error in my XCode results window:

"Unknown class 'CustomCombo' in nib file. using 'NSObject' instead."

I spent the next half hour trying to figure out why Interface Builder didn’t know about this class: it was there in the class inspector, and the ui widgets were correctly set up to use that new class, and all the hookups in the UI also seemed correct.

Well, I finally figured out: when I first developed the class, I had some errors in it, but wanted to test a few things out in my program elsewhere. So I had unchecked it in XCode, telling the IDE not to compile and link it.

Thus, when Cocoa tried to load the NIB file, it couldn’t find the definition for the class, and just put in NSObject instead.

Here’s to hoping that this blog entry saves somebody that 30 minutes I spent on that one.

PHP Tricks and Traps II: Variable Expansion and Regular Expressions

Thu, 01 Mar 2007 19:17:48 -0800

Thanks to Keith from the UK for pointing out something odd in my book that doesn’t seem to work as it did in earlier versions of PHP:

If you have a regular expression (I use the POSIX ones almost exclusively since they’re UTF-8 aware whereas the Perl ones were not when last I inquired), and you want to set a range for the number of matches on a particular expression you can use the syntax:


$expr = '[a-zA-Z]{5,50}';        // matches between 5 (incl) and 50 (incl) letters

Now, the problem is: what if you want to have the number of characters in the range be PHP variables that you can set in a configuration file or some such thing? Your first attempt, and what I used in my book, might be:


$expr = "[a-zA-Z]\{$min,$max}";        // double quotes for var expansion

And you would get a wonderfully annoying error message from the PHP engine:


Parse error: syntax error, unexpected ',', expecting '}' in Filename on line 5

No amount of backslashes will fix this problem. It turns out that the PHP parser consumes { and } characters when performing complex variable expansion, so …. all you have to do is add an extra set around each of the variables you wish to expand. PHP leaves the other two alone:


$expr = "[a-zA-Z]{{$min},{$max}}";        // extra { }s are consumed.

And what you are left with is a wonderfully working regular expression.

Taking the Fun out of Buying a new Computer

Thu, 01 Mar 2007 19:17:48 -0800

In normal times, buying a new computer is a rather fun experience. In addition to the endorphin rush caused by plonking down a huge wad of cash for such a small – but often 好看 (that’s Chinese for good looking) – piece of hardware, you get to take it home and play with it and discover all the new and fun things that are different about the new toy. Even if it’s still running Microsoft Windows, your new vendor has probably come up with some new set of stuff to include with the machine. Or, far more common with me, my last attempt to finally come up with the “ultimate organisation of my hard disk™” was a miserable failure, and I’m excited about trying out something new.

So, imagine my disappointment when I recently upgraded my 15” Powerbook G4 to a shiny new 15” Macbook Pro.

In short, my upgrade experience was:

Plug in new computer, run through annoying Apple registration sequence (Tip: If you indicate in one of the first steps that the computer will not be connected to the Internet, you can make it so that the registration will not be sent).
Run migration wizard from the old Powerbook to the new Macbook Pro (this involves clicking a few buttons and connecting the two machines by FireWire).

Yes, about an hour later (35GB of iTunes and image data takes a while to transfer from one 5400RPM laptop drive to another), my new machine, appeared exactly the same as the old one. My desktop, my language settings, all my files, SSH keys, svn and source enlistments, and more.

What’s worse, all of the software on the old machine was Universal Binaries, so I didn’t even need to install any new software. It if weren’t for the fact that JustLooking compiled in less than 30 seconds instead of well over 2 minutes, I might not have even noticed right away.

I did also notice after a bit of experimenting that I can finally watch HDTV movies (720p and 1080i) at more than 2 frames a second, and that instead of being completley unable to play Civ IV, I can now play it at 1900×1200 with all features and whizzy 3D goodies turned on. I’ve also noticed that the fabled heat problems of the MBP have not materialised on my machine. After a copule of hours of gaming, the machine is no warmer than the G4.

However, the upgrade itself was a complete and utter disappointment. No temporary disruption in productivity, nor any hours spent trying to figure out how to migrate all of my data over. In short, no excuse to do anything other than just go back to work.

Thanks Apple. Jerks.

Rotating an NSImage object in Cocoa

Thu, 01 Mar 2007 19:17:46 -0800

I recently started working on an image viewing program for Mac OS X using Cocoa, and one of the features I decided to add was the ability to rotate images in 90° increments. I did some searching on the internet, and found a few things:

Some other people who were struggling with the same thing and posted some code snippets.
Code for a rotation function that supports arbitrary rotation and scaling.
Apple Documentation on Drawing

Neither of the first two was exactly what I wanted—the first didn’t quite work, while the second was too complicated and, in order to support arbitrary rotation, created an NSImage object that was way too large.

So, after reading the Apple Cocoa Drawing Guide on Using Transforms, I came up with the following code to rotate an NSImage object 90°. Please note that this function ONLY supports a clockwise or counterclockwise 90° rotation. It also returns an object that must be released by the callER. I’m not 100% sure if this is standard Objective-C/Cocoa methodology, as I’m still a bit new to all this stuff.

But, without any further ado, here is the rotation function:


- (NSImage *)rotateIndividualImage: (NSImage *)image clockwise: (BOOL)clockwise
{
    NSImage *existingImage = image;
    NSSize existingSize;

    /**
     * Get the size of the original image in its raw bitmap format.
     * The bestRepresentationForDevice: nil tells the NSImage to just
     * give us the raw image instead of it's wacky DPI-translated version.
     */
    existingSize.width = [[existingImage bestRepresentationForDevice: nil] pixelsWide];
    existingSize.height = [[existingImage bestRepresentationForDevice: nil] pixelsHigh];

    NSSize newSize = NSMakeSize(existingSize.height, existingSize.width);
    NSImage *rotatedImage = [[NSImage alloc] initWithSize:newSize];

    [rotatedImage lockFocus];

    /**
     * Apply the following transformations:
     *
     * - bring the rotation point to the centre of the image instead of
     *   the default lower, left corner (0,0).
     * - rotate it by 90 degrees, either clock or counter clockwise.
     * - re-translate the rotated image back down to the lower left corner
     *   so that it appears in the right place.
     */
    NSAffineTransform *rotateTF = [NSAffineTransform transform];
    NSPoint centerPoint = NSMakePoint(newSize.width / 2, newSize.height / 2);

    [rotateTF translateXBy: centerPoint.x yBy: centerPoint.y];
    [rotateTF rotateByDegrees: (clockwise) ? - 90 : 90];
    [rotateTF translateXBy: -centerPoint.y yBy: -centerPoint.x];
    [rotateTF concat];

    /**
     * We have to get the image representation to do its drawing directly,
     * because otherwise the stupid NSImage DPI thingie bites us in the butt
     * again.
     */
    NSRect r1 = NSMakeRect(0, 0, newSize.height, newSize.width);
    [[existingImage bestRepresentationForDevice: nil] drawInRect: r1];

    [rotatedImage unlockFocus];

    return rotatedImage;
}

The function works great and correctly rotates the incoming NSImage 90 degrees. However, People working with JPEGs should probably not save these data to disk in place of existing images, as the function works on the decompressed bits and resaving them can lose more image data. To truly solve this problem, you should use a “Lossless JPEG Image Rotation” algorithm.

Announcing Payjacks, an Object-Oriented PHP/Ajax Web Application Framework

Thu, 01 Mar 2007 19:17:46 -0800

I am happy to announce the immediate availability of Payjacks, currently at version 0.2.0. Payjacks is a PHP/Ajax web application framework I’ve written using the object-oriented features in PHP5+.

Payjacks can be downloaded here:

http://chipmunkninja.com/download/payjacks-0.2.0.tar.gz

What is Payjacks?

Payjacks is an object oriented PHP-Ajax web application framework I’ve written to help write robust and organised web applications. It was designed to require a minimal amount of effort to get your own web application up and running, while helping with such tasks as accessing a (MySQL, currently) database or providing a framework for sending asynchronous Ajax requests back to the server.

Payjacks uses many of the new object-oriented features in PHP 5 to do its work, and handles most of the details required to run a robust web application.

Some of the main features are:

Fully object-oriented.
Support for various types of Asynchronous Ajax requests, including submitting forms asynchronous or simply sending your own data strings asynchronously
Robust error handling and exception handling, allowing for customisable error pages and otherwise helping you help your users when things go wrong.
Support for URL processing. Thus, instead of having URLs such as http://mysite.blabhalblahblah.com/SomePageName.php?showarticle=getname you can have your site do things like: http://mysite.blhablahblah.com/showarticle/name. This helps make your application easier to use for power users and helps with search engine indexing.
The ability to create simple pages if you just want to test things out and don’t want the hassle of setting up a web application.
A full set of samples to show you how to set things up.
The framework has been designed from day one with web application security in mind.
Full support for UTF-8 and localisable web sites.

When should I use Payjacks?

I use Payjacks for almost all of my “web page writing” these days. You can use the SimplePage class to just throw out a single PHP/HTML page, or you can use the WebApplication class and template off one of the samples to create more robust web applications.

Since Payjacks takes care of so many of the details that many web application authors simply neglect (error handling, URL processing, security details), using it gives you nearly everything you want except for the actual HTML of your pages.

If you want to use asynchronous Ajax requests, Payjacks helps to make this very easy too.

What does Payjacks not do?

Payjacks does not help with generation of well-styled HTML or page content. It merely gives you the framework in which to place your content. There is nothing preventing you from designing and developing perfectly secure and robust yet hideously ugly web applications using Payjacks. It is for this reason that graphic designers earn their paycheques. Even my blog site, http://chipmunkninja.com is very rudimentary and clearly shows the limits of my design skills.

Payjacks Requirements

In order to run Payjacks, you need:

PHP 5.0 or later. I’ve tested on 5.0.[45] and 5.1.4. PHP needs to have mbstring and mbregex built-in for any sites not using strict Latin1 pages.
A web server. If you want this web server to support the web application system that Payjacks is built around, it must have some sort of URL re-writing capability.
On Apache, this is mod_rewrite, and can easily be compiled into your Apache build. I have tested and run this on Apache 1.3.xx and 2.0.yy.

For IIS, there are a number of mod_rewrite clones, ranging from very free and open source to very expensive and not open source.

Payjacks is currently version 0.2.0 and should be considered development quality only right now. I plan on making some minor changes to function signatures in the next few versions (mostly to get rid of some parameters that I thought I would need for localisation, but have since realised were the wrong way to go about the problem).

If you have any bugs, comments, or questions, please do not hesitate to contact me at marcwan@chipmunkninja.com.

When Apples go Bad

Thu, 01 Mar 2007 19:17:45 -0800

Last night, after a nice weekend of varying activities (yoga, studying, yoga, sleeping), the plan was to lie in bed and read some Apple Developer Connection documentation for a while and generally learn more about some stuff I’ve been working on lately.

So, it was with some dismay that I sat down with Samantha’s PowerBook G4, clicked on Safari and got … nothing. The icon bounced twice and then stopped. System logs showed nothing, and the application appeared to be all fine in its folder in /Applications.

Uuuuuh. Now what? Oh yes, the System Console, in /Application/Utilities. It gave me the following very helpful message:

2006-08-20 22:32:29.825 Safari[249] Unable to load nib file: MainMenu, exiting

For those not familiar with Mac OS X (or NeXTStep) development, NIB files are files that contain the structure and connections used by the various user Interface elements used by OS X Cocoa applications. These are usually only modified by the Interface Builder application used to develop applications.

So what was wrong with mine? Some searching around the Intarwebs suggested that it might be a permissions thing. So I fired up the Disk Utility and that said that a bunch of permissions were all wrong (how did that happen?) and fixed them.

Still no dice with Safari. There was another button on the Disk Utility to “Verify Disk”. That did not go well. Lots of red text and scary messages, and finally “exiting”.

So, in the end, I had to actually boot the PowerBook into single user mode (when the machine is booting up, hold down CMD/Apple + S), and run

/sbin/fsck -fy

to get everything working again. It took running this program twice to get everything fixed, and even then, one of the sub-nib files in the MainMenu.nib/ directory for Safari.app was still corrupted, and had to be copied from somewhere else. Luckily, we’ve got another PowerBook G4 here in Beijing, or I would have had to harrass somebody overseas for the file.

None of this is particularly new, shocking, or difficult to figure out, especially for those who have run various flavours of Unix for some time. I’ve just grown so used to my PowerBook being extremely reliable (in three years of owning a Dell Laptop running XP, I never once had half the average uptime I get on my PowerBoook) that it’s so shocking when there is a problem, I’m taken off guard.

There is an interesting point to be made here though, and that is that there no way Samantha would have been able to figure out and do this by herself. She still would have had to go to an Apple Geek Lab (or whatever they’re called) – of which there are none here in Beijing – and get somebody to look at it for her, or beg a friend (who might not have known what s/he was doing and suggested “wiping and reinstalling everything”). As nice as it can be to use, personal computing software still has a long way to go.

Program Execution in PHP: exec, system, passthru, and shell_exec, oh my!

Thu, 01 Mar 2007 19:17:45 -0800

PHP is a sufficiently rich programming environment that it is not common that I truly need to execute external programs on the server on which it executes. However, every once in a while, this situation does come along, and for these, it is important to understand the options that PHP provides, what their differences are, and their relative strengths and weaknesses.

There are four primary choices for executing external programs in PHP:

The system function.
The exec function.
The shell_exec function or its syntactic analogue, the backtick operator, ( ` ).
the passthru function.

There is, in fact, a fourth option in PHP 5, using a new set of functions all prefixed with proc_, such as proc_open and proc_close, but these are quite advanced, and beyond the scope of this article.

We will first discuss the usage of these three functions in their most basic forms.

The system() Function

The system function in PHP takes a string argument with the command to execute as well as any arguments you wish passed to that command. This function executes the specified command, and dumps any resulting text to the output stream (either the HTTP output in a web server situation, or the console if you are running PHP as a command line tool). The return of this function is the last line of output from the program, if it emits text output.

For example:


// Windows Users replace "ls ..." with "dir ..."
$lastLine = system('ls /Users/marcwan');
echo "<br/>LastLine: $lastLine<br/>\\n";

The output of this command on my system is:


Desktop Documents Download Library Movies Music Pictures
  Public Sites bin blah.php books make1.png make1.tiff media src
  yici1.png yici1.tiff
LastLine: yici1.tiff

The return value of the function will be FALSE if the function completely fails to execute.

In addition to the return value of the system function, however, there is also the question of the return status of the program being executed (just as functions in PHP all have return values (NULL if you don’t explicitly choose one), all commands executed in most operating systems also have return values, typically a 32bit integer value). For those programs you are executing whose return status you wish to know, you can pass a second argument to the system function, which tells PHP where to put this value.

For example:


// define this variable here so that it can actually be used in the system fn
$returnValue = -1;
system('ls /Users/marcwan', $returnValue);
echo "Return Value: $returnValue<br/>\\n";

The output of this program will be:


Desktop Documents Download Library Movies Music Pictures
  Public Sites bin blah.php books make1.png make1.tiff media src
  yici1.png yici1.tiff
Return Value: 0

For most modern operating systems, a return value of 0 is considered a success, and other values indicate failure. Some programs will have a range of values to indicate the exact way in which they failed while others will just return 1 or -1 to indicate that they were unsuccessful.

The exec() Function

The system function is quite useful and powerful, but one of the biggest problems with it is that all resulting text from the program goes directly to the output stream. There will be situations where you might like to format the resulting text and display it in some different way, or not display it at all.

For this, the exec function in PHP is perfectly adapted. Instead of automatically dumping all text generated by the program being executed to the output stream, it gives you the opportunity to put this text in an array returned in the second parameter to the function:


// you define this variable here so that it exists for the call to exec
$output = null;

// Windows users: 'dir c:\\' or something similar
exec('ls /Users/marcwan', $output);
echo "<pre>" . var_export($output, TRUE) . "</pre>\\n";

Once again, on my machine, the following output is printed:


array (
  0 => 'Desktop',
  1 => 'Documents',
  2 => 'Download',
  3 => 'Library',
  4 => 'Movies',
  5 => 'Music',
  6 => 'Pictures',
  7 => 'Public',
  8 => 'Sites',
  9 => 'bin',
  10 => 'blah.php',
  11 => 'books',
  12 => 'make1.png',
  13 => 'make1.tiff',
  14 => 'media',
  15 => 'src',
  16 => 'yici1.png',
  17 => 'yici1.tiff',
)

Of course, instead of just directly emitting this output, we could instead choose to process it and only emit those files with a ’.png’ extension, for example:


foreach ($output as $fileName)
{
  if (strtolower(substr($fileName, -4)) == '.png')
    echo "<br/>$fileName\\n";
}

And once again, you can get the return code from the exec function as follows:


$output = null;
$returnValue = -1;
exec('ls /Users/marcwan', $output, $returnValue);
echo "Return Value: $returnValue<br/>\\n";
echo "<pre>" . var_export($output, TRUE) . "</pre>\\n";

The shell_exec() Function

Most of the programs we have been executing thus far have been, more or less, real programs¹. However, the environment in which Windows and Unix users operate is actually much richer than this. Windows users have the option of using the Windows Command Prompt program, cmd.exe (See Figure 1). This program is known as a command shell.

Figure 1— The Windows Command Prompt

Similarly, Unix (and Mac OS X) users can run a command shell such as sh, bash, or csh (see Figure 2). These are typically quite advanced interactive systems that allow elabourate scripts such as for x in *.png; do echo $x; done to be executed.

Figure 2— bash on Mac OS X

On both Windows and Unix, the commands for the various command shells can be put into files and executed by these programs. On Windows, convention has it that these script file have the extension .cmd while on Unix it is not uncommon to see .sh.

For those sitautions where we want to run these shell scripts from within PHP, we can use the shell_exec function, as follows:



// lists all .bat files anywhere in the system
$output = shell_exec('FOR /R C:\\ IN (*.bat) DO echo %x');   // Windows

// lists all .png files in /Users/marcan only
$output = shell_exec('for x in /Users/marcwan/*.png; do echo $x; done'); // Unix

echo "<pre>" . var_export($output, TRUE) . "</pre>\\n";

On my system this produces:


'/Users/marcwan/make1.png
/Users/marcwan/yici1.png
'

PHP also has a syntactic operator called the backtick operator which does the exact same thing as the shell_exec function. Instead of calling the shell_exec function, you simply wrap the command in the backtick character, `, which is very different from the single quote character '.



// lists all .bat files anywhere in the system
$output = `FOR /R C:\\ IN (*.bat) DO echo %x`; // Windows

// lists all .png files in /Users/marcan only
$output = `for x in /Users/marcwan/*.png; do echo $x; done`; // Unix

echo "<pre>" . var_export($output, TRUE) . "</pre>\\n";

It is worth noting that I avoid this operator at all times as it is not easy to spot in code, and I like potential security trouble spots to be as visible as humanly possible.

The passthru() Function

One fascinating function that PHP provides similar to those we have seen so far is the passthru function. This function, like the others, executes the program you tell it to. However, it then proceeds to immediately send the raw output from this program to the output stream with which PHP is currently working (i.e. either HTTP in a web server scenario, or the shell in a command line version of PHP).

When would this be useful? When you are working with image files and want to execute a program to show or otherwise manipulate these files. A most simple example would be as follows:


/**
 * Windows:
 */
header('Content-Type: image/bmp');
passthru('type c:\\Windows\\zapotec.bmp');

/**
 * Unix
 */
header('Content-Type: image/jpg');
passthru('cat /Users/marcwan/yici1.png');

The result will be an image in your browser. This is extremely powerful if you want to use some programs to resize or otherwise manipulate image files and then just dump the output to the client. There are a number of interesting utilities that come with various JPEG source code distributions that do exactly this and are powerful tools.

Are We Done Yet?

Armed with this basic knowledge, we have a one major thing left to deal with before we go off into the real world and begin using these functions: Security.

Allowing users to specify any portion of the string you pass to the exec, system, shell_exec, or passthru functions is a huge security risk, and you need to plan carefully in advance before allowing any of this.

For example, the following code is simply a recipe for disaster:


$output = shell_exec('cat /webserver/info/user_files/' . $_POST['user_info_file']);
echo $output

If a malicious user were to set the value of the user_info_file post variable to bob; cat /etc/passwd. The output of both files would now be dumped, which is definitely not in your best interests.

There are two key ways which we will get around this:

We will design our web applications carefully for security. If we can design the web application in such a way that only we choose the file names and programs that execution functions work with, we have much less to worry about in terms of security. We can also do our best to isolate any files that we work with and try to limit damage whenever something unexpected occurs.
For those cases where we decide that user input is absolutely unavoidable, we will filter the user input extremely heavily and make sure it is as safe as possible before passing it to any of these functions. I always do the following:
- Absolutely forbid the use of any slash characters, whether it be forward ( / ) or backward ( \ ). I also forbid any user strings from beginning with X:, where X could be a valid drive letter on the system (Windows only, of course).
- Use the escapeshellarg function to further make the parameters safer. I.e. $output = shell_exec("cat /web/users/info/" . escapeshellarg($processedUserInfoFile));

A way in which we are helped by the operating system is that most web servers operate as a user with restricted permission, and thus can only do things that user is permitted to do.

One last security note we should mention here is that if your php.ini file has safe_mode set to On, then the shell_exec function and analagous backtick operator is not available.

Conclusion

We have covered a lot of ground in this article, but we should now have a solid understanding of the various execution functions available to us in PHP, as well as how they differ. While we should be extremely concerned about the security of running programs external to our web server, we can do so in a reasonably safe and controlled manner, which can only be a good thing for our web applications and their customers.

—

¹ It is worth noting that dir, type, and many other commands on Windows are not programs, but actualy commands built into the cmd.exe program. Fortunately the system execution functions are smart enough to know to execute cmd.exe for you if you ask to execute one of them.

StripTags 1.0 Released

Thu, 01 Mar 2007 19:17:44 -0800

Download version 1.0 of StripTags for PHP5

After some further development over the last couple of weeks, I have released version 1.0 of the StripTags class for PHP.

This class is designed to replace the strip_tags function in PHP, which does not work particuarly well. It serves to help website authors avoid cross-site-scripting (XSS) attacks in user-created content, for sites such as blogs or forums where users can enter entries, articles, or comments.

You can read more about the class and XSS in general in the following article:

Helping Prevent XSS Attacks in PHP5

The big new feature change in this version of the class is the ability to find XSS attacks injected via unicode-enrypted attributes, such as:


<IMG SRC=&#106;&#97;&#118;&#97;&#115;&#99;&#114;&#105;&#112;&#116;
      &#58;&#97;&#108;&#101;&#114;&#116;&#40;&#39;&#88;&#83;&#83;&#39;&#41;>

We now successfully find these and neutralise them by inserting extra junk in the attribute string so that they are not processed by client browsers.

Please note that this class is not a 100% complete solution to XSS. We do not handle all of the ways that XSS can be achieved through CSS and other forms of style (and thus always recommend that you not permit users to enter STYLE elements or “style” attributes on other elements). Solving this problem requires significant amount of work and effort, and I believe that if you want to give users that degree of input control, you should have them use a Wiki-language engine such as Textile.

The README and INSTALL documents have full information on how to use the class as well as what it does and does not do.

As always, please feel free to email me with any questions, comments, or bug reports. I’ll fix the latter as quickly as I can.

Download version 1.0 of StripTags for PHP5

Google Blocked in China

Thu, 01 Mar 2007 19:17:43 -0800

Living in China for the last few months has been surprisingly hassle-free Internet-wise. In the living room here I’ve got a 768k DSL connection, and the government blocking of sites has not been too big of a deal, given that I often don’t do much more than read digg.com or check email.

The lack of the BBC has been a bit annoying, but I can get the New York Times (by changing DNS servers) and The Economist which compensate quite nicely.

This changed last Thursday when, suddenly, Google disappeared. It had worked pretty much flawlessly before that, and Google Mail was never an issue.

There have been periodic outages of sites and things before, but this has been going on for four days now and, more ominously, google.cn works perfectly fine. Perhaps weirdest is that it does work if you keep trying to get to the site. Having to reload the page 20 times each time you want to check mail is a bit annoying, but better than nothing.

Suddenly promoted to the top of the “To do” list? Setting up a VPN via SSH.

Hotmail blocked in China

Related to this is that Hotmail is also proving problematic here in China because of blocking of various stages of Microsoft’s Microsoft Live.

A simple workaround for this is to use the mobile phone version of hotmail, which will at least let you check mail. I just visit the URL:

http://mobile.msn.com/hm/Folders.aspx

New RSS Feeds

Thu, 01 Mar 2007 19:17:43 -0800

The RSS feeds on the Chipmunk Ninja site have been changed slightly. For compatibility, the old rss/tech.rss and rss/personal.rss feeds will continue to work, but there are now three new feeds:

rss/tech	Tech Articles
rss/personal	Personal Articles and Stories
rss/all	All Chipmunk Ninja Blog Entries

I have also fixed up the RSS output, particular the XML headers and the way tags are exposed through the <category> element.

As always, please let me know if there are any problems.

Troubles with Asynchronous Ajax Requests and PHP Sessions

Thu, 01 Mar 2007 19:17:42 -0800

As I sit here watching “The Muppets Take Manhattan” in Spanish in the middle of a Costa Rican thunderstorm, I find my mind drifting back to a recent project where I spent a day debugging a frustratingly annoying problem: A user would visit the web application I was working on, and after a given page was loaded, all of the session data associated with their visit would be suddenly gone. The user would no longer be logged into the site, and any changes they made (which were logged in session data) were lost.

I spent tonnes of time in the debugger (while at times unreliable and frustrating on huge projects, the Zend debugger is still an invaluable aid for the PHP application developer) and kept seeing the same thing: the session data were simply being erased at some point, and the storage in the database would register ’’ as the data for the session.

It was driving me crazy. I would sit there in the debugger and go through the same sequence each time:

Debug the request for the page load.
Debug the request for the first Ajax request that the page load fired off.
Debug the request for the second Ajax request that the page load simultaneously fired off.
Debug the request for the third Ajax request that the page initiated

In retrospect, looking at the above list, it seems blindingly obvious what I had been running into, but it was very late in a very long contract, and I blame the fatigue for missing what now seems patently obvious: A race condition.

For those unfamiliar with what exactly this is, a race condition is seen most often in applications involving multiple “threads of execution” – which include either separate processes or threads within a process – when two of these threads (which are theoretically executing at the same time) try to modify the same piece of data.

If two threads of execution that are executing more or less simultaneously (but never in exactly the same way, because of CPU load, other processes, and chance) try to write to the same variable or data storage location, the value of that storage location depends on which thread got there first. Given that it is impossible to predict which one got there first, you end up not knowing the value of the variable after the threads of execution are finished (in effect, “the last one to write, wins”) (see Figure 1).

Normally, when you write web applications in PHP, this is really not an issue, as each page request gets their own execution environment, and a user is only visiting one page at a time. Each page request coming from a particular user arrives more or less sequentially and shares no data with other page requests.

Ajax changes all of this, however: suddenly, one page visit can result in a number of simultaneous requests to the server. While the separate PHP processes cannot directly share data, a solution with which most PHP programmers are familiar exists to get around this problem: Sessions. The session data that the various requests want to modify are now susceptible to being overwritten by other ones with bad data after a given request thinks it has written out updated and correct data (See Figure 2).

In the web application I was working on, all of the Ajax requests were being routed through the same code that called session_start() and implicitly session_write_close() (when PHP ends and there is a running session, this function is called). One of the Ajax requests would, however, set some session data to help the application “remember” which data the user was browsing. Depending on the order in which the various requests were processed by the server, sometimes those data would overwrite other session data and the user data would be “forgotten”.

As an example of this problem, consider the following example page, which when fully loaded, will execute two asynchronous Ajax requests to the server.


<?php
// race1.php
// part 1
session_start();

//
// We will prime the session data with a simple value here.
// One of the Ajax requests going to race2.php will clobber
// this value periodically.
//
$_SESSION['fudgecicle'] = 'eeeek!';

?>
<html>
<head>
  <title>Race Condition Demo Page</title>
</head>

<script>
/* part 2 */
/**
 *=-------------------------------------------------------=
 * getNewHTTPObject
 *=-------------------------------------------------------=
 * This function is here just to create a new
 * XmlHttpRequest object.
 */
function getNewHTTPObject()
{
    var xmlhttp;

    /** Special IE only code ... */
    /*@cc_on
      @if (@_jscript_version >= 5)
          try
          {
              xmlhttp = new ActiveXObject("Msxml2.XMLHTTP");
          }
          catch (e)
          {
              try
              {
                  xmlhttp = new ActiveXObject("Microsoft.XMLHTTP");
              }
              catch (E)
              {
                  xmlhttp = false;
              }
         }
      @else
         xmlhttp = false;
    @end @*/

    /** Every other browser on the planet */
    if (!xmlhttp && typeof XMLHttpRequest != 'undefined')
    {
        try
        {
            xmlhttp = new XMLHttpRequest();
        }
        catch (e)
        {
            xmlhttp = false;
        }
    }

    return xmlhttp;
}

/**
 *=-------------------------------------------------------=
 * onLoadFunction
 *=-------------------------------------------------------=
 * We call this function when the page loads, and it starts
 * two asynchronous Ajax requests to race2.php.
 */
var xml1;
var xml2;
function onLoadFunction()
{
    xml1 = getNewHTTPObject();
    xml2 = getNewHTTPObject();

    xml1.open('GET', 'http://localhost/race2.php?req1', true);
    xml2.open('GET', 'http://localhost/race2.php?req2', true);

    xml1.onreadystatechange = handleResponse1;
    xml2.onreadystatechange = handleResponse2;

    xml2.send('');
    xml1.send('');
}

/**
 *=-------------------------------------------------------=
 * handleResponse1
 *=-------------------------------------------------------=
 * This handles the response from the first ajax request.
 * It puts the returned string in the <div> element
 * with th eid 'fudgecicles'
 */
function handleResponse1()
{
    if (xml1.readyState == 4)
    {
        document.getElementById('fudgecicles').innerHTML =
            xml1.responseText;
    }
}

/**
 *=-------------------------------------------------------=
 * handleResponse2
 *=-------------------------------------------------------=
 * This handles the response from the second ajax request.
 * We don't actually care about this, so we just ignore
 * the results.
 */
function handleResponse2()
{
    // we don't care about this response
}
</script>

<!--
     part 3

     This page just executes the onLoadFunction() call and
     contains a single <div> where the result will be
     placed.
  -->
<body onLoad='onLoadFunction();'>
  <div id='fudgecicles'>Waiting for response from Server</div>
</body>
</html>

The code is divided into three main sections:

The first contains the call to session_start() and opens the HTML headers.
The second contains the Javascript code to execute the asynchronous requests to the server. The biggest function, getNewHTTPObject() is used to create new objects. The onLoadFunction() is executed when the page finishes loading and starts the ball rolling, while the other two functions are simply used to wait for and handle the responses and results from the asynchronous requests.
In the final section, we just write out the <body> section of the document, which contains a single <div> element to hold the results and an attribute on the <body> element to make sure that the onLoadFunction() is called when the document finishes loading.

The asynchronous Ajax requests are then made to race2.php and are processed by the following code, which can handle two different Ajax work requests:


<?php
// race2.php

session_start();

//
// if they ask for req1, then just return the current value
// of the $_SESSION['fudgecicles'] data.  If they ask for
// req2, then have this session simulate a bogus value by
//  setting $_SESSION['fudgecicles'] to ''.
//
// NOTE:  the two for loops are to simulate the various
//        Ajax processes doing lots of stuff, and force
//        the processor to interrupt the processes
//        periodically to let the other work.  This helps
//        the race condition occur more.
//
if (isset($_GET) and isset($_GET['req1']))
{
    // waste time
    for ($x = 0; $x < 10000; $x++)
        strpos('asdfasdf', 'tttttttttttttttasdfioap');

    if (isset($_SESSION['fudgecicle'])
        and $_SESSION['fudgecicle'] != '')
    {
        $retString = $_SESSION['fudgecicle'];
    }
    else
        $retString = '(empty)';
}
else if (isset($_GET) and isset($_GET['req2']))
{
    // waste time
    for ($x = 0; $x < 10000; $x++)
        strpos('asdfasdf', 'tttttttttttttttasdfioap');

    $retString = '';
    $_SESSION['fudgecicle'] = '';
}

header('Content-Type: text/plain');
echo $retString;
?>

This PHP script handles the two request types differently, and creates the race condition by having the second request type req1 set the sesion data to ’’. (In a real world application, you might have accidentally had this request set some value you thought was meaningful).

If you install the two files race1.php and race2.php on your server, and then load race1.php into your borwser, you will periodically see that the test string is set after the page is completely loaded, and other times it will be “(empty)”, indicating that the second Ajax request has clobbered the value.

Now that we are aware of this problem and how it can manifest itself, the next question is, of course, how do we solve it? Unfortunately, I think this is one of those problems best solved by avoiding it. Building in logic and other things into our web application to lock the threads of execution (i.e. individual requests) would be prohibitively expensive and eliminate much of the fun and many of the benefits of asynchronous requests via Ajax. Instead, we will avoid modifying session data when we are executing multiple session requests.

Please note that this is much more specific than saying simply that we will avoid modifying session data during any Ajax request. Indeed, this would be a disaster: In a Web 2.0 application, we are mostly likely using Ajax for form submission and updating the state of the user data (i.e. session data) as the data are processed and we are responding to the changes. However, for those requests we are using to update parts of pages dynamically, we should be careful to avoid modifying the session data in these, or at least do so in a way that none of the other requests are going to see changes in their results depending on these session data.

Ajax requests and session data do not have be problematic when used together: With a little bit of care and attention, we can write web applications that are powerful, dynamic, and not plagued by race condition-type bugs.

Got other suggestions or solutions to this problem? Add a comment or mail me, and I’ll update the article.

PHP Book Addenda I

Thu, 01 Mar 2007 19:17:42 -0800

As I sat down to edit “Core Web Application Programming with PHP and MySQL”, I would sometimes find errors in the text so blindingly obvious and stupid that I would question whether or not I was truly qualified to write such a book. And yet, after talking with some other people who write books (and recalling days when I wrote huge amounts of code), it seems that this is all common and with much proof-reading and the hard work of some friendly reviewers, I was able to write a book of extremely high quality.

Of course, that just meant I would be even more devastated when the first technical errors WERE found in the book.

There have been a couple, but they’re not that killer serious.

Chapter 21 Error

In Chapter 21, where the book discusses writing your own output handler, the constant in the $_SESSION array to check is HTTP_ACCEPT_ENCODING, without the letter ‘S’ on the word accept.

The Accompanying Source Code

There are a couple of errors in the source code, the most glaring of which DID get fixed, but never made it on to the shipping CD ROM (d’oh!). In the SimpleBlog sample, in the file lib/entrymanager.inc, the class DBManager is accidentally misspelled DBMananager. Just fix it and change it back to the correct spelling and the sample will compile fine under PHP 5.0.x (x <= 4).

The other problem with the samples is that some new things have appeared in newer versions of PHP 5.0.y (y >= 5). PHP now defines a class called InvalidArgumentException, which conflicts directly with the class I have defined using the same name. The easy fix for this problem is to simply change the name of the class slightly, to something like MyInvalidArgumentException or some such thing.

To save you the hassle of tracking down and fixing all of these problems, I have put a new copy of the book source code up on the chipmunkninja.com servers. You can download these from here: phpwasrcupdate_2005-12-01.zip.

As always, if you see any other problems or errors in the book, or just want to comment on it, please feel free to drop me some mail.

I remain chagrined, but I’ll get over it.

The Best Game You're not Playing

Thu, 01 Mar 2007 19:17:41 -0800

You turn the corner, walk to the end of the hall, and open the door. Inside, a giant orange “D” rears its ugly head to attack. You panic and kick the dog! (Argh! You didn’t mean to do that!) It yelps at you, but still attacks the dastardly beast on your behalf. Your sword is useless, and you are gravely wounded.

You have only one option left – that scroll that you found lying in the pile of dust in that last room. You’re not sure what it is, but you’re running out of time and it may just save your neck. You read it and …

All your armour falls off and turns into dust. The orange “D” proceeds to eat you for breakfast. Game over.

Welcome to another average game of the best game you’re not playing: Nethack. Inspired by an earlier similar game called Rogue, Nethack has been an active project for over 20 years, being worked on by some well known names such as Jay Fenlason and Jon Payne.

One of the biggest reasons you’re probably not playing it? It’s entirely ASCII based. You are a small ”@” character (assuming you’re humanoid) fighting off little “d”s (dogs and jackals), “o”s (goblins, orcs, and Uruk-Hai), and “D”s (dragons), oh my!. Walls are made with plain old |, -, and + ASCII characters, and you use the cursor keys to move around in the dungeon. Only on some platforms, such as Microsoft Windows, do you even know you’re fighting a brown “c” (coatl – tough, but survivable) versus a yellow “c” (a cicatrice – bad, bad, and more bad).

The current version is 3.4.3, and supports a remarkably wide variety of platforms and architectures, including Unix, Windows, and my trusty OS X-based powerbook. The game is controlled by keyboard sequences and are usually single characters, sometimes requiring you to hold down the CTRL or ALT keys first. Thus, instead of having to type “pick up scroll, pick up potion”, you can just hit ”,” and select the items you want to grab.

In the game, in a way familiar to anybody who’s played games like Dungeons & Dragons, you are an adventure-seeking character, working your way down through dungeons, mazes, Gehennom, and various elemental planes, to find the Amulet of Yendor, take it from the body of the Wizard of Yendor, who is not at all inclined to let it go, and carry it out to the real world again. Along the way, you will have to defeat monsters, the undead, ghosts of players past, and grab as much loot as you can. More monsters and loot mean more points, which is good.

To give the game more variety, you can choose from a number of character types, including wizards, knights, valkyries, samurai, tourists, and barbarians. These all have various strengths and weaknesses, such as different affinities for magic, healing, fighting, or otherwise doing cool things. As you use different character types and die (often) and try again, you’ll learn which ones are better at which things, and how to make the best of all their skills.

Most impressive are the dizzying number of different levels and special places this game offers. You will go through traditional looking dungeon levels, pass through special floors with puzzles and other tricks and traps, and then proceed to other special floors with neat surprises at the end.

Make no mistake: the game is hard. Between random chance and random levels, your game can go south, quickly. But in those games where you do everything right and things go your way, you can do gloriously well and get tantalisingly far along in the game—usually only to do something incredibly stupid and kill yourself (I have done this so many times it is embarrassing).

All of this combines to make Nethack one of the most replayable games I have ever seen. I buy other games, play them, and either finish them or get bored of doing the same tedious crap again and again. I always end up coming back to Nethack, to get my butt kicked. If you are a hard core pure gamer, you can figure out how the game works by trail and error (and frequently dying). For those less patient, you can check out the source code (eeek!) or one of the many Nethack Spoilers sites still viewable to this day on the Intarwebs.

So before you sit and mock the lowly ASCII game player from behind your texture mapping 5GHz overclocked-requiring fully light-sourced bouncing bits of anatomy first-person shooter games, give Nethack a try. I bet you’ll get hooked.

Nethack sources and many binaries are available from the Nethack website, www.nethack.org. The game installs trivially on most supported platform.