Coding

Infyblogs dashboard

I just finished Stephen Few‘s book on Information Dashboard Design. It talks about what’s wrong with the dashboards most Business Intelligence vendors (Business Objects, Oracle, Informatica, Cognos, Hyperion, etc.), and brings Tuftian principles of chart design to dashboards.

So I took a shot at designing a dashboard based on those principles, and made this dashboard for InfyBLOGS.

Infyblog dashboard

You can try for yourself. Go to http://www.s-anand.net/reco/
Note: This only works within the Infosys intranet.

  1. Right click on the "Infyblog Dashboard" link and click "Add to Favourites…" (Non-IE users — drag and drop it to your links bar)
  2. If you get a security alert, say "Yes" to continue
  3. Return to InfyBLOGS, make sure you’re logged in (that’s important) and click on the "Infyblog Dashboard" bookmark
  4. You’ll see a dashboard for your account, with comments and statistics

The rest of this article discusses design principles and the technology behind the implementation. (It’s long. Skim by reading just the bold headlines.)

Dashboards are minimalist

I’ll use the design of the dashboard to highlight some of the concepts in the book.

I designed the dashboard first on Powerpoint, keeping these principles in mind.

  1. Fits in one screen. No scrolling. Otherwise, it isn’t a dashboard.
  2. Shows only what I need to see. Ideally, from this dashboard, I should receive all the information I need to act on, and no more.
  3. Minimises the data-ink ratio. Don’t draw a single pixel that’s not required.

The first was easy. I just constrained myself to one page of PowerPoint, though if you had a lot of toolbars, the viewing area of your browser may be less than mine.

The second is a matter of picking what you want to see. For me, these are the things I look for when I log into InfyBLOGS:

  1. Any new comments?
  2. Any new posts from my friends?
  3. What’s new? What’s hot?

Then I dig deeper, occasionally, into popular entries, popular referrers, how fast the blogs are growing, etc. So I’ve put in what I think are the useful things.

The third is reflected in the way some of this information is shown. Let me explain.

Keep the charts bare

Consider the graphs on the right. They look like this.

Notice the wiggly line to the right. It’s a graph called sparkline, and was introduced by Edward Tufte. Sparklines are great to show trends in a compact way. Forget the axes. Forget the axes labels. Forget the gridlines. The text on the left ("visitors per day") tells you what it is. The number (10475) is the current value. And the line is the trend. Clearly the number of visitors has exploded recently, from a relatively slow and flat start. The labels and axes aren’t going to tell you much more.

Boldly highlight what’s important

The most important thing here, the title, is on top. It’s pure black, in large font, positioned right on top, and has a line segregating it from the rest of the page.

The sections are highlighted by a bigger font, different colour, and a line, but the effect is a bit more muted.

The numbers on the right are prominent only by virtue of size and position. If anything, the colour de-emphasizes them. This is to make sure that they don’t overwhelm your attention. (They would if they were in red, for instance.)

The number 10475 is carefully set to occupy exactly two line spaces. That alignment is very important. The small lines are at a font size of 11px, and line spacing is 1.5. So at a font size of 2 x 11px x 1.5 = 33px, the size of the large number covers exactly two rows.

The labels, such as "visitors" or "sites" are in blue, bold. Nothing too out of the way, but visible enough that they stand out.

The "View more…" links just use colour to stand out. They’re pretty unimportant.

The bulk of the text is actually made of links, unlike traditional links, they’re not underlined and they’re not blue. It would just add to the noise if everything where underlined. But on mouseover, they turn blue and are underlined, clearly indicating that they’re links.

I’ve used four mechanisms to highlight relative importance: position, size, colour and lines. There are many more: font, styling, boxes, underlining, indentation, etc.

The purpose of good design is to draw attention to what’s important, to direct the flow of the eye in a way that builds a good narrative. Don’t be shy of using every tool at your disposal in doing that. While on the topic, the Non-Designer’s Design Book is a fantastic and readable book on design for engineers.

Always use grids to display

I can’t say this any better than Mark Boulton of subtraction.com, in his presentation Grids are Good. Grids are pervasive in every form of good design. It’s a fantastic starting point as well. Just read the slideshow and follow it blindly. You can’t go wrong.

This dashboard uses 12-grid. The page is divided into two vertically. The left half has mostly text and the right half has statistics and help. There are 3 blocks within each, and they’re all the same size (alignment is critical in good design). Two of the blocks on the left are subdivided into halves, while the bottom right "Links and help" section is subdivided into three. Well, it’s easier to show it than explain it:

Picture of grid

Copy shamelessly

The design for this dashboard was copied in part from WordPress 2.7’s new dashboard, part from the dashboards on Stephen Few’s book, and part from the winners of the 2008 Excel dashboard competition.

Most of these designs are minimalist. There’s no extra graphics, jazzy logos, etc. that detract from the informational content. This is an informational dashboard, not a painting.

Good design is everywhere. View this presentation on How to be a Creative Sponge to get a sense of the many sources where you can draw inspiration from. You’re much much better of copying good design than inventing bad ones.

You can hack the dashboard yourself

I’ve uploaded the source for the dashboard at http://infyblog-dashboard.googlecode.com/

Please feel free to browse through it, but don’t stop there! Go ahead and tweak it to suit what you think should be on an ideal dashboard. I’ll give access to the project to anyone who asks (leave a comment, mail me or call me at +44 7957 440 260).

Please hack at it. Besides, it’s great fun learning jQuery and working on a distributed open source project.

Now for some notes on how this thing works.

Bookmarklets bring Greasemonkey to any browser

This is implemented as a bookmarklet (a bookmark written in Javascript). It just loads the file http://www.s-anand.net/infyblog-dashboard.js which does all the grunt work. This is the code for the bookmarklet.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
// Create a new a script element
var s = document.createElement('script');
 
// Set the source of the script element to my bookmarklet script
s.setAttribute('src','http://www.s-anand.net/infyblog-dashboard.js');
 
// Add the script element to the head
// This does the the equivalent of:
// <head><script src="..."></script></head>
document.getElementsByTagName('head')[0].
appendChild(s);
 
// Return void. Not sure why, but this is required
void(s);

This form of the bookmarklet is perhaps it’s most powerful use. It lets you inject any script into any site. If you want to change it’s appearance, content, anything, just write it as a bookmarklet. (Think of it as the IE and Safari equivalent of Firefox’s Greasemonkey.)

So if you wanted to load jQuery into any page:

  1. Change the URL in line 5 above to the jQuery URL and add it as a bookmark. (Or use Karl’s jQuery bookmarklet. It’s better)
  2. Go to any page, say the IMDB Top 250 for instance
  3. Click on the bookmarklet. Now your page is jQueryified. (If you used Karl’s bookmarklet, it’ll tell you so.)
  4. On the address bar, type "javascript:alert($(‘table:eq(10) tr’).length)" and press enter.
  5. You should see 252, the number of rows on main table of the IMDB Top 250

Notes:

jQuery is a powerful Javascript library

Firstly, you don’t want to program Javascript without a library. You just don’t. The state of browser incompatibilities and DOM usability is just to pathetic to contemplate. Stop learning DOM methods. Pick any library instead.

Going by popularity, you would do well to pick jQuery. Here’s a graph of searches on Google for the popular Javascript libraries, and jQuery leads the pack.

Legend: jquery prototype dojo ext yui

Google Trends showing jQuery as the dominant library

I picked it in early 2007, and my reasons were that it:

  1. Is small. At that time, it was 19KB and was the smallest of the libraries. (Today it’s 18KB and has more features.)
  2. Makes your code compact. It lets you chain functions, and overloads the $() function to work with DOM elements, HTML strings, arrays, objects, anything.
  3. Doesn’t pollute the namespace. Just 2 global variables: jQuery and $. And you can make it not use $ if you like.
  4. Is very intuitive. You can learn it in an hour, and the documention is extremely helpful.
  5. Is fully functional. Apart from DOM manipulation, it covers CSS, animations and AJAX, which is all I want from a library.

These days, I have some additional reasons:

  1. It’s extensible. The plugin ecosystem is rich.
  2. It’s hosted at Google, and is really fast to load. Most often, it’s already cached in your users’ browser.
  3. John Resig is a genius. After env.js and processing.js, I trust him to advance jQuery better than other libraries.

So anyway, the first thing I do in the dashboard script is to load jQuery, using the same code outlined in the bookmarklet section above. The only addition is:

1
2
3
4
5
script.onload = script.onreadystatechange = function() {
    if (!this.readyState||this.readyState=='loaded'||this.readyState=='complete'){
        callback();
    }
};

This tells the browser to run the script "callback" once the script is loaded. My main dashboard function runs once jQuery is loaded.

InfyBLOGS has most statistics the dashboard needs

Thanks to posts from Utkarsh and Simon, I knew where to find most of the information for the dashboard:

The only things I couldn’t find were:

  • Friends posts. I’d have liked to show the titles of recent posts by friends. Yeah, I know: blogs/user_name/friends has it, but thanks to user styles, it’s nearly impossible to parse in a standard way across users. I’d really like an RSS feed for friends.
  • Interests posts. Would be cool to show recent posts by users who shared your your interest.
  • Communities posts. Again, I’d have liked to show the recent posts in communities, rather than just the names of communities.

Using IFRAMEs, we can load these statistics onto any page

Let’s say we want the latest 10 comments. These are available on the comments page. To load data from another page, we’d normally use XMLHTTPRequest, get the data, and parse it — perhaps using regular expressions.

1
2
3
$.get('/tools/recent_comments.bml').function(data) {
    // Do some regular expression parsing with data
});

But from a readability and maintainability perspective, regular expressions suck. A cleaner way is to use jQuery itself to parse the HTML.

1
2
3
4
$.get('/tools/recent_comments.bml').function(data) {
    var doc = $(data);
    // Now process the document
});

This works very well for simple pages, but sadly, for our statistics, this throws a stack overflow error (at least on Firefox; I didn’t have the guts to try it on IE.)

So on to a third approach. Why bother using Javascript to parse HTML, when we’re running the whole application in a browser, the most perfect HTML parser? Using IFRAMEs, we can load the whole page within the same page, let the browser do the HTML parsing.

1
2
3
4
5
6
$('<iframe src="/tools/recent_comments.bml"></iframe>')
    .appendTo('body')
    .load(function() {
        var doc = $(this).contents();
        // Now process the document
    });

This way, you can read any page from Javascript within the same domain. Since both the bookmarklet and statistics are on the InfyBLOGS domain, we’re fine.

jQuery parses these statistics rather well

Now the doc variable contains the entire comments page, and we can start process it. For example, the comments are on the rows of the second table in the page. (The first table is the navigation on the left.). So

doc.find('table:eq(1) tr')

gets the rows in the second table (table:eq(0) is the first table). Now, we can go through each row, extract the user, when the comment was made, links to the entry and to the comment.

1
2
3
4
5
6
7
8
doc.find('table:eq(1) tr').slice(0,8).map(function() {
    var e = $(this),
        user = e.find('td:eq(0) b').text(),
        when = e.find('td:eq(0)').text().replace(user, ''),
        post = e.find('td:eq(1) a:contains(Entry Link)').attr('href'),
        cmnt = e.find('td:eq(1) a:contains(Comment Link)').attr('href'),
        // return HTML constructed using the above information
    });

Google Charts API displays dynamic charts

Another cool thing is that we can use Google Charts to display charts without having to create an image beforehand. For instance, the URL:

http://chart.apis.google.com/chart?cht=lc&chs=300x200&chd=t:5,10,20,40,80

shows an image with a line chart through the values (5,10,20,40,80).

Google chart example

The dashboard uses sparklines to plot the trends in InfyBLOG statistics. These are supported by Google Charts. We can extract the data form the usage page and create a new image using Google Charts that contains the sparkline.

If you want to play around with Google Charts without mucking around with the URL structure, I have a relatively easy to use chart constructor.

Always use CSS frameworks

Libraries aren’t restricted to Javascript. CSS frameworks exist, and as with Javascript libraries, these are a no-brainer. It’s not worth hand-coding CSS from scratch. Just build on top of one of these.

The state of CSS frameworks isn’t as advanced as in Javascript. Yahoo has it’s UI grids, Blueprint‘s pretty popular and I found Tripoli to be nice, but the one I’m using here is 960.gs. It lets you create a 12-grid and 16-grid on a 960-pixel wide frame, which is good enough for my purposes.


At nearly 2,500 words, this is the longest post I’ve ever written. It took a day to design and code the dashboard, but longer to write this post. Like Paul Buchheit says, it’s easier to communicate with code. I’ve also violated my principle of Less is More. My apologies. But to re-iterate:

Please hack it

The code is at http://infyblog-dashboard.googlecode.com/. You can join the project and make changes. Leave a comment, mail me or call me.

To Python from Perl

I’ve recently switched to Python, after having programmed in Perl for many years. I’m sacrificing all my knowledge of the libraries and language quirks of Perl. The reason I moved despite that is for a somewhat trivial reason, actually. It’s because Python doesn’t require a closing brace.

Consider this Javascript (or very nearly C or Java) code:

1
2
3
4
5
6
var s=0;
for (var i=0; i<10; i++) {
    for (var j=0; j<10; j++) {
        s = s + i * j
    }
}

That’s 6 lines, with two lines just containing the closing brace. Or consider Perl.

1
2
3
4
5
6
my $s = 0
foreach my $i (1 .. 10) {
    foreach my $j (1 .. 10) {
        $s = $s + $i * $j
    }
}

Again, 6 lines with 2 for the braces. The $ before the variables also drops readability just a little bit. Here’s Python:

1
2
3
4
s = 0
for i in xrange(1, 10):
    for j in xrange(1, 10):
        s = s + i * j

On the margin, I like writing shorter programs, and it annoys me to no end to have about 20 lines in a 100-line program devoted to standalone closing braces.

What I find is that once you’ve really know one language, the rest are pretty straightforward. OK, that’s not true. Let me qualify. Knowing one language well out of C, C++, Java, Javascript, PHP, Perl, Python and Ruby means that you can program in any other pretty quickly — perhaps with a day’s reading and a reference manual handy. It does NOT mean that you can pick up and start coding with Lisp, Haskell, Erlang or OCaml.

Occasionally, availability constrains which programming language I use. If I’m writing a web app, and my hosting provider does not offer Ruby or Python, that rules them out. If I don’t have a C or Java compiler on my machine, that rules them out. But quite often, these can be overcome. Installing a compiler is trivial and switching hosting providers is not too big a deal either.

Most often, it’s the libraries that determine the language I pick for a task. Perl’s regular expression library is why I’ve been using it for many years. Ruby’s HPricot and Python’s BeautifulSoup make them ideal for scraping, much more than any regular expression setup I could use with Perl. Python Image Library is great with graphics, though for animated GIFs, I need to go to the GIF89 library in Java. And I can’t do these easily with other languages. Though each of these languages boast of vast libraries (and it’s true), there are still enough things that you want done on a regular basis for which some libraries are a lot easier to use than others.

So these days, I just find the library that suits my purpose, and pick the language based on that. Working with Flickr API or Facebook API? Go with their default PHP APIs. Working on AppEngine? Python. These days, I pick Python by default, unless I need something quick and dirty, or if it’s heavy text processing. (Python’s regular expression syntax isn’t as elegant as Perl’s or Javascript’s, mainly because it isn’t built into the language.)

To get a better hang of Python (and out of sheer bloody-mindedness), I’m working through the problems in Project Euler in Python. For those who don’t know about Project Euler,

Project Euler is a series of challenging mathematical/computer programming problems that will require more than just mathematical insights to solve. Although mathematics will help you arrive at elegant and efficient methods, the use of a computer and programming skills will be required to solve most problems.

Each problem has been designed according to a "one-minute rule", which means that although it may take several hours to design a successful algorithm with more difficult problems, an efficient implementation will allow a solution to be obtained on a modestly powered computer in less than one minute.

It’s a great way of learning a new programming language, and my knowledge of Python is pretty rudimentary. At the very least, going through this exercise has taught me the basics of generators.

I’ve solved around 40 problems so far. Here are my solutions to Project Euler. I’m also measuring the time it takes to run each solution. My target is no more than 10 seconds per problem, rather than the one-minute, for a very practical reason: the solutions page itself is generated by a Python script that runs each script, and I can’t stand waiting for more than that to refresh the page each time I solve a problem.

Bound methods in Javascript

The popular way to create a class in Javascript is to define a function and add methods to its prototype. For example, let’s create a class Node that has a method hide().

1
2
3
4
5
6
var Node = function(id) {
    this.element = document.getElementById(id);
};
Node.prototype.hide = function() {
    this.style.display = "none";
};

If you had a header, say <h1 id="header">Heading</h1>, then this piece of code will hide the element.

1
2
var node = new Node("header");
node.hide();

If I wanted to hide the element a second later, I am tempted to use:

3
4
var node = new Node("header");
setTimeout(node.hide, 1000);

… except that it won’t work. setTimeout has no idea that the function node.hide has anything to do with the object node. It just runs the function. When node.hide() is called by setTimeout, the this object isn’t set to node, it’s set to window. node.hide() ends up trying to hide window, not node.

The standard way around this is:

3
4
var node = new Node("header");
setTimeout(function() { node.hide()}, 1000);

I’ve been using this for a while, but it gets tiring. It’s so easy to forget to do this. Worse, it doesn’t work very well inside a loop:

1
2
3
4
for (var id in ["a", "b", "c"]) {
    var node = new Node(id);
    setTimeout(function() { node.hide(); }, 1000);
}

This actually hides node "c" thrice, and doesn’t touch nodes "a" and "b". You’ve got to remember to wrap every function that contains a function in a loop.

1
2
3
4
5
6
for (var id in ["a", "b", "c"]) {
    (function() {
        var node = new Node(id);
        setTimeout(function() { node.hide(); }, 1000);
    })();
}

Now, compare that with this:

1
2
3
for (var id in ["a", "b", "c"]) {
    setTimeout((new Node(id)).hide, 1000);
}

Wouldn’t something this compact be nice?

To do this, the method node.hide must be bound to the object node. That is, node.hide must know that it belongs to node. And when we call another_node.hide, it must know that it belongs to another_node. This, incidentally, is the way most other languages behave. For example, on python, try the following:

>>> class Node:
...     def hide():
...             pass
...
>>> node = Node()
>>> node
<__main__.Node instance at 0x00BA32D8>
>>> node.hide
<bound method Node.hide of <__main__.Node instance at 0x00BA32D8>>

The method hide is bound to the object node.

To do this in Javascript, instead of adding the methods to the prototype, you need to do two things:

  1. Add the methods to the object in the constructor
  2. Don’t use this. Set another variable called that to this, and use that instead.
1
2
3
4
5
6
7
var Node = function(id) {
    var that = this;
    that.element = document.getElementById(id);
    that.hide = function() {
        that.element.style.display = "none";
    }
};

Now node.hide is bound to node. The following code will work.

8
9
var node = new Node("header");
setTimeout(node.hide, 1000);

I’ve taken to using this pattern almost exclusively these days, rather than prototype-based methods. It saves a lot of trouble, and I find it makes the code a lot compacter and easier to read.

Downloading online songs

You know those songs on Raaga, MusicIndiaOnline, etc? The ones you can listen to but can’t download?

Well, you can download them.

It’s always been possible to download these files. After all, that’s how you get to listen to them in the first place. What stopped you is security by obscurity. You didn’t know the location where the song was stored, but if you did, you could download them.

So how do you figure out the URL to download the file from?

Let’s take a look at MusicIndiaOnline first. When you click on a song, it pops up in a new window. You can’t figure out the address of that window because the address bar is hidden, but you can get it by hovering over the link before clicking, or by right-clicking and copying the link location. It’s always something like this:

http://www.musicindiaonline.com/p/x/FAfgqz0HzS.As1NMvHdW/

It always has the same structure: http://www.musicindiaonline.com/p/x/something/. Let’s call that the something the song’s key.

Now, what this page does is play a .SMIL file. An SMIL file is a text file that has a list of songs to play. This file is always stored at

http://www.musicindiaonline.com/g/r/FAfgqz0HzS.As1NMvHdW/play2.smil

(Notice that the key remains the same.) If you type this into your browser, you’ll get a text file that you can open in Notepad. You’ll find that it has a bunch of lines, two of which are interesting. There’s one that reads:

<meta name="base" content="http://205.134.247.2/QdZmq-LL/">

The content="…" part gives you the first part of the song’s address. Then there’s a line that reads:

<audio src="ra/film/…"/>

The src="…" part gives you the rest of the address. Putting the two together, you have the full URL at which to download the song.

Except, they’re a bit smart about it. These songs are meant to be played on RealPlayer, and not downloaded. So if you try to access the page using your browser, you get a 404 Page not found error. But if you typed the same page into RealPlayer, you can hear the song play.

To actually download the song, you need to fool the site into thinking that your browser is RealPlayer. So first, you need to get a good browser like Firefox. Then download the User Agent Switcher add-on. Change your user agent to "RMA/1.0 (compatible; RealMedia)" and try the same song: you should be able to download it.

Let me summarise:

  1. Right-click on the song you want to play, and copy the URL
  2. Extract the key. In the URL http://www.musicindiaonline.com/p/x/FAfgqz0HzS.As1NMvHdW/ the key is FAfgqz0HzS.As1NMvHdW
  3. Open http://www.musicindiaonline.com/g/r/<your_key>/play2.smil in your browser. Open it with Notepad
  4. Switch the user agent on your browser to "RMA/1.0 (compatible; RealMedia)"
  5. Put the content="…" and audio src="…" pieces together to form the URL
  6. Type the URL and save the file

I’ve automated this in my music search engine. So if you go to the Hindi, Tamil or any other online music page and click on the blue ball next to the song, you’ll see a menu with a download option. The download opens a Java program that does these steps for you and saves the song in your PC.

So now, you’re probably thinking:

  1. How did he figure this out?
  2. What about other sites?
  3. How does that Java program work?
  4. How do I listen to these on my iPod?

How did I figure this out?

Fiddler.

I believe a good measure of a tool’s power is it’s ability to be the one-word answer to a relatively broad question. For example, "Where can I find more about something?" "Google it." "How do I improve my pictures?" "Photoshop it".

Fiddler’s like that. "How do I find out what I’m downloading?" "Use Fiddler".

It’s a proxy you can install on your machine. It automatically configures your browsers when you run it. Thereafter, it tells you about all the HTTP requests that are being sent by your machine. So if you patiently walk through the logs, you’ll find all the URLs that MusicIndiaOnline or any other site uses, as well as the other headers (like User-Agent) that are needed to make it work.

What about other sites?

I’ll list a couple here.

Smashits:

  1. Right-click on the song you want to play, and copy the URL
  2. View the source and hunt for the URL fragment "player/ra/ondemand/launch_player.cfm?something".  The key is is something.
  3. Open http://ww.smashits.com/player/ra/ondemand/playlist.asp?6;giftsToIndia1.rm;1,something in your browser, using Notepad
  4. Switch the user agent on your browser to "RMA/1.0 (compatible; RealMedia)"
  5. Type in what’s inside the src="…" in your browser and save the file

Oosai:

  1. Right-click on the song you want to play, and copy the URL
  2. View the source and hunt for the URL fragment onclick="setUsrlist(something)".  The key is is something.
  3. Open http://www.oosai.com/oosai_plyr/playlist.cfm?sng_id=something in your browser, using Notepad
  4. Switch the user agent on your browser to "RMA/1.0 (compatible; RealMedia)"
  5. Type in the URL that you see in Notepad and save the file.

Try figuring out the others yourself.

How does the Java program work?

It does most of these steps automatically. The applet itself is fairly straightforward, and you can view it here. It takes two parameters: db, which indicates the server from which to download (M for MusicIndiaOnline, S for Smashits, etc.) and num, which is the key.

But in order for an applet to be able to download files from any site, and to be able to save this on your PC, it needs to be a signed applet. Since Sun offers a tool to sign JAR files, this wasn’t much of an issue.

There is one pending problem with Windows Vista, however. Signed applets can’t save files anywhere on Vista. They are saved in a special folder. This is being fixed, but until then, if you use Internet Explorer on Vista, you probably will be unable to find your saved files.

How do I listen to these on my iPod?

The saved files are in RealMedia format. Your best bet is to convert them to MP3 files using MediaCoder. Then transfer them using iTunes.

Keyword searches as a Web command line

Andre’s mentions dumping Google Chrome because of lack of extension support, especially Ubiquity, and lists 15 useful Ubiquity commands.

If you haven’t seen Ubiquity, you should. It’s a great extension that transforms your browser into an Internet command prompt. It is modelled on the Enso Launcher, which is a great piece of work by itself.

I wasn’t quite prepared to let go of Chrome that easily. On Task Manager, seeing 10 Chrome processes, the largest of which takes up 60MB, is a lot more comforting, psychologically, than 1 Firefox process taking up 300MB. (I rarely hit my 1GB RAM limit, so it shouldn’t matter either way. Yet, the spendthrift in me keeps watching.)

So the question is, can I do all the items on his list without using Ubiquity?

Let’s pick the easiest. Google search. If you typed "g some words" on Ubiquity, you get the Google search results for "some words". But you already have that. If you have Firefox, typing any words on the address bar automatically does a Google search for you. On Internet Explorer, it search live.com, but you can easily change that by installing the Google Toolbar.

But the great thing is that this can be customised. On Firefox, click on the down arrow icon next to the search box and select "Manage Search Engines…" to see a list of your search engines. Select the one you want to use, click on "Edit Keyword…" and select the keyword you want. For instance, I’ve typed "google".

Manage Search Engines Add Keywords

So when I type "google some words" on the address bar (not the search bar, the address bar) I get search results for "some words". These are called keyword searches.

On Firefox, you add your own search engines, but you do that using bookmarks. Press Ctrl-Shift-B (Organize Bookmarks) and create a New Bookmark. You can type in any URL in the location field. If you type "%s" as part of the URL, that will be replaced by the search string. So for instance, using a location http://en.wikipedia.org/wiki/Special:Search?search=%s and the keyword "wiki" will do a Wikipedia search for "Harry Potter" if you type "wiki Harry Potter" on the address bar.

It works on Internet Explorer as well, even with version 6. The easiest way is to download TweakUI. Go to Internet Explorer – Search. Click on the Create button. Type in a keyword (called Prefix) and a URL. If you type "%s" as part of the URL, that will be replaced by the search string.

TweakUI

On Google Chrome, get to the Options (what, no shortcut key?) and in the Basics tab, click the Manage button. Here, you can click on "Add" to add a search engine.

Chrome

So that takes care of all the basic searches: Google, Amazon, IMDB, Wikipedia, etc.

Can we go further? Item 8 on the list caught my attention:

Twit. As much as I love full-featured Twitter clients like TweetDeck, nothing beats the simplicity of hitting Ctrl-Space and typing twit [message] to so_and_so, or sending a selection of text using twit this to so_and_so. At the moment, there’s no way to receive tweets or ping Twitter for new messages.

I don’t use Twitter, but I do use Identi.ca, and I would like something like this. Right now, I’m using Google Talk to update identi.ca. Two problems. I don’t like chatting, and logging on exposes me to a lot of distraction. Secondly, I’d rather not have to open an application just for this. Something in the browser would be perfect. But is it possible?

Identi.ca (and Twitter, and most micro-blogging services) let you update via e-mail. So if I could write a program that would mail identi.ca, I should be done. So I did that with a Perl script.

my $q = new CGI;
open OUT, "|/usr/sbin/sendmail -t";
print OUT join "\n",
    "From: my.authenticated.id@gmail.com",
    "To: 1234567890abcdef@identica.com",
    "Subject: \n\n",
    $q->param('q');

So if I placed this at www.s-anand.net/identica (no, I haven’t placed it there), I just need to create a keyword search with a prefix Tidentica" that points to www.s-anand.net/identica?q=%s. Then I can type "identica Here is a message that I want to post" on the address bar, and it gets posted.

Actually, if you can write your own programs, the possibilities are endless. If you’re looking for someone to host this sort of thing for free, Google’s AppEngine may be a reasonable point to start.

But the real power of this comes with Javascript. Those URLs that you saw for keyword searches? Those can be Javascript URLs. So item 9 on the list

Word count. As a student of copywriting, I’m frequently curious about an article’s word length. Highlighting the desired text and entering word count into Ubiquity will give you just that.

… might just be possible.

It’s easy to get the selection. The following snippet gives you the current selection. (Tested in IE 5.5 – 8, Firefox 3 and Google Chrome. Should work for Opera, Safari.)

document.selection ? document.selection.createRange().text :
window.getSelection ? window.getSelection().toString() : ""

To get the word count, just split by white space, and count the results:

s = document.selection ? document.selection.createRange().text :
    window.getSelection ? window.getSelection().toString() : "";
alert(s.split(/\s+/).length + " words")

Now, this whole thing can be made into a keyword search. Let’s call it count. If I go to the address bar and type "count it", I want to use count the words in the selection. If I typed "count some set of words here", I want to count the words in "some set of words here". Here’s how to do that.

javascript:var s = "%s";
if (s == "it") {
  s = document.selection ? document.selection.createRange().text :
      window.getSelection ? window.getSelection().toString() : "";
}
alert(s.split(/\s+/).length + " words");

Now, put all of this in one line and add it as your keyword search. Try it!

(Note: You need to replace { curly braces } with %7B and %7D in Google Chrome. It interprets curly braces as a special command. Also, Chrome replaces spaces with a +, so the word count will always return 1 if you search for "count some set of words here".)

You could use selections to search as well. If you wanted to Google your selection, just use:

javascript:var s = "%s";
if (s == "it") {
  s = document.selection ? document.selection.createRange().text :
      window.getSelection ? window.getSelection().toString() : "";
}
location.replace("http://www.google.com/search?q=" + s)

Typing "google it" will search for your selected words on Google. "google some words" will search for "some words" on Google.

I’ve configured these keyword searches on my browser to:

  • Share sites. Typing "share google" adds the page to Google Reader, "share delicious" posts it to del.icio.us, "share digg" diggs the page, etc.
  • Send mail from the address bar. Typing "mail someone@gmail.com sub:This is the subject. Rest of the message" in the address bar will send the mail out. (Of course, you need to have created a mail gateway. I’ll try and share this shortly.)
  • Add entries to my calendar. Typing "remind Prepare dinner at 8pm" adds a reminder to my calendar to prepare dinner at 8pm.
  • Highlight parts of a page. Typing "highlight it" highlights what I’ve selected on the page. Even after I remove the selection, the highlighting stays. Typing "highlight some phrase" highlights all occurrences of "some phrase" in the entire document. The colours change every time you use it on a page, so you can search for multiple words and see where how they’re distributed.
  • Replaces tables with charts. Typing "chart it" with a table selected replaces the table with a chart. Typing "chart it as pie" or "chart it as scatter" changes the chart type.

You could actually take any bookmarklet and convert it into a keyword search. Which means that practically anything you can do on Javascript can be convert into a command-line-like syntax on the address bar.

So there it is! You can pretty much have a web command line. I wonder if we could add UNIX-pipes-like functionality.

Caching pages on Apache

I don’t use any blogging software for my site. I just hand-wired it some years ago. When doing this, one of the biggest problems was caching.

Consider each blog entry page. Each page has the same template, but different content. Both the template and content could be changed. So ideally, blog pages should be served dynamically. That is, every time someone requests the page, I should look up the content, look up the template, and put them together.

I did that, and within a few days outgrew my hosting service‘s CPU usage limit. Running such a program for every page hit is too heavy on the CPU.

One way around this is to create the pages beforehand and serve it as regular HTML. But every time the template changes, you need to re-generate every single page. I had over 2,500 pages. That would kill the CPU usage if I changed the template often.

At that point, I did a piece of analysis. Do I really need to regenerate all 2000 blog entries? Wouldn’t the 80-20 rule apply? The Apache log confirmed that 20% of the URLs were accounting for 76% of the hits. So I’d be wasting my time regenerating all the pages every time I changed the template.

Graph: 20% of URLs account for 76% of hits

So based on this, I decided to dynamically cache the pages. When a page is requested for the first time, I create the page and save it in a cache. The next time, I’d just serve it from the cache. If the template changes, I just need to delete the cache. This way, I only generate pages that are requested, and they’re only generated once.

OK, so that’s the background. Now let me get to how I did it.

I wrote a Perl script, blog.pl, that would generate a page in the html folder whenever it is called. Next, I changed Apache‘s .htaccess to run this program only if the page did not exist in the html folder.

# Redirect to cache first
RewriteCond %{REQUEST_FILENAME} !-f
RewriteRule ^([^/]*)\.html$       html/$1.html

# If not found, run program to create page
RewriteCond %{REQUEST_FILENAME} !-f
RewriteRule ^html/([^/]*)\.html$  blog.pl?t=$1

The first block redirects Apache to the cache. The second block checks if the file exists in the cache. If it doesn’t, the Apache redirects to the program. The program creates the page in the cache and displays it. Thereafter, Apache will just serve the file from the cache.


This Apache trick can be used in another way. I keep files organised in different folders to simplify my work. But to visitors of this site, that organisation is irrelevant. So I effectively merge these folders into one. For example, I have a folder called a in which I keep my static content. I also have this piece of code:

RewriteCond %{REQUEST_FILENAME} !-f
RewriteRule ^([^/]+)$   a/$1

If any file is not found in the main folder, just check in the a/ folder. So I can access the file /a/hindholam.midi as /hindholam.midi as well.

This can be extended to a series of folders: either as a cascade of caches, or to merge many folders into one.

JPath – XPath for Javascript

XPath is a neat way of navigating deep XML structures. It’s like using a directory structure. /table//td gets all the TDs somewhere below TABLE.

Usually, you don’t need this sort of a thing for data structures, particularly in JavaScript. Something like table.td would already work. But sometimes, it does help to have something like XPath even for data structures, so I built a simple XPath-like processor for Javascript called JPath.

Here are some examples of how it would work:

jpath(context, “para”) returns context.para
jpath(context, “*”) returns all values of context (for both arrays and objects)
jpath(context, “para[0]”) returns context.para[0]
jpath(context, “para[last()]”) returns context.para[context.para.length]
jpath(context, “*/para”) returns context[all children].para
jpath(context, “/doc/chapter[5]/section[2]”) returns context.doc.chapter[5].section[2]
jpath(context, “chapter//para”) returns all para elements inside context.chapter
jpath(context, “//para”) returns all para elements inside context
jpath(context, “//olist/item”) returns all olist.item elements inside context
jpath(context, “.”) returns the context
jpath(context, “.//para”) same as //para
jpath(context, “//para/..”) returns the parent of all para elements inside context

Some caveats:

  • This is an implementation of the abbreviated syntax of XPath. You can’t use axis::nodetest
  • No functions are supported other than last()
  • Only node name tests are allowed, no nodetype tests. So you can’t do text() and node()
  • Indices are zero-based, not 1-based

There are a couple of reasons why this sort of thing is useful.

  • Extracting attributes deep down. Suppose you had an array of arrays, and you wanted the first element of each array.
    Column Selection
    You could do this the long way:
    for (var list=[], i=0; i < data.length; i++) {
        list.push(data[i][0]);
    }
    

    ... or the short way:

    $.map(data, function(v) {
        return v[1];
    })

    But the best would be something like:

    jpath(data, "//1")
    
  • Ragged data structures. Take for example the results from Google's AJAX feed API.
    {"responseData": {
     "feed": {
      "title": "Digg",
      "link": "http://digg.com/",
      "author": "",
      "description": "Digg",
      "type": "rss20",
      "entries": [
       {
        "title": "The Pirate Bay Moves Servers to Egypt Due to Copyright Laws",
        "link": "http://digg.com/tech_news/The_Pirate_Bay_Moves_Servers_to_Egypt_Due_to_Copyright_Laws",
        "author": "",
        "publishedDate": "Mon, 31 Mar 2008 23:13:33 -0700",
        "contentSnippet": "Due to the new copyright legislation that are going ...",
        "content": "Due to the new copyright legislation that are going to take...",
        "categories": [
        ]
       },
       {
        "title": "Millions Dead/Dying in Recent Mass-Rick-Rolling by YouTube.",
        "link": "http://digg.com/comedy/Millions_Dead_Dying_in_Recent_Mass_Rick_Rolling_by_YouTube",
        "author": "",
        "publishedDate": "Mon, 31 Mar 2008 22:53:30 -0700",
        "contentSnippet": "Click on any \u0022Featured Videos\u0022. When will the insanity stop?",
        "content": "Click on any \u0022Featured Videos\u0022. When will the insanity stop?",
        "categories": [
        ]
       },
       ...
      ]
     }
    }
    , "responseDetails": null, "responseStatus": 200}
    

    If you wanted all the title entries, including the feed title, the choice is between:

    var titles = [ result.feed.title ];
    for (var i=0, l=result.feed.entries.length; i<l; i++) {
        titles.push(result.feed.entries[i].title;
    }
    

    ... versus...

    titles = jpath(result, '//title');
    

    If, further, you wanted the list of all categories at one shot, you could use:

    jpath(result, "//categories/*")
    

In search of a good editor

It’s amazing how hard it is to get a good programming editor. I’ve played around with more editors/IDEs than I care to remember: e Notepad++ NoteTab SciTE Crimson Editor Komodo Eclipse Aptana

There are four features that are critical to me.

  • Syntax highlighting. Over time, I’ve found this to increase readability dramatically. Look at this piece of code with and without syntax highlighting:
    Syntax Highlighting
    Doesn’t the structure of the document just jump out with syntax highlighting? Anyway, I’ve gotten used to that.
  • Column editing. I want to be able to do this:
    Column Editing
    Being able to type across rows is incredibly useful. I use it both for programming as well as to complement data-processing on Excel.
  • Unicode support. I often work with non-ASCII files, particularly in Tamil. Unicode support comes in handy when debugging pages for my songs site.
  • Auto-completion. This is 10 times more productive than having to look up the manual for each function.
    AutoCompletion

(Oh, and it’s got to be free too. Except for e Text Editor, all the others qualify.)

The problem is, none of the browsers that I’ve looked at support all of these features.

Editor Syntax highlighting Column editing Unicode support Auto-completion
e Text Editor Yes Yes No Yes
Crimson Editor Yes Yes No No
Notepad++ Yes No Yes No
NoteTab-Lite No No No No
SciTE Yes No Yes Yes
TextPad Yes No Yes No
UltraEdit Yes No No ?
Aptana Yes No Yes Yes
Eclipse Yes No Yes Yes
Komodo Yes No Yes Yes

Wikipedia has a more in-depth comparison of text editors.

Actually, there’s another parameter that’s pretty important: responsiveness. When I type something, I want to see it on the screen. Right that millisecond. With some of the features added by these editors, there’s so much bloat that it often takes up to one second between the keypress and the refresh. That’s just not OK.

I’ve settled on Crimson Editor as my default editor these days, simply because it’s quick and has column editing. (Column editing on e Text Editor is a bit harder to use.) When I am writing Unicode, I switch over to Notepad++. For large programs, I’m leaning towards Komodo right now, largely because Eclipse is bloated and Aptana was slow. (Komodo is slow too. Maybe I’ll switch back.)

There’s many other things on my “would love to have” features, like regular-expression search and replace, line sorting, code folding, brace matching, word wrapping, etc. Most of those, though, are either not too important, or most browsers already have them.

Well, there’s the sad thing. I’ve been hunting for a good text editor for over 10 years now. May someone write a lightweight IDE with column editing.

Automating Internet Explorer with jQuery

Most of my screen-scraping so far has been through Perl (typically WWW::Mechanize). The big problem is that it doesn’t support Javascript, which can often be an issue:

  • The content may be Javascript-based. For example, Amazon.com shows the bestseller book list only if you have Javascript enabled. So if you’re scraping the Amazon main page for the books bestseller list, you won’t get it from the static HTML.
  • The navigation may require Javascript. Instead of links or buttons in forms, you might have Javascript functions. Many pages use these, and not all of them degrade gracefully into HTML. (Try using Google Video without Javascript.)
  • The login page uses Javascript. It creates some crazy session ID, and you need Javascript to reproduce what it does.
  • You might be testing a Javascript-based web-page. This was my main problem: how do I automate testing my pages, given that I make a lot of mistakes?

There are many approaches to overcoming this. The easiest is to use Win32::IE::Mechanize, which uses Internet Explorer in the background to actually load the page and do the scraping. It’s a bit slower than scraping just the HTML, but it’ll get the job done.

Another is to use Rhino. John Resig has written env.js that mimics the browser environment, and on most simple pages, it handles the Javascript quite well.

I would rather have a hybrid of both approaches. I don’t like the WWW::Mechanize interface. I’ve gotten used to jQuery‘s rather powerful selectors and chainability. So I’ll tell you a way of using jQuery to screen-scrape offline using Python. (It doesn’t have to be Python. Perl, Ruby, Javascript… any scripting language that can use COM on Windows will work.)

Let’s take Google Video. Currently, it relies almost entirely on Javascript. The video marked in red below appears only if you have Javascript.

The left box showing the top video uses Javascript

I’d like an automated way of checking what video is on top on Google Video every hour, and save the details. Clearly a task for automation, and clearly not one for pure HTML-scraping.

I know the video’s details are stored in elements with the following IDs (thanks to XPath checker):

ID What’s there
hs_title_link Link to the video
hs_duration_date Duration and date
hs_ratings Ratings. The stars indicate the rating and the span.Votes element inside it has the number of people who rated it.
hs_site The site that hosts the video
hs_description Short description

So I could do the following on Win32::IE::Mechanize.

use Win32::IE::Mechanize;
my $ie = Win32::IE::Mechanize->new( visible => 1 );
$ie->get("http://video.google.com/");
my @links = $ie->links
# ... then what?

I could go through each link to extract the hs_title_link, but there’s no way to get the other stuff.

Instead, we could take advantage of a couple of facts:

  • Internet Explorer exposes a COM interface. That’s what Win32::IE::Mechanize uses. You can use it in any scripting language (Perl, Ruby, Javascript, …) on Windows to control IE.
  • You can load jQuery on to any page. Just add a <script> tag pointing to jQuery. Then, you can call jQuery from the scripting language!

Let’s take this step by step. This Python program opens IE, loads Google Video and prints the text.

# Start Internet Explorer
import win32com.client
ie = win32com.client.Dispatch("InternetExplorer.Application")
 
# Display IE, so you'll know what's happening
ie.visible = 1
 
# Go to Google Video
ie.navigate("http://video.google.com/")
 
# Wait till the page is loaded
from time import sleep
while ie.Busy: sleep(0.2)
 
# Print the contents
# Watch out for Unicode
print ie.document.body.innertext.encode("utf-8")

The next step is to add jQuery to the Google Video page.

# Add the jQuery script to the browser
def addJQuery(browser,
    url="http://jqueryjs.googlecode.com/files/jquery-1.2.4.js"
 
    document = browser.document
    window = document.parentWindow
    head = document.getElementsByTagName("head")[0]
    script = document.createElement("script")
    script.type = "text/javascript"
    script.src = url
    head.appendChild(script)
    while not window.jQuery: sleep(0.1)
    return window.jQuery
 
jQuery = addJQuery(ie)

Now the variable jQuery contains the Javascript jQuery object. From here on, you can hardly tell if you’re working in Javascript or Python. Below are the expressions (in Python!) to get the video’s details.

# Video title: "McCain's YouTube Problem ..."
jQuery("#hs_title_link").text()
 
# Title link: '/videoplay?docid=1750591377151076231'
jQuery("#hs_title_link").attr("href")
 
# Duration and date: '3 min - May 18, 2008 - '
jQuery("#hs_duration_date").text()
 
# Rating: 5.0
jQuery("#hs_ratings img").length
 
# Number of ratings '(8,288 Ratings) '
jQuery("#hs_ratings span.Votes").text()
 
# Site: 'Watch this video on youtube.com'
jQuery("#hs_site").text()
 
# Video description
jQuery("#hs_description").text()

This wouldn’t have worked out as neatly in Perl, simply because you’d need to use -> instead of . (dot). With Python (and with Ruby and Javascript on cscript), you can almost cut-and-paste jQuery code.

If you want to click on the top video link, use:

jQuery("#hs_title_link").get(0).click()

In addition, you can use the keyboard as well. If you want to type username TAB password, use this:

shell = win32com.client.Dispatch("WScript.Shell")
shell.sendkeys("username{TAB}password")

You can use any of the arrow keys, control keys, etc. Refer to the SendKeys Method on MSDN.