S Anand

In October 1997, Ram, my manager at IBM, strolled over to my desk and asked if I would like to visit the US. I'd never been there before. The impulse was to say "Yes". But...

I'd written the CAT exam once before. Didn't get through. Applied once again. But thanks to my diligence, I'd given the wrong residence address, and never got my admission card, and didn't bother following it up. This would be my third "attempt". And I didn't want to goof it up again. (I didn't get through that one either, as it turned out.)

"Ram, I need to be back on Dec 11th."

"Mmm... I think we should be back by then."

"NO MATTER WHAT!"

He smiled, and said "OK. We'll be back by Dec 11th NO MATTER WHAT." He thought I was going to get married or something.

It was quite warm in Bangalore, so I set out with a T-shirt and formal trousers. As I was leaving, my landlord and landlady (very nice people, and in retrospect, very far-sighted) pulled me in and said, "Have some snacks. You'll feel hungry on the way."

I tried my protests. They'll feed me on the plane. I'm already carrying some food. I have cash to buy stuff. I'm fat and dieting. Didn't matter. I still ended up carrying a fairly hefty package. "And this is for Kallol." Another package. A colleague travelling with me was an ex-roommate as well. I just hoped I wouldn't exceed 27 kgs.

It was a KLM flight that would halt at Amsterdam. We were to land early morning in Amsterdam, take a connecting flight to Boston, and then over to Charlotte. We'd reach Charlotte by night, in time for the class next day.

The flight itself was uneventful, except for my first non-vegetarian bite.

And then the fun began.

Breakfast was done by around 5:00am local time. The captain announced that we were near Amsterdam, fasten your seatbelts.

5:30am. No landing.

6:00am. No landing. When I pulled the shutters up, we were still flying over clouds.

7:00am. No landing.

7:30am. The captain announces that due to bad weather at Amsterdam, we would not be able to land there. We were being diverted to Cologne.

Not having been on any long-haul flights before, I wasn't even worried. It was a KLM connecting flight. KLM would do something. But for feeling a bit hungry, things were fine.

At around 11:00am, the plane began its descent. We were amidst clouds, though. For quite a while... and the plane kept descending...

Until, all of a sudden, I could see the ground about 20 feet from the plane! The fog dense enough to be indistinguishable from clouds. (Or at least, I couldn't tell the difference.) Lucky the pilot managed to land, and I'm surprised he even tried.

8:00am. We're still in the plane, waiting.

9:00am. Hungry. No one has told us anything yet.

9:30am. We're all asked to get down. Delighted, we all got off, ready to board the next plane...

... only to be herded off into a glass building on the terminal, where our luggage was waiting for us. No problem. Pick up luggage. Wait.

10:00am. All the flight staff had cleared the terminal. And, looking out of the glass walls, we could see our plane taking off! There was a fair bit of confusion (and mild panic) in the room, but being the suave software engineers that we were, we stay put and relaxed.

11:00am. Still in the glass building. No flight has landed or taken off. Worse, no human in sight. I mean it: not a single human in sight other than us KLM passengers in this deserted terminal. We're still hungry.

12:00noon. My snacks finally come out. We all have a bite. That turned out to be our lunch.

12:30pm. Some official enters the building and is mobbed. The closest we could get to him (or her?) was about 50m behind many hundreds of raised heads.

12:45pm. Official vanishes. We ask around if anyone knows more than we do. No one seems to.

1:30pm. Another official enters. Vanishes after a few minutes.

2:00pm. Finally, word gets around that we'll be travelling via bus to Amsterdam. Clearly we'd missed our connecting flight. We'd be put in to the same flight the next day.

2:10pm. We hear a lot of activity. People start streaming out of the building. We try to join in the rush.

2:20pm. Ahead of us, we see a guy checking passports. Now, none of us had a German visa. Presumably it was OK, but in any case, we were entering Germany without a valid visa. The official stamped my passport without question.

2:30pm. We exit the airport. The temperature was 0 degrees C. I was still in my T-shirt. My warm clothes were packed. That day, I learnt two lessons. One, never keep all your warm clothes inaccessibly in the check-in baggage. (I had my check-in baggage. But it was packed, and if I opened it, I can't put it back in. Besides, we were being herded into a bus: not much chance of hanging around to open a suitcase.) Two, it's actually possible to get a headache from the cold. For 15 freezing minutes, we stood on the road waiting for the bus, and enjoying the pleasures of our first day on European soil.

2:45pm. Bus arrives. Mob tries to enter bus. Half of our group manages to get through. I am left behind. Fortunately, next bus is only 5 minutes behind.

7:00pm. Bus finally arrives at Schipol airport. We're herded out to the KLM counter. By now, it's been well over 24 hours since my last full meal.

7:30pm. We're told we'll get a hotel to stay in, and our flight is confirmed for the next day. At this point, we're famished. So we exchanged some currency, and decided to buy some food. I picked a green apple. This happened to be my first green apple. No one had told me that apples could taste sour. (While on that topic, I must mention apple pies. I love apple pies in India. I hate apple pies in London. I suspect it's the red versus green apples.)

7:31pm. I take one bite. Another bite. Have a funny feeling in my stomach. Burning sensation. And at that point, I collapsed. Physically. Just dropped on the floor and had to be pulled up.

8:00pm. Finally reach the hotel. Not entirely sure how. I'm too tired for anything but milk, so I get a glassful and go to sleep.

PS: We finally reached Charlotte a day late. Fortunately, we didn't miss much.

Apparantly, most passengers on the flight complained to KLM and received gifts / free miles of a substantial magnitude. We didn't know of that till much later.

This remains my only trip to Germany till date. My passport still holds an entry stamp without a visa.

We did get the bonus of spending half a day in Amsterdam, which is a rather nice place. Again, without a visa.

• Arrested in Paris
• I am a vegetarian
• You only eat plants
• Earthquake and impact

It's amazing how hard it is to get a good programming editor. I've played around with more editors/IDEs than I care to remember: e Notepad++ NoteTab SciTE Crimson Editor Komodo Eclipse Aptana ...

There are four features that are critical to me.

• Syntax highlighting. Over time, I've found this to increase readability dramatically. Look at this piece of code with and without syntax highlighting:
  
  Doesn't the structure of the document just jump out with syntax highlighting? Anyway, I've gotten used to that.
• Column editing. I want to be able to do this:
  
  Being able to type across rows is incredibly useful. I use it both for programming as well as to complement data-processing on Excel.
• Unicode support. I often work with non-ASCII files, particularly in Tamil. Unicode support comes in handy when debugging pages for my songs site.
• Auto-completion. This is 10 times more productive than having to look up the manual for each function.
  

(Oh, and it's got to be free too. Except for e Text Editor, all the others qualify.)

The problem is, none of the browsers that I've looked at support all of these features.

Wikipedia has a more in-depth comparison of text editors.

Actually, there's another parameter that's pretty important: responsiveness. When I type something, I want to see it on the screen. Right that millisecond. With some of the features added by these editors, there's so much bloat that it often takes up to one second between the keypress and the refresh. That's just not OK.

I've settled on Crimson Editor as my default editor these days, simply because it's quick and has column editing. (Column editing on e Text Editor is a bit harder to use.) When I am writing Unicode, I switch over to Notepad++. For large programs, I'm leaning towards Komodo right now, largely because Eclipse is bloated and Aptana was slow. (Komodo is slow too. Maybe I'll switch back.)

There's many other things on my "would love to have" features, like regular-expression search and replace, line sorting, code folding, brace matching, word wrapping, etc. Most of those, though, are either not too important, or most browsers already have them.

Well, there's the sad thing. I've been hunting for a good text editor for over 10 years now. May someone write a lightweight IDE with column editing.

• JPath - XPath for Javascript
• Automating Internet Explorer with jQuery
• Statistically improbable phrases on Google AppEngine update
• Statistically improbable phrases on Google AppEngine
• Chaining functions in Javascript

XPath is a neat way of navigating deep XML structures. It's like using a directory structure. /table//td gets all the TDs somewhere below TABLE.

Usually, you don't need this sort of a thing for data structures, particularly in JavaScript. Something like table.td would already work. But sometimes, it does help to have something like XPath even for data structures, so I built a simple XPath-like processor for Javascript called JPath.

Here are some examples of how it would work:

Some caveats:

• This is an implementation of the abbreviated syntax of XPath. You can't use axis::nodetest
• No functions are supported other than last()
• Only node name tests are allowed, no nodetype tests. So you can't do text() and node()
• Indices are zero-based, not 1-based

There are a couple of reasons why this sort of thing is useful.

• Extracting attributes deep down. Suppose you had an array of arrays, and you wanted the first element of each array.
  
  You could do this the long way:

  for (var list=[], i=0; i < data.length; i++) {
      list.push(data[i][0]);
  }
  

  ... or the short way:

  $.map(data, function(v) {
      return v[1];
  })

  But the best would be something like:

  jpath(data, "//1")
  

• Ragged data structures. Take for example the results from Google's AJAX feed API.

  {"responseData": {
   "feed": {
    "title": "Digg",
    "link": "http://digg.com/",
    "author": "",
    "description": "Digg",
    "type": "rss20",
    "entries": [
     {
      "title": "The Pirate Bay Moves Servers to Egypt Due to Copyright Laws",
      "link": "http://digg.com/tech_news/The_Pirate_Bay_Moves_Servers_to_Egypt_Due_to_Copyright_Laws",
      "author": "",
      "publishedDate": "Mon, 31 Mar 2008 23:13:33 -0700",
      "contentSnippet": "Due to the new copyright legislation that are going ...",
      "content": "Due to the new copyright legislation that are going to take...",
      "categories": [
      ]
     },
     {
      "title": "Millions Dead/Dying in Recent Mass-Rick-Rolling by YouTube.",
      "link": "http://digg.com/comedy/Millions_Dead_Dying_in_Recent_Mass_Rick_Rolling_by_YouTube",
      "author": "",
      "publishedDate": "Mon, 31 Mar 2008 22:53:30 -0700",
      "contentSnippet": "Click on any \u0022Featured Videos\u0022. When will the insanity stop?",
      "content": "Click on any \u0022Featured Videos\u0022. When will the insanity stop?",
      "categories": [
      ]
     },
     ...
    ]
   }
  }
  , "responseDetails": null, "responseStatus": 200}
  

  If you wanted all the title entries, including the feed title, the choice is between:

  var titles = [ result.feed.title ];
  for (var i=0, l=result.feed.entries.length; i<l; i++) {
      titles.push(result.feed.entries[i].title;
  }
  

  ... versus...

  titles = jpath(result, '//title');
  

  If, further, you wanted the list of all categories at one shot, you could use:

  jpath(result, "//categories/*")
  

• JPath - XPath for Javascript
• Automating Internet Explorer with jQuery
• Statistically improbable phrases on Google AppEngine update
• Statistically improbable phrases on Google AppEngine
• Chaining functions in Javascript

Most of my screen-scraping so far has been through Perl (typically WWW::Mechanize). The big problem is that it doesn't support Javascript, which can often be an issue:

• The content may be Javascript-based. For example, Amazon.com shows the bestseller book list only if you have Javascript enabled. So if you're scraping the Amazon main page for the books bestseller list, you won't get it from the static HTML.
• The navigation may require Javascript. Instead of links or buttons in forms, you might have Javascript functions. Many pages use these, and not all of them degrade gracefully into HTML. (Try using Google Video without Javascript.)
• The login page uses Javascript. It creates some crazy session ID, and you need Javascript to reproduce what it does.
• You might be testing a Javascript-based web-page. This was my main problem: how do I automate testing my pages, given that I make a lot of mistakes?

There are many approaches to overcoming this. The easiest is to use Win32::IE::Mechanize, which uses Internet Explorer in the background to actually load the page and do the scraping. It's a bit slower than scraping just the HTML, but it'll get the job done.

Another is to use Rhino. John Resig has written env.js that mimics the browser environment, and on most simple pages, it handles the Javascript quite well.

I would rather have a hybrid of both approaches. I don't like the WWW::Mechanize interface. I've gotten used to jQuery's rather powerful selectors and chainability. So I'll tell you a way of using jQuery to screen-scrape offline using Python. (It doesn't have to be Python. Perl, Ruby, Javascript... any scripting language that can use COM on Windows will work.)

Let's take Google Video. Currently, it relies almost entirely on Javascript. The video marked in red below appears only if you have Javascript.

I'd like an automated way of checking what video is on top on Google Video every hour, and save the details. Clearly a task for automation, and clearly not one for pure HTML-scraping.

I know the video's details are stored in elements with the following IDs (thanks to XPath checker):

So I could do the following on Win32::IE::Mechanize.

use Win32::IE::Mechanize;
my $ie = Win32::IE::Mechanize->new( visible => 1 );
$ie->get("http://video.google.com/");
my @links = $ie->links
# ... then what?

I could go through each link to extract the hs_title_link, but there's no way to get the other stuff.

Instead, we could take advantage of a couple of facts:

• Internet Explorer exposes a COM interface. That's what Win32::IE::Mechanize uses. You can use it in any scripting language (Perl, Ruby, Javascript, ...) on Windows to control IE.
• You can load jQuery on to any page. Just add a <script> tag pointing to jQuery. Then, you can call jQuery from the scripting language!

Let's take this step by step. This Python program opens IE, loads Google Video and prints the text.

# Start Internet Explorer
import win32com.client
ie = win32com.client.Dispatch("InternetExplorer.Application")

# Display IE, so you'll know what's happening
ie.visible = 1

# Go to Google Video
ie.navigate("http://video.google.com/")

# Wait till the page is loaded
from time import sleep
while ie.Busy: sleep(0.2)

# Print the contents
# Watch out for Unicode
print ie.document.body.innertext.encode("utf-8")

The next step is to add jQuery to the Google Video page.

# Add the jQuery script to the browser
def addJQuery(browser,
    url="http://jqueryjs.googlecode.com/files/jquery-1.2.4.js"):

    document = browser.document
    window = document.parentWindow
    head = document.getElementsByTagName("head")[0]
    script = document.createElement("script")
    script.type = "text/javascript"
    script.src = url
    head.appendChild(script)
    while not window.jQuery: sleep(0.1)
    return window.jQuery

jQuery = addJQuery(ie)

Now the variable jQuery contains the Javascript jQuery object. From here on, you can hardly tell if you're working in Javascript or Python. Below are the expressions (in Python!) to get the video's details.

# Video title: "McCain's YouTube Problem ..."
jQuery("#hs_title_link").text()

# Title link: '/videoplay?docid=1750591377151076231'
jQuery("#hs_title_link").attr("href")

# Duration and date: '3 min - May 18, 2008 - '
jQuery("#hs_duration_date").text()

# Rating: 5.0
jQuery("#hs_ratings img").length

# Number of ratings '(8,288 Ratings) '
jQuery("#hs_ratings span.Votes").text()

# Site: 'Watch this video on youtube.com'
jQuery("#hs_site").text()

# Video description
jQuery("#hs_description").text()

This wouldn't have worked out as neatly in Perl, simply because you'd need to use -> instead of . (dot). With Python (and with Ruby and Javascript on cscript), you can almost cut-and-paste jQuery code.

If you want to click on the top video link, use:

jQuery("#hs_title_link").get(0).click()

In addition, you can use the keyboard as well. If you want to type username TAB password, use this:

shell = win32com.client.Dispatch("WScript.Shell")
shell.sendkeys("username{TAB}password")

You can use any of the arrow keys, control keys, etc. Refer to the SendKeys Method on MSDN.

• JPath - XPath for Javascript
• Automating Internet Explorer with jQuery
• Statistically improbable phrases on Google AppEngine update
• Statistically improbable phrases on Google AppEngine
• Chaining functions in Javascript

I've added some interactivity to the Statistically improbable phrases application. You can now:

• Filter out stopwords
• Dynamically filter infrequent words and commonly used words
• Dynamically play with the contrast and font size

• JPath - XPath for Javascript
• Automating Internet Explorer with jQuery
• Statistically improbable phrases on Google AppEngine update
• Statistically improbable phrases on Google AppEngine
• Chaining functions in Javascript

I read about Google AppEngine early this morning, and applied for an invite. Google's issuing beta invites to the first 10,000 users. I was pretty convinced I wasn't among those, but turns out I was lucky.

AppEngine lets you write web apps that Google hosts. People have been highlighting that it give you access to the Google File System and BigTable for the first time. But to me, that isn't a big deal. (I'm not too worried about reliability, and MySQL / flat files work perfectly well for me as a data store.)

What's more interesting unlike Amazon's EC2 and S3, this is free up to a certain quota. And you get a fair bit of processing power and bandwidth for free. One of the reasons I've held back on creating some apps was simply because it would take away too much bandwidth / CPU cycles from my site. (I've had this problem before.) Google quota is 10 GB of bandwidth per day (which is about 30 times what my site uses). And this is on Google's incredibly fast servers It also offers 200 million megacycles a day. That's like a dedicated 2.3 GHz processor -- better, because this is the average capacity, not peak capacity. The only restriction that really worries me is that only 3 apps are allowed per developer.

So I decided to give a shot at publishing some code I'd kept in reserve for a long time. You may remember my statistical analysis of Calvin & Hobbes. For this, I'd created a script in Perl that could generate SIPs for any text. This is based on (a somewhat limited) 23MB corpus of ebooks that I had. I'd wanted to put that up on my website, but ...

AppEngine only uses Python. So the first task was to get Python, and then to learn Python. The only saving grace was that I was just cutting-and-pasting most of the time. Google wasn't helping:

Anyway, the site is up. You can view it at sip.s-anand.net for now. Just type a URL, and it'll tell you the improbable words in that site.

Visit sip.s-anand.net

Technical notes

I realise that these are statistically improbable words, not phrases. I'll get to the phrases in a while.

The logic is simple:

• Get the frequency of words in a corpus. I pre-generated this file. It has over 100,000 words.
• Get the URL as text. Rather than muck around with Python, I decided to use the W3 html2txt service.
• Convert the text to words. Splitting text into words is tricky. For now, I'm simply assuming that any group of letters is a word, and anything that's not a letter is a word delimiter.
• Find the relative frequency (improbability) of words. This is the frequency in the URL divided by the frequency in the corpus, normalised (i.e. scale it so that the maximum value is 1.0).
• Create a tag cloud. I use the word frequency as the size and the improbability as the colour. You need a bit of mathematical jugglery to get the pattern right. Right now, I'm taking the 6^th root of the improbability and the logarithm of the frequency to get a reasonably smooth tag cloud.

The source code is here.

Update: 12-Apr-2008. I've added some interactivity. You can play with the contrast and font size, the filter out common or infrequent words.

Update: 22-Apr-2008. Added concordance. You can click on a word and see the context in which it appears.

• JPath - XPath for Javascript
• Automating Internet Explorer with jQuery
• Statistically improbable phrases on Google AppEngine update
• Statistically improbable phrases on Google AppEngine
• Chaining functions in Javascript