I’m quite busy on a project right now, and don’t get time to write long articles. So for a while, I’m going to stick to short notes on interesting stuff.
Peter Bregman has a very interesting piece on Why You Should Encourage Weakness. It boils down to a choice: do you focus on on improving strengths or minimising weaknesses? Conventional performance evaluations focus on the latter. I very strongly support Bregman’s view on this. The weakness isn’t why you hired the person! Unless it’s killing the organisation, just leave them to focus on their strengths.
Google Analytics has a fairly interesting API that I hadn’t explored until recently. Picked up Advanced Web Metrics with Google Analytics and learnt that you can track outbound clicks, page load times, Javascript events and error logs, almost anything at all using Google Analytics. You can also mirror the logging on your local server using pageTracker._setLocalRemoteServerMode()
It’s amazing how much of a dependency there is on individuals writing IT systems. Reminds me of that Dilbert strip:
A few weeks ago, I was trying to figure out in what happens when there are multiple promotions. (Our client is a retailer.) I mean, if there’s a phone that costs £100 and there are 2 promotions: 10% off on phones and £10 off on phones. Do you apply the 10% off first and pay £80 or the £10 off and pay £81?
Funnily enough, the organisational answer is, “I don’t know.” The person who determined the logic is no longer with the firm. The person who wrote the code was a contractor and moved on to another project. The vendor hadn’t gotten around to documenting the code. Sure, the code’s there, and you just had to read it to figure out what it does. But no human knew what it was supposed to do.
Last week, there was a decision to rewrite some code that was 10 years old. A colleague who wasn’t quite involved in this work said, “I’m going to have to set aside 2-3 weeks for this. I wrote this stuff when I was a developer. The docs have vanished. The business owners have vanished. I’m the only one who has any clue on what it’s supposed to do.”
This week, we were trying to figure out how their store locator system works. After fiddling around with Fiddler, and seeing that it used Microsoft Virtual Earth, I was able to figure out that it identified stores near a location using a simple JSON API. But can we get the documentation around that? Nope. Tough luck. Nobody knows how it works any longer.
Personally, I don’t think this is unusual. We forget. Companies forget. But it’s usually good if what we forget is derivable. That’s how I got through my high-school physics exams: not by remembering stuff, but by being able to derive the stuff from a few principles.
Organisations can do the same. But to be able to do that, you need to have commonly understood principles. As Fred Brooks put it in The Mythical Man Month,
I contend that conceptual integrity is the most important consideration in system design. It is better to have a system omit certain anomalous features and improvements, but to reflect one set of design ideas, than to have one that contains many good but independent and uncoordinated ideas.
One of the biggest enemies of conceptual integrity is growth. Too many people too soon, and the important decisions are taken by people who’ve never had a long chat about things. There’s another reason not to grow too fast.
With allthediscussion around URL shorteners, Diggbar, blockingit, and the rev=canonical proposal, I decided to implement a URL shortening service on this blog with the least effort possible. This probably won’t impact you just yet, but when tools become more popular and sophisticated, it would hopefully eliminate the need for tinyurl, bit.ly, etc.
Since the blog runs on WordPress, every post has an ID. The short URL for any post will simply be http://www.s-anand.net/the_ID. For example, http://s-anand.net/17 is a link to post on Ubuntu on a Dell Latitude D420. At 21 characters, it’s roughly the same size as most URL shorteners could make it.
The code is easy: just one line added to index.php:
Writing a program to draw or change slides is sometimes easier than doing it manually. To change all fonts on a presentation to Arial, for example, you’d write this Visual Basic macro:
Sub Arial()
For Each Slide In ActivePresentation.Slides
For Each Shape In Slide.Shapes
Shape.TextFrame.TextRange.Font.Name = "Arial"
Next
Next
End Sub
If you didn’t like Visual Basic, though, you could write the same thing in Python:
import win32com.client, sys
Application = win32com.client.Dispatch("PowerPoint.Application")
Application.Visible = True
Presentation = Application.Presentations.Open(sys.argv[1])
for Slide in Presentation.Slides:
for Shape in Slide.Shapes:
Shape.TextFrame.TextRange.Font.Name = "Arial"
Presentation.Save()
Application.Quit()
Save this as arial.py and type “arial.py some.ppt” to convert some.ppt into Arial.
Let’s break that down a bit. import win32com.client lets you interact with Windows using COM. You need ActivePython to do this. Now you can launch PowerPoint with
The Application object you get here is the same Application object you’d use in Visual Basic. That’s pretty powerful. What that means is, to a good extent, you can copy and paste Visual Basic code into Python and expect it to work with minor tweaks for language syntax, just please make sure to learn how to update python before doing anything else.
So let’s try to do something with this. First, let’s open PowerPoint and add a blank slide.
# Open PowerPoint
Application = win32com.client.Dispatch("PowerPoint.Application")
# Create new presentation
Presentation = Application.Presentations.Add()
# Add a blank slide
Slide = Presentation.Slides.Add(1, 12)
That 12 is the code for a blank slide. In Visual Basic, you’d instead say:
Slide = Presentation.Slides.Add(1, ppLayoutBlank)
To do this in Python, run Python/Lib/site-packages/win32com/client/makepy.py and pick “Microsoft Office 12.0 Object Library” and “Microsoft PowerPoint 12.0 Object Library”. (If you have a version of Office other than 12.0, pick your version.)
This creates two Python files. I rename these files as MSO.py and MSPPT.py and do this:
import MSO, MSPPT
g = globals()
for c in dir(MSO.constants): g[c] = getattr(MSO.constants, c)
for c in dir(MSPPT.constants): g[c] = getattr(MSPPT.constants, c)
This makes constants like ppLayoutBlank, msoShapeRectangle, etc. available. So now I can create a blank slide and add a rectangle Python just like in Visual Basic:
Incidentally, the dimensions are in points (1/72″). Since the default presentation is 10″ x 7.5″ the size of each page is 720 x 540.
Let’s do something that you’d have trouble doing manually in PowerPoint: a Treemap. The Guardian’sdata store kindly makes available the top 50 banks by assets that we’ll use for this example. Our target output is a simple Treemap visualisation.
We’ll start by creating a blank slide. The code is as before.
import win32com.client, MSO, MSPPT
g = globals()
for c in dir(MSO.constants): g[c] = getattr(MSO.constants, c)
for c in dir(MSPPT.constants): g[c] = getattr(MSPPT.constants, c)
Application = win32com.client.Dispatch("PowerPoint.Application")
Application.Visible = True
Presentation = Application.Presentations.Add()
Slide = Presentation.Slides.Add(1, ppLayoutBlank)
import urllib2, csv
url = 'http://spreadsheets.google.com/pub?key=phNtm3LmDZEOoyu8eDzdSXw&output=csv&range=B2:C51'
# Open the URL using a CSV reader
reader = csv.reader(urllib2.urlopen(url))
# Convert the CSV into a list of (asset-size, bank-name) tuples
data = list((int(s.replace(',','')), b.decode('utf8')) for b, s in reader)
I created a simple Treemap class based on the squarified algorithm — you can play with the source code. This Treemap class can be fed the data in the format we have, and a draw function. The draw function takes (x, y, width, height, data_item) as parameters, where data_item is a row in the data list that we pass to it.
def draw(x, y, w, h, n):
# Draw the box
shape = Slide.Shapes.AddShape(msoShapeRectangle, x, y, w, h)
# Add text: bank name (asset size in millions)
shape.TextFrame.TextRange.Text = n[1] + ' (' + str(int(n[0]/1000 + 500)) + 'M)'
# Reduce left and right margins
shape.TextFrame.MarginLeft = shape.TextFrame.MarginRight = 0
# Use 12pt font
shape.TextFrame.TextRange.Font.Size = 12
from Treemap import Treemap
# 720pt x 540pt is the size of the slide.
Treemap(720, 540, data, draw)
Try running the source code. You should have a single slide in PowerPoint like this.
The beauty of using PowerPoint as the output format is that converting this into a cushioned Treemap with gradients like below (or changing colours, for that matter), is a simple interactive process.
The Random Quotes Generator is a simple tool that creates quotes by mixing up words on a web page. The results are often funny, but sometimes surprisingly insightful.
Yes, this is the equivalent of a million monkeys typing Shakespeare, except that they’re using the works of Shakespeare as a starting point. And it doesn’t have to be Shakespeare. It could be you or your friends.
To try it out, visit this page, select the link and “Add to Favorites” or drag it into your browser’s bookmark toolbar.. Then go to any web page that has a lot of text, and click the link to generate random quotes.
The net will find monetization models of theater and sporting events before them. Indeed, there has to be some way to create websites that do other than Advertising. The expected drop in internet advertising will rapidly lose its value and its impact, for reasons that can easily be understood.
For the technically minded, Programming Pearls has a section on Generating Text that explains the concept. The bookmarklet uses an Order-2 word-level Markov chain. Translated into English, what that means is: I look at every pair of words in and find out what word is likely to follow that.
For example, in the Generating Text page, the pair of words “we can” are followed by the words “extend”, “also”, “get” and “write” with equal probability. We pick one randomly (say “also”) and write “we can also”. Then we look at the word pair “can also”, see what word follows that, pick one at random, and so on.
This is Order-2 because we pick pairs of words. And it’s word-level rather than letter-level because we use words instead of letters as the basic building blocks.
When you’re trying it out, make sure that the page is large enough. If not, you may find that the page’s content is reproduced verbatim.
The bookmarklet is built on top of the excellent Readabilitybookmarklet by Arc90, which helps identify the main content to be randomized.
Creating motion charts in Excel is a simple four-step process.
Get the data in a tabular format with the columns [date, item, x, y, size]
Make a “today” cell, and create a lookup table for “today”
Make a bubble chart with that lookup table
Add a scroll bar and a play button linked to the “today” cell
For the impatient, here’s a motion chart spreadsheet that you can tailor to your needs. For the patient and the puzzled, here’s a quick introduction to bubble and motion charts.
What is a bubble chart?
A bubble chart is a way of capturing 3 dimensions. For example, the chart below could be the birth, literacy rate and population of countries (X-axis, Y-axis and size). Or the growth, margin and market cap of companies.
It lets you compare three dimensions at a glance. The size dimension is a different from the X and Y axes, though. It’s not easy to compare differences in size. And the eye tends to focus on the big objects. So usually, size is used highlight important things, and the X and Y axes used to measure the performance of these things.
If I were to summarise bubble charts in a sentence, it would be: bubble charts show the performance of important things (in two dimensions). (In contrast, Variwide charts show the same on one dimension.)
Say you’re a services firm. You want to track the productivity of your most expensive groups (“the important things”). Productivity is measured by 2 parameters: utilisation and margin. The bubble chart would then have the expense of each group as the size, and its utilisation and contribution as the X and Y axes.
What is a motion chart?
Motion charts are animated bubble charts. They track the performance of important things over time (in two dimensions). This is chart with 4 dimensions. But not all data with 4 dimensions can be plotted as a motion chart. One dimension has to be time, and another has to be linked to the importance of the item.
Motion charts were pioneered by Hans Rosling and his TED Talk shows you the true power of motion charts.
How do I create these charts?
Use the Motion Chart Gadget to display any of your data on a web page. Or use Google Spreadsheets if you need to see the chart on a spreadsheet: motion charts are built in.
If you or your viewer don’t have access to these, and you want to use Excel, here’s how.
1. Get the data in a tabular format
Get the data in the format below. You need the X, Y and size for each thing, for each date.
Date
Thing
X
Y
Size
08/02/2009
A
64%
11%
1
08/02/2009
B
14%
33%
2
08/02/2009
C
78%
55%
3
08/02/2009
D
57%
73%
4
08/02/2009
E
39%
32%
5
08/02/2009
F
40%
81%
6
09/02/2009
A
64%
12%
1
09/02/2009
B
14%
33%
2
09/02/2009
C
78%
56%
3
09/02/2009
D
57%
73%
4
09/02/2009
E
39%
32%
5
09/02/2009
F
40%
81%
6
…
…
…
…
..
To make life (and lookups) easier, add a column called “Key” which concatenates the date and the things. Typing “=A2&B2” will concatenate cells A2 and B2. (Red cells use formulas.)
Date
Thing
Key
X
Y
Size
08/02/2009
A
39852A
64%
11%
1
08/02/2009
B
39852B
14%
33%
2
08/02/2009
C
39852C
78%
55%
3
08/02/2009
D
39852D
57%
73%
4
…
…
…
…
…
…
2. Make a “today” cell, and create a lookup table for “today”
Create a cell called “Offset” and type in 0 as its value. Add another cell called Today whose value is the start date (08/02/2009 in this case) plus the offset (0 in this case)
Offset
0
(Just type 0)
Today
08/02/2009
Use a formula: =STARTDATE + OFFSET
Now, if you change the offset from 0 to 1, “Today” changes to 09/02/2009. By changing just this one cell, we can create a table that holds the bubble chart details for that day, like below.
This is a simple Insert – Chart. Go through the chart types and select bubble. Play around with the data selection until you get the X, Y and Size columns right.
4. Add a scroll bar and a play button linked to the “today” cell
Now for the magic. Add a scroll bar below the chart. Excel 2007 users: Go to Developer – Insert and add a scroll bar. Excel 2003 users: Go to View – Toolbars – Control Toolbox and add a scroll bar
Right click on the scroll bar, go to Format Control… and link the scroll bar to the “Offset” cell. Now, as you move the scroll bar, the value in the offset cell will change to reflect it. So the “today” cell will change too. So will the lookup table. And so will the chart.
Next, create a button called “Play” and edit its code. Excel 2007 users: Right click the button, go to Developer – View Code. Excel 2003 users: Right click the button and select View Code.
Type in the following code for the button’s click event:
DeclareSub Sleep Lib "kernel32" (ByVal dwMilliseconds AsLong)
Sub Button1_Click()
Dim i AsIntegerFor i = 0 To 40: ' Replace 40 with your range
Range("J1").Value = i ' Replace J1 with your offset cell
Application.Calculate
Sleep (100)
NextEndSub
Declare Sub Sleep Lib "kernel32" (ByVal dwMilliseconds As Long)
Sub Button1_Click()
Dim i As Integer
For i = 0 To 40: ' Replace 40 with your range
Range("J1").Value = i ' Replace J1 with your offset cell
Application.Calculate
Sleep (100)
Next
End Sub
Now clicking on the Play button will give you this glorious motion chart in Excel:
One of the reasons I moved to WordPress was the ability to write posts offline, for which I use Windows Live Writer most of the time. The beauty of this is that I can preview the post exactly as it will appear on my site. Nothing else that I know is as WYSIWYG, and it’s very useful to be able to type knowing exactly where each word will be.
The only hitch is: if you write your own WordPress theme, Live Writer probably won’t be able to detect your theme — unless you’re an expert theme writer.
I hunted on Google to see how to get my theme to work with Live Writer. I didn’t find any tutorials. So after a bit of hit-and-miss, I’m sharing a quick primer of what worked for me. If you don’t want to go through the hassle, you can always call on professionals who are adept at services like professional custom website design.
Open any post on your blog (using your new theme) and save that as view.html in your theme folder. Now replace the page’s title with {post-title} and the page’s content with {post-body}. For example:
{post-title}
{post-body}
This is the file Live Writer will be using as its theme. This page will be displayed exactly as it is by Live Writer, with {post-title} and {post-body} replaced with what you type. You can put in anything you want in this page — but at least make sure you include your CSS files.
To let Live Writer know that view.html is what it should display, copy WordPress’ /wp-includes/wlw-manifest.xml to your theme folder and add the following lines just before </manifest>.
WebLayout
WebLayout
Live Writer searches for wlmanifest.xml in the <link rel="wlmanifest"> tag of your home page. Since WordPress already links to its default wlwmanifest.xml, we need remove that link and add our own. So add the following code to your functions.php:
function my_wlwmanifest_link(){echo'';}
remove_action('wp_head','wlwmanifest_link');
add_action('wp_head','my_wlwmanifest_link');
function my_wlwmanifest_link() { echo
'';
}
remove_action('wp_head', 'wlwmanifest_link');
add_action('wp_head', 'my_wlwmanifest_link');
That’s it. Now if you add your blog to Live Writer, it will automatically detect the theme.
This morning, I was watching an episode of Finley the Fire Engine in which one of the trucks had hiccups. Reminded me of this Calvin & Hobbes — especially Hobbes’ remark in the second strip.
By curious coincidence, just a day after my post on client side scraping, I had a chance to demo this to a client. They were making a contacts database. Now, there are two big problems with managing contacts.
Getting complete information
Keeping it up to date
Now, people happy to fill out information about themselves in great detail. If you look at the public profiles on LinkedIn, you’ll find enough and more details about most people.
Normally, when getting contact details about someone, I search for their name on Google with a “site:linkedin.com” and look at that information.
Could this be automated?
I spent a couple of hours and came up with a primitive contacts scraper. Click on the link, type in a name, and you should get the LinkedIn profile for that person. (Caveat: It’s very primitive. It works only for specific URL public profiles. Try ‘Peter Peverelli’ as an example.)
It uses two technologies. Google AJAX Search API and YQL. The search() function searches for a phrase…
From this result, it displays all the <LI> tags which have a class and a <H3> element inside them (that’s what the //li[@class][h3] XPath does).
The real value of this is in bulk usage. When there’s a big list of contacts, you don’t need to scan each of them for updates. They can be automatically updated — even if all you know is the person’s name, and perhaps where they worked at some point in time.