S Anand, Author at S Anand

Storytelling: Part 1

August 29, 2012 August 27, 2012 / Business realities, How I do things / 2 Comments

In a number of sessions I’ve been to, people ask analysts to make their results more interesting – to tell stories with them. I’m co-teaching a course, part of which involves telling stories with data. So this got me thinking: what is a story? How does one teach storytelling to, let’s say, an alien?

Consider this mini-paper.

ABSTRACT: Meter readings exhibit spikes at slab boundaries. We also
find significant evidence of improbably events at round numbers.

Electricity shortage is a serious problem in most Indian states. Part
of this problem is due to the inaccuracy of reporting procedures used
in monitoring meter readings. Our focus here is not to document or
experimentally determine the degree of inaccuracy. We have adopted a
data driven approach to this problem and attempt to model the extent
of inaccuracy using basic statistical analysis techniques such as
histograms and the comparison of means.

Our dataset comprises of the frequency analysis 12-month dataset
containing monthly meter readings of 1.8 million customers in the
State of Andhra Pradesh.

We find that a histogram of these readings shows unexpectedly high
values at the slab boundaries: 50 (+45.342%, t > 13.431), 100
(+55.134%, t > 16.384), 200 (+33.341%, t > 15.232), and 300
(+42.138%, t > 19.958).

We also detected spikes at round numbers: 10 (+15.341%, t > 5.315),
20 (+18.576%, t > 6.152), 30 (+11.341%, t > 4.319).

The statistical significance of every deviation listed above is over
99.9%. Further, every deviation has a positive mantissa. This leads us
to confidently declare the existence of a systematic bias in the meter
readings analysed.

You’re probably thinking: “I know why he’s put this example here. It must be a bad one. So, what a rotten paper it must be!”

Well, not quite. It’s a good piece of analysis. I did it myself and there’s a fair bit of effort and care behind these short paragraphs.

The trouble is, if I read it out to my daughter, she’d say “What?” and not understand a word. My wife’d say “So what?” and not care a bit. I might as well not have written it.

It’s like that Zen thing: If a tree falls in a forest and no on hears it, does it make a sound?

If you did a piece of analysis, and no one understands or cares about it, why did you do it in the first place?

Why do you do it?

That last question is important: why do we analyse?

Sometimes, we do it for fun. The knowledge is beautiful. Knowing Tetris is NP-Complete is rewarding, even though my colleague sarcastically remarked, “Thank God! I’m sooo relieved now that I know that Tetris is NP whatever.” If that’s the case with you, great. Write the analysis any which way you’ll enjoy.

Sometimes, we do it because we’re forced to. In class. At work. Wherever. But that’s another way of saying “I don’t know why I’m doing it.” In that case, I’d gently recommend watching 3 Idiots.

Most often, we do it to share knowledge and drive actions. In that case, if no on understands it, or does anything with it, why do it?

Keep it simple

We prerajulisation of Farhanitate flagellated with ...

Would your audience understand that? Or are you just scared that simple words indicate a simple mind?

I was once afraid. 15 years ago, when writing a paper on IBM India’s competitive advantage for the CXOs, I was worried about it being too simple. I didn’t know anything about management. So I filled it with jargon. They politely nodded when I presented it, but I wasn’t fooling anyone. If there’s no content, jargon doesn’t help.

Unfortunately, it’s become polite to accept jargon as a substitute for substance. Why were they not ripping me apart? Or at least, kindly asking me what on earth I wanted to say?

My friend Manoj did that. In his nice, humble way, he asked, “But Anand, what does this mean?” When I explained it to him, I found I didn’t have a clue. He was OK with that. He just wanted to make sure he hadn’t missed something.

(That’s the technique I use these days. Ask people to explain things clearly. It’s OK if they’re just lost in jargon. I just want to make sure I haven’t missed something.)

Don’t cloak your ignorance. No one will think less of you. In the long run, you’ll learn more, and won’t need the jargon.

Part 2 of the article will talk about focusing on people and actions; storylining and the pyramid principle; and the structure of messages.

Colour spaces

August 27, 2012 August 27, 2012 / Coding, Visualisation / 1 Comment

In reality, a colour is a combination of light waves with frequencies between 400-700THz, just like sound is a combination of sound waves with frequencies from 20-20000Hz. Just like mixing various pure notes produces a new sound, mixing various pure colours (like from a rainbow) produces new colours (like white, which isn’t on the rainbow.)

Our eyes aren’t like our ears, though. They have 3 sensors that are triggered differently by different frequencies. The sensors roughly peak around red, green and blue. Roughly.

It turns out that it’s possible to recreate most (not all) colours using a combination of just red, green and blue by mimicking these three sensors to the right level. That’s why TVs and monitors have red, blue and green cells, and we represent colours using hex triplets for RRGGBB – like #00ff00 (green).

There are a number of problems with this from a computational perspective. Conceptually, we think of (R, G, B) as a 3-dimensional cube. That’d mean that 100% red is about as bright as 100% green or blue. Unfortunately, green is a lot brighter than red, which is a lot brighter than blue. Our 3 sensors are not equally sensitive.

You’d also think that a colour that’s numerically mid-way between 2 colours should appear to be mid-way. Far from it.

This means that if you’re picking colours using the RGB model, you’re using something very far from the intuitive human way of perceiving colours.

Which is all very nice, but I’m usually in a rush. So what do I do?

I go to the Microsoft Office colour themes and use a colour picker to pick one. (I extracted them to make life easier.) These are generally good on the eye.
Failing that, I pick something from http://kuler.adobe.com/
Or I go to http://colorbrewer2.org/ and pick a set of colours
If I absolutely have to do things programmatically, I use the HCL colour scheme. The good part is it’s perceptually uniform. The bad part is: not every interpolation is a valid colour.

Style of blogging

August 27, 2012 August 27, 2012 / How I do things / 1 Comment

Until 2007, my blog was mostly just linking to stuff I found interesting on the Web. Since 2007, I’ve tried to write longer articles, mostly based on my own experiences.

At the moment, that’s unsustainable. Right now, being in a startup, I doing more stuff than I ever have in the past. (That does not mean working more hours, by the way.)

My posts, going forward, are likely to be smaller, less original, but hopefully more frequent.

Is Protocol buffers worth it?

August 1, 2012 August 15, 2012 / Coding / 7 Comments

Google’s Protocol Buffers is a “language-neutral, platform-neutral, extensible mechanism for serializing structured data – think XML, but smaller, faster, and simpler”

XML is slow and large. There’s no doubting that. JSON’s my default alternative, though it’s a bit large. CSV’s ideal for tabular data, but ragged hierarchies are a bit difficult.

I was trying to see if Protocol Buffers would be smaller and faster, at least when using Python. I took JSON as the base, and checked the write speed, read speed and file sizes. Here’s the comparison:

Protocol Buffers are 17 times slower to write and almost 6 times slower to read than JSON files. File sizes are smaller, but then, all it takes is a simple gzip operation to compress the JSON files even smaller. Reading json.gz files is just 2% slower than JSON files, and writing them is only 4 times slower.

The code base is at https://bitbucket.org/sanand0/protobuftest

On the whole, it appears that GZipped JSON files are smaller, faster, and just as simple as Protocol Buffers. What am I missing?

Update: When you add GZipped CSV to the mix, it’s twice as fast as GZipped JSON to read: clearly a huge win. It’s only slightly slower to write, and but compresses a tiny bit more than JSON.

Audio data URI

July 17, 2012 July 17, 2012 / Coding / 1 Comment

Turns out that you can use data URIs in the <audio> tag.

Just upload an MP3 file to http://dataurl.net/#dataurlmaker and you’ll get a long string starting with data:audio/mp3;base64...

Insert this into your HTML:

<audio controls src=”data:audio/mp3;base64...”>

That’s it – the entire MP3 file is embedded into your HTML page without requiring additional downloads.

This takes a bit more bandwidth than the MP3, and won’t work on Internet Explorer. But for modern browsers, and small audio files, it reduces the overall load time – sort of like CSS sprites.

So, on my bus ride today, I built a little HTML5 musical keyboard that generates data URIs on the fly. Click to play.

Recent Tamil Songs Quiz

May 21, 2012 January 25, 2025 / Quizzes / 42 Comments

After a long break, here’s another quiz, featuring relatively recent Tamil songs. Can you guess which movie they are from?

Don’t worry about the spelling. Just spell it like it sounds, and the box will turn green.

Donations for Sanskrit College

April 27, 2012 June 13, 2012 / Links / 1 Comment

The following article appeared in The Times of India earlier this month.

The institute is struggling for funds. Please contribute, if you could, by calling +91 44 24985320 or via PayPal.

Sanskrit centre struggles to stay alive

The Kuppuswami Sastri Research Institute attached to the Sanskrit College in Mylapore is in doldrums because of lack of government patronage.

The Institute, one of the three involved in Sanskrit research in the country, has been surviving on private donations. With not enough resources, the management is unable to pay the faculty the benefits of the sixth pay commission.

Institute director V Kameswari said the Union government stopped its financial support in 1995, after which it has been solely dependent on donations. “The institute has a trove of rare palm leaf manuscripts and books not just about Sanskrit literature but also on architecture, fine arts, geography, history and astronomy in Sanskrit,” says Kameswari.

The two other such institutes are the R G Bandarkar Sanskrit Institute in Pune and the Ganganath Jha Sanskrit Institute in Allahabad. “We have requested a onetime grant from the Union planning commission and also annual assistance from the Rashtriya Sanskrit Sansthan, but are yet to get any support,” says K S Balasubramanian, deputy director of the institute. The plan panel had given grants to the Mumbai Asiatic Society and Kolkata-based Asiatic Society.

The institute was getting about 10 lakh till 1995 but due to a misunderstanding between the government-appointed members of the governing committee and the management, the aid was stopped. Today, there are 24 scholars at the institute, most of them women doing their PhDs. “Scholars from across the country and world visit the institute. We send out publications to many foreign universities and they in turn send their publications which are preserved here,” says Kameswari.

The institute was started as a private non-profit organisation in 1944 in memory of Kuppuswami Sastri, a renowned Sanskrit scholar. It has a library with books on astronomy, architecture, fine arts, mathematics, Vedas, Puranas, Upanishads and various branches of science.

“A private entrepreneur made a donation with which we have air-conditioned the library. The palm-leaf manuscripts in the library are 600 to 1,000 years old. Many of them are in Grantha script. We also have books on Jainism that speak about solving mathematical equations and explain geographical concepts,” says Kameswari, who is worried about keeping the ancient language alive.

Downloading songs from YouTube

March 15, 2012 July 31, 2012 / How I do things / 15 Comments

Five years ago, I built a song search engine – mainly because I needed to listen to songs. Three years ago, I stopped updating it – mainly because I stopped listening to songs actively, and have been busy since. For those of you who have been using my site for music: my apologies.

These days, I don’t really find the need to download music. YouTube has most of the songs I need. Bandwidth is pretty good too even when on the move.

But when I do need to download music, this is my new workflow.

Find the song on YouTube. (Misspellings are still an issue, but you’ll usually find what you need)
Download the video. Keepvid is the simple option. youtube-dlis the geek’s option (for multiple downloads)
Use VLC – the swiss-army knife of media – to convert the video into an MP3.

That last step requires a bit of explaining. It’s very simple once you know how, but it took me a few months to get it right. So here goes.

Select the Convert / Save option in the Media menu.

Click on Add to open file you want to convert. You can pick a track from an disk as well if you want to rip an audio CD or a DVD.

Choose the file.

Click on Convert / Save.

Type the destination filename. Make sure you type the full file name, and not just the name of the folder.

Select the output format you want under Settings – Profile. You can tweak the bitrate with the settings button, but I usually don’t bother.

When you click on the Start button, the file will be converted or the CD will be ripped. You’ll see the position marker move fairly fast.

The only problem I have with this method is that I can’t seem to do batch conversions easily enough with the GUI. Does anyone have any other workflow they like?

Update (31 Jul 2012): Aditya Sengupta suggests the following: (should’ve guessed VLC would have something up its sleeve)

vlc -I dummy $FILENAME --no-sout-video --sout "#transcode{acodec=mp3,5Dab=AUDIO_BITRATE,channels=2}:std{access=file,mux=raw,dst=$NAME.mp3}" vlc://quit

Correlating subjects

February 12, 2012 February 12, 2012 / Visualisation / 8 Comments

A question from Dorai get me thinking: does being good at maths help in programming?

I don’t have a personal view. But since Reportbee has data on the Class 12 examination results for the last three years, we thought we could do a bit of analysis.

Here’s the correlation of the scores of various subjects with Computer Science.

Correlation	Subject
0.79	CHEMISTRY
0.79	PHYSICS
0.75	ENGLISH
0.75	MATHEMATICS
0.72	LANGUAGE
0.67	BIOLOGY
0.66	ECONOMICS
0.66	COMMERCE
0.65	ACCOUNTANCY
0.56	HISTORY
0.52	GEOGRAPHY

It almost breaks neatly into four groups.

Physics & Chemistry, both of which have a correlation of 0.79, and clearly are the most correlated with Computer Science
Maths, English & Language, which have a correlation of 0.72 – 0.75
Biology, Economics, Commerce and Accountancy, which hover at around 0.66
History & Geography, which are 0.52 – 0.56

The results in 2010 are almost exactly the same.

Correlation	Subject
0.78	PHYSICS
0.78	CHEMISTRY
0.75	ENGLISH
0.75	MATHEMATICS
0.73	LANGUAGE
0.67	ACCOUNTANCY
0.65	ECONOMICS
0.65	COMMERCE
0.64	BIOLOGY
0.60	GEOGRAPHY
0.55	HISTORY

I’m not sure what it is that leads to this kind of correlation. In fact, the full correlation between every pair of subjects (for 2011) is below:

What inferences would you draw from this?

And what do you think is the reason for this?

The three Rs

January 23, 2012 January 23, 2012 / Links / 13 Comments

Reading, wRiting and aRithmetic are the 3 ‘R’s that are taught at school. I was thinking about their relevance today.

Reading continues to be relevant. The volume of information available today is more than before. So you need to read faster AND smarter. (If there was one good thing that came out of my IIM coaching classes, it was the ability to read fast, and making it subconscious.)

But I wouldn’t say the same of writing. In the last 10 years, I have typed several hundred more pages than I’ve written. So have all my friends.

Yesterday, I was at a bank with a relationship manager as he was taking notes in paper and pen. I do the same on occassion. I looked at his notes later. I could not understand a single word. “Don’t worry, sir, I can read it. I’ll type it out and mail you,” he said. And he did.

Writing seems to have become a device for personal memory, not communication. He’s faster at writing than typing, perhaps. Or note taking is more convenient on paper. But for communication, he still prefers a typed format. So do I, and most other people.

Perhaps writing will fade. Perhaps not. I don’t know. But what I do know is that typing has become more important than writing. Yet, writing is taught more at school than typing.

(A broader aspect of writing, though, is expressing oneself. That will remain important, of course.)

The third R is aRithmetic. When I was 12, I could multiply four-digit numbers in my head reasonably well. I could recite 50 digits of Pi. I could do long division. Today, I can’t. Nor can my friends. Nor have we needed to. A good feel for the numbers has helped, but not the actual mechanics of the calculations.

We had an undergraduate course in statistics that taught us how to solve a linear regression problem. That skill went completely unused. I’ve never since used regression without a computer. We had a graduate course in statistics that taught us how to INTERPRET the results of a linear regression. That was worth it’s weight in gold.

This is not a critique of the three Rs. Rather, an attempt to re-interpret them. It’s about comprehension, expression and computation. Two decades ago, it was reading, writing and arithmetic. Today, it’s reading, typing and computing.

Computers will grow more powerful. It may be worth planning for it. Teaching the ability to use them can go a long way. A tool like Excel for general purpose computing gives incredible power in the hands of people. It’s worth training children for that.

If I oversimplified, I’d say children must learn typing and Excel.

Over the next few years, this is something I plan to work on. Making sure schools and parents do this. Any suggestions or leads you may have are welcome!

S Anand