Coding

Releasing modified mosquitoes precisely

At PyCon Indonesia, I spoke about a project we worked on with the World Mosquito Program.

The World Mosquito Program (WMP) modifies mosquitoes with a bacteria — Wolbachia. This reduces their ability to carry deadly viruses. (It makes me perversely happy that we’re infecting mosquitoes now 😉.)

Modifying mosquitoes is an expensive process. With a limited set of “good mosquitoes”, it is critical to find the best release points that will help them replicate rapidly.

But planning the release points took weeks of manual effort. It involved ground personnel going through several iterations.

So our team took high-resolution satellite images, figured out the building density, estimated population density based on that, and generated a release plan. This model is 70% more accurate and reduced the time from 3 weeks to 2 hours.

More details at the Gramener website.

The slides for the talk are below.

Programming Minecraft with Websockets

Minecraft lets you connect to a websocket server when you’re in a game. The server can receive and send any commands. This lets you build a bot that you can … (well, I don’t know what it can do, let’s explore.)

Minecraft has commands you can type on a chat window. For example, type / to start a command and type setblock ~1 ~0 ~0 grass changes the block 1 north of you into grass. (~ means relative to you. Coordinates are specified as X, Y and Z.)

Minecraft grass block

Note: These instructions were tested on Minecraft Bedrock 1.16. I haven’t tested them on the Java Edition.

Connect to Minecraft

You can send any command to Minecraft from a websocket server. Let’s use JavaScript for this.

First, run npm install ws uuid. (We need ws for websockets and uuid to generate unique IDs.)

Then create this mineserver1.js:

const WebSocket = require('ws')
const uuid = require('uuid')        // For later use

// Create a new websocket server on port 3000
console.log('Ready. On MineCraft chat, type /connect localhost:3000')
const wss = new WebSocket.Server({ port: 3000 })

// On Minecraft, when you type "/connect localhost:3000" it creates a connection
wss.on('connection', socket => {
  console.log('Connected')
})

On Minecraft > Settings > General > Profile, turn off the “Require Encrypted Websockets” setting.

Run node mineserver1.js. Then type /connect localhost:3000 in a Minecraft chat window. You’ll see 2 things:

  1. MineCraft says “Connection established to server: ws://localhost:3000”
  2. Node prints “Connected”

Now, our program is connected to Minecraft, and can send/receive messages.

Minecraft chat connect

Notes:

  • The Python equivalent is in mineserver1.py. Run python mineserver1.py.
  • If you get an Uncaught Error: Cannot find module 'ws', make sure you ran npm install ws uuid.
  • If you get an “Encrypted Session Required” error, make sure you turned off the “Require Encrypted Websockets” setting mentioned above.
  • To disconnect, run /connect off

Subscribe to chat messages

Now let’s listen to the players’ chat.

A connected websocket server can send a “subscribe” message to Minecraft saying it wants to “listen” to specific actions. For example, you can subscribe to “PlayerMessage”. Whenever a player sents a chat message, Minecraft will notify the websocket client.

Here’s how to do that. Add this code in the wss.on('connection', socket => { ... }) function.

  // Tell Minecraft to send all chat messages. Required once after Minecraft starts
  socket.send(JSON.stringify({
    "header": {
      "version": 1,                     // We're using the version 1 message protocol
      "requestId": uuid.v4(),           // A unique ID for the request
      "messageType": "commandRequest",  // This is a request ...
      "messagePurpose": "subscribe"     // ... to subscribe to ...
    },
    "body": {
      "eventName": "PlayerMessage"      // ... all player messages.
    },
  }))

Now, every time a player types something in the chat window, the socket will receive it. Add this code below the above code:

  // When MineCraft sends a message (e.g. on player chat), print it.
  socket.on('message', packet => {
    const msg = JSON.parse(packet)
    console.log(msg)
  })

This code parses all the messages it receives and prints them.

This code in is mineserver2.js. Run node mineserver2.js. Then type /connect localhost:3000 in a Minecraft chat window. Then type a message (e.g. “alpha”) in the chat window. You’ll see a message like this in the console.

{
  header: {
    messagePurpose: 'event',        // This is an event
    requestId: '00000000-0000-0000-0000-000000000000',
    version: 1                      // using version 1 message protocol
  },
  body: {
    eventName: 'PlayerMessage',
    measurements: null,
    properties: {
      AccountType: 1,
      ActiveSessionID: 'e0afde71-9a15-401b-ba38-82c64a94048d',
      AppSessionID: 'b2f5dddc-2a2d-4ec1-bf7b-578038967f9a',
      Biome: 1,                     // Plains Biome. https://minecraft.gamepedia.com/Biome
      Build: '1.16.201',            // That's my build
      BuildNum: '5131175',
      BuildPlat: 7,
      Cheevos: false,
      ClientId: 'fcaa9859-0921-348e-bc7c-1c91b72ccec1',
      CurrentNumDevices: 1,
      DeviceSessionId: 'b2f5dddc-2a2d-4ec1-bf7b-578038967f9a',
      Difficulty: 'NORMAL',         // I'm playing on normal difficulty
      Dim: 0,
      GlobalMultiplayerCorrelationId: '91967b8c-01c6-4708-8a31-f111ddaa8174',
      Message: 'alpha',             // This is the message I typed
      MessageType: 'chat',          // It's of type chat
      Mode: 1,
      NetworkType: 0,
      Plat: 'Win 10.0.19041.1',
      PlayerGameMode: 1,            // Creative. https://minecraft.gamepedia.com/Commands/gamemode
      Sender: 'Anand',              // That's me.
      Seq: 497,
      WorldFeature: 0,
      WorldSessionId: '8c9b4d3b-7118-4324-ba32-c357c709d682',
      editionType: 'win10',
      isTrial: 0,
      locale: 'en_IN',
      vrMode: false
    }
  }
}

Notes:

Build structures using chat

Let’s create a pyramid of size 10 around us when we type pyramid 10 in the chat window.

The first step is to check if the player sent a chat message like pyramid 10 (or another number). Add this code below the above code:

  // When MineCraft sends a message (e.g. on player chat), act on it.
  socket.on('message', packet => {
    const msg = JSON.parse(packet)
    // If this is a chat window
    if (msg.body.eventName === 'PlayerMessage') {
      // ... and it's like "pyramid 10" (or some number), draw a pyramid
      const match = msg.body.properties.Message.match(/^pyramid (\d+)/i)
      if (match)
        draw_pyramid(+match[1])
    }
  })

If the user types “pyramid 3” on the chat window, draw_pyramid(3) is called.

In draw_pyramid(), let’s send commands to build a pyramid. To send a command, we need to create a JSON with the command (e.g. setblock ~1 ~0 ~0 grass). Add this code below the above code:

  function send(cmd) {
    const msg = {
      "header": {
        "version": 1,
        "requestId": uuid.v4(),     // Send unique ID each time
        "messagePurpose": "commandRequest",
        "messageType": "commandRequest"
      },
      "body": {
        "version": 1,               // TODO: Needed?
        "commandLine": cmd,         // Define the command
        "origin": {
          "type": "player"          // Message comes from player
        }
      }
    }
    socket.send(JSON.stringify(msg))  // Send the JSON string
  }

Let’s write draw_pyramid() to create a pyramid using glowstone by adding this code below the above code:

  // Draw a pyramid of size "size" around the player.
  function draw_pyramid(size) {
    // y is the height of the pyramid. Start with y=0, and keep building up
    for (let y = 0; y < size + 1; y++) {
      // At the specified y, place blocks in a rectangle of size "side"
      let side = size - y;
      for (let x = -side; x < side + 1; x++) {
        send(`setblock ~${x} ~${y} ~${-side} glowstone`)
        send(`setblock ~${x} ~${y} ~${+side} glowstone`)
        send(`setblock ~${-side} ~${y} ~${x} glowstone`)
        send(`setblock ~${+side} ~${y} ~${x} glowstone`)
      }
    }
  }

This code in is mineserver3.js.

  • Run node mineserver3.js.
  • Then type /connect localhost:3000 in a Minecraft chat window.
  • Then type pyramid 3 in the chat window.
  • You’ll be surrounded by a glowstone pyramid.
Minecraft glowstone pyramid

Notes:

  • The Python equivalent is in mineserver3.py. Run python mineserver3.py.
  • The “requestId” needs to be a UUID — at least for block commands. I tried unique “requestId” values like 1, 2, 3 etc. That didn’t work.

Understand Minecraft’s responses

For every command you send, Minecraft sends a response. It’s “header” looks like this:

{
  "header": {
    "version": 1,
    "messagePurpose": "commandResponse",                  // Response to your command
    "requestId": "97dee9a3-a716-4caa-aef9-ddbd642f2650"   // ... and your requestId
  }
}

If the command is successful, the response has body.statusCode == 0. For example:

{
  "body": {
    "statusCode": 0,                  // No error
    "statusMessage": "Block placed",  // It placed the block you wanted
    "position": { "x": 0, "y": 64, "z": 0 }   // ... at this location
  },
}

If the command failed, the response has a negative body.statusCode. For example:

{
  "body": {
    "statusCode": -2147352576,        // This is an error
    "statusMessage": "The block couldn't be placed"
  },
}

To print these, add this to socket.on('message', ...):

    // If we get a command response, print it
    if (msg.header.messagePurpose == 'commandResponse')
      console.log(msg)

This code in is mineserver4.js.

  • Run node mineserver4.js.
  • Then type /connect localhost:3000 in a Minecraft chat window.
  • Then type pyramid 3 in the chat window.
  • You’ll be surrounded by a glowstone pyramid, and the console will show every command response.

Notes on common error messages:

  • The block couldn't be placed (-2147352576): The same block was already at that location.
  • Syntax error: Unexpected "xxx": at "~0 ~9 ~-1 >>xxx<<" (-2147483648): You gave wrong arguments to the command.
  • Too many commands have been requested, wait for one to be done (-2147418109): Minecraft only allows 100 commands can be executed without waiting for their response.
  • More error messages here.

Wait for commands to be done

Typing “pyramid 3” works just fine. But try “pyramid 5” and your pyramid is incomplete.

Minecraft incomplete pyramid

That’s because Minecraft only allows up to 100 messages in its queue. On the 101st message, you get a Too many commands have been requested, wait for one to be done error.

{
  "header": {
    "version": 1,
    "messagePurpose": "error",
    "requestId": "a5051664-e9f4-4f9f-96b8-a56b5783117b"
  },
  "body": {
    "statusCode": -2147418109,
    "statusMessage": "Too many commands have been requested, wait for one to be done"
  }
}

So let’s modify send() to add to a queue and send in batches. We’ll create two queues:

  const sendQueue = []        // Queue of commands to be sent
  const awaitedQueue = {}     // Queue of responses awaited from Minecraft

In wss.on('connection', ...), when Minecraft completes a command, we’ll remove it from the awaitedQueue. If the command has an error, we’ll report it.

    // If we get a command response
    if (msg.header.messagePurpose == 'commandResponse') {
      // ... and it's for an awaited command
      if (msg.header.requestId in awaitedQueue) {
        // Print errors 5(if any)
        if (msg.body.statusCode < 0)
          console.log(awaitedQueue[msg.header.requestId].body.commandLine, msg.body.statusMessage)
        // ... and delete it from the awaited queue
        delete awaitedQueue[msg.header.requestId]
      }
    }
    // Now, we've cleared all completed commands from the awaitedQueue.

Once we’ve processed Minecraft’s response, we’ll send pending messages from sendQueue, upto 100 and add them to the awaitedQueue.

     // We can send new commands from the sendQueue -- up to a maximum of 100.
     let count = Math.min(100 - Object.keys(awaitedQueue).length, sendQueue.length)
     for (let i = 0; i < count; i++) {
       // Each time, send the first command in sendQueue, and add it to the awaitedQueue
       let command = sendQueue.shift()
       socket.send(JSON.stringify(command))
       awaitedQueue[command.header.requestId] = command
     }
     // Now we've sent as many commands as we can. Wait till the next PlayerMessage/commandResponse

Finally, in function send(), instead of socket.send(JSON.stringify(msg)), we use sendQueue.push(msg) to add the message to the queue.

This code in is mineserver5.js.

  • Run node mineserver5.js.
  • Then type /connect localhost:3000 in a Minecraft chat window.
  • Then type pyramid 6 in the chat window.
  • You’ll be surrounded by a large glowstone pyramid.
  • The console will print messages like setblock ~0 ~6 ~0 glowstone The block couldn't be placed because we’re trying to place duplicate blocks.
Minecraft glowstone pyramid

How to extend Markdown with custom blocks

One problem I’ve had in Markdown is rendering a content in columns.

On Bootstrap, the markup would look like this:

<div class="row">
  <div class="col">...</div>
  <div class="col">...</div>
</div>

How do we get that into Markdown without writing HTML?

On Python, the attribute lists extension lets you add a class. For example:

This is some content
 {: .row}

… renders <p class="row">This is some content</p>.

But I can’t do that to multiple paragraphs. Nor can I next content, i.e. add a .col inside the .row.

Enter markdown-customblocks. It’s a Python module that extends Python Markdown. This lets me write:

::: row
     ::: col
       Content in column 1
     ::: col
       Content in column 2
 ::: row
     ::: col
       Content in column 1
     ::: col
       Content in column 2

This translates to:

<div class="row">
  <div class="col">Content in column 1</div>
  <div class="col">Content in column 2</div>
</div>

Better yet, we can create our own custom HTML block types. For example, this code:

from markdown import Markdown
from customblocks import CustomBlocksExtension

def audio(ctx, src, type="audio/mp3"):
    return f'''<audio src="{src}" type="{type}">
      {ctx.content}
    </audio>'''

md = Markdown(extensions=[
    CustomBlocksExtension(generators={
        'audio': audio
    }),
])

… lets you convert this piece of Markdown:

md.convert('''
::: audio src="mymusic.ogg" type="audio/ogg"
    Your browser does not support `audio`.
''')

… into this HTML:

<audio src="mymusic.ogg" type="audio/ogg">
  Your browser does not support <code>audio</code>.
</audio>

markdown-customblocks is easily the most useful Python module I discovered last quarter.

Create SVG with PowerPoint

With Office 365, PowerPoint supports SVG editing. This is really powerful. It means you can draw in PowerPoint and render it on the web — including as interactive or animated visuals.

For example, the SVG in this simulator was created just with PowerPoint.

The process is simple. Draw anything. Select any shapes and right-click. Select Save As Picture… and choose SVG.

For example, you can use PowerPoint to create Smart Art, export it as SVG, and embed it into a page. See this example on CodePen.

The SVG is fairly well structured and easy to edit. The code generated for these 2 simple shapes:

… is quite straight-forward — just two SVG shapes.

<rect x="125.5" y="185.5" width="107" height="107" stroke="#2F528F" stroke-width="1.33333" stroke-miterlimit="8" fill="#4472C4"/>
<path d="M243.5 292.5 297 185.5 350.5 292.5Z" stroke="#2F528F" stroke-width="1.33333" stroke-miterlimit="8" fill="#4472C4" fill-rule="evenodd"/>

I was worried about the lack of SVG authoring tools in Windows. (InkScape is not usable, and Adobe’s tools are complex and expensive.) PowerPoint fits perfectly.

lxml is fast enough

Given the blazing speed of Node.js these days, I expected HTML parsing to be faster on Node than on Python.

So I compared lxml with htmlparser2 — the fastest libraries on Python and JS in parsing the reddit home page (~700KB).

  • lxml took ~8.6 milliseconds
  • htmlparser2 took ~14.5 milliseconds

Looks like lxml is much faster. I’m likely to stick around with Python for pure HTML parsing (without JavaScript) for a while longer.

In [1]: from lxml.html import parse

In [2]: %timeit tree = parse('reddit.html')
8.69 ms Âą 190 Âľs per loop (mean Âą std. dev. of 7 runs, 100 loops each)
const { Parser } = require("htmlparser2");
const { DomHandler } = require("domhandler");
const fs = require('fs');
const html = fs.readFileSync('reddit.html');
const handler = new DomHandler(function (error, dom) { });

const start = +new Date();
for (var i = 0; i < 100; i++) {
  const parser = new Parser();
  parser.write(html);
  parser.end();
}
const end = +new Date();
console.log((end - start) / 100);

Note: If I run the htmlparser2 code 100 times instead of 10, it only takes 7ms per loop. The more the number of loops, the faster it parses. I guess Node.js optimizes repeated loops. But I’m only interested in the first iteration, since I’ll be parsing files only once.

Geocoding in Excel

It’s easy to convert addresses into latitudes and longitudes into addresses in Excel. Here’s the Github project with a downloadable Excel file.

This is via Visual Basic code for a GoogleGeocode function that geocodes addresses.

Function GoogleGeocode(address As String) As String
    Dim xDoc As New MSXML2.DOMDocument
    xDoc.async = False
    xDoc.Load ("http://maps.googleapis.com/maps/api/geocode/" + _
        "xml?address=" + address + "&sensor=false")
    If xDoc.parseError.ErrorCode <> 0 Then
        GoogleGeocode = xDoc.parseError.reason
    Else
        xDoc.setProperty "SelectionLanguage", "XPath"
        lat = xDoc.SelectSingleNode("//lat").Text
        lng = xDoc.SelectSingleNode("//lng").Text
        GoogleGeocode = lat & "," & lng
    End If
End Function

Github page-only repository

Github offers Github Pages that let you host web pages on Github.

You create these by adding a branch to git called gh-pages, and this is often in addition to the default branch master.

I just needed the gh-pages branch. So thanks to YJL, here’s the simplest way to do it.

  1. Create the repositoryon github.
  2. Create your local repository and git commitinto it.
  3. Type git push -u origin master:gh-pages
  4. In .git/config, under the [remote "origin"] section, add push = +refs/heads/master:refs/heads/gh-pages

The magic is the last :gh-pages.

The most popular scientific Python modules

I just scraped the scientific packages on pypi. Here are the top 50 by downloads.

Name Description Size Downloads
numpy NumPy: array processing for numbers, strings, records, and objects. 2000000 133076
scipy SciPy: Scientific Library for Python 7000000 33990
pygraphviz Python interface to Graphviz 99000 22828
geopy Python Geocoding Toolbox 32000 18617
googlemaps Easy geocoding, reverse geocoding, driving directions, and local search in Python via Google. 69000 15135
Rtree R-Tree spatial index for Python GIS 495000 14370
nltk Natural Language Toolkit 1000000 12844
Shapely Geometric objects, predicates, and operations 93000 12635
pyutilib.component.doc Documentation for the PyUtilib Component Architecture. 372000 10181
geojson Encoder/decoder for simple GIS features 12000 9407
GDAL GDAL: Geospatial Data Abstraction Library 410000 8957
scikits.audiolab A python module to make noise from numpy arrays 1000000 8856
pupynere NetCDF file reader and writer. 16000 8809
scikits.statsmodels Statistical computations and models for use with SciPy 3000000 8761
munkres munkres algorithm for the Assignment Problem 42000 8409
scikit-learn A set of python modules for machine learning and data mining 2000000 7735
networkx Python package for creating and manipulating graphs and networks 1009000 7652
pyephem Scientific-grade astronomy routines 927000 7644
PyBrain PyBrain is the swiss army knife for neural networking. 255000 7313
scikits.learn A set of python modules for machine learning and data mining 1000000 7088
obspy.seisan SEISAN read support for ObsPy. 3000000 6990
obspy.wav WAV(audio) read and write support for ObsPy. 241000 6985
obspy.seishub SeisHub database client for ObsPy. 237000 6941
obspy.sh Q and ASC (Seismic Handler) read and write support for ObsPy. 285000 6926
crcmod CRC Generator 128000 6714
obspy.fissures DHI/Fissures request client for ObsPy. 1000000 6339
stsci.distutils distutils/packaging-related utilities used by some of STScI’s packages 25000 6215
pyopencl Python wrapper for OpenCL 1000000 6124
Kivy A software library for rapid development of hardware-accelerated multitouch applications. 11000000 5879
speech A clean interface to Windows speech recognition and text-to-speech capabilities. 17000 5809
patsy A Python package for describing statistical models and for building design matrices. 276000 5517
periodictable Extensible periodic table of the elements 775000 5498
pymorphy Morphological analyzer (POS tagger + inflection engine) for Russian and English (+perhaps German) languages. 70000 5174
imposm.parser Fast and easy OpenStreetMap XML/PBF parser. 31000 4940
hcluster A hierarchical clustering package for Scipy. 442000 4761
obspy.core ObsPy – a Python framework for seismological observatories. 487000 4608
Pyevolve A complete python genetic algorithm framework 99000 4509
scikits.ann Approximate Nearest Neighbor library wrapper for Numpy 82000 4368
obspy.imaging Plotting routines for ObsPy. 324000 4356
obspy.xseed Dataless SEED, RESP and XML-SEED read and write support for ObsPy. 2000000 4331
obspy.sac SAC read and write support for ObsPy. 306000 4319
obspy.arclink ArcLink/WebDC client for ObsPy. 247000 4164
obspy.iris IRIS Web service client for ObsPy. 261000 4153
Orange Machine learning and interactive data mining toolbox. 14000000 4099
obspy.neries NERIES Web service client for ObsPy. 239000 4066
pandas Powerful data structures for data analysis, time series,and statistics 2000000 4037
pycuda Python wrapper for Nvidia CUDA 1000000 4030
GeoAlchemy Using SQLAlchemy with Spatial Databases 159000 3881
pyfits Reads FITS images and tables into numpy arrays and manipulates FITS headers 748000 3746
HTSeq A framework to process and analyze data from high-throughput sequencing (HTS) assays 523000 3720
pyopencv PyOpenCV – A Python wrapper for OpenCV 2.x using Boost.Python and NumPy 354000 3660
thredds THREDDS catalog generator. 25000 3622
hachoir-subfile Find subfile in any binary stream 16000 3540
fluid Procedures to study geophysical fluids on Python. 210000 3520
pygeocoder Python interface for Google Geocoding API V3. Can be used to easily geocode, reverse geocode, validate and format addresses. 7000 3514
csc-pysparse A fast sparse matrix library for Python (Commonsense Computing version) 111000 3455
topex A very simple library to interpret and load TOPEX/JASON altimetry data 7000 3378
arrayterator Buffered iterator for big arrays. 7000 3320
python-igraph High performance graph data structures and algorithms 3000000 3260
csvkit A library of utilities for working with CSV, the king of tabular file formats. 29000 3236
PyVISA Python VISA bindings for GPIB, RS232, and USB instruments 237000 3201
Quadtree Quadtree spatial index for Python GIS 40000 3000
ProxyHTTPServer ProxyHTTPServer — from the creator of PyWebRun 3000 2991
mpmath Python library for arbitrary-precision floating-point arithmetic 1000000 2901
bigfloat Arbitrary precision correctly-rounded floating point arithmetic, via MPFR. 126000 2879
SimPy Event discrete, process based simulation for Python. 5000000 2871
Delny Delaunay triangulation 18000 2790
pymc Markov Chain Monte Carlo sampling toolkit. 1000000 2727
PyBUFR Pure Python library to encode and decode BUFR. 10000 2676
collective.geo.bundle Plone Maps (collective.geo) 11000 2676
dap DAP (Data Access Protocol) client and server for Python. 125000 2598
rq RQ is a simple, lightweight, library for creating background jobs, and processing them. 29000 2590
pyinterval Interval arithmetic in Python 397000 2558
StarCluster StarCluster is a utility for creating and managing computing clusters hosted on Amazon’s Elastic Compute Cloud (EC2). 2000000 2521
fisher Fast Fisher’s Exact Test 43000 2503
mathdom MathDOM – Content MathML in Python 169000 2482
img2txt superseded by asciiporn, http://pypi.python.org/pypi/asciiporn 443000 2436
DendroPy A Python library for phylogenetics and phylogenetic computing: reading, writing, simulation, processing and manipulation of phylogenetic trees (phylogenies) and characters. 6000000 2349
geolocator geolocator library: locate places and calculate distances between them 26000 2342
MyProxyClient MyProxy Client 67000 2325
PyUblas Seamless Numpy-UBlas interoperability 51000 2252
oroboros Astrology software 1000000 2228
textmining Python Text Mining Utilities 1000000 2198
scikits.talkbox Talkbox, a set of python modules for speech/signal processing 147000 2188
asciitable Extensible ASCII table reader and writer 312000 2160
scikits.samplerate A python module for high quality audio resampling 368000 2151
tabular Tabular data container and associated convenience routines in Python 52000 2114
pywcs Python wrappers to WCSLIB 2000000 2081
DeliciousAPI Unofficial Python API for retrieving data from Delicious.com 19000 2038
hachoir-regex Manipulation of regular expressions (regex) 31000 2031
Kamaelia Kamaelia – Multimedia & Server Development Kit 2000000 2007
seawater Seawater Libray for Python 2000000 1985
descartes Use geometric objects as matplotlib paths and patches 3000 1983
vectorformats geographic data serialization/deserialization library 10000 1949
PyMT A framework for making accelerated multitouch UI 18000000 1945
times Times is a small, minimalistic, Python library for dealing with time conversions between universal time and arbitrary timezones. 4000 1929
CocoPy Python implementation of the famous CoCo/R LL(k) compiler generator. 302000 1913
django-shapes Upload and export shapefiles using GeoDjango. 9000 1901
sympy Computer algebra system (CAS) in Python 5000000 1842
pyfasta fast, memory-efficient, pythonic (and command-line) access to fasta sequence files 14000 1836

Auto reloading pages

After watching Bret Victor’s Inventing on Principle, I just had to figure out a way of getting live reloading to work. I know about LiveReload, of course, and everything I’ve heard about it is good. But their Windows version is in alpha, and I’m not about to experiment just yet.

This little script does it for me instead:

(function(interval, location) {
  var lastdate = "";
  function updateIfChanged() {
    var req = new XMLHttpRequest();
    req.open('HEAD', location.href, false);
    req.send(null);
    var date = req.getResponseHeader('Last-Modified');
    if (!lastdate) {
      lastdate = date;
    }
    else if (lastdate != date) {
      location.reload();;
    }
  };
  setInterval(updateIfChanged, interval);
})(300, window.location)

 

It checks the current page every 300 milliseconds and reloads it if the Last-Modified header is changed. I usually include it as a minified script:

<script>(function(d,c){var b="";setInterval(function(){var a=new
XMLHttpRequest;a.open("HEAD",c.href,false);a.send(null);
a=a.getResponseHeader("Last-Modified");if(b)b!=a&&
c.reload();else b=a},d)})(300,window.location)</script>

There are no dependencies on any library, like jQuery. However, it requires that the file be on a web server. (It’s easy to fix that, but since I always run a local webserver, I’ll let you solve that problem yourself.)

Windows XP virtual machine

Here’s the easiest way to set up a Windows XP virtual machine that I could find.

(This is useful if you want to try out programs without installing it on your main machine; test your code on a new machine; or test your website on IE6 / IE7 / IE8.)

  1. Go to the Virtual PC download site. (I tried VirtualBox and VMWare Player. Virtual PC is better if you’re running Windows on Windows.)
    image
    If you have Windows 7 Starter or Home, select “Don’t need XP Mode and want VPC only? Download Windows Virtual PC without Windows XP Mode.”
    If you have Windows Vista or Windows 7, select “Looking for Virtual PC 2007?”
  2. Download it. (You may have to jump through a few hoops like activation.)
  3. Download Windows XP and run it to extract the files. (It’s a 400MB download.)
  4. Open the “Windows XP.vmc” file – just double-clicking ought to work. At this point, you have a working Windows XP version. (The Administrator password is “Password1”.)
  5. Under Tools – Settings – Networking – Adapter 1, select “Shared Networking (NAT)”
    image

That’s pretty much it. You’ve got a Windows XP machine running inside your other Windows machine.

Update (18 Sep 2012): I noticed something weird. The memory usage of VMWindow and vpc.exe is tiny!

image

Between the two processes, they take up less than 30MB of memory. This is despite the Windows XP Task Manager inside the virtual machine showing me 170MB of usage. I’ve no clue what’s happening, but am beginning to enjoy virtualisation. I’ll start up a few more machines, and perhaps install a database cluster across them.