Outgoing

Python 3.0rc2 speed

Update: Python 3.1 is substantially faster.  See this later post.

--Th

We run a lot of Python code and have been following the Python 3.0 development with interest.  3.0 is the new incompatible version of Python.  The new version is still in testing with a final version expected in December 2008.  It comes with a script (2to3) that can transform much of 2.x Python into 3.0 Python.  I have run the script on small pieces of code, and for those it does all you could expect it to, which is a pretty good translation.

Since one of the things we use Python for is processing large amounts of XML code(e.g. 120 million 3K chunks) , we worry about I/O speeds and XML processing speed (we use cElementTree that comes with Python for our XML support).

So I ran a small test.  I took 10,000 XML records (79 megabytes).  Read them in and parsed them using cElementTree's fromstring method.

The 2.5.1 code takes 3.8 seconds on my PC running 64-bit Vista.

In 3.0, opening the file with open('filename', encoding='utf-8') and passing the (Unicode) strings to cElementTree: 17.1 seconds (4.5 times as long as 2.5.1).

Opening the file as a binary file, and converting the input to a 3.0 string (strings are now Unicode) cut the run time from 17.1 seconds to 5.0.

Avoiding the conversion into a string by passing a bytearray to cElementTree cut the time down to 4.4 seconds.

We were disappointed in the speed of doing things in a standard manner, especially the time it took to read in UTF-8 data and return Unicode.  Our 2.x programs are constantly worrying about whether a string is in Unicode or not and it would be great to just work in Unicode.  But while we could probably live with slightly slower code to get that, when you have a program that runs all night, you don't want to see it take four times as long to run.

Maybe the final release will be faster.

Michael Watkins seems to have had quite a different experience.  My timings on cElementTree in 3.0 show it parsing XML just slightly slower than in 2.5.1.

--Th

Australia and New Zealand

Eric Lease Morgan and I recently were the guests of the National Library of Australia at their 2008 Forum, held at the Powerhouse Museum.  I talked about authority files, primarily VIAF and WorldCat Identities.  Eric talked about the future of catalogues.

This was my first trip to Australia and I was very impressed with Sydney, the National Library (which has a lot of things going on) and Australians in general.  Luckily I got a chance to rehearse my talk in front of a large group at the National Library in Canberra and was able to adjust my talk to Australia a little better.

I was also pleased that the museum (named Powerhouse because it is housed in a building that was once a power house) was able to connect their catalog up to WorldCat Identities just before the conference.  Here's an example of a page with links to WorldCat Identities on it.  If you scroll down, in the right-hand column there is a 'Person' section, and clicking on any of the [view on WorldCat] links will bring up a corresponding WorldCat Identities record.  I used Bob Carr in my talk, and his page, though not perfect, is a pretty good example.  Seb Chan manages Web services at the museum and is a great advocate for the advantages of linking between institutions such as museums and libraries.  They took all the names they thought they might link to Identities and searched for links in batch.  This lets them know before a page displays whether there are any outside links, and who they are to.

One of the adjustments to my talk was to substitute Phar Lap for Man 'o War as an example of animals represented in Identities.  Phar Lap was born in New Zealand and you can see his skeleton in the Te Papa Museum of New Zealand in Wellington.  You can see the outside of Phar Lap in the Melbourne Museum (which I didn't get to) and his heart in the National Museum in Canberra (I was in Canberra, but didn't get a chance to go to the museum).  Australians and New Zealanders take horse racing seriously!  If you ever get to New Zealand, I can recommend the train trips available.  Both the TranzAlpine and TranzCoastal have spectacular scenery.

--Th