Twisted

Although I try to do my Python programming with out-of-the-box (batteries included) Python, there are a lot of excellent Python libraries that are not included in the standard distribution.  XSLT support comes to mind as an obvious omission.  Python does come with a simple HTTP server which I've used extensively, but there is no getting around that its capabilities are fairly limited.  A couple of years ago we looked at the Twisted framework from Twisted Matrix Laboratories.  It looked good and seemed to work, but was obviously still under development and a bit hard to understand.

Well, Twisted is still under development, but it's become much easier to understand, since there is now a book (Twisted Network Programming Essentials) about it.

Twisted is more than just HTTP, it's a framework for asynchronous network programming in Python, but all I've used is the HTTP support, which is pretty good.  One of the problems with Python's BaseHTTPServer is that it isn't asynchronous.  Unless you go to the trouble of adding threads, requests are processed one at a time.  Twisted's approach to this is Deferreds.

Deferred objects are quite easy to use.  As an example, calling the twisted.web.client.getPage method with a URL returns a Deferred to which you can add call-back methods for success and failure.  When the requested page is ready it gets passed to the appropriate call-back for processing.  In my application I am accepting SRU searches, turning them into calls to a back-end SRU server, and then consolidating the back-end responses into my response.  Plus, of course, being a standard HTTP server responding to file requests (twisted.web has a static.File class that neatly encapsulates the expected functionality).

The twisted.web part of Twisted is being replaced by twisted.web2.  I haven't tried it out, mainly because twisted.web is more stable and what Abe Fettig's book describes.  Twisted claims to be able to support quite high traffic loads, and nothing I've seen argues against that.  Moving to a package like this gives us lots of new features, such as a standard way of creating daemon processes, Apache-like logging and, I hope, a robustness that would take a lot of work to achieve otherwise.  It took me a couple of days to learn enough Twisted to get my 200-line server ported, which ended up slightly shorter with more functionality.

As expected in Python, everything worked the same under both Windows XP and Linux, although running the scripts provided is easier in Linux.  Fettig's book is very good, although the constant repetition of the headings 'How Do I Do That?' and 'How Does That Work?' gets a little wearing.  I do wish it was a little longer.  I think by the time he got to Chapter 11 (Services, Processes, and Logging) the author may have gotten a little tired and the chapter is too short.  The documentation on the Twisted site leaves much to be desired -- I was glad I had the book next to me just to get it installed on Linux (as usual, the Windows installation was more automated).

--Th

Update (2006 February 6)

Since I wrote this I ran into this review of the book by someone that really knows Twisted.  The reviewer makes it very clear that twisted.web is not the way to write Web applications in Twisted.  With that disclaimer, here's the code I wrote.  It won't run for you as it depends too much on my environment, and it's a bit of a hack, but it might be of interest to David (see comment below).

Comments

[Edward Vielmetti]

thanks for the review Thom.

What are you using for a Z39.50 library for python? We picked up on Aaron Lav's zoom implementation for the superpatronbot (Brian Kerr is doing the integration work).

[Thom]

We're not doing any Z39.50 in Python. Our Z39.50 code is in Java, but all our recent development has been using SRU, which is easy in straight Python.

--Th

[David]

Hi. I am also interested in Twisted as a web service in python but looking for any kind of examples using twisted.web.client. Is your application open source that I could look at it to see how you are using Twisted. Many thanks.