Now for something a little different:
Since retiring from OCLC I don't do a lot with library metadata, but I've recently had some fun exploring astronomical images, which come with their own data/metadata format FITS, the Flexible Image Transport System.
Everyone that wants to share astronomy data uses FITS. It was developed in the early 1980's and has a strong FORTRAN flavor in the how the data is stored. Having processed the variable length fields inherent in MARC records with FORTRAN I can appreciate the attractiveness of fixed length blocks, arrays of binary data and 80-byte card images to the engineers/scientists of the time.
One of the wonders of our time is all the astronomical work that is being done, and that within a year or two of the observations much of the data is publicly available. The image at the head of this post came from the Hubble Legacy Archive which has an interface that will allow you to search by name and star catalog numbers, select the type of image you are interested in, and view previews of images before downloading the FITS file.
Of course most of the fun in working with the images is writing some of the code that makes it possible. There are lots of programs available that will help you look at FITS files, such as FITS Liberator which will 'liberate' FITS images into something that Photoshop can process. Those are nice, but farther away from bare metal than I like to be. So I wrote a little program in J that does some rudimentary processing with FITS data. J (download it here) is a slightly obscure (but actively used and maintained) language derived from APL. Possibly more accurately J evolved from APL in an effort led by Kenneth Iverson, the inventor of APL. While it does take some initial effort to become proficient in array languages such as J, it is remarkable how much can be done in a few characters. Admittedly those few characters may take some deciphering, but so would the much longer code they replace. In some ways it reminds me of trying to use a new alphabet, such as Cyrillic. At first the script is confusing or actually misleading, but once learned they just become letters.
I got introduced to APL in the late 1970's when I first joined OCLC. At the time OCLC ran on Sigma computers from Xerox/Honeywell. Xerox tried to compete with IBM in the early 70's and APL that was one of the few languages available on Sigma machines (in general we did most things in CP-V assembler which was really quite nice). Their APL was clunky and slow and OCLC didn't have an APL terminal, but it worked and I used it to do some research into how people were using search keys on the OCLC system (not so well!).
J can be described as a fusion of APL and Backus's FP. It is open source, easy to install, does not require a special alphabet and the things it can do with arrays are amazing, if not always immediately obvious. One of the things I like about it is the brevity of the code. Having experimented with compact code in Python (Z39.50 client on a t-shirt), it is surprisingly easy to work with dense code because you can see so much of it at once. J does invite a certain amount of points-free coding, a style that confuses me at times, but can be quite elegant. Map-reduce, is another style of functional programming that can be difficult at first to get comfortable with, but turns out to be very powerful. We used map-reduce extensively at OCLC, so I came to J with some familiarity of that aspect of the language.
The code that produced the image at the top of the post can be found at github.com/ThomasBHickey/JFits. It consists of about 60 lines of code in two files and has a couple of sample FITS image samples as well. Since it is one of my first J programs, I asked the J programming forum to take a look at it, and they came back with a number of suggestions, so most of the clever code probably came from them. I've tried to keep the code reasonably straightforward so it might be worth a look, but if you are interested in astronomy, it isn't that hard to explore what's available without doing any programming at all. Don't expect the images to look just like the ones you see published, however. Those have had a fair amount of processing (often in Photoshop) to tease out the most pleasing parts and suppress many of the instrument artifacts that seem to be in all the images.
--Th
Comments