Outgoing

VIAF as linked data

Update 2010 May:  See a more recent post about VIAF and linked data: http://outgoing.typepad.com/outgoing/2010/05/viafs-new-linked-data.html.

As promised in an earlier post, VIAF (the Virtual International Authority File) is now available as linked data. For those not familiar with VIAF, it is a merger of nearly 20 national-level name authority files (currently we are only dealing with personal names).  There are some 9.5 million personae described in VIAF and have established more than 4 million links between the files.

To us linked data means:

• URIs for everything
• HTTP 303 redirects for URIs representing the personae our metadata is about 
• HTTP content negotiation for different data formats
• An RDF view of the data
• A rich a set of internal and external links in our data

Here are some example URIs for Lorcan Dempsey in VIAF:

• The canonical VIAF record: http://viaf.org/viaf/3697804
• A MARC -21 version: http://viaf.org/viaf/36978042.m21
• UNIMARC: http://viaf.org/viaf/36978042.unimarc
• RDF: http://viaf.org/viaf/36978042.rdf
• The persona being described: http://viaf.org/viaf/36978042.rwo

In addition to using the extensions (e.g. m21), HTTP content negotiation can be used to get the various versions of the record. 

The canonical URI currently returns an XML version of the VIAF record (which gets rendered into HTML in the browser via XSLT), but in the future we expect to return HTML.  As more of the authority files contributing to VIAF become available on the web we will be including more links to external sites.

For RDF we are currently returning a fairly rudimentary Friend of a Friend (FOAF) record.

The design and implementation of VIAF's linked data was done by Ralph LeVan.

--Th

Python 3.1.1

Last November I did a short post about Python 3.0's speed reading in some XML and parsing it, finding it quite a bit slower than 2.5.  We haven't started switching to 3.0, partly because of that, but I noticed in some recent release notes that the I/O in 3.1 has been redone to increase its speed.  So, I thought it was time to redo the test.

I couldn't find the old code and test file, but the idea was pretty simple: read in chunks of UTF-8 encoded XML and have CElementTree parse it.  I'm happy to report that Python 3.1.1 does this faster on my 64-bit Vista machine than Python 2.5.2.

For testing I just read in 16 lines of XML that made up a 217 MByte file.  In 3.1.1 I indicated the encoding during file opening as 'UTF-8' and passed the resulting (Unicode) strings directly into CElementTree's parsestring method.  In 2.5.2 the text was read in as a standard string and passed unchanged to parsestring (as UTF-8).

Python 3.1.1: ~15.3 seconds

Python 2.5.2: ~22.8 seconds

Of course we do a lot more with Python than just read in XML and parse it, but until we got past this basic speed issue it was hard to even think about converting to 3.X.

--Th

More ships in WorldCat Identities

I've written before about ships in WorldCat Identities, but never tried to find all the ships for which there are Identity pages.  Recently Peter McCracken suggested that ShipIndex.org might link to appropriate Identities pages (and vice-versa), so we went ahead and tried to locate all the ships in Worldcat.

Not surprisingly, identifying all the ships took a little work.  The task reminds me of trying to find fictitious characters in WorldCat, and we used a similar approach looking at the parenthetical qualifier that describes the name.  For instance the Confederate sloop Alabama is controlled as Alabama (Screw sloop).  The trouble is that there are quite a few different types of ships and boats, from the Albatross (Steamer) to Zingarella (Bark).  Here is the basic list we are currently using:

• barge
• bark
• barque
• boat
• brig
• carrier
• catamaran
• corvette
• cruiser
• destroyer
• dredge
• freighter 
• frigate
• galleon
• ironclad
• ketch
• packet
• schooner
• ship
• sloop
• steamer
• submarine
• sweeper
• tender
• trawler
• trimaran
• tug
• vessel
• whaler
• yacht

I'm sure there are more words that describe ships, so please let me know if you notice anything missing.

Of course you have to be careful not to include airships or boat seaplanes or ship replicas or townships or Steamboat Springs, so I also have a list of patterns to skip, and then a list of overrides for the skips.

Altogether I found just under 40,000 Identity pages for ships in WorldCat Identities.  About 6,700 of them have an LC/NACO authority record associated with the name.  Approximately 11,000 had two or more citations and 6,000 had three or more.

How successful we will be in linking the databases remains to be seen, but it was interesting to try to find all the ships.

Here are the regular expressions (in Python):

#Always accept these:

simpleRE = re.compile(r'\(battle.?ships?\)|\(.*cargo.?ships?\)|\(cruise.?ships?\)|\(motor.?ships\)|\(passenger.?ships?\)|\(steam.?ships?\)|\(war.?ships?\)|\(ships?\)', re.IGNORECASE)

#Skip these

skipRE = re.compile(r'\(airship\)|\(air.ship\)|\(.*boat seaplane.*\)|\(.*boat work.*\)|\(.*fellowship.*\)|\(.*friendship.*\)|\(.*govenership.*\)|\(.* key.*\)|\(.*partnership.*\)|\(.* replica.*\)|\(.*ship[a-z].*\)|\(.*springs.*\)|\(.*township.*\)|\(.*twonship.*\)|\(.*voivodeship.*\)', re.IGNORECASE)

# The main regular expression

shipsRE = re.compile(
    r'\(.*barges?\b\)|\(.*bark\b\)|\(.*barkentine.*\)|\(.*barque.*\)|\(.*boat.*\)|\(.*brig\b\)|\(.*carrier.*\)|\(.*catamaran.*\)|\(.*corvette.*\)|\(.*cruiser.*\)|\(.*destroyer.*\)|\(.*dredge.*\)|\(.*freighter.*\)|\(.*frigate.*\)|\(.*galleon.*\)|\(.*ironclad.*\)|\(.ketch.*\)|\(.*packet.*\)|\(.*schooner.*\)|\(.*ship.*\)|\(.*sloop.*\)|\(.*steamer.*\)|\(.*submarine.*\)|\(.*sweeper.*\)|\(.*tender.*\)|\(.*trawler.*\)|\(.*trimaran.*\)|\(.*tugs?\b\)|\(.*vessel.*\)|\(.*whaler.*\)|\(.*yacht.*\)', re.IGNORECASE)

--Th