Outgoing

VIAF and OpenSearch

Update May 2010:  The URLs shown below will work better if you add a 'viaf/', e.g. http://viaf.org/viaf/search... instead of http://viaf.org/search.

Ralph LeVan recently added OpenSearch capability to the Virtual International Authority File (VIAF).  Since the VIAF site is built on top of SRU, the easiest way to that was to use proposed SRU extensions. Here is Ralph's description of it:

Currently, an SRU request generates a searchRetrieveResponse.  The SRU committee has proposed the addition of a new parameter to an SRU request that can indicate to the server that the client would like some other form of response.  For OpenSearch, that means an RSS response.  We have implemented the new parameter and have succeeded in providing OpenSearch access to VIAF.

Here is an example of an SRU query for the name “Dempsey” in VIAF: http://viaf.org/search? query=cql.any+all+ "dempsey"+ &maximumRecords=100 &stylesheet=xsl/results.xsl &sortKeys=holdingscount.  This returns an SRU searchRetrieveResponse, though that may not be obvious to you because an XSLT stylesheet was used to render that response into HTML.  To turn that search (or any SRU search of VIAF) into an OpenSearch query simply add &http:accept=application/rss%2bxml to the end of the URL.

http://viaf.org/search?query=cql.any+ all+"dempsey"+ &maximumRecords=100 &stylesheet=xsl/results.xsl &sortKeys=holdingscount &http:accept=application/rss%2bxml

Support for RSS is built into most browsers and clicking on that link should get you a screen with the results of the search and options to subscribe to that search.  There’s no point in using the subscription option yet, but sometime in the near future you’ll be able to use that search and see only the records that have changed in the results.

OpenSearch is a popular protocol for search aggregators, such as A9.  In addition, we are now providing a hint to browsers when they display VIAF records that the database is available via OpenSearch, so you may see VIAF offered as a potential search engine in your browser’s search box.  OpenSearch requires an OpenSearch Description Document.  We have one at http://viaf.org/allFieldsSearch.xml.  All you have to do is point an OpenSearch client and that URL and it will know everything it needs to search VIAF.

One of the weaknesses of OpenSearch is that it doesn’t support a query grammar.  In our OpenSearch implementation we had to pick an SRU/CQL query that we could drop OpenSearch search terms into.  We chose the generic “All Fields” search that we expose on the VIAF home page.  Other pre-defined queries could be supported with other Description Documents, but we don’t have enough experience with exposing multiple Description Documents to know how the clients will expose the various search options made available.

Our SRU server, which has built in support for Lucene and DSpace databases (SOLR coming soon) is available as open source at Google Code (http://code.google.com/p/oclcsrw/).  The new feature to support the http:accept parameter will be committed to Google Code soon.

If you have any questions about this, contact Ralph LeVan ([email protected])

Ralph

New PURL Server running

As promised earlier, we did in fact install a new version of the PURL server early on July 27, 2009.  We had a couple of surprises, but they have been fairly minor, and with some adjustments we are now completely transitioned to the new software.

The software is open source and was written for us by Zepheira.  The original specifications for the new software allowed for some variation in the exact operation of PURLs.  Our thinking was that part of the reason we were redoing the software was to make improvements to PURLs, and we wanted to impose as few restrictions on that as we could.  As we attempted the transition, however, reality collided with our best intentions and we found that changes that seemed quite reasonable had too great an impact on existing users.  Working through those issues took quite a bit of work and a lot of testing, but I think we finally managed it.

Probably the most important new feature in PURLs is the ability to create advanced PURLs for which you can control the HTTP status code (e.g. the new 303-See Other redirection code used in the semantic Web).  You can also delete PURLs.  Once you have done this the PURL is tombstoned so there is a record in the PURL server, even though the PURL no longer is available for redirection.

In case you are interested, here are some of the problems we encountered:

• Non-standard characters.  Some characters (e.g. the dollar sign $) are valid is some parts of a URI, but not in others.  We still have a few unusual PURLs with valid but unusual characters we aren't handling correctly.
• Undocumented features.  There were some cases where the old PURL server would try to 'do the right thing' even though the 'right thing' was neither documented nor expected, for instance treating a PURL as a partial indirect even though it was not.  In almost all cases we were able to fix these problems by more conventional PURL techniques.

• DNS problems.  There was at least one case where someone had redirected a domain name to purl.org.  We have actually used that trick in-house, but did not realize anyone else was doing it.  The new software is more aware of where requests originate than the old system and blocks this sort of thing.  In the case in question we were able to accommodate them, but it isn't something we are encouraging.

The DNS problem in particular shows how difficult it can be to thoughly test this type of software.  PURLs are very much embeded in the Web and it is difficult to predict all the ways various PURLs intereact.  We did get better at it with (rather painful) experience, however.

Since purl.org resolves something close to two million PURLs/day (about 20/second), we do everything we can to avoid disrupting that flow, and had only minimal interruptions in this transition.

--Th