Outgoing

VIAF Infrastructure

I thought some people might be interested in the infrastructure that supports VIAF.

There are two distinct aspects of it.  The first is the transformation of the raw data into a database and the second the online system that users interact with.

Essentially all the code that gets the data ready to be indexed is written in Python.  Other than the standard Python (2.7) it is all our own code, no added libraries. Most of the processing is done using map-reduce using a system written by Jenny Toves (in Python of course), who also wrote much of the matching code for VIAF.  We started using map-reduce shortly after Google wrote about the technique, gradually improving and extending our system as VIAF and WorldCat processing got larger and larger.

However Hadoop has overtaken us!  We are in the midst of moving to a new machine set up for running Hadoop, porting the VIAF build software to it along with some FRBR processing. Most of the code will remain in Python, but will be using streaming map-reduce support of Hadoop, which is really quite nice. In addition to becoming a standard part of OCLC's production software, Hadoop has a lot of nice job control and monitoring features that are more comprehensive than we were ever able to add to our home-built framework, and it gives us access to HBase.  

The current machine we build VIAF on is a 33-node Beowulf cluster with 132 cores, 4GB of RAM/core and about 1.5TBytes of disk/node. When we first got it a few years ago that seemed like a lot of disk, but we have managed to use most of it. The new Hadoop machine has 40 8-core nodes plus a name node with 12 cores for a total of 332 cores. Each of the 40 8-core machines has 3 2TByte disks and 32GBytes of RAM, so it has several times the capacity of the old cluster, including 10 times the network speed, going from 1 Gbit to 10.  The old cluster disk was fully raided, but the new cluster depends mainly on HDFS for reliability (3x replication).

We could probably process the 120 million bibliographic and 30 million authority source records into VIAF clusters in about a day's processing on the old machine. Normally that gets spread out over several days and has to compete with other jobs.

One of longest processes in building VIAF is the final indexing stage that right now runs as a single processes. As we move to the new Hadoop machine, we are using Hadoop to parallelize that, removing a bottleneck.

The online system runs on a standard Linux box. Over the next few months we expect to move that to a more robust production environment with better monitoring, support and redundant servers. The indexing and server software is all Java, running in Apache Tomcat.  We use Pears, an open source OCLC indexing engine written by Ralph LeVan. Ralph has added linked data support, multi-lingual options and AtomPub support to what is a fairly thin layer of URL rewrite rules and XSLT scripts on top of an SRU system, also built by Ralph.

--Th

 

Just the Links, Please

We haven't done much with JSON in VIAF yet, but Ralph came up with a new feature for VIAF for which JSON seemed a natural fit: requesting a view of a VIAF cluster that just shows the links.  Here's the JSON for this new view of the Mark Twain cluster, http://viaf.org/viaf/50566653/justlinks.json: 

 

{ "viafID":"50566653",
    "BAV":["ADV10188047"],
    "BIBSYS":["x90056487"],
    "BNE":["XX945992"],
    "BNF":["http://catalogue.bnf.fr/ark:/12148/cb11927291n"],
    "DNB":["118624822"],
    "EGAXA":["vtls000823270"],
    "JPG":["500020427"],
    "LAC":["000002392","0105C0556"],
    "LC":["340758"],
    "NDL":["00459304"],
    "NKC":["jn19981002263"],
    "NLA":["000035028957"],
    "NLIara":["000160730"],
    "NLIcyr":["000156897"],
    "NLIheb":["000175478"],
    "NLIlat":["000133341"],
    "NUKAT":["vtls000634910"],
    "PTBNP":["46269"],
    "RERO":["vtls000164542"],
    "RSL":["nafpn-000083946"],
    "SELIBR":["98057"],
    "SUDOC":["027171876"],
    "VLACC":["000002392"],
    "WKP":"Mark_Twain"}

We return an array for each source because some may have multiple IDs (e.g. LAC in the example).  The array labels are VIAF's abbreviations (e.g. http://viaf.org/authorityScheme/LAC) for each of the source files, with WKP standing for Wikipedia.

For those of you that prefer content-negotiation to mangling URIs, the mime-type 'application/json+links' 'application/vnd.oclc.links+json' should also work.

Already built into VIAF is the ability to go from the source ID to the VIAF cluster: http://viaf.org/viaf/sourceID/SUDOC|027171876 will take you to the VIAF cluster http://viaf.org/viaf/50566653, and http://viaf.org/viaf/sourceID/SUDOC|027171876/justlinks.json will give you just the links for that page.

If you want ALL the links out of VIAF then visit the VIAF Dataset page at http://viaf.org/viaf/data.  It has a pointer to a file that lists each of the 27 million VIAF ID to source ID links.

We're having fun here at VIAF Central!, especially Ralph LeVan who thought of and implemented this.

--Th

 Update: 2012.05.09  changed mime type (old one should still work)

VIAF Dataset

VoID The VIAF dataset is now available for public consumption!  http://viaf.org/viaf/data describes and links to the files involved and describes how we expect the ODC-By license to be applied.  We are not sure just how popular the files will be, so if the site appears slow, please stop downloading and come back later.  From my machine here at OCLC my browser is estimating 20-30 minutes to download the larger files, from my home it was double that.

For more about this, see a previous post: http://outgoing.typepad.com/outgoing/2012/04/viaf-developments.html.

One question that has come up was whether it would be possible to incorporate VIAF identifiers and information into a dataset that is released under a CC-0 license.  The short answer is yes.  Here's a longer answer:

• We would like to see acknowledgement of VIAF as a source somewhere on your site
• We encourage the use of VIAF URI's where appropriate and they can be considered acknowledgement
• Incorporation of those VIAF URI's and associated information from VIAF should not prevent you from releasing your dataset under CC-0, since the URI's are considered sufficient acknowledgment

--Th