Entire Library of Congress

One often hears statements like 'transmitting the Library of Congress in 15 minutes' or 'equivalent of the Library of Congress on a key fob.'  I suppose I've contributed to that by putting all of WorldCat on an iPod (not hard to do, but the iPod isn't up to interacting with it yet), but how realistic is LC on a key fob?

The Library of Congress is larger than I thought.  The site claims 29 million books, 2.7 million recordings, 12 million photographs, 4.8 million maps and 58 million manuscripts!  Sometimes people equate a volume to a megabyte (the typical novel is around that), but more realistically, you'll need a scanned image of those pages, around 100Kbytes/page.  At 500 pages/volume that gives us about 50 megabytes per volume.  At 500 megabytes/recording, 2 megabytes/photo, 5 megabytes/map and 50 megabytes/manuscript I get: 30m x 50mb + 3m x 500mb + 12m x 2mb + 5m x 5mb +60m x 50m = 6 petabytes.  This doesn't include video.  At 5 gigabytes/video, it only takes 300,000 videos to match the scanned size of all the books, so lets call the collection an even 10 petabytes.  This is quite a bit larger than the size people often use, but more realistic.

My son recently put a terabyte together for around $1,000.  I just bought a one-gigabyte Memory Stick for $100 (about 100x the cost of disk).  10 petabytes is 10,000 terabytes and 10,000,000 gigabytes, so we've got something between 4 and 7 orders of magnitude in cost reduction to wait for.  Assuming the cost of storage continues to decline at 50% per year, it takes about three years to get a 10x reduction in price.  So it will be 12 years before you could stuff the contents of the Library of Congress on your personal storage server, 21 years before you could get it on your key fob.

That's not too bad.  15-20 years is a long way to extrapolate, but the cost decline has held up for the last 40.  I got asked the same question in 1970 though, and didn't have any easy answer then as to what it would mean.  We were talking microfilm, but it was still the same question.  Of course I suppose the contents of the Library of Congress fall into the "limited canon of book-centered knowledge", so the interest in this has probably already declined.  Still, it does give you an idea of the capabilities we might have and how hard it's going to be for publishers to keep everyone from having a copy of everything they come in contact with.

--Th

Update 2011 June 30: It's been six years since this was originally posted. Using the rough 3 years/10-fold decrease we should have seen a 100-fold decline in storage prices. Looks like you can get about two terabytes in a USB external drive for about $100, or a factor of 20 rather than 100. SD cards are now in the $1/gigabyte, a 100-fold reduction in 6 years, pretty much on target.

So, $1,000 now gets you 20 terabytes of disk, but we need 10 petabytes, or 500x20TB. Call that 2.5 orders of magnitude at 3/years/magnitude gives 7-8 years left at the original rate, and more like 12 years at the current rate of decline. That is the same wait predicted 6 years ago, especially if, as one of the comments suggests, LC continues to grow their collection. The $100 key fob at 100 gig, still needs needs a 100,000-fold drop in price, but only 15 years might get us there!

Update 2015 May: Ten years since this was first posted, and the exponential drop in prices seems to have slowed.  USB sticks in general seem to have stopped growing, although you can get a 1 terabyte stick fro about $800. Amazon is offering at least one 128 GByte Micro SD card for $20 (oops, see update below).  Those little cards are 165 cubic millimeters (15x11x1).  A cubic foot could hold about 154 thousand of them, or 20 million gigabytes, probably enough to hold LC, even with some growth, especially since image compression algorithms have improved substantially over the last ten years.  That would be heavy to pickup, but it would about as portable as the first portable PCs!

But 50 thousand 128 GByte Micro SD cards would cost $1 million.  To get to a $100 we're still looking for 4 orders of magnitude, or 12 more years at best.  5 terabyte disk drives are now about $130 so $1,000 gets you nearly 40 TBytes on disk, half of what it cost four years ago, but not nearly as fast a drop as predicted.  Disk is still cheaper than flash memory, but only by a factor of 10. If we wanted to give up on the $100 stick and targeted a $1000 hand held device (which would be disk today), we are only three orders of magnitude away from have LC in our hand!  Maybe another 10 years?

As several of the comments have mentioned, we're going to be depending on telecommunications for access to collections this size for some time to come.  One thing that has changed is that there are now lots of places with storage systems big enough to hold 20 petabytes without any problem, so hosting such a collection is now technically feasible.

Update to the 2015 May update:

Turns out that $20 128Gbyte Micro SD card was bogus.  It pretends to be 128G, but really will only store 8G.  About the cheapest real Micro SD cards right now are $30 for 64GBytes, triple what is assumed above.  --Th

 

--Th

Comments

[Andy Havens]

I read an article awhile ago -- I think it was in "Wired" -- saying that data storage technology has actually been outstripping Moore's Law pretty significantly. I tried to find the exact article on Wired's site, but was unable to do so.

In searching around, though, I found this neat site: http://www.littletechshoppe.com/ns1625/winchest.html on the history of the cost of data storage.

Now expense does not, of course, translate directly into size. So it may take that 15-20 years to get the LOC on a key fob. What may happen more quickly, though, with the advent of high-speed, wireless Internet access being upon us, is the ability to access the entire LOC from a device the size of a key fob, even if the data isn't stored right there.

Which is, of course, kinda better. Since as soon as you put all that data on any kind of media, it's immediately out-of-date. And you can't connect with it to other libraries. Or other stuff.

Here are the related questions about connectivity, then: When is my wireless phone going to be so small that it gets worn as an ear-ring or implanted in my jawbone? When is the screen/monitor/HUD for the device going to be embedded in my glasses or something that can project an image right onto my retina? When is the speaker going to be the size of a hearing aid? When will it all be so small and cheap that the devices themselves are disposable?

Don't laugh... any of us over the age of 30 should think about how bizarre the words "disposable camera" would have sounded in 1975.

[Thom]

That's a great site. (The URL didn't display right in one of my browsers. Here it is again as a link.) Size and price seem so highly correlated that they are almost the same thing for storage. I too have seen articles showing the exponential drop in the price of storage has actually accelerated (to everyone's surprise), but I'd hate to base projections on that continuing. In truth, it will be a bit surprising to see the current exponential drop in price continue for 25 more years, but as I said, it's happened before.

--Th

[Lisle]

OK, sign me up. I want the LC keyfob that seamlessly connects and updates and interacts with the mothership in DC and anywhere else I want and the monitor in my glasses (rimless, please, no thank you laser surgery) and the input device wherever and I am willing to wait 20 years, no problem. It sounds so scifi and Gibson and Stephenson seemed so out there fifteen years ago. But hey, you are giving me hope! The intellectual property battles are what we can't predict.

[Raphael]

I put 'Library of congress size' into google and got this article.

3 years later, and $100 will get you a 4Gb stick. (The biggest flash drive is supposedly around the 1.5Tb mark - but $30k is a bit much)

$600-700 will get you a 2Tb HDD (Biggest I could find) With $1000 maybe you could manage 3Tb?

It's interesting to run into the article 3 years on and see changes in size and cost...

Right on target.

Good article

[David Morton]

Interesting analysis, although as always they date quickly. A 64 GB USB drive can now be got for US$ 150.
My real point though is the Library of Congress receives 22,000 new items each day and catalogs 10,000 (I just looked at their facts pages), with a total of almost 140 million items. Therefore, I suspect their growth rate is fast eough that USB drive storage technology, or anything we mere mortals can easily use, will NEVER catch up to the Library of Congress.

[Walker Hall]

Enjoyed the read! As early as November or December 2008, there have been occasional brief deals for 1.5TB drives as low as $125.

To the comment above mine: even if the Library of Congress is adding 32,000 new items daily, their data is still just increasing arithmetically, while computer storage is increasing exponentially. So, unless the Library of Congress begins doubling exponentially (extremely unlikely), then USB drives and EVERY digital storage device us mortals create, will DEFINITELY catch up and far surpass the Library of Congress. It's not a question of IF, but rather WHEN. It's simple math.

[Joe]

I just thought I'd mention I saw a TB external HDD for about $120 today and it's been roughly 3.5 years since that PB was put together for $1,000 :)

[Monty Cantsin]

Far too late to reasonably reply, but that won't stop me. Andy Havens, the example of imagining a disposable camera c. 1975 is not a great one as the technology for a disposable camera has been around for as long as film really - a camera just being a box with a hole in it more or less. The advent of the disposable camera was just due to the imagination of the film industry, realizing that they could reuse the chassis when the film was developed. For that matter it's pretty easy to imagine constructing a truly disposable camera for a few dollars more than the price of the film; this is not particularly a function of technology though it helps to have injection-molded plastic I guess.

[Chris]

"21 years before you could get it on your key fob."

The library of congress on a key fob. Has anyone thought how mind boggling this is? That's several centuries of data that can be tossed around in the palm of your hands. Out of this world lol