In the meantime, I've returned to contemplating how best to construct my dream project: a digital library of all things from the days of microcomputing past, with a particular emphasis on amateur content and making it maximally possible to examine and actually run the programs in the collection.
Of course, there are a great many projects and archives out there with overlapping aims, notably TOSEC. There are a few things that I find dissatisfying about these in general: lack of meta-information, limited searchability, emphasis on commercial releases, and not much in the way of source code or encouraging further development. Rather than being mostly a huge collection of old games you might like to replay, I'd like a retrocomputing library to put items in a cultural context and encourage interaction with them -- even foster continued development.
This has been a bee in my brainsteam for a number of years, and along the way I've tried out some great open source or freely redistributable digital library packages out, chief among them being FedoraCommons and Greenstone. There are a couple of reasons why they miss the mark for me for a project like this:
- Metadata construction: There are a lot of items to categorize, and I find that existing system tend to have a large cost (time cost per item) associated with the metadata. Some of it is the cognitive load of trying to pound square pegs into a round hole (especially true of MARC, Dublin Core, and the like), and some of it is interface sluggishness. To minimize time per item, I need a flexible, loosely-defined format that I can manipulate in a text editor.
- Item transformations: The core of the library I envision would be to transform the stored items in various ways. Suppose the item in question is a game program written in Commodore BASIC. The library interface ought to be able to give you that program as a disk image, tape image, binary file, text listing, pretty-printed text listing, or possibly even loaded up into an in-browser emulator. While Fedora has provisions for this sort of thing, I need some flexibility of implementation.
- Auto-indexing: Another core feature that I'm after is the ability to harvest indexing data directly from software items. As a basic principle, if a string appears on the screen in the course of a game, it ought to be discoverable using that string -- ditto with code snippets, graphic assets, etc.
- Schema changes: As a system like this develops, I can expect the metadata to change in format and capabilities a number of times as I discover what's really important, aspects that I'd overlooked, and so forth. When that happens to a system built on top of a fully normalized relational database, the results tend to be a mass of awkward bandaids or painful rewrites.
As a result of these, I'm strongly drawn to a homegrown NoSQL approach built around protocol buffers and BigTable (or the like). There's a well-defined text representation of protocol buffers, and they're designed especially for the purpose of extending them while maintaining backward compatibility with existing data. Dropping a SQL representation of the data allows me to think more easily in terms of a distributed hosting model -- and while the data will be noisier and less compact, I'd rather have that than constantly fight with a rigid schema that doesn't quite fit the item at hand.
(I also have to admit that as a professional developer, I'm drawn to solutions that involve me building things, and I'm a fan of these technologies in part because they're a big part of my day-to-day programming life.)
At this point, I've got a draft of some initial proto buffer definitions for metadata, and I'm going to begin trying to organize some items to see whether it's fast and expressive enough to build a library around. With luck, my next post will share those definitions...
No comments:
Post a Comment