Constructing a library of microcomputer software

This has been a silent space for the past little while.  It's not another case of "blog starts, ten minutes pass, blog ends," but rather waiting on formal approval for making some software freely distributable.

In the meantime, I've returned to contemplating how best to construct my dream project: a digital library of all things from the days of microcomputing past, with a particular emphasis on amateur content and making it maximally possible to examine and actually run the programs in the collection.

Of course, there are a great many projects and archives out there with overlapping aims, notably TOSEC.  There are a few things that I find dissatisfying about these in general: lack of meta-information, limited searchability, emphasis on commercial releases, and not much in the way of source code or encouraging further development.  Rather than being mostly a huge collection of old games you might like to replay, I'd like a retrocomputing library to put items in a cultural context and encourage interaction with them -- even foster continued development.

This has been a bee in my brainsteam for a number of years, and along the way I've tried out some great open source or freely redistributable digital library packages out, chief among them being FedoraCommons and Greenstone.  There are a couple of reasons why they miss the mark for me for a project like this:

1. Metadata construction: There are a lot of items to categorize, and I find that existing system tend to have a large cost (time cost per item) associated with the metadata.  Some of it is the cognitive load of trying to pound square pegs into a round hole (especially true of MARC, Dublin Core, and the like), and some of it is interface sluggishness.  To minimize time per item, I need a flexible, loosely-defined format that I can manipulate in a text editor.
2. Item transformations: The core of the library I envision would be to transform the stored items in various ways.  Suppose the item in question is a game program written in Commodore BASIC.  The library interface ought to be able to give you that program as a disk image, tape image, binary file, text listing, pretty-printed text listing, or possibly even loaded up into an in-browser emulator.  While Fedora has provisions for this sort of thing, I need some flexibility of implementation.
3. Auto-indexing: Another core feature that I'm after is the ability to harvest indexing data directly from software items.  As a basic principle, if a string appears on the screen in the course of a game, it ought to be discoverable using that string -- ditto with code snippets, graphic assets, etc.
4. Schema changes: As a system like this develops, I can expect the metadata to change in format and capabilities a number of times as I discover what's really important, aspects that I'd overlooked, and so forth.  When that happens to a system built on top of a fully normalized relational database, the results tend to be a mass of awkward bandaids or painful rewrites.

As a result of these, I'm strongly drawn to a homegrown NoSQL approach built around protocol buffers and BigTable (or the like).  There's a well-defined text representation of protocol buffers, and they're designed especially for the purpose of extending them while maintaining backward compatibility with existing data.  Dropping a SQL representation of the data allows me to think more easily in terms of a distributed hosting model -- and while the data will be noisier and less compact, I'd rather have that than constantly fight with a rigid schema that doesn't quite fit the item at hand.

(I also have to admit that as a professional developer, I'm drawn to solutions that involve me building things, and I'm a fan of these technologies in part because they're a big part of my day-to-day programming life.)

At this point, I've got a draft of some initial proto buffer definitions for metadata, and I'm going to begin trying to organize some items to see whether it's fast and expressive enough to build a library around.  With luck, my next  post will share those definitions...

Oddities & idiosyncrasies of old code

Since I last posted, I've been digitizing the listings from Rainbow Magazine (a magazine exclusively for the TRS-80 Color Computer, a 6809-based system that I have a particular fondness for).  This is part of a long-running project of mine to gather and organize as much 8-bit material as possible, with a particular emphasis on preserving source code and amateur/hobbyist content.  I plan to write more about this project in the future and would love for it to become a genuine online museum someday.

What prompted this post, though, is a program from Rainbow's second issue (August 1981) that epitomizes part of what I'm getting at when I talk about the fascinating weirdness of programs from the early microcomputer age.  The program is a simple one: just a short application that produces loan amortization schedules (the code is at the end of the post).  It's the error handling that's special; when you're asked whether the payment schedule is based on months or years, here's the screen you get if you enter the wrong input:

The code that produces this screen takes up about 10% of the program.  No other input is validated at all, and the error message isn't even telling the truth: any input starting with "M" is taken as "MONTH", and any input starting with "Y" is taken as "YEAR".

I haven't got the slightest idea what motivated the author to devote a substantial chunk of a program in a magazine to an error screen that berates the user like this, but it's also not exactly rare for the time.  It's silly and unprofessional and you're not likely to see the like these days, but it's also thoroughly endearing and a reminder that behind the code was a person, every bit of as much as bundle of contradictions and neuroses as the rest of us.  And that's why I think it deserves to be preserved as a record of that computing culture.

(Code below the cut.)

70s, 80s, and 90s computer magazines, oh my!

In writing my first post, I was ecstatic to find the Internet Archives have launched a collection of old computer magazines, especially because I had to shed most of my collection (including complete runs of Compute!'s Gazette and RUN, and lots of issues from Kilobaud and Byte in their heyday) when moving from Indiana to California a few years ago.

So why am I such a gleeful collector of these things?  Well, first of all, they're just about the best record of computing past as practiced by actual people that we've got.  The advertisements let us track the rise and fall of various products and their prices, and the articles were generally by enthusiastic hobbyists rather than academic computer scientists.  Sure, it's a little embarrassing on some level to read about someone's implementation of bubble sort in interpreted BASIC, but it's a record of a time in which people had more of a sense that their computer was theirs to manipulate as they pleased -- and the fun of getting the bitty box to do anything outweighed the desire to do it by the most elegant means possible.

But the core attraction for me are the program listings: most of these magazine, particularly in the U.S., contained type-in programs (usually in BASIC, occasionally in assembly, C, Pascal, or Logo).  In general, this wasn't high-quality software... but it was also a key factor in my learning how to program.  The instruction manual that came with the computer was like a foreign language textbook, but the listings were like actual translation exercises.

First of all, you'd always make a number of typos when entering a listing -- and tracking down those typos was a great crash course in debugging, made easier and easier as you developed an understanding of the underlying language.  After a while, you weren't just typing; you were actually reading the code as you entered the program.

Second of all, that listing was just a starting point.  Surely that game could be more fun with a extra couple of features -- let's try to add them!  When you're eight, that generally translates to antics like adding the word "barf" to PRINT statements, but after a while, that turns into adding high score tables, replacing the graphics, designing levels... all of the things that are still good training for budding developers today.  (And it was quite easy to thereby exhaust main memory, leading to some early education in optimization and shrinking code.)

This is something that I think we're lacking enough of today: a pool of relatively simple but entertaining programs that are meant to be modified by the end user.  The open source movement is one I wholeheartedly support (and I've contributed some projects), but most packages are of a complexity and sophistication that makes them unsuitable for the beginner.

BASIC, of course, is a fairly wretched language for serious development, particularly the dialects used by 80s micros, but there are a number of modern languages especially designed to be approachable and foster the exchange of programs.  I'm particularly fond of Processing, which I used as the foundation of my introductory programming class.

Stumbling across this cache has given me some motivation to move ahead with a project that I've been wanting to tackle for a while: taking some 80s computer books and magazines and recasting the listings into a more modern environment -- probably Python, as it's wide-spread, easy to learn, and relatively civilized.

Expect to see more posts on the topic in the near future, as I plan to dig into some of the old Color Computer magazines shortly.  (I've always been particularly fond of the CoCo, and that's a topic for another post...)

An introduction of sorts

Welcome to Bittyboxen!

I've been addicted to computers since I was small enough that I needed to use the thumb and middle fingers of my hand as a splint to make my index finger stiff enough to push the buttons.  It certainly wasn't the colorful graphics that attracted me: we're talking about the dawn of the 1980s here, and I later had a TRS-80 Color Computer for years without ever seeing its video output in color.

No, it was the idea that these things could be programmed that captivated me.  The other objects I knew of tended to do just one thing, but computers could be made to do anything you wanted.  That was sheer, unmitigated magic.  (In theory, of course.  My desires and imagination far exceeded my budding programming talents, and I was far better at creating instruction screens for games that sounded cool than actually finishing them.)

And it was more than just that you could program these things yourself: there was a culture associated with microcomputers that seemed to be all about showing off what you'd gotten your computer to do and sharing that expertise with others.  I witnessed that culture first through magazines (99'er, Compute!, Rainbow, Hot CoCo, Timex/Sinclair User, I miss you all) and delightfully gonzo books from TAB, Sams, and Hayden Press, as well as tagging along with my father to innumerable amateur radio shows.  (I actually enjoyed laboriously typing in BASIC program lists from these magazines -- waiting for the inevitable corrections in the next month's issue, not so much.)

In 1986 or so, I got my hands on a modem for the first time: a 300-baud Volksmodem that ran off a 9-volt battery and couldn't autodial, but a modem just the same.  I was soon spending as much time as I could get away with calling local BBSes, which were full of other people fascinated by these marvelous things (as opposed to the other kids at school, who largely regarded even having a computer as somewhat suspect).  That culture involved not only showing off and sharing your own knowledge, but also other people's knowledge, whether they liked or not -- which is to say, it was also my introduction to cracking, warez, reverse-engineering, etc.  Eventually, I got Usenet access through a local business that ran a dialup BBS on its VAX, figured out how to drop to shell, and began to explore the Internet... but I digress.

The purpose of this blog is to celebrate both the culture that surrounded these early micros and the hardware itself -- and hopefully to spread more of that anything you want spirit to a wider audience.

So, yes, this is yet another retrocomputing blog, but with a bit different emphasis than usual.  I'm not interested in reminiscing over the great 8-bit games we all remember -- Seven Cities of Gold and Impossible Mission were great, but that territory is well-covered already.  I'm more interested in writing about these topics:

• Digital preservation: We're not likely to lose the commercially successful programs of the 80s, but what about the hodge-podge of quirky, off-beat, and bizarre amateur productions that make up the long tail?  Short-lived disk magazines, graphics demos, text adventures in BASIC... these are a window into history and culture just as much, or more, than the big-budget productions.  I want to preserve that long tail and make it discoverable.
• Technical details: I love reverse-engineering, disassembling, and otherwise pulling things apart, and I've got a hankering to analyze some of the classic games of the era down to the assembly-language level.  Ultima III was a wonderful game -- so let's study its code, not just look at its screen shots!
• Retro projects: It's probably no surprise by this point that I'm into emulation of classic software and hardware, and I plan on blogging about the progress of my various projects here.  (For example, I'm working on a multi-player engine for the classic Scott Adams adventure games, written in Go.)
• Sharing technical knowledge: I also hope it's clear by now that I don't much cotton to a priesthood of information: programming, computers, and DIY culture are for everyone, and everyone is worthy.  I've taught introductory programming at the college level with some reasonable success, and I hope to use this space as a forum for spreading how-to knowledge about hacking microcomputers to anyone who's interested, including folks who didn't grow up with these computers.
• Building community: I can be long-winded, but I wouldn't be writing these words if I didn't hope to help connect retrocomputing enthusiasts to one another and meet other Makers.  There's no NIH syndrome to be found here, and I'm more than happy to highlight other people's spiffy projects, solicit guest tutorials, etc.  Commenting is both encouraged and welcome -- please be good to one another, is all that I ask.

So, dust off that old Commodore 64, CoCo, Apple ][, Atari, TI-99/4A, Sinclair, etc., and let's go on a journey together!