I’ve learned from #code4lib that Erik Hatcher is working on a Ruby library that will index MARC (so he has a start on a MARC mapping in a subversion repository). Are there other sources for seeing how people have mapped their MARC (or, even, how they’ve cleaned up their data — I know CDL has a date normalizing library). Is a site where this sort of information could be shared something that other people would find useful (and do our ILS contracts allow us to share it in some generic form)?

Currently, ruby-marc uses REXML, a pure Ruby XML library. Since REXML is built into Ruby, it is convenient. I was curious, though, how much of a performance boost there would be from using Libxml2. Here are the results of my very informal test (using some HCL MARC data):

	User	System	Total	Real
XMLReader [old]:	24.300000	0.030000	24.330000	25.607547
XMLReader [new]:	3.180000	0.010000	3.190000	3.231896
XMLWriter [old]:	38.960000	0.060000	39.020000	41.017238
XMLWriter [new]:	11.950000	0.050000	12.000000	12.607114

Both XMLWriter times include the new XMLReader reading records in from a source file. As a record is read in, it is written out to a new file. This is just intended to get an inkling of what the difference between the two versions might be (not to be a formal benchmark). Lower numbers are better.

So, in reimplementing, I completely rewrote the reader. It just reads from a file and returns MARC::Record objects. What is being used to read the XML is completely swappable with anything else.

With the writer, I changed the encode method so that it now takes an option specifying which library should be used (REXML is the default still). Since the method is public, I figured someone is probably using those REXML Documents returned and their code would break if I returned a Libxml Document instead. The write method, on the other hand, now uses Libxml by default.

I haven’t checked in any of these changes yet (since I haven’t passed them by Ed and don’t know whether they should be incorporated), but I have validated that the existing tests still pass just fine.

The speed improvements are pretty nice. If an extra dependency can be tolerated it would be nice to have the performance boost. The only other caveat is I used the 0.4.0pre01 version of Libxml-Ruby. It might be desirable to wait until the final 0.4.0 release.

Anyway, I’ll get Ed’s opinion on all this sometime this next week. Right now, it is just a fun experiment.

I don’t think we have an answer to this question yet because noone that I am aware of has moved the complete MARC structure into RDF. So, to me, the question of how many triples are there in a MARC record is a bit like the guessing game: “How many jellybeans are there in the jellybean jar?” I’m sure someone out there knows the answer to how many possible units of information are there in a MARC record (you’d have to break out all the encoded info from the control fields too), but this is a bit different than triples because the relationships between those units have to be represented as well.

It would also, perhaps, be more instructive to know how many triples there are in your average MARC record (and how many on the high end). This is related to the question, “Which parts of the MARC record do we actually use?” Bill Moen’s MCDU group is doing some interesting work in this area.

As for triples in your average MARC record, my guess would be a hundred (keep in mind all the subfields and the number of bytes in each control fields if that seems like a lot to you). Okay, that’s totally off the top of my head with no real logic at all, but it doesn’t seem too far off to me either (in fact it might be a bit low). So, using this completely bogus guess, a database of 10 million triples would represent a database of a hundred thousand records. I think it is worth noting that different triple-per-record counts would affect all parts of the system.

It’s funny that I’ve been contemplating all this RDF stuff again. I first looked into it when we were developing XOBIS. I think Leigh Dodds’ question was the right when he asked: “What does RDF give us that XML can’t?” Well, he said “XML or a relational database” but I think the database part isn’t really the question since we don’t have to put XML into a relational database (I can certainly understand the frustrations of developers who have had to map one XML model after another into a relational db).

His answers seem to fall into three categories to me. The first is the database/modelling question, the second is easily combining disparate data, and the third is the ability to do Semantic Web-like things (any time someone starting talking about machines “inferencing” I lump it into the last category (right or wrong)). The first seems to be to be more directed at relational databases, the second seems to me to be handled by choosing RELAX NG and NRL. The third I remain skeptical about.

I don’t mean to trivialize these issues by categorizing them in this way. I don’t think XML offers a complete solution to MARC yet either (this doesn’t mean we shouldn’t keep looking and working on one though). I guess this is just my last sweep, before leaving the RDF stuff behind (though I have emailed the AustLit people to find out a little more about their caching system), to sort of make sense of it.

Since RDF isn’t something I work with in my daily life, I’ve probably already spent much more time than I should taking another look it at (and, yes, though I’ve been arguing a point I’ve also been genuinely looking). Maybe in another five years I’ll poke my head up again and see if there is any more out there. Back to work…

My own post (far from thoughtful, unfortunately… hmmm, am I blog people?) commented that MARC and XML are the first layer (both are the structure in which information is passed around). Then I sort of merged the next two layers, commenting that a 245 has no real importance apart from that assigned to it by cataloging rules (e.g., the idea of title and the rules that govern recording a title both seem, to me, to be under the control of cataloging rules (both are AACR2 governed)).

Lorcan Dempsey also commented that Dublin Core focused on the middle layer (and this caused several possible third layer possibilities to pop up). The first layer, of course, is handled by XML. The whole thing made me think a bit about XOBIS.

We have assumed the encoding layer would be XML. We focused on the second layer and explicitly said, when appropriate, that a particular issue would be handled by the third layer. The interesting difference between DC and XOBIS is that DC started as a community effort… committees were set up, vested interests consulted, etc. XOBIS is just an experiment that attempts to ask and answer (or, rather, give one possible answer to) some interesting questions, but we haven’t really tried to drum up a community around it.

We have been told that we should do this, but apart from giving presentations on it we haven’t. I think, in part, this is because we are more interested in the questions than in doing the work needed to make a standard (perhaps I should just speak for myself on that though).

We have always taken the approach that we are experimenting (this gives us a bit more leeway in our approach and delivery). This is not to say that we shouldn’t tackle the nuts and bolts, but that we’ve worked from the perspective that we didn’t have limitations (so that we would not be restricted by them). In the end, though, one still has to deal with them in order to produce something useful.

Interestingly, Dick has just finished a study of the tags used by Lane Library. Now that I am no longer there, though, I find myself wondering (once again) about an XSLT stylesheet that goes from MARC to XOBIS (my earlier attempts at tranforming MARC into XOBIS were done in Java).

Lane and Dick do a lot of special things with their records (making the transformation easier). I’m now wondering about what will be involved with creating a transformation for a standard MARC record (without all of Lane’s special enhancements). Once I get a little more free time (after April 15th), I’d like to take a shot at it.

Anyway, I got all excited for nothing! When I get an accessible copy, my first plan is to convert them into XML and make them searchable via eXist. Eventually, though, I’d like to convert the records into XOBIS and experiment with retrieval and navigation of the records. I thought about requesting them long ago when I first heard about the snapshot, but the license restricts any changes to the records’ content so they cannot be upgraded or modified… they are essentially a snapshot of the HCLC’s cataloging records at the time Berman left. At the time I first heard of them that was enough to discourage my interest. But now that I’m thinking about writing an aut navigator, the records would be useful (even without being able to modify the content — perhaps specifically because I can’t modify the content).