My first thought after digesting the conversation was, “Hey, wait, I’m not even talking about native XML databases at code4libcon!” My proposal is about using XQuery in the digital library realm. True, we are using a native XML database here, but just because one uses XQuery doesn’t mean s/he is using a native XML database. You can use XQuery just as easily with DB2 or Oracle’s database (or files on the file system).

The one thing that native XML databases and XQuery do have in common, though, is that they let you interact with your data directly — it doesn’t have to be deconstructed into another structure and then reconstructed when you want the whole thing back out again (in the case of XQuery being used over a relational database, that (de|re)construction takes place invisibly in the database layer).

But, is this a good thing? Ed kept saying he didn’t see any data modeling going on with native XML databases.

There is data modeling going on with native XML databases, I’d suggest, but it happens on the metadata side of things. Andrew Nagy made this observation recently on the code4lib mailing list when he noted how poorly just putting MARCXML into a native XML database performs.

This is because putting MARCXML into a native XML database makes MARCXML the data model. MARC was intended for concise transfer, not for working with the data… it was assumed by the architects of MARC (I believe and hope) that MARC would be reconstructed into something else before anyone tried to do anything meaningful with it (for what it is worth, this only partially happens in the library world).

XQuery allows the developer to work more nimbly with the data models s/he is given (instead of mapping them into other data models that match the database s/he has chosen to use). So, what are these data models? They are the XML metadata standards being created by the different knowledge communities. Unsurprisingly, to use these data models, the developer needs to know them (i.e., having programmers in our libraries is a better idea than contracting out to people not in the profession).

Are the people creating these (meta)data models working to make accessing the data easier? That could be one critique leveled at native XML databases (from the perspective of the developers)… if you aren’t doing the data modeling, can you trust the people who are?

It’s not that bad though (put down that gun you cynical library developers); keep in mind that XQuery isn’t a fulltext query language. Think of it more as a database query language (even though there doesn’t have to be a database).

To use XQuery in the digital library world, in my opinion, you still need to use a fulltext indexer (like Lucene or the type built into many XML-enabled databases). Indices may be used through proprietary extensions to the XQuery language (indicated by a different namespace) or through separate processes which feed the XQuery engine (as in a pre-processing stage).

For what it is worth, there is a fulltext extension to the XQuery spec that is being written to take advantage of these external indices, but it is not really out there in the world yet. In the meantime, even if our metadata models (e.g., MARCXML) aren’t the best, we can still create and use indices that provide an intelligent view of the data.

One nice thing about working directly with the data/data models you receive is that there are less “moving parts” to fix when things change (in other words, less things to get in the way) — because we all know digital library standards don’t change, right?

Rather than go through the process of re-mapping to the database’s structure, you only need to modify the parts of your code that deal with the parts of the metadata that have changed. You’d have to do this with the other option too… just because you have a standard way of normalizing data doesn’t mean that all data is structured in the same way (in terms of how you get at the pieces you want).

I could mention, I guess, some reasons why I like native XML databases in my presentation, but I’m not sure this is a good idea. I think it may distract from the beauty that is XQuery. I’m also hoping Andrew Nagy will cover this territory in his presentation comparing different native XML databases. One of XQuery’s strengths is that it is database agnostic; I shouldn’t stray from that.

For the record, the Ed Summers that appears in this post is not the real Ed (despite the conversation actually happening), but one of my own conception for rhetorical purposes only.