For the upcoming Code4Lib preconference, I thought about forming an XQuery group. I ended up joining the Java group, though, because there aren’t any native HTTP libs in XQuery (so I’d have to do that as an extension in Java anyway). I still think doing an XQuery group would be fun though.

For instance, one nice feature of XQuery is that is allows you to be as strongly or loosely typed as you’d like. Take off all the “as …” statements from the XQuery and it still works just fine (it just won’t be so picky about what you pass into (or return from) its functions).

Recently, I’ve found myself on both sides of this fence; when working with a little bit of throw-away Java code, I’ve found myself wishing for a little of Ruby’s loose typing. On the other hand sometimes, when experimenting with Ruby, I mutter to myself: “Why can’t this just be strongly typed so I know what to expect and do?”

XQuery really gives you the best of both worlds. This isn’t to say XQuery can do everything those other languages can (it can’t… and far from it). But, if you are working with XML (and want to focus on the data rather than the data’s source) I can’t think of a nicer language to use. It will be interesting to watch XQuery grow as a programming language.

So anyway… since my marc2solr.xq is written as a module you’ll need to call it from something else. This little XQuery (also here) works fine from Saxon (pass in the location of a MARCXML file on the file system as $input):

xquery version "1.0";

import module
  namespace marc2solr = "http://lisforge.net/ns/marc2solr"
  at "marc2solr.xq";

declare variable $input external;

marc2solr:add-records(doc($input))

Interesting… I recently changed the URL of my worklog from “/worklog” to “/weblog” (I’ve set up redirects in my .htaccess file and, now, everything that was pointing to “/worklog” should go to its new home). What is interesting, though, is that my worklog’s new home used to be the home of my personal weblog (that has a new location now too obviously). Stay with me… this is where it gets interesting…

Search for The Bruised Edge (the name of my personal journal) in Google and “Kevin’s Worklog” now shows up as the number one hit. This is because, I assume, the worklog has replaced the location of the journal. There is nothing in the worklog, though, that mentions the journal (also keep in mind the Google results that are returned have been updated to show the worklog’s name (not TBE’s)).

What this means is that Google is finding the phrase “The Bruised Edge” in it cache and ranking my worklog as the number one hit because of data in it’s cache, not the current data that it has (and even though the worklog has nothing to do with the journal aside from replacing it). All this brings me back to the Google patent (titled appropriately enough: Information retrieval based on historical data).

It seems that this historical approach to data is how Google intends to defeat the abuse of its ranking system by spammers (who have latched onto weblogs and the linking aspect of Google’s algorithms to drive up their Google rankings). For more on this new approach on Google’s part, see the Buzzle story that brought the patent to my attention.

The Buzzle story was posted, earlier today, to one of the lists to which I subscribe… I can’t remember which one. It is all very interesting (the conditions used to determine relevance (or should we say take a stab at relevance)). Those Google folks are frighteningly smart.

It reminds me of when they spoke at Stanford. I didn’t see them, but Dick reported that one of the most interesting things from the talk was that they said: once you had tons of data it was amazing the types of things you could do. The same algorithms that wouldn’t return good results with smaller sets worked much better when the data set was massive. It seems once you get past a certain point, you get a new perspective.

I wonder if the same is true for library organizations (like OCLC) that just have reams and reams of data?

I think we, at Lane, have side-stepped most of the misconceptions for our own search, at this point at least, because it doesn’t aspire to be any more than an overview of what is provided by each of the individual engines. We pass the users on to the source rather than try to digest (munge together) all the search results into one interface. Hopefully, we are just trying to act as tool that lets the user choose where to go rather than actually tells them this source or this result is the best and what “you” are actually seeking. When it comes to searching, we shouldn’t be an aggregator but, rather, a facilitator.