Let me take you back, dear readers, to last Saturday night when, prior to attending a party for a friend's birthday, some friends of ours met up at our apartment (warning - a peek into Rand's life, upcoming). We drank some Rosé and were standing around in the kitchen when the topic of co-occurrence came up.

This shouldn't seem as unnatural as it initially sounds, because the friend I was speaking to (and her boyfriend) both work for a major tehcnology firm here in the Seattle area. She specifically works in natural language processing (NLP), and has some knowledge about SEOmoz and what I'm hoping to eventually achieve here with regards to topic extraction and on-topic calculations.

I explained to my friend the concept of co-occurrece, using examples from the room we were in - the kitchen. Naturally, these included Cinnamon Toast Crunch, Lucky Charms & Crispix (the cereals that currently sit atop my refrigerator).

I explained the following scenario:

Let's say I have two documents - one has the phrases "Cinnamon Toast Crunch" and "Lucky Charms", while the other has the phrase "Cinnamon Toast Crunch" and "Crispix". I want to discover which of these two documents is more relevant to the search term - "Cinnamon Toast Crunch", in other words, which is the Cinnamon Toast Crunchiest among my two documents. In order to do this, I could rely on a huge, searchable corpus of text, the www, as my guide.

By using Yahoo! (as they exhibit the most stable results of the major SEs), I could see how many documents contained the phrases "Cinnamon Toast Crunch" and "Lucky Charms" and the term "Crispix". I could then see how many documents contained both "Cinnamon Toast Crunch" and "Lucky Charms" as well as how many contained "Cinnamon Toast Crunch" and "Crispix". From this data, I could determine whether Yahoo! believes "Lucky Charms" or "Crispix" to be more associated (semantically connected) with "Cinnamon Toast Crunch".

This isn't my idea - I first learned about it from my friend Dr. Garcia. His papers in this area have certainly made the SEO world a more interesting place to be, and my kitchen conversations more in-depth. The definitive article on keyword co-occurrence, including the equations for its calculation are available from his site.

To wrap up, my NLP friend agreed that this could be a great method for determing the connectivity of terms, assuming that the data retrieved by the search engine was accurate. She did, however, note that in NLP, term vectors are the more commonly used technique for understanding and processing documents, but she'll get back to me this week on how the rest of her team feels about co-occurrence.

My big goals at SEOmoz for the future center around building two tools - one that can calculate how "on-topic" a particular document is, and another that can, given a document, determine it's primary subject matter and map that to a large, hierarchical ontology of concepts. Wish me luck! These projects are clearly eating into my social time :)