It's never easy to admit to being dead wrong, but in this instance, it would be doing the SEO community a great disservice not to make my mistakes public.
In a thread at SEOChat - More on Term Weight..., I abuse members of the notion that keyword density can be used to calculate the importance of a term or phrase in a given document. I propose that, much as the IR community does, we instead use term weight to conduct this kind of analysis and help to better 'optimize' our keyword use in documents. That suggestion, as it turns out, won't work either.
My friend, Dr. Garcia, who many know as Orion from SEW, posted a few great follow-ups to the thread explaining why term weight calculation of a single document probably won't help you create a better optimized document for a global search engine. His reasoning is simple - search engines use term vectors, which are calculated across the millions of documents they index, to form ideas about context and term use. Therefore, the importance of a single keyword term or phrase in a document cannot be measured without access to this additional data. It's also impossible to measure a document without conducting linearization, which one can do through automated software.
I was worried that there would be no alternative method for "optimizing", but thankfully, Orion has some ideas as to how to achieve a scientific method of keyword use in a document. It's called on-topic analysis and he's posted about it in the past at SEW and has a good article on the subject at Miislita (his site).
On-topic analysis can't be done entirely by machine (unless you're a search engine). It requires human involvement to help categorize and sort topics from broad to narrow. It also uses Dr. Garcia's infamous c-index calculations to measure the connections between terms in a document to their use across the www. I'm in the process of ordering up a report from him today and will post what he allows me to share so we can all learn a little more about a real way to optimize keyword usage for SEO.
NOTE: To learn more about term vectors and the "whys" behind this approach, Dr. Garcia recommends a book called Information Retrieval: Algorithms and Heuristics, which he's reviewed here.
Everett - My advice would still be to make your most targeted keyword more used than any other on the page, but what Orion is noting is that search engines will look at more than just local term weight.
However, by making that term more frequent than any others, you make it obvious that it is the term you're targeting. The other key, for the moment, is to make sure that the text around it (the rest of the page) is well-written and complimentary. In other words, don't write an article about marshmallows, throw in 4 instances of the term "mortage loans" and expect it to rank well.
Thanks for the comments, Rand. I found the thread at SEO Chat difficult to follow. This blog entry helps a lot.
Thanks Randfish -- I am a BIG fan of Orion and read everything he writes -- The Algorithms and Heuristics is an excellent book with all the gory details, and I do mean GORY.
Virually all of Orions articles reference this book -- R. Baeza-Yates and B. Ribeiro-Neto; Modern Information Retrieval, which is the 'advanced' version of the Algorithms and Heuristics
Which, BTW, is also recommended by Matt Cutts in his recent interview with Aaron Wall
I'm cornfuzed...
The new paradigm of putting your KW on the page simply "more often" than any other word, regardless of what that means to KWD, really made sense to me.
In light of your mature admission here (don't you wish our US President could do the same?), does that paradigm shift mentioned above still hold true? Or should we revert back to KWD analyzers?
BTW - I write naturally first, and figure out the KW stats later.