One of the best ways I have found to think about the optimization process and to answer questions like, "Why does this site rank above mine?", are to to think like a search engineer. Imagine that your goal is to provide the most relevant results possible to the end users while punishing spammers and obvious optimization efforts.

My first experiment was to think of ways I would fight the SEOs. When I sat down to brainstorm, I came up with several ideas about how I might try to expand on the ability to detect artificial optimization efforts and punish them:

• Track SEOs using the toolbar and attempt to identify the sites they optimize by tagging their link searches, site: command, visits to SEO forums, etc. From there, one could discount any links that are from sites the SEO (now tagged by their IP address) has visited.
• Watch for sites that are linked to by SEO forum signatures.
• Track sites that are searched for using the "site:", "allinanchor", "link", etc. command on a frequent basis (of course, SEOs also use these on their competition).
• Watch for unnatural domain names, i.e. most-keyword-laden-website.com
• Look for sites that use over-optimization in on-page factors (keyword stuffing)
• Detect artificial or unnatural text by using advanced text analysis tools - readability indices, semantic analysis, stop word frequency, etc.
• Find sites who are primarily linked to with reciprocal links.
• Watch the domain registration information of TLDs to see if a known SEO/spammer has registered the domain.

Obviously, some of these ideas are probably being implemented (over-optimization, reciprocal links), while others are a pipe dream. What's valuable is to be able to sit down and think from your opponents perspective. Imagine every possibility and you'll have a good idea of how far search technology could be taken. Remember too - if you can think of it, they probably have thought of it.