The poor results for SEO got me thinking about the ways in which engineers would measure the quality and relevance of the search results for a query. I came up with a few quick ideas, but I'd love to hear feedback and ideas. Perhaps a small article/project can bubble up from this brew.

Psychology of Searcher

  • Informational vs. Transactional query (looking to research or buy)
  • Navigational vs. Search query (going to a specific place or browsing for the right place)
  • Broad scope vs. Narrow scope (researching a category or searching for an exact item)
  • Open-Ended vs. Specific-Minded (Is open to receiving multiple viewpoints or wants only ideas that fit into their biases)

Types of Results

  • Delivers an exact answer
  • Provides broad information on the topic
  • Links out to many other specific resources on the subject
  • Requires interaction to receive an answer
  • Amalgamates multiple sources
  • Provides opinion, fact or both

Actual Search Metrics

  • Query terms matching terms on the page
  • Measuring the "authority" via links
  • Measuring relevance via anchor text in links
  • Ascribing trust via link sources
  • Discarding results that attempt to manipulate the system

The incredible gap between how a human would answer a question (or provide results) vs. how a search engine does it seems insurmountable. Looking at the above, it's impressive that search engines function as well as they do!

A good challenge might be to ask experts in a field to deliver the top 5 results for a query (given 24 hours) vs. a search engine given the same query. Comparing those results against what the major 4 engines have to offer might be truly fascinating...