For the next few weeks, my blog posts will primarily consist of re-authoring and re-building the Beginner's Guide to Search Engine Optimization, section by section. You can read more about this project here.

Part III: Why Search Engine Marketing is Necessary

While many readers of this document may have overcome their skepticism about the need for and value of search marketing (and specifically, organic search engine optimization), it's entirely likely that others in your organization, company, network or client meetings may have differing opinions. Thus, this section is provided to help explain the need for proactive search engine optimization.

One of the most common issues I hear from folks on both the business and technology sides of a company goes something like this:

No smart engineer would ever build a search engine that requires websites to follow certain rules or principles in order to be ranked or indexed. Anyone with half a brain would want a system that can crawl through any architecture, parse any amount of complex or imperfect code and still find a way to return the best and most relevant results, not the ones that have been "optimized" by unlicensed search marketing experts.

Initially, this can seem like a tough obstacle to overcome, but the more you're able to explain details and examine the inner-workings of the engines, the less powerful this argument becomes.

Limitations of Search Engine Technology

The major search engines all operate on the same principles, as explained in Part I. Automated search bots crawl the web, following links and indexing content in massive databases. But, modern search technology is not all-powerful. There are technical limitations of all kinds that can cause immense problems in both inclusion and rankings. I've enumerated some of the most common of these below:

Spidering and Indexing Problems

• Search engines cannot fill out online forms, and thus any content contained behind them will remain hidden.
• Poor link structures can lead to search engines failing to reach all of the content contained on a website, or allow them to spider it, but leave it so minimally exposed that it's deemed "unimportant" by the engines' index.
• Web pages that use Flash, frames, Java applets, plug-in content, audio files & video have content that search engines cannot access.
  

Interpreting Non-Text Content

• Text that is not in HTML format in the parse-able code of a web page is inherently invisible to search engines.
• This can include text in Flash files, images, photos, video, audio & plug-in content.

The "Tree Falls in a Forest" Effect

This is perhaps the most important concept to grasp about the functionality of search engines & the importance of search marketers. Even when the technical details of search-engine friendly web development are correct, content can remain virtually invisible to search engines. This is due to the inherent nature of modern search technology, which rely on the aforementioned metrics of relevance and popularity to display results.

The "tree falls in a forest" adage postulates that if no one is around to hear the sound, it may not exist it all - and this translates perfectly to search engines and web content. The major engines have no inherent gauge of quality or notability and no potential way to discover and make visible fantastic pieces of writing, art or multimedia on the web. Only humans have this power - to discover, react, comment and (most important for search engines) link. Thus, it is only natural that great content cannot simply be created - it must be marketed. Search engines already do a great job of promoting high quality content on popular websites or on individual web pages that have become popular, but they cannot generate this popularity - this is a task that demands talented Internet marketers.

NOTE: All of the above issues will be covered in greater detail in later sections that deal more directly and technically with barriers to search engine visibility and rankings.

The Competitive Nature of Search Rankings

Take a look at any search results page and you'll find the answer to why search marketing, as a practice, has a long, healthy life ahead:

Yahoo!, Google & Live Search Results Side-by-Side

10 positions, ordered by rank, with click-through traffic based on their relative position & ability to attract searchers. The fact that so much traffic goes to so few listings for any given search means that there will always be a financial incentive & monetary reward for search engine rankings. No matter what variables may make up the algorithms of the future, websites and businesses will contend with one another for this traffic and the branding, marketing & sales goals it provides.

A Constantly Shifting Search Landscape

When search marketing began in the mid-1990's, manual submission, the meta keywords tag and keyword stuffing were all regular parts of the tactics necessary to rank well. In 2004, link bombing with anchor text, buying hordes of links from automated blog comment spam injectors and the construction of inter-linking farms of websites could all be leveraged for traffic. In 2007, social media marketing, paid link networks and vertical search inclusion are mainstream methods for conducting search engine optimization.

The future may be uncertain, but in the world of search, change is a constant. For this reason, along with all the many others listed above, search marketing will remain a steadfast need in the diet of those who wish to remain competitive on the web. Others have mounted an effective defense of search engine optimization in the past, but as I see it, there's no need for a defense other than simple logic - websites and pages compete for attention and placement in the search engines, and those with the best knowledge of and experience with these rankings will receive the benefits of increased traffic and visibility.

While much of the last week's news centered around the Toolbar PageRank affair (for a smart perspective, read Greg), there has been plenty of other exciting stuff from other sources:

• Obviously, a big story is that Microsoft is putting a load of money into Facebook, and Live search will almost certainly be a part of that site in the near future. Also - am I the only one who read about OpenSocial and yawned? (FYI - an oft-overlooked trait of mine is that I'm compelled to link to any post featuring a photo of the writer smoking a pipe)
  
• Could Salesforce Ideas actually be a useful Digg clone? I think it might well be - that style's definitely worked well over at YCombinator News (and sends us healthy traffic whenever we get a listing). 
• John Mu notes that Googlers in the Webmaster Help forums were quite inactive over September. I blame Susan. Hopefully, we'll see more activity there in the future - it could be a great resource for site owners (and take a lot of weight off Matt's burdened shoulders) if they could staff it (not that I can't empathize - premium Q+A has been eating a ton of time here at the mozplex, too - over 500 threads in knowledge base now). BTW - Mystery Guest and I had a great time in Stockholm meeting John (and a couple other Googlers) - more on that later this week.
  
• Neil had some brilliant linkbait on the difference between marketing, PR, advertising and personal branding. And Mystery Guest got a present from him last week that made her heart happy - thanks, Neil :)
• If Rebecca were writing this, she'd tell us all to read Daniel Scocco's article on Copyblogger - "But shes''' not".
• Want to get insider-y with SEOmoz? Check out Will Critchlow's interview with our own Scott Willoughby.
• Congratulations to Thomas Bindl & Refined Labs, who've taken on some funding (from these guys) to build what sounds like very, very cool software for web marketers.
• Cranium has a character with a familiar occupation :)

As always, feel free to add your news and links.