Are those pesky search engine robots crawling your site too much? Does the sight of even one result from a "site:" command at the engines boil your blood? Are you tired of seeing visitor after visitor trickle into your website from referrers like "Google", "Yahoo" and "MSN". Well worry no more, because below, I've listed the top 10 ways to keep those spiders far, far away.
#10 - IP Delivery
Not just for spammers anymore - there are now dozens of legitimate uses for delivering content based on the IP address of the visitor. One of the best things you can do for spiders is to put California (and Washington) on your list of non-served states. Maybe it's because you don't ship your wacky weeds there, or maybe you've got issues with far left politics. Either way, keeping IPs from these geographies pointed to a junk page will ensure that pesky crawlers turn tail and run.
#9 - Duplicate Content
If you and ten dozen of your best friends all scrape from already existing content and run it through your whiz-bang text re-organizer, you're sure to get crawled, right? Wrong! Thanks to advances in duplicate content detection, you can use this as a great way to go on the ban list.
#8 - Reject Inbound Links, Just Submit!
Why bother getting links to your site when you can just submit to the engines on a daily basis. This tactic is recommend by many of the top ranking sites for "SEO" and "SEM" so why not take their advice?
#7 - Buy Your First Thousand Links
When your first thousand links are part of a link manipulation scheme, you can really boost your chances of being ignored by Google. Yahoo! and MSN though may still give you the crawl (and even the rankings), so make sure to spam report yourself (or wait a few weeks, savvy competitors can help save you the trouble).
#6 - Use 100% Pure, Grade A Flash
Everyone's favorite animations can help you solve your spider infestation, too. Try an all-Flash portal and watch your crawling activity shrivel up and disintegrate.
#5 - Text is for Losers
Images, rich media, video, and those beautiful midi files are what make the web great, so don't mess around with text in HTML.
#4 - Update Once a Decade
Search engines love fresh content, so starve them of it. If you update your website before 2010, you're probably on the wrong path.
#3 - Splash/Intro Pages
Everyone knows that splash pages are fun and user-friendly, but did you know they can help keep out spiders, too? It's true - just try forcing a spider to click a link in a Flash file or an image map and watch the magical results.
#2 - No Content Without User Action
Love AJAX? Javascript? Then this one's for you. Until a user clicks, hovers or draws a unicorn with their cursor, don't show them anything (and make sure to keep the code in a separate file to be called by your scripts).
#1 - The Beauty of NoIndex
Just one handy tag in your robots.txt or meta robots tag and all your problems are solved. If you're not wanting to do it yourself, just leave your server relatively unsecure and someone else can do it for you.
I hope it's not too early in the morning for a strong dose of irony :)
Wait, irony? I was writing these down...crap.
O HAY here's another tip: Make sure to put the pancake astronaut on your front page!
Kat,
LOL!!! That was random.
Irony? You must be American. There's nothing ironic about it. The word you were looking for is "sarcastic". Why is that American's can't tell the difference between the two...
The sarcasm is there, too, yes. But, the irony is that you're reading a blog about how to rank better in the search engines and are, instead, getting a post on how to decimate your rankings - hence, irony. :)
Having just been back from a few weeks abroad, I can empathise with your views on Americans, but we're not all morons.
It seems dictionary.com agrees with your use of the word too - so perhaps Americans simply use the word a very different way to us Brits. So it's not a lack of understanding as such, but simply culteral differences. I'll withdraw my statement :)
Pulled this from the Oxford, which I believe is British: noun (pl. ironies) 1 the expression of meaning through the use of language which normally signifies the opposite, typically for humorous effect. 2 a state of affairs that appears perversely contrary to what one expects.How do the Brits use it? I'd hate to stumble on my next trip 'cross the pond.
Hey Rand,
Good little list here. Any recommendations on how to work around a static intro page? The link through to the site is an image of the company.
Should we just recommend that they remove the intro page? It appears the PageRank of the site is being distributed to the intro page and not the homepage.
Thanks,
Rise
Hilarious, one line into number 10 and I was laughing in hard enough for my boss to stroll over.
I'm diggin' it, just for the newbies.
Hmmm, Rand you should have titled this blog "arachnophobia"!
My sister is scared of spiders -- especially those big, black, and furry ones.
LOL... nice one! I think that they are in the wrong order though...