Tonight I'm back from Hong Kong and China, sitting at my computer in lovely, cool, wonderfully-clean-tap-water-spouting Seattle. There's a lot to come on Chinese search marketing from myself, Si and David Temple (who generously offered coverage of the conference for the blog), but, in the meantime, I know we need to get back to basics here at SEOmoz, so I thought a good, old-fashioned, illustrated post on enterprise search strategy would be just the ticket.
Enterprise sized sites range between 10,000 - 10 million pages in size. Good examples would be sites like AllBusiness.com, Answers.com and Findlaw (full disclosure - these 3 are SEOmoz clients) and others like Yahoo! Media Group, Expedia, New York Times, Amazon and Forbes.com. For most of these, one of two problems exist in the realm of link strategies:
- Inaccurate distribution of internal "link juice"
- Not enough inbound links (resulting in not enough "link juice" to go around)
Let's take a look at how we can tackle these tough issues.
Above is an illustration of the link juice distribution issue. Imagine that each of the tiny pages above represents from 50-1000 pages in an enterprise-sized site. Some areas like blogs, articles, tools, popular news stories, etc. might be receiving more than their fair share of inbound link love and internal link attention. Other areas, often business-centric and sales-centric content, tends to fall by the wayside. How do we fix it?
It's simple, at least in principle. We have the link-rich pages spread the wealth to their link bereft brethren. As easy as this looks, in execution, it can be incredibly complex. Inside an architecture of several hundred thousand or million pages, it can be nearly impossible to identify link-rich and link-poor pages, nevermind adding code that helps to distribute juice equitably.
The answer, sadly, is labor-intensive from a programming standpoint. Enterprise site owners need to develop systems to track inbound links and/or rankings and build bridges (or, to be more consistent with the above illustration, spouts) that funnel juice between the link rich and link poor. Typically, since interfacing directly with an analytics system like Omniture or Indextools is very difficult, a secondary layer of analytics might need to be installed to help control the tracking.
An alternative is simply to build a very flat site architecture that relies on relevance or semantic analysis (several enterprise-focused site search and architecture firms offer these). This strategy is more in line with the search engines' guidelines (though slightly less perfect) and certainly far less work intensive.
Interestingly, the rise of massive weight given to domain authority over the last 2-3 years appears to be an attempt by the search engines to overrule potentially poor internal link structures (as designing websites for PageRank flow really doesn't serve users particularly well) and reward sites who have massive authority, inbound links, and trust.
Let's move on to problem #2 - enterprise-sized sites that have a low ratio of links to content. These sites are hungry for link growth, but even the most massive linkbait campaigns often isn't enough to make a large dent. Enterprise sites don't need a few thousand links pointing at a few dozen pages, they need hundreds of thousands of links pointing at millions of pages. Achieving this is a true challenge, but I have a few suggestions.
- Don't use link brokers or link purchases
- Don't rely on sitewide link buys or "partnerships" from other large sites (technically, you'll be passing the PageRank, but sitewides don't carry their 2002 levels of value)
- Don't count on a manual link building campaign of any kind to bail you out
- DO think about your site design, usability, accessibility and site architecture - do they appeal to your users and, in this case more importantly, potential Linkerati?
- DO think about your content strategies - does it appeal to Linkerati?
Management in these cases generally need to make two large, sometimes painful decisions or risk becoming irrelevant in comparison to link-savvier and more link-magnetic competitors.
First off, it's critical to look at the design & layout of your content pages. Do you overuse pagination? advertising? banners? poor color choices? 1998-style design? Don't forget how important the presentation of material can be - the best dish you've ever had may be at a run-down street vendor in Taipei, but the new Frank Gehry designed restaurant in New York will be getting far more press, even if the cuisine is subpar.
The second tough task is thinking about content generation - how it's done, how it's controlled, how it's managed and how it's executed. The smart manager will provide guidelines, suggestions and templates to the creative team and then get out of the way. Good writers/producers will take these tools and queues and execute truly inspiring and link-worthy ideas.
At the end of the day, you've got to have some help from these guys:
Without them, you're up search engine ranking creek (way up in the dozens that is) without a paddle (link love).
BTW - I didn't have time to discuss specific enterprise link tactics, but if you have some to share, I'd love to hear them.
My answer to problem #1 is to write custom software to identify the link rich and the link poor pages. Dealing with large volume of pages is practically impossible without partial or full automation.
The server log files contain all the information necessary to do this. A few log parsing scripts can do the trick. The hardest part would be to classify the pages to identify which link rich pages should link to which link poor pages. This can be automated too with basic vector space similarity, co-citation, affinity, clustering or similar approaches. The easiest way to insert the links is to treat them as ads and inject the ad block before the closing body tag.
For problem #2 it is very difficult and unrealistic to expect a site with hundreds of thousands or millions of pages to start from scratch. It is also very hard to get massive natural links to commercial pages. I think it is better to leave content writing teams create related non-commercial content that includes links to the commercial content--maybe as ads. In my opinion, the best strategy is the first Rand mentions: to pass the link love from the link rich to the link poor. Addressing the first problem is the key.
Hamlet - excellent point about using a parsing of log files to grab link-rich/link-poor data.
We often leave the heavy lifting to the analytics packages but the server logs are a gold mine of information.
Funny I have one of my programmers working on this right now... pulling info from log files for organic traffic, and combining PR info...
Though once done we will have a procedure in place when any new content/pages are added to the site there is a weighing of what internal pages will link to it.
I'd combine that with spider data for these pages too, and throw in some weighting between PR and spiderfrequency...
Good ideas.... thanks for the tip
I just posted a detailed explanation of the logic behind such log parsing scripts. I will try to find some time to actually implement them. Separating link-rich from link-poor pages sounds like a great idea for a tool to offer for free.
Batista,
that was an excellent post but I found some terms are beyond my knowledge... such as the matrix and mapping the pages with IDs. Can you simplify a little more, that would be a great help.
Thanks, kichus. I am working on the code and followup article. I promise a more detailed explanation.
Done!
Very good post. I happened to come across it a few days ago before finding it here and had already bookmarked it to reread it. I'll be looking forward to the follow up post as well.
While linking to link poor pages as ads may be one option, there's a better one.
You can essentially find related pages to the link rich page and link
to them, in the 'read more here' kind of way.
Alternatively, showing related pages (3, 5, 7?) on link rich pages might help.
Perhaps combine this with page tagging, not unlike blog posts, where pages could be tagged by the content providers, possibly even a primary and secondary tagging methodology or even assigning weights of importance.
Now you can sort out which pages are high or low in traffic and linking as well as which other pages they should be related with. This tagging and weighting will allow flexibility as well as page content changes over time.
great post Rand. As always the illustrations are fab, I particularly loved the two triangles, "here you go buddy!" lol.
I do SEO for a big recruitment site and have exactly the problem described above, all the links seem to be going to the homepage (which is really a landing page, yikes!). It started off as a "normal" size recruitment size but is now a "beast" (global routing, 5 continents, 5 different recruitment areas..it's messy). The only way I see to solve the SEO AND usability issues is to totally re structure the site and start from scratch.
Lisa,
I've often found that restructuring is very necessary too, even on very large corporate sites. It's often not about improving a site's internal link structure, but about creating one...
Hmm... while dealing with a large sites restructuring is a headache if you are dealing it for the whole site. If you can segment the whole site to several subsections - like a career site can be segmented as IT careers, Engineering careers, and so on - and restructure each, probably you can create micro-sites for each which has unique internal navigation (unique left and sub-top navigation for example, with a common Site wide navigation tab at the top most) and a master navigation at the Top.
just my $0.20
I think the issue trickles down from the large corporate sites to small business sites though obviously not to the same extent. Most of my clients are small business owners with far less than the pages we're talking about here, but the first thing I always want to do is reowrk the architecture of the site.
Of course it's much easier to restucture a small to medium site, but the issue of poor structure is still there.
I think it's simply what happens when seo isn't a consideration during site development.
well its up to you, if you think you can restruct full website i can not see you can give a big favor to you than that, as i always have issue with contenet driven massive and messy websites, some time you even can not divide it into different sector.... best of luck
The presentation on this in Xiamen was excellent. Some notes from the presentation. Remember these are notes, they're not spell or grammar checked and are slanted towards my own sites.
Segment Searchers, create sections and information for each of them, IE: for an Insurance Site segment users into searches for
Quotes, Price, Vendors, Functionality, Location, General
By researching exactly what these searchers are looking for on sites we can create content that is useful for them and helps send them into a buying process. We should have content created for keywords these searchers are typing in.
Create checklists for in site audits – By creating short checklists of things to look for in content, incoming links, link purchasing etc. we can bench mark our progress and identify problems.
Identify important Landing pages and Benchmark those pages – Basically an important idea behind creating good landing pages. Looking at how those landing pages lead to conversions or don’t can help your site improve. It’s also important to notice whether these pages start to rank, or haven’t gained the ranking you were looking for.
Combine Information from SEM and SEO – By using information from SEM campaigns you can improve both campaigns.
Planning, What can we do now that will have quick lasting effects – A general idea instead of creating new large ideas, do the things that can have quick and lasting effects first.
Organize - Analyze - educate - execute - track your results – A strategy for businesses which have larger staffs.
One thing I'm interested in when it comes to this field is the trade-off between doing the best you can with what you have versus large-scale rewrites / rebuilds. Doing that is going to cost a fortune and determining ROI is going to be hard (almost impossible in advance!).
Nevertheless, is it one of those situations where the longer you leave it, the more painful it becomes and the less benefit you get from doing it? In other words, do some of these enterprise-type sites need to adapt (painfully) now or die?
Attacking problems tactically is probably a good start and I'm sure will generate positive ROI, but many of these sites have fundamental problems that are going to be addressed.
I would be interested to know where in this process SEOmoz find themselves being consulted (do you get asked "should we start over?", "we are starting over, who should do the build, and will you guys do the SEO consulting to make it work?", "we're starting over, these guys are building it and we'd like you to do SEO" or (worst) "we started over, these guys built it, and now it's just as bad from an SEO stand-point, heeeeeeeelp!").
Hi Will,
Many of my clients don't even realize that the expensive makeover they just purchased was worse than useless. They bring me in as an afterthought (optimized copy) and I have to tell them that unless they lose the frames, use some CSS, optimize their page structure and so on, they won't really benefit much from great copy. It's painful for both sides, but i have to do it or otherwise they'll be asking me further down the road, why they aren't getting stellar results from the copy. Sigh.
I have much to say on this topic as my role revolves around enterprise SEO but time constraints and other minor distractions mean I'll jot it down as one more thing-to-do-when-free (or inebriated).
For most of these, one of two problems exist in the realm of link strategies:
I would however, add that you should preface these problems with the obvious question;
"Which parts of the site do I want Google to find and rank?"
In a linking strategy, defining areas you do not want to link to is just as important as deciding which sections you would like to link (rank).
In some cases, enterprise sites may not want to dilute their link equity by linking from 'stronger' sections to 'poor quality' sections. For example:
Are just a few sections of a site you may not want to link to.
In days of yore, we once decided to tackle the 'not enough internal links' issue on one site by going crazy with internal linking. We created hundreds of new cross-linked category pages by pulling all the fields we had out of our product database. These new pages would in turn link back to our products.
End result; millions of new pages... in the supplemental index. On the rare occasion our new category pages were indexed (due to IBLs higher-up in the hierarchy), they sometimes outperformed our product pages and did not pass on their link juice very effectively. In other words, an absolute disaster which required a thorough 'de-indexing' of the crosslinked category pages and resulted in the creation of thematic link 'silos' (BC™ hee). So we learned the hard way that focused internal linking + domain authority + long tail = domination of low competition industries.
Of course, that's a story in itself...
Great comment, as usual, I'd say Shor :)
This is basically the trick you can pull in just about any non english industry, given your domain has enough authority... Internal link structure is the driving force between some of the sites I now manage, and of the things I've learned in managing them is to NOT be afraid to put a noindex, follow tag on pages you'd rather not see ranking :)
I have a love/hate relationship with noindex, follow. Good for things like RSS feeds, annoying when trying to do competitive analysis.
Also, am I the only one who feels a slight twinge of guilty when reciprocal thumbing (maybe recip thumbing should carry diminished weight!)
Hehe, guilt, what's that? :P
That's a very thought-provoking post, Rand. It illustrates the problem that big companies have as opposed to small and mid-sized companies.
I think a key element in strategy for any enterprise is to focus, focus, focus. Make sure you're the best in the world at doing something for a particular niche of customers. In other words have a USP that makes you very special for those customers. That makes the content much more powerful and provides many more hooks for the linkerati.
So many big companies forget that need for a USP. They assume that being big and spending money on being visible will do the trick. Perhaps that worked for Web 1.0 but it's not true for Web 2.0 and will be increasingly less true as the Internet further evolves.
Fantastic post Rand.
I think what your post highlights is that the larger your site becomes the less 'traditional' SEO you need to do (or even should do). e.g. you can't feasibly create a linking strategy for every single page you want to rank. Therefore you have to look at the site architecture and the technical build to ensure that it is link-friendly and also that the internal link-juice is passed intelligently.
In a nutshell, the SEO you do for a small site is less important for a large site. What IS still relevant is the knowledge gained and the underlying principles which you then need to apply to the technical build and the site architecture.
This is something I'm starting to see more and more with larger clients. I now demand (where possible) that I am involved in the whole web development life-cycle for any site which I will be optimising. Making sure that the architecture and build are correct makes your life a whole lot easier later on.
Great job Rand and welcome back! I look forward to more of this kind of thing.
Great post Rand and welcome back....
Clearly you have identified a number of issues with corporate/large sites....having worked in a large corporate environment, I often found a problem with departmental heads not understanding the web enough or SEO, as a result, often they make decisions that have no benefit for SEO or usability. Its only when they see such poor results, they start "throwing the toys out of the pram"....why is my area of the site receiving low traffic etc..... :-)....
I exeprienced this when working with a large uk based insurance company.
A great article....
Thanks
I'm at exactly this stage with a client; they're small, but they have a dynamic site with what amounts to 10's of thousands of pages. We've been successful in improving their SEO significantly, and they have an old and fairly respected site, but the ratio of links to pages is fairly low. I've been exploring a couple of options (and would love to hear opinions from folks in similar situations):
I'd love to hear other ideas. While this is an event site, the same issues (and potential solutions) would apply to a dynamically-driven product site.
Rand,
and what do you say about preventing low value pages from being indexed ?
You keep link juice in important categories / pages and you aren't wasting link equity ...
Welcome back Rand. I assume you'll be in Seattle at least for a few days While SMX is going on.
Do you think cleaning out some of the poor link juice pages from the index is a viable solution to any degree? I saw where John Scott recently pruned v7n and saw an overall improvement after having done so.
I'm not thinking you'd necessarily want to remove pages from the site, but could removing some from search engine indexes be a reasonable solution to increasing search traffic?
I would think some of the pages don't have links in because they might not be all that link worthy. Could the link juice possibly be put to better use than trying to prop up pages that may not be so deserving?
Amazing post Rand. Truly this is a major problem for many large sites i.e. portals who have archived content for many years.
Just curious, we have many pages which have login check before entering. Will that affect the indexing of the pages inside that directory. I was wondering whether Googlebot or any other search engine bot would be able to index those pages.
Any suggestions are welcome!
Once again great post Rand.
Hey Rand. I know this is an old post but I just found it for the first time :) One of the effective strategies you can use for sites this big, is to take better advantage of existing links. Typically enterprise sites already have hundreds, if not thousands of links, pointing to their site. Often times the link text is not optimal based on the targeted keywords for the page being linked to. Contacting those Webmasters and thanking them for the link and politely requesting a small change in the link text, can be very effective.
I have followed this strategy in the past only to find out the webmaster for whatever reason changed my link to one of my competitors? So test this with a few of your inbound link partners and see what happens!
I'm very fortunate at the moment in that I am never ever intending on working with or on a enterprise site.
I'm interesting in knowing how many SEOmoz users and/or members actually work with an enterprise sites. Also, How they manage to work around such a large site with such complex navigation.
Thanks Rand for the overview though.
a great post hand up to you man, i love this post and looking forward to implement some if not all of these jinks in my next website which is due very soon... i found a very useful articles where different seo expert discuss differnt stratigies to get back links and which is good and which is bad stratigies for baklinking. Views of SEO expert on link building
Thankfully I haven't gotten to that point yet. I'm just now convincing the client that CSS isn't evil, so I'm glad this article came out before I made a fool of myself with a medium sized business.
Interesting post Rand.
I appreciate that sites need to build back links in order to compete in the search engines but don't you think some sites are overdoing it and crossing the line into the realms of black hat spam?
Take moneyexpert.com for example, looks like they've paid for a few million off-topic back links to get themselves to the top for the search term "loans" on Google.
As your post suggests "dont use link brokers or purchase links" I wondered what you thought of this. Are moneyexpert.com benefiting from unethical methods, is it worth reporting them to Google?
MoneyExpert.com have finally been dropped by Google for the term credit cards. So, perhaps Google have stopped being so lazy!
I think for the larger companies it means a restructuring or rethinking of their online marketing strategies. Focus should be on obtaining deeplinking backlinks to ensure juice to the deeper pages and a solid site structure to ensure good internal themeing. Your post is excellent as usual and has made me think about my suggestions I have made to customers recently.
Great stuff. All my clients' have sites with less than 10,000 pages, but good to know how you would logically think through the issue.
I liked Hamlet's idea also.
C'mon Rand, give us some more link stuff! :)
Hi Rand,
Great post. Having worked on many enterprise level implementations, I think we can not ignore the impact of technology on these sites. Sites that are designed using J2EE or .Net technologies do not lend themselves to simple SEO techniques. Our top 10 enterprise clients have serious problems with their pages getting indexed with Google or Yahoo. The other problem is what you mentioned where a client that offers over 50,000 products will struggle with creating content that truly can attract inbound links.
Khalid
I think one of the most important actions for enterprise sites in this respect is to not only give their editors some more freedom, but also the public. I know, it's a classic web 2.0 paradigm, but I do think it works. You create content, facilitate the discussion and let people create content by commenting. This not only increases content (relevance), but also loyalty and incoming links when done right. (Just look at the discussions going on here, I've linked to SEOmoz countless times already and many others here as well probably)
Some of the large enterprise sites still seem pretty ignorant when it comes to web 2.0 concepts. Many of them are trying to leverage visitor-generated content, but it's usually done clumsily. In order to build an effective web 2.0 site I think a lot of managers/ departmental heads need to get off their high horse and listen to people that know about the interweb (like what Shahid says). It's difficult I know, but the Behemoths of the off-line world need to slim down and smart up to live on in the online world.
Talk about a high horse....;)
Very nice post Rand. My focus has always been on small to mid sized businesses so I don't have a ton of experience on very large enterprise type sites. Some very thought provoking content here. Thanks for the post!
Scott
Interesting topic Rand. I'd say site/link structure is critical for these sites. Assuming the structure is sound and there aren't any inherrent problems with onpage factors, how much link juice is passed along by sheer number of pages?
Also, why not add a script that checks PR for a page when requested and if the PR is low, pull a link from an authority page? It could be as simple as a "You may also be interested in [link here]" just above the footer. If the pulled link is somewhat random that could also show onpage change which would keep the Googlebot around more (if that's an issue and for low PR pages I would think it is).
If you'd do that on page request, I think you might be astonished on such large sites about how many pages never get requested at all ;)
Why would that be? It's just a dynamic function so it shouldn't slow load time down... especially if the programmer knows what they are doing.
That's not what I meant, I meant that if you do it on request, you would never find the pages and create links to the page that are NEVER requested...
Got it. Good point.
:D
You could still do it in a more permanent manner.
I'm just starting to wrap my head around this level of sites and found this very interesting... a bit of seeing the forest for the trees challenge.
Starting in design and quickly incorporating SEO at the design level and then beyond, I've always been a huge proponent in the idea that design, accessibility, usability and SEO are all connected. This illustrates what Wellwrittenwords touched on about how critically important, perhaps even more so, this is at the enterprise level where making improvements later are even more challenging if not impossible and sites covering this much ground need to be as efficient as possible to get every last drop of SEO goodness.
This reminds me of a saying I read a long time ago:
Seems that there is a point where a site becomes too large to "tackle." Rather, it needs to be broken down, at least in approach, into smaller pieces... perhaps incorporating the concept of micro-sites into larger sites where key sections are identified and built up as main pages to focus links, distributing that link love within their sphere of influence.
My company is wanting to get involved with larger sites, and this is fair warning of the challenges involved. Thanks for putting a finger clearly on what I had vaguely suspected: we'll need to invest the farm on a powerful creative team for content, and we'll have to get the architecture right from the start... or else!
Instead of creating a flat structure, why not create a pyramid one?
Identify main site themes and create category/theme pages.
Find intermediate-importance pages for these categories/subcategories and link to all the related pages from them.
Essentially, you'll be flowing the link weight to most important pages, anyway. Especially, if you provide 'read more links' to related pages on all link-rich pages of the site.
Getting links to less important pages can be done with link baiting and content promotion.
Great Post
You stated, "BTW - I didn't have time to discuss specific enterprise link tactics, but if you have some to share, I'd love to hear them."
Please add this to your things to address in the future! :-) As you can see everyone is very interested in this subject matter. I would love to hear your thoughts on the subject and tactics you think would work well.
With larger sites I have found the biggest problem are the basic SEO practices are not being meet. They are usually breaking all the rules such as dynamic URL with horribly URL structure. Duplicate content issues, duplicate meta-data and other basic SEO no no’s.
Many times you have to focus on the basics to get the site up to speed. In addition, to catching up on the basics, you have to deal with lots of different departments and managers depending on the size of the site it could be hundreds. Whereas a small to medium size site you only have to deal with an owner, project leader and a web developer. The idea is to be as organized as you can be with well thought out action items.
For me many times dealing with big sites it comes down to training, training and more training. By focusing and teaching ways to implement SEO best practices into the organization at all the right levels you now get other people involved and turn the project into a team event. I find making them aware of the competition gets them involved and how you’re ranking. Be sure to share your reporting, when they start seeing progress there is nothing more powerful. Next I would have to say it would be working with the developers/programmers to automate as much as the process as possible using templates and database optimization.
As for linking, I have noticed many of these larger sites tend to scary bloggers with big copy-write notices. This causes bloggers to be afraid to grab a paragraph and link off to the main article. They should promote this type of linking. I believe they should always be working in the direction of creating more dimension to their page using graphics, video or some form of interactive content. Engaging content tends to be more link worthy.
Anyhow this was a great post and Ienjoyed the graphics! Now I'm going to go to my blog and link to it! See how it works! :-)
Killer graphics.
Really killer.