Hi Mozzers, I've recently had to deal with several indexing problems that a few clients were experiencing. After digging deeper into the problems, I figured I'd write a post for Moz to share my experience so others don't have to spend as much time digging for answers to indexation problems. All it means is that your site, or parts of it, are not getting added to the Google index, which means that nobody will ever find your content in the search results.
Identifying Crawling Problems
Start your investigation by simply typing site:yoursite.com into the Google search bar. Does the number of results returned correspond with the amount of pages your site has, give or take? If there's a a large gap in the number of results versus the actual number of pages, there might be trouble in paradise. (Note: the number given by Google is a ballpark figure, not an exact amount). You can use the SEO Quake plugin to extract a list of URLs that Google has indexed. (Kieran Daly made a short how-to list in the Q&A section on this).
The very first thing you should have a look at is your Google Search Console dashboard. Forget about all the other tools available for a second. If Google sees issues with your site, then those are the ones you'll want to address first. If there are issues, the dashboard will show you the error messages. See below for an example. I don't have any issues with my sites at the moment, so I had to find someone else's example screenshot. Thanks in advance, Neil :)
The 404 HTTP status code is most likely the one you'll see the most. It means that whatever page the link is pointing to, cannot be found. Anything other than a status code of 200 (and a 301 perhaps) usually means there's something wrong, and your site might not be working as intended for your visitors. A few great tools to check your server headers are URIvalet.com, the Screaming Frog SEO Spider, and Moz Pro's Site Crawl (take a free trial for the full experience).
Fixing Crawling Errors
Typically these kinds of issues are caused by one or more of the following reasons:
- Robots.txt - This text file which sits in the root of your website's folder communicates a certain number of guidelines to search engine crawlers. For instance, if your robots.txt file has this line in it; User-agent: * Disallow: / it's basically telling every crawler on the web to take a hike and not index ANY of your site's content.
- .htaccess - This is an invisible file which also resides in your WWW or public_html folder. You can toggle visibility in most modern text editors and FTP clients. A badly configured htaccess can do nasty stuff like infinite loops, which will never let your site load.
- Meta tags - Make sure that the page(s) that's not getting indexed doesn't have these meta tags in the source code: <META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">
- Sitemaps - Your sitemap isn't updating for some reason, and you keep feeding the old/broken one in Webmaster Tools. Always check, after you have addressed the issues that were pointed out to you in the webmaster tools dashboard, that you've run a fresh sitemap and re-submit that.
- URL parameters - Within the Webmaster Tools there's a section where you can set URL parameters which tells Google what dynamic links you do not want to get indexed. However, this comes with a warning from Google: "Incorrectly configuring parameters can result in pages from your site being dropped from our index, so we don't recommend you use this tool unless necessary."
- You don't have enough Pagerank - Matt Cutts revealed in an interview with Eric Enge that the number of pages Google crawls is roughly proportional to your Pagerank.
- Connectivity or DNS issues - It might happen that for whatever reason Google's spiders cannot reach your server when they try and crawl. Perhaps your host is doing maintenance on their network, or you've just moved your site to a new home, in which case the DNS delegation can stuff up the crawlers access.
- Inherited issues - You might have registered a domain which had a life before you. I've had a client who got a new domain (or so they thought) and did everything by the book. Wrote good content, nailed the on-page stuff, had a few nice incoming links, but Google refused to index them, even though it accepted their sitemap. After some investigating, it turned out that the domain was used several years before that, and part of a big linkspam farm. We had to file a reconsideration request with Google.
Some other obvious reasons that your site or pages might not get indexed is because they consist of scraped content, are involved with shady link farm tactics, or simply add 0 value to the web in Google's opinion (think thin affiliate landing pages for example).
Does anyone have anything to add to this post? I think I've covered most of the indexation problems, but there's always someone smarter in the room. (Especially here on Moz!)
You can add to all the above:
If anyone can add more, please do so!
Very vaild points but I would highly agree with the site’s page time load. If you have a website that takes much time to load! You will face the issues with indexing, specially the deeper pages (in case of e commerce product pages!)
Agreed - I have personally experienced the effects of a template update to an ecommerce site that has caused thousands of pages to be de-indexed or go into the supplementary index of Google. Within a very short time of resolving the issue the pages indexed shot straight back up!
Thats why we always recommend an SEO audit prior to lauch of new design launches. A tool like IIS seo toolkit quickly reveals such indexation problems. yesterday revealed a relauched website where anything but normal useragents where sent in infinite 302 redirect loops.
Actually improper redirects based on useragent or ipadress, geolocation shoul be on the list.
Oh, oh! I just remembered another one:
- Very Bad publicity in the press (online and offline) and bad reputation at all - literally hundreds of angry users, gazillion complaints, low ratings almost everywhere...
Thank you for the list, I will use them to check my site.
Thumbs up for putting Matt Cutts head on stuff, one of my favorite pasttimes :)
lol i just add it on Pinterest! I call the Google's Spider Man! :p
The picture is really good. :))
That' what I was going to mention in my comment, loved the Matt Cutts picture, he deserve this... And infact it would be much better if you had placed a bald head of matt cutts there. :)
The two biggest reasons I run into are flash, and content being loaded via Javascript that is uncrawlable. Both will completely isolate entire sections of your site if that flash/js is the only place they're linked from.
Easy solution (without redesigning the site) is to just link to those pages in straight HTML from elsewhere on the site. This wouldn't be the best solution if the JS or Flash is a sitewide issue.
I've had the same problem a bunch of times with different clients.
JS and Flash shouldn't be used in linking to other pages at all, they have their place in the front end development, like animations, aplication etc., but not linking to other pages.
You are an idiot. js isn't used to link to different pages. it's used to supply dynamic content on the same page. Aka Ajax. js can also supply dynamic content by hiding and un-hiding html in a single file making it look as if it's a different page when it's using the same fail to load all of the content. My site works in this way.
https://www.poplinkd.com
Disagreement is fine, but there's no need for name-calling. JavaScript can be used to trigger links - for example, some sites embed onclick events in anchor (<a>) tags. In some cases, especially when that code is poorly written, it can cause crawl issues.
Wrong! Simply using HTML to link to other pages on your website that are loaded with flash or javascript will not help at all when it comes to indexing. One thing you guys need to realize is that search engines, including Google, don't just index links, they index HTML and the content that is displayed within that HTML specifically. If your pages contents are loading from within tons of javascripts then forget your content being indexed at all. Sure you could link to the other pages, but even then the web crawlers will see nothing but javascript and completely ignore most of what it cannot read. Another major issue I've come across is the fact that people are not tying their content to their title, headers, and meta tags. Your meta keywords, descriptions, page title, and headers should be descriptive of the actual content of your web pages, if not, then don't expect much of a page rank. Avoid buttons for navigation altogether, text links are always the best! Even if the buttons are not flash based, they still cause problems because the only thing the web crawlers can see is your alt tag, and this is the only way for them to reference a button at all. Web crawlers cannot see or read the text on a button, period! Also, as I said, search engines do infact index your content, this means the actual text, images, etc.Google more often than not will provide an exact match to someones search queries. For example, if I type in Google search "Free Flash Arcade Games", Google will respond with an exact match if one is available. Stick with HTML, properly reference your content with your headers, and use proper headings (h1, h2, etc). As well, if your code is bad, then your page rank will be BAD! Period! Google takes into great consideration the quality of your content and code. If your web pages are full of bad grammar and poor HTML markup, then you will be placed at the bottom of the list, unless your pages content fits in exact reference to the search query. One more thing, NO broken links! This kills page rank too. All web based content should be placed inside of HTML, not javascript. You can place javascript, flash, etc, in HTML but not the other way around. Otherwise, forget you ever had a website to begin with. I don't know everything, I'm 24 years old, and an avid web designer and computer specialist. Believe it or not, I have over 30,000 hours of experience in the field. Hard work builds character. And while no one is perfect, and none of us know it all, it annoys me that so many in the IT industry know so little. You only get back what you put in.
Yeah, and have you ever launched a site, it doesn't rank, and then you realize that you forgot to make sure that the WP privacy wasn't turned on?
...Not me. I just heard about it.
Thanks..needed a laugh! :)
Yeah, I've heard about that one too come to think of it :)
Yeah, some guy did that once. Crazy.
I actually LOL'd at this! Very funny. Reminds me of the Veggie Tales skit about picking your nose and your bike having a banana seat. Took me back man, TOOK ME BACK! *sniff!*
I have a brand new site that is not indexed after 7 days now. I have never waited longer than 1 or 2 days tops to get indexed.
The domain has a clean history, and I'm not doing anything wacky with it. I have submitted the sitemap to WMT and that looks good.
There are no errors, because there is no crawl yet. I even asked Google to index me in WMT and nothing is happening.
I've tried a few techniques to get Google to notice me that have worked 100% of the time really fast until now. Nothing.
Not only am I indexed in both Yahoo and Bing, but I am ranking too. I know...who cares, but it just doesn't make any sense.
For those who may say "add quality content'...yea yea, It's a normal site with "quality" content. I've submitted some pretty lame sites before and still got indexed in a matter of hours, so I don't think that is the issue.
I've read a few times now about people saying "build links, Google doesn't index sites with no links" but that doesn't make any sense.
1) I've never had this problem before on a site with no links.
2) If you have to "build" links to get indexed, isn't that defeating what Google supposedly wants? I thought they didn't want people to build their own links.
They want natural links, right? But if my site is not indexed, it will not show up anywhere on any search for any keyword. So nobody will SEE my quality content and link to me.
SOMETHING IS FISHY.
I really don't want to wait anymore for this site to index, as ironically it is a site that I am doing a 100% white-hat test on, and so far....the test sucks.
I am here and read this post for exactly this kind of thing. I have a newer site in wordpress that is now indexed but some of the problem sites pages are not indexing well. The root is indexed and I checked all the things clues etc.
I did notice that google had indexed page one of items in a list but the paginated content on deeper pages was not indexed ! So I removed the pagination. So that was something I noticed.
I wonder if wordpress has a funcky thing going on as it seems to index much better than my problem joomla site. I know under the hood joomla vs wordpress is nonsense because its html regarless of what makes it.
Good points Bob.
Here are some more points that need to be focused:
Apart from de-indexation. For the keywords going down:
Do you have real evidence, that there is a relation between:
- traffic on pages
- social shares
- ranking
? It sound logical. But does Google really care of this?
With sunny regards from Germany,
Sebastian
From my expeireince the most common problems for blocked domains in Google are:
- Google Quality Penalty in the past for the domain: domain blocked.
- Excessive Black Hat on the domain by prior party: domain blocked.
- Domain part of a link networking/ low quality: domain blocked.
- Flash based content issues preventing the access of crawlers to the domain.
- CMS based settings placing a block on Google.
My favourite trick is, when using Wordpress, to disallow search engines when I first set the site up and start populating content. Once I'm finished I generally forget that I haven't gone to the privacy screen to re-enable access and then (2 weeks later) start to wonder why no pages are showing up yet. Such fun. :P
Hey Bob - Awesome article I found when researching resources for resolving Google Indexing Issues. I noticed this article is a little aged now although still very relevant I wanted to share another recent post I wrote that might be helpful as well to those finding this article:
https://raleighleslie.com/how-to-troubleshoot-google-indexing-issues/
Really Great Information. A site of my client was not indexing and caching then I read your post and it had the problem of meta robots. I fixed that and After it get indexed and cached. Thanks for sharing this stuff
Hi Dear, Why don't index my home page in Google. before some day again change My home page content and Again submit XML sitemap in webmaster tool. Now why don't index my home page. please say to me. How to I can My home page index in Google. My Web site name is optimationbd.com
This would be a good question for Moz's Q&A section at https://moz.com/community/q
So I go into Google Webmaster tools and the sitemap show up, but it says -- in the column under "pages indexed". So I google site:mywebsite.com but the search results show all of the pages that are in the sitemap. So my question is -- are my pages being indexed or not?
If your pages are appearing in search engine then its mean your site is indexed
That picture of Matt Cutts in spiderman is awesome!
I agree...
Very nice write up about what to do with crawl errors. Certain sites do get crawl from the same google bots nonstop and it takes a heavy toll on resources. What do you suggest in this situation?
Had an interesting one recently in which the client had managed to upload a sitemap for one of their other sites which saw all the rankings starting to fall away and pages getting dropped out of Google, we simply uploaded the right sitemap, submitted a reconsideration request and they were back in action in a few weeks.
Yes. That sounds familiar to me. We have several wordpress blogs running in one installation. And most plugins for sitemap generation do not handle such multi-domain-installations. The result is -> one sitemap (containing the entries of one domain) for all domains.
Very unpretty ;-)
With sunny regards from Germany,
Sebastian
Hi guys I work for an audio company, and they have the hardest time getting indexed. Most of the time if your searching for a product.. Google will show a sites from all over the world, and at times site not even related to the search before they show us. the site is www.muzeekworld.com we are working on getting product pages some original content currently.. Is there anything wrong with the code that you guys see? :) Any help it greatly appreciated
Also, if a page is orphan, then it's one of the few reasons why a page isn't being crawled or indexed! What you guys think?
hi Please have a look www.bollywoodshowbiz.com everything is right with in my websites. But last couple of days my new pages are not indexing. I have check all the thing. Why? any other ideas. or is my content quality is low ?
Friend - Can you please help me with my new website www.hotels.evmastu.com. I have created this about 2 weeks back and added 109 Pages after hell lot of efforts but google is just showing 7. I am really disappointed with this. It would be great if you can have a look into my website and advice me few steps i should take for getting indexed well.
Hi! Our Q&A forum is a great place to ask questions that are specific to your site.
Well the classic: rename one folder in your webdir.
I.e. rename your blog suburl to /blog2 instead of /blog , now you got 1000 defective blog pages
rename your shop to /shopnew instead of /shop , now you got 5000 defective shop pages
rename your .htaccess rewrites and base them on ip/referer in a bad way, now whole website is inaccessible.
Don't use htaccess like this, never make referer rules it is lame and stupid.
People will be able to get to your site if you make it or not!
Great post, now i need to short out my blog problem why its not indecing in google
Why does these URLs are not getting indexed or crawled by Google?
https://www.thomascook.in/tcportal/travel-insurance/single-trip-worldwide
https://www.thomascook.in/tcportal/travel-insurance/single-trip-asia
https://www.thomascook.in/tcportal/travel-insurance/student-international-travel-insurance
Can someone please help me determine the possible reason for this?
I was just looking out for this issues for my new site and i found the reasons why some of my web pages of Akash Roofing Sheets is not indexing on Google. Thanks a lot for posting this useful article and found some useful links which was very much helpful to me.
My site isn't being indexed on Google, even when doing a site:http//etc. search. The direct link to the website works fine, as do all the pages, however WBT is telling me that each page has an Error 500. Tried calling host but no luck, they said "try calling Google"
Hi Bob! Very helpful list you got here. I will surely use them to check my site. I will also share this post to my friends for them to check their site. Thank you for sharing this :)
Thanks Bob but im not OK with the page rank thing , in 2014 Google can not control the fake page rank . by the fake page rank i dont mean the redirect one or giving a pagerank with some linkin and 302 tricks , by fake page rank i mean the whole new algorithms like "Huge Tags Algorithms" by siavash . this new methods destroyed the Pagerank things for webmasters . i have a website with 50.000 new visitors from google.com everyday , and my pr is 3 . i also have a pr5 website that is real but with reaaaal bad indexes and low google visitors . if you focus on some real important things you will get what you want from your content . any way , thanks for your post. :)
I have seen most of time people do mistakes like meta noindex tag & block from robots.txt file.
I'm troubled, my website last index date was 20th Oct, 2014. I have checked all the given points and all are used in the right way. Before one month back website crawling and indexing on regularly after 6-7 days.
The major changes I made integration "WordPress Yoast" on the place replacement of "All in One SEO" plugin last month.
Where I checked all the above setting and work update according.
Would you please analysis my website and help me to get what is the problem here? I submit request from "Google webmaster" tool to index my site 2 times, but nothing found.
Now I need expert help. Please do that. Thanks!
website is: www.orionesolutions.com, please help me out from this issue.
Check GWT to see if there's an issue. Try re-crawling manually through GWT - I had a quick look at the site and can't see an obvious issue that would cause Google to stop indexing.
Maybe also get rid of this META tag: <meta name="revisit-after" content="5 days"/> - I don't think Google uses this but it might cause a problem.
Looking at your https://www.orionesolutions.com/post-sitemap.xml - the last post about RHEL6 isn't in there. That's also the only one I can't find directly in Google - the rest seems to have been indexed just fine.
Thanks Rob.. Now site is crawl-able by itself. and where I checked there are around 40 site that was nor crawling from the same 20th and 21st Oct.
Rob, There is one another problem that I am going to face, XML sitemap is activate from back-end, but on front side error of 404 not found page coming:( . website is same: orionesolutions.com.
Would you help me out from this please.
I have site for my Fort Worth church ( Texas ) https://www.michaeloministries.org and I used to ranked for keyword "non denominational church" then I changed the title and h1 and description to Fort Worth Church, google has crawled my site several times since but I don't rank at all. My site doesn't appear in all 35 pages.
Any ideas what is going on?
This is a great question to ask the community in our Q&A forum! - https://moz.com/community/q
I think I am a victim of inherited issues. I registered my site just september 2016 but archive.org says my site was registered in 2012
my website is not show in google plz help me webmaster some error
severe health issues are found in your property
important page removed
Hi
Thanks for this wonderful article. I have a website mcdeals.in and it is indexed, but I see that google indexes the site less often. There is new content added to the site every few hours but google take 2-3 days to index the pages again, and this makes the content obsolete. While I see other sites with same or similar content are being scanned by google every few hours.
Thanks in advance for any help or suggestion. FYI: This is a six months old site and I still hardly have much traffic on the site (if this makes any difference)
Thanks
Sandy
I have Ecommarce website. My webmaster tools sitemap submission report says I have 102,628 urls submitted and 31,998 indexed and its dcreasing. Under the index status tab in webmaster tools it says I have 31,998 pages indexed. Doing a site: search with Google it says I have 34,100 results and this results also decreasing. I am checking those on regular basis. site: search and number of indexing pages in webmaster decreasing continuously. It is decreased from 56,392 to 34,100. Please share your suggestions
[Link removed by editor.]
my website was working fine in webmastertools until I added a certificate and redirect all my traffic in .htaccess to https://
I added all the https variants to webmaster tools, i created a new https sitemap and uploaded it (no errors)
just updated robots.txt to: https:// from https:// to see if that makes any diffference
I have created quite a few new pages as I SEO'd my site but they arent getting indexed.
It says pending - then goes to processed and URLS submitted = 29 but index says -
Hello,
I'm Vishal. I am facing a indexing problems with my website www.indusdiva.com. I checked the Webmaster tool, it gives 404 error for the sitemap. To my guess it is because we are re-directing the same website to indusdiva.in if accessed in India. Is this creating the indexing problem or something else? Also how often I should update my XML sitemap for an e-commerce website?
Ya its the great help I think, if we people go through proper process then success is must. Thanks
Thank you for such a wonderful post but
I have a serious problem with my website indexing. Actually there are TWO.
1> My home page never gets indexed instead other page gets indexed.
2> Whenever i search my website in Google search as flywoosh.com, I get meta tags of my website BUT the link is of other some other. WHAT SHOULD I DO(Facing serious business losses due to this ) :(
Hi BOB,
I have added my blog to google webmaster as "www" and "non www"
"non-www" account showing google have indexed 350 out of 375 but
"www" showing that sitemap submitted and pending.
what should in do now
my blog address BSmartDeaL
Hi bsmartdeal! Just a heads-up—you may get more of a response if you post in our Q&A forum. And if you search for this topic, you actually may find that your question has already been answered there. ;)
Hi,
I am posting here out of total desperation to find an answer as to why Google refuses to index a group of web pages on a web site.
I have been developing web sites for more than 10 years, I understand such as meta data, site-maps, good HTML structure, Google's web-master guidelines etc & have many existing websites with 100% page index count.
The company I work for runs several on-line stores which I maintain and have relatively few issues from a Google perspective & from a small targeted UK market get an average of 500 visits per day.
My boss asked me if we could add some textual content to cover many of the basics that technical sales staff are asked repeatedly by customers. I suggested that we make this web site responsive, light weight, high text/code ratio and that the content is relevant, original and of good quality. Basically a small wiki about the various technologies that our products 'solenoid valves' employ along with common reference technical data. No problems were envisaged & I was given the go ahead.
Although the site is still a work in progress we had sufficient pages and content to register the site with Google via my webmaster tools account, submited a sitemap and monitored the sites indexing.
Within about 3 weeks Google had index 22 of 24 pages and all was looking OK. Then checking webmaster tools I see that the pages indexed dropped to 1. I left things alone for another two weeks, but the pages indexed remained at 1, so I resubmitted the sitemap and Google started indexing pages again. Index count rose to 22 pages again, but then the day after dropped back to 1.
The site can be seen at : https://www.solenoid-valves.com
Here are the key points about the web page content of this site:
Light weight: page load size of less than 80 Kb.( home page ~ 32Kb)
High text/code ratio: 70% - 80%
Custom javascript library created for the site.
Responsive design.
Vertical page navigation.
Pretty URL's
Very few if any outbound links & the same inbound links the site has had for several years.' genuine good quality'
The store that had been on the domain was moved to solenoid-valves.com/store and the e-commerce pages have since been indexed as expected, yet the new web page content seems to be poisonous to Google.
To make sure that moving the store to tld/store/ was not the issue I decided to create a separate web site on a new domain using the same framework I had used for the web site component of solenoid-valves.com. This can be seen at https://coax-valve.com. I went through the normal process of submiting this new site along with sitemap etc. Indexing occurred as normal of all 6 pages submitted within 3 weeks. Then the page indexed count dropped to 1, exactly as it had done with the other domain. I resubmitted the sitemap, index count rose to 6 then 3 days after went to 1.
Out of desperation I decided to just leave things alone for a month. The situation is unchanged and after having released over 200+ successful web sites over the past 10 years I am left having no idea why Google is treating these 2 new sites as it has.
This whole saga has been ongoing now for more than 8 months and I have no idea why. I have made many posts in SEO type forums / blog's, but no one has been able to give me an answer as to why Google is behaving the way it is with these new sites.
Having become so involved with this issue, spending so much time researching and testing to find clues I am a little concerned that the issue may be a simple one that I have overlooked while attempting to resolve this issue. I have been a software engineer since 1972 and this is the first time I have not been able to identify a code problem.
I throw open the challenge to the Moz community, for now I have just left things as they are.
Regards.
Jonathan Parker
UPDATE: At present the meta description is automatically generated from the first 200 words of page content. Have changed this on one page to use unique meta description content. I'll update this post if anything changes.
Hi Jonathan! Just FYI, you might be able to get more responses if you post this to our Q&A Forum. :)
I am having some issues with my site at https://www.velynews.com it's a news aggregator website, but i find it difficult to get it indexed on Google. I want to know what's likely to be wrong. Could it be the nature of the website?
I have also a problem, It's big for me.
My website's some urls are even cached or indexed properly, and after i saw 3-5 days it's not indexed or cached by google, Why? I could not understand.
I have checked robots.txt, Not problems here.
also checked pages not duplicate.
Pls suggest me....
As, you mentioned that your pages get de-indexed by Google, it means, Google found something wrong either with your page's content or with other things. If you will chek your Google webmaster account, then you must see few crawling errors there. So that you can identify what's wrong with your page content.
I hope it help you.
8."Inherited issues"! well, how do we check if a domain had a bad rep (spam or such) before we buy it!
archive.org is a great place to start
I have some problems in my site TechAFee
Google does'nt indexing my new pages.
Hi there! This is the sort of question that is much more likely to get a response in our Q&A forum. I recommend asking there. :)
After reading whole post, I am sure that I have implemented all things which are compulsory for indexing of any site but still my site is out of Google index. Even, I have fetch my site in Google index via Google Console but still, site is out of index. However my site is indexed at other search engines. So I am not sure why this happening with me. I want to share my site site URL with moz SEO expert. Here is the site url https://www.massalaexpress.com/. I shall remain thankful for better solution.
Hi! I highly suggest heading to our Q&A forum in order to get your specific questions about your website answered by our community of experts. :)
Sometimes the easiest answer is the right one. These are some of the first things you should check if your site isn't indexing properly or has been deindexed.
I have 10 URLs in my sitemap, Google only indexes one page...please help me
Index, No Follow link have to be used carefully by SEOs to target the right traffic to right landing pages.
Great post.
Good stuff - just dealt with the oppostie problem, overindexation.
Hi I have a website hindi-comedy.com , have over 200 links and site:query also shows 165 links but actually 4 links are indexed other never come in search result , I have tried going over my pages again n again but never understood the issue why arnt they being indexed in google.
if anybody can give an idea where the problem lies ?
Check your webmaster tool to confirm how many pages are indexed. after that create a sitemap and submit it to Google or you can also fetch your links in webmaster tool to get index
Great Efforts. just let me focus on one more tip.
First, you need to make sure they're able to crawl your pages. Search engine spiders favour a few simple things:
• Clean XHTML or HTML code
• A lack of on-page JavaScript (put it in an external file and use an include call)
• An absence of inline CSS styles: put them in an external CSS file instead
• Non-dynamic URL paths
This was the rules to know how you can get your website indexed by the big search engines.
Best Regards,
Pillar Joe
https://www.smarttouch.me/en/
You Guys/Gals are awesome and your blogs and posts/comments very informative .. Thanks a bunch from Gary @ Chicago Illinois
p.s anyone know how you contact a Google Recruiter ? Would love to speak with one.
One reason that I have seen at least 3 times in my career: The web developer (or company) for reasons unknown to humanity changes robots.txt and places a nofollow for the whole website!
I think most SEOs have at one time accidently placed an invalid disallow rule in the robots.txt! Learn from your mistakes!
I love these checklist style posts. Very actionable information.
Great summary of what should be no-brainer stuff!
Another reason why a page would not get indexed is that it could 'offer' too little to the visitor and Google seems not to bother indexing it. A simple example is a form on top of the page and below that some content, it seems that Google is not a big favourite of this.
A simple solution is also the most obvious one: have more content above the form, and you will increase the chances of the page getting indexed.
Never had an issue with sites not being indexed, but the seemingly uncanny thing that I run into from time to time is a new post that I have published will not appear in search results in the normal ranking area. I can't see what I have done, or perhaps not done. Good post on what not to do when creating a new site.
Great post and Matt Cutts has never ever looked so cool.
I recently changed my website from html to word press. I installed site map plugin and submitted site map in google webmasters. Webmaster says submitted but its still pending. Old Html files which i found in crawl errors, I have fetched those to new links and now that there are no crawl errors. It had been 2 days since i had submitted site map. Its not getting indexed. Please help me
well written, great post.
my website is not getting indexed by google how can i config the robots.txt of my wp site
Thank you for this article. It's nothing new. But good to have this checklist for those bad things that happen in life ;-)
With sunny regards from Germany,
Sebastian
I like this post, crisp and neat. My question is are these the only reasons why a site's webpages might not get indexed or are these some of the major reasons? What about a site that has built absolutely no links ever since it was launched. Does it get indexed anyway?
My Website www.mulberryebags.co.uk There is some problem about crawl. When i type cache:website url then 404 error comes. and my site not crawl by google. please help me to solve this.
Valueable information. My site is about internet and how to stuff. At first my pages were greatly indexed. But Now it's not. I'll check my site according to this instructions. My site is https://HowBees.com.
Thanks for the article. I recently had a ton of problems with a site not being indexed AT ALL by Google after a site redesign. It was indexed before the redesign, but stopped showing up afterward. This article helped me pinpoint some problem areas, but I also wrote a post that explained some of the crazy stuff that happened to my site and how I was able to resolve it. https://www.webdesignetc.co/problems-getting-indexed-by-google/
Really Great Info. It helped me in a client's project that was not indexed and cached due to meta robots issues. Thanks for sharing.
So I go into Google Webmaster tools and the sitemap.xml file shows up OK, but it says "--" (or nothing) in the column under "pages indexed". So I google site:mywebsite.com but the search results show all of the pages that are in the sitemap.xml file. So my question is -- are my pages being indexed or not?
I think you kinda answered your own question there :)
Mine is # Blogger Sitemap generated on 2013.04.05User-agent: Mediapartners-GoogleDisallow: /searchAllow: /Sitemap: https://www.mysite.com/atom.xml?redirect=false&start-index=1&max-results=500
is there any mistake in it because google is not reffering any visitor to us frm many months
I am having the same problam with my site www.aloeverahq.com Yet SEO Quake says Google has indexed 16 pages which is roughly correct from the sitemap I submitted however In webmster tools it says only 2 pages indexed, but I can't find one page when I search for my keywords. I'm hoping I have'nt overlooked something really silly by mistake like a noindex nofollow meta tag or something haha.
Any advice would be cool.
Thanks
Jamie.
I have recently submitted my feeds to feedagg.com. And now I believe that was an awful mistake I have committed,after which All my subsequent posts are automatically feed to feedagg.com and google is not indexing it because of duplicate content. Refer my blog at https://blogsandyou.com.
To deactivate account in feedagg you need to mail them which I guess will take its own time.So all that I could do is change my feed address for further trouble. I would highly appreciate for suggestion on "How i can reclaim my posts" or if possible complaint against feedagg.com.
Great post Bob, this is a great checklist to go through any time there are issues.Something that has worked for me in addition to this is using the "Fetch as Google" feature under the Health menu. Once the page is fetched, you then have the option to "Submit to Index" and you can submit just that page or the page and all linked pages. You only get a certain number of these so you obviously it has to be used sparingly.
I've used this numerous times to get pages indexed that have been repeatedly passed up on the sitemap (e.g. new landing pages hint hint, nudge, nudge).
I like to add a little hierarchy to my LPs as well so that I can support fatter keywords and long-tail at the same time. If I fetch the parent LP and have the child pages listed at the bottom (just a few mind you, not a massive amount) in a "other pages you might find interesting" or "topics related to this page" section, I've been successful not only getting the current LP picked up, but also the child pages. Probably because it's now being reinforced through internal contextual linking + they're in the sitemap. More check-marks for Google.
Anyone else had success with this? Maybe even just fetching the page as Google but stopping short of submitting to index has some value?
Thanks Bob, I found this pretty useful in dealing with some of my own indexing issues.
Quick question - Can not keeping an XML sitemap updated cause a DECREASE in pages indexed as long as all those URLs have not changed? I have noticed a steep decline (and I'll be updating the sitemap today just in case), but I would have expected the amount of pages indexed to remain the same.
Well, I had an issue on my site and I couldn't find the answer here, my site was indexing for the past 2 years without any problems (I submit 30 links - all of them were indexed), now I removed a couple links and create a new links then generate a new sitemap.xml and submit it to Google, Google indexed only 1 of 31 link.
After researching I figured it out that Google changed the game, (sitemap.xml) it is not important any more, Google says on the below link, "Sitemaps provide an extra way for us to understand what pages you have on your site, and can help speed up the discovery of new and updated pages on your site. But neither HTML nor XML Sitemaps replace the normal crawling process. Having a Sitemap will not increase your ranking in Google's search results pages—but it will not reduce it, either"
So whether you submit your stiemap.xml or not, it will not increase or decrease your rank on Google cause now Google using "Auto Spiders" to Crawl all your site links, and for this reason, sometimes Google give me an crawl error (404) that the page not found which this page I never created it because simply Google reading everything even if it was 10 years ago. so, I am not worry about sitemap.xml any more.
Refer to Google Webmaster Central:
https://sites.google.com/site/webmasterhelpforum/en/faq--sitemaps-malware--other-issues/#sitemap-penalty
Thanks;
I have same problem is my site not index by google this is my site https://www.dogfoodsecretsreview.ga/ and my site educate about Dog Food Secrets Review . please answer me thank you so much
[Links removed by editor.]
I just wanted to say thank you for this post, I improve greatly my site! :)
Has anyone used the Godaddy tool for creating site maps for websites? I think they have helped with some of the time consuming parts of SEO. What do you think?
Thanks for your marvelous post.I am working on a website and the website has been crawling nicely by google but problem is some url like 1.https://www.aldictionary.com/definition/english-to-...
2.https://www.aldictionary.com/definition/english-to-... ,I think where google see the error on the link google can't index the link.So how I can solve the problem?
Kindly help me.
[Links removed by editor.]
Hi there! This question is much better suited to our Q&A forum. You're much more likely to get an answer there. :)