Google Webmaster Tools and Blocked URLs
A Case Study in the Subtle Meanings Behind Webmaster Tools Verbiage
Website “health checks” are one of the standard services we provide to our online marketing and SEO clients. The purpose of a health check is similar to that of a check-up in your doctor’s office: to alert you to any serious problems that can and should be caught as early as possible to prevent catastrophic problems later on.
Some basic health checks don’t require advanced experience to do, and a lot of in-house webmasters and even website hobbyists can keep their finger on the pulse of their site this way.
When we try to help one of our clients become a more active partner in managing their online marketing, we will occasionally introduce them to one of Google’s most powerful customer support mechanisms: Google Webmaster Tools. You can perform some of your own Website “health checks” using Webmaster Tools, which is the primary tool we use for determine pain points in any website.
However, if you indeed are one of those intrepid DIY’s, there is a note of caution that you need to hear: Google Webmaster Tools can be notoriously open to misinterpretation. Let’s take for example a recent request from a client to explain why they were showing a bunch of blocked URLs in Webmaster Tools. They were understandably concerned. Wouldn’t you be, if you opened up Webmaster Tools and saw the following:
But hold on there, you’ve run into one of the first gotchas of peering into the Webmaster Tools dashboard, especially if you only log in once in a while. There might be a dire warning, but check the date. Note the next screen shot:
In the case of this real world example, we can see that the notice is 2 weeks old. So how do we find out if it’s still valid?
Simple, navigate to the “Health” tab and click on the (appropriately named) “Blocked URLs” tab.
So now we have another intimidating warning. This notice says that Googlebot was blocked from 35 URLs, which in this case is basically the whole site. Not only that, but notice that date, I don’t know what it means but it’s dated yesterday, so this is really bad, right?
Once again, not necessarily. Notice the little question mark next to the “Blocked URLs” column heading. Well, if it’s too small here’s a larger image, and also I’ve hovered over the question mark so we can reveal its secrets:
So now we know: Blocked URLs are any that have been blocked in the last 90 days. Again, this shows that the problem might exist, or it might already be an obsolete issue. There’s one last place to go for us to render a pretty fair decision on this, namely the “Crawl Errors” tab. What we see eases our worries, although there’s still some head scratching that needs to be endured:
So here we see the report from Google, showing only one lonely “Access Denied” error. If we really had 35 pages blocked, each one would show up as triggering an error.
The head-scratching comes in when we try to reconcile these screens. Why does the Blocked URLs tab show 35 errors in the last 90 days, but this tab, even though it is covering the same 90 days, only shows one. And not only that, but if you try to find that error in the blue line where error events are tracked, you won’t. The blue dotted line shows 0 errors on each of the days in this period.
So the conclusion of this is simply not to immediately jump to a conclusion based on Google Webmaster Tools warnings or errors. Take the time to look for dates and details. In the long run, you’ll be ahead.