Google stopped caching my website - http 4xx error!

I was taken by surprise when I noticed that I was not receiving any traffic from Google lately. Our website used to come at the top for home warranty searches, but not any more! Websites with no credible content and lower page ranking than mine were showing up in Google search, but not ours. Now I was really worried that Google may have blocked my website, however, my page ranking was intact, so there must be something else going on.

Fortunately, I had Google Webmaster Tools configured for this website. I peeked into GWT for an explanation. There were some http errors but it did not have much details other than “4xx error” and a link to http status codes list as shown here (click on the image to expand the view) -

My next stop was to perform health check of the website and to check log files for errors. Everything looked good and no signs of a problem. At that point I realized that I do not have 24×7 monitoring on my website, so if my host server site was down, I will never know about the problem. I went ahead and set up monitoring using the free mon.itor.us service and set up alerts to notify me if my website goes down as a precaution.

Next I contacted my web hosting company and explained the problem to them. Alas! they knew what was going on… Apparently my hosting company Linksky had blocked all bots for few days to handle excessive crawling happening due some other sites hosted on the same server. After making several unsuccessful crawling attempts, Google removed my site from its cache and also from search results.

My web hosting company moved our site to another server which can better handle such issues quickly after I reported the issue. However, they did not act pro-actively and as a result my SERP position in Google is gone and it may take a while to get back to the same level. They should have blocked bot requests on only those high traffic sites, not all.

If you encounter this error, please make sure your web hosting company has not blocked Googlebots. It is always a good idea to to set up external monitoring using many services available online. This is a shortlist of some of those service and their basic plan is free

siteuptime.com
mon.itor.us
uptrends.com
checkwebsite.org

Posted in WebDesign | No Comments »