DailyLit Blog

Overeager Bot!

October 14th, 2007 Albert

You can safely ignore this post if you are not using gmail or google reader. It turns out that google is so eager to index the entire web that they use information from RSS feeds and gmail to do so. The Googlebot — the program which “crawls” the web to add pages to google’s index — was accessing pages that were linked to from within our emails and RSS feeds, causing subscriptions to be randomly suspended and resumed. As soon as we discovered this we instructed the Googlebot not to touch these pages. We apologize for any inconvenience this may have caused.

Entry Filed under: Service