The Google Crawler is Fast

Posted on by

It took the Google crawler all of 15 minutes to index my last post about PowerEdge Ubuntu RAID monitoring.  That is utterly ridiculous.  It boggles my mind that they can so quickly index, compute PageRank, and cache everything in that little amount of time.  Perhaps they don’t do all of that right away.  Maybe they assume a fresh post has a high PageRank.  Maybe they really do compute the PageRank on the fly.  Regardless, I’m totally amazed.

  • http://sunilgarg.com/ Sunil Garg

    15 minutes is fast, but Google’s job is made pretty easy by WordPress’ use of XML-RPC pings.

  • http://www.alexloddengaard.com Alex Loddengaard

    I suppose what I’m most impressed with is Google’s ability to index and compute PageRank in such little amount of time. Regardless, thanks for the link, Sunil!

  • http://sunilgarg.com/ Sunil Garg

    What exactly would they need to calculate in terms of PageRank?

    If they know that your post is extremely fresh, couldn’t they operate under the assumption that no one else has linked to it yet (so initial PageRank is close to zero, or maybe some average of your older posts), and wait until they see incoming links in future posts from across the web?

    Their indexing speed is nonetheless quite impressive.