Most of us who are engaged in the serious practice of Site optimization are focused on the routine visits of Google crawlers – while sites with regular content updates love the FB aka ‘FreshBots’.
But, as the web is moving towards a higher bandwidth era and a more multi-media content - capable of being consumed and distributed – the data collection task has to be performed by multiple crawlers. Google utilizes a suit of crawlers for its ‘Universal Search’ …. The Graphics below gives a family picture of the Google crawlers.
