Working at Incapsula gives us a bird’s-eye view of the bot traffic landscape. Amongst the innumerable creatures roaming those fields, few are as intriguing as Googlebot – a web crawler that facilitates knowledge exchange between billions of humans, influencing our perceptions, preferences and imaginations in more ways than we can even comprehend.
Over the years, many efforts have been made to better understand Googlebot’s behavior and motives, but many have fallen short. At this time, we are pleased to share with you some of our insights on the matter. Presented in the infographic below are our findings on the first face of Googlebot – what we call “Dr. Crawlit.”
However, no overview of Googlebot activity would be complete without a mention of Googlebot imposters, most of whom assume Googlebot’s identity to gain privileged access to websites and online information.
Every day millions of these “evil twins” are used for DDoS attacks, hacking, spam, content theft and many other shady activities. The details of these malicious escapades that paint the event logs of Incapsula’s security services are presented below as Dr. Crawlit’s alter-ego, “Mr. Hack.”
Methodology
For purposes of this study, we observed over 400 million search engine visits to 10,000 sites, resulting in more than 2.19 billion page crawls over a 30 day period.
Information about Googlebot impostors (a.k.a., Fake Googlebots) comes from inspection of over 50 million Googlebot impostor visits, as well as findings from our ‘DDoS Threat Landscape’ report, published earlier this year.
About Incapsula
Incapsula’s cloud-based Application Delivery service enables businesses to simplify their IT operations and reduce costs by consolidating multiple appliances and services into a single cloud solution. Enterprises get best-of-breed security, load balancing, failover and a global CDN, without having to deploy, manage and integrate separate products.
