MaxMind, which provides GeoIP info for global solution providers, is incorrectly identifying Googlebot. Websites worldwide depend on this data to restrict access from unwanted regions, such as Russia and Iran (in light of global political situations), meaning sites are unintentionally hampering their search engine presence.
MaxMind integrates with business websites to provide the IP locations of site visitors, alongside additional layers of security.
MaxMind’s GeoIP database is used by Cloudflare and other content delivery networks and security services. Cloudflare is used by 7.59 million active websites, with 122,313 from the UK alone, making it an incredibly popular security tool. When you consider the enormity of those numbers, and that many businesses around the world block Russia in Cloudflare, it becomes increasingly apparent that many will be affected by MaxMind’s GeoIP problem.
Here’s an example of Cloudflare’s documentation outlining their use of MaxMind:
Google sends Googlebot to crawl websites and update its indexes. After crawling pages, websites can be found and matched to relevant Google search queries.
MaxMind provides invaluable mapping information about visitors, including these regular Google crawlers, identifying their IP addresses to help you understand where they are visiting from.
However, MaxMind is registering these Googlebot IP addresses as from Russia and Iran. This is problematic, as recent hostile activity has resulted in governments across the world, including the UK and US, issuing sanctions against Russia. Many businesses have also therefore chosen to block traffic from these countries, simply done in a tool like Cloudflare.
So, by MaxMind’s GeoIP incorrectly recognising Google’s crawlers from Russia, your site could be actively and negatively impacted in Google search results.
For one of our eCommerce clients at magic42, Google Search Console alerted us that it was being blocked from crawling both mobile and desktop versions of the website. This provided IP addresses for which Googlebots were being blocked.
Curious over the alert, we investigated the problem by looking at our integrated version of MaxMind’s GeoIP, GeoLite2. When inputting the IP addresses from Google Search Console, MaxMind revealed they were from Russia, Iran and further afield. Upon further investigation, it was clear that those IP addresses were actually Google’s crawlers, identifiable from their IPs. This meant that MaxMind’s GeoIP database had identified them incorrectly and therefore could be restricting access, in turn impacting search rankings.
It is vital that MaxMind’s GeoIP shows the correct country of origin for Google’s crawlers, otherwise this will continue to hamper search engine performance.
At the moment, the best way to correct the IP issue is to inform MaxMind. You can do this and make sure MaxMind shows the correct Country Code if you submit a correction request. Alternatively, you can send a request to correct a GeoIP location in MaxMind.
You can also contact Cloudflare support when MaxMind is showing the correct country code, but not the CF-IPCountry request header.
Fingers crossed this can be resolved in the near future to increase site visibility.
