Many people have noticed a new googlebot user agent in their log files recently. This new entry in the log will look like Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html). Way back in March, googleguy over at webmasterworld forum mentioned that they were testing this new user agent and would roll it out when they believed it make a smooth transition. Apparently, six months later, that time has come.
He discussed the purpose of this by saying: "The primary reason for this is that some web servers assume that unless a user agent is IE, Netscape/Mozilla, or maybe Opera, that your browser wont support JavaScript, frames, etc. As Googlebot gets better over time, it gets closer to a regular user and browser in our ability to handle features like that. This user-agent change would let vanilla webservers assume Googlebot is more like a regular user, while still allowing site owners to recognize that Google is visiting and still providing clear contact info in case of questions or issues. We would still respect Googlebot in robots.txt, so the overwhelming majority of webmasters wouldnt have to change anything at all... most site owners probably wouldnt even notice the difference."
So if you were wondering what this new user agent in your log files was all about, now you know!