From my webstats:

Robots/Spiders - Bandwidth
MSNBot - 26.86 MB
Googlebot - 2.56 MB
Referrer links from an Internet Search Engine:
Google - 685
MSN - 14

This obviously ain’t a bandwith problem, but somehow it feel as an abuse from MSN.

Can’t resists…. urge…. block….MSN……

What should I do?



User-agent: msnBot
Disallow: /
User-agent: *
Disallow: /albums/

It’s seems to be important to put the User-agent: * part at the end, as some bots might not look further if it’s at the beginnining.

As I don’t trust MS to behave, I added something to my apache config also.

SetEnvIf User-Agent "msnbot" badbots
Order Allow,Deny
Allow from all
Deny from env=badbots