nextthing.org » robots.txt Adventure nextthing.org · Originally posted on Pinboard

Great article following up on a robots.txt survey. The linked HTTP headers survey is amusing, too.