CCBot is a Nutch-based web crawler that makes use of the Apache Hadoop project.
Bot identifies itself with the following User-Agent string: CCBot/1.0
Bot honor robots.txt directives - Crawl-Delay, Disallow and Allow. Support the NOFOLLOW meta-tag.