Cobweb
JSON →Cobweb is a web crawler that can use Resque to cluster crawls for large sites, with a statistics monitoring interface.
Cobweb is a web crawler that can use Resque to cluster crawls for large sites, with a statistics monitoring interface.