{"library":"simplecrawler","type":"library","category":null,"description":"simplecrawler is an event-driven web crawler for Node.js (v1.1.9, stable, low release cadence). It provides flexible queue and cache mechanisms with extensible backends, automatic robots.txt respect, and link discovery. Differentiators vs alternatives like node-crawler or puppeteer: lightweight, uses EventEmitter, supports freezing/defrosting queues to disk, and preserves binary data via buffers. Suitable for archiving, analysis, and large-scale crawling.","language":"javascript","status":"active","version":"1.1.9","tags":["javascript","simple","crawler","spider","cache","queue","simplecrawler","eventemitter"],"last_verified":"Sun Jun 07","install":[{"cmd":"npm install simplecrawler","imports":["const Crawler = require('simplecrawler');\nconst crawler = new Crawler('http://example.com');","import Crawler from 'simplecrawler';","const queue = crawler.queue;"]},{"cmd":"yarn add simplecrawler","imports":[]},{"cmd":"pnpm add simplecrawler","imports":[]}],"homepage":"https://github.com/simplecrawler/simplecrawler","github":"https://github.com/simplecrawler/simplecrawler","docs":null,"changelog":null,"pypi":null,"npm":"simplecrawler","openapi_spec":null,"status_page":null,"smithery":null,"compatibility":null}