Robots.txt Parser

The `robots-txt-parser` library provides a lightweight, promise-based solution for parsing `robots.txt` files efficiently in Node.js environments. It is currently at version 2.0.3, offering features such as comprehensive wildcard support in rules, configurable caching of `robots.txt` content, and flexible asynchronous operations via both promises and traditional callbacks. This package is specifically designed for developers building web crawlers, scrapers, and other automated bots that must adhere to website crawling policies. Key differentiators include its focus on Node.js, a clear API for determining URL crawlability, retrieving sitemaps, and managing crawl delays. The project maintains a stable release cadence, with the 2.x major version being actively supported since late 2018. Users can configure critical parameters such as the default user agent string and how the parser evaluates scenarios where allow/disallow rules are balanced.