Robots.txt Parser

`robots-parser` is a JavaScript/TypeScript library designed to parse `robots.txt` files according to the draft specification. As of version 3.0.1, it provides robust support for directives such as `User-agent`, `Allow`, `Disallow`, `Sitemap`, `Crawl-delay`, and `Host`, including advanced features like wildcard (`*`) and end-of-line (`$`) matching for paths. The library maintains an active development status, with recent releases addressing critical bug fixes (e.g., HTTPS URL port handling in 3.0.1) and improving compatibility (e.g., using global URL object in 3.0.0, adding TypeScript definitions in 2.4.0). Its primary differentiator is its adherence to the specification and comprehensive feature set for accurately determining URL crawlability for different user agents, making it a reliable choice for web crawlers and SEO tools.