{"library":"robot-directives","title":"Robot Directives Parser","description":"The `robot-directives` package (current stable version 0.4.0) provides a focused utility for parsing and interpreting robot directives found within HTML `<meta name=\"robots\">` tags and `X-Robots-Tag` HTTP headers. It allows developers to programmatically determine a crawler's allowed or disallowed actions based on these instructions, such as `noindex`, `nofollow`, `noarchive`, and `unavailable_after`. The library handles the cascading logic of multiple directives, user-agent specific rules, and resolves conflicts based on a `restrictive` default (mimicking Googlebot's behavior). It explicitly differentiates itself by not handling the underlying HTML parsing, requiring users to extract meta tag content themselves. While not on a rapid release cycle, the package offers a stable API for its specialized parsing tasks, including a comprehensive set of constants for all standard robot directives and static methods for general utility.","language":"javascript","status":"maintenance","last_verified":"Wed Apr 22","install":{"commands":["npm install robot-directives"],"cli":null},"imports":["const RobotDirectives = require('robot-directives');","const { NOINDEX } = RobotDirectives;","const { isBot } = RobotDirectives;"],"auth":{"required":false,"env_vars":[]},"quickstart":{"code":"const RobotDirectives = require('robot-directives');\n\n// Instantiate with default options\nconst robots = new RobotDirectives({\n  // Example: Override default userAgent if needed\n  // userAgent: 'Googlebot/2.1 (web crawler) (+http://www.google.com/bot.html)',\n  // Example: Override current time for 'unavailable_after' testing\n  // currentTime: () => new Date('jan 1 2025').getTime()\n});\n\n// Add directives from an X-Robots-Tag HTTP header\nrobots.header('googlebot: noindex, nosnippet');\n// Add directives from HTML meta tags\nrobots.meta('robots', 'noarchive,nofollow');\nrobots.meta('bingbot', 'unavailable_after: 1-Jan-3000 00:00:00 EST');\n\n// Check specific directives\nconsole.log('Is nofollow?', robots.is(RobotDirectives.NOFOLLOW));\n// Expected: true\n\nconsole.log('Is noindex for Googlebot?', robots.is(RobotDirectives.NOINDEX, { userAgent: 'Googlebot' }));\n// Expected: true\n\nconsole.log('Is noarchive?', robots.is(RobotDirectives.NOARCHIVE));\n// Expected: true\n\n// Check for a directive that is not present\nconsole.log('Is index?', robots.is(RobotDirectives.INDEX));\n// Expected: false\n\n// Check if 'unavailable_after' has passed (example: assuming current time is after 3000)\nconsole.log('Is noindex for Bingbot after 3000?', robots.is(RobotDirectives.NOINDEX, {\n  currentTime: () => new Date('Jan 2 3000').getTime(), // Set current time past the unavailable_after date\n  userAgent: 'Bingbot/2.0'\n}));\n// Expected: true\n\n// Use static helper function\nconsole.log('Is \"googlebot\" a recognized bot name?', RobotDirectives.isBot('googlebot'));\n// Expected: true\nconsole.log('Is \"mycustombot\" a recognized bot name?', RobotDirectives.isBot('mycustombot'));\n// Expected: false","lang":"javascript","description":"Demonstrates how to instantiate `RobotDirectives`, add directives from HTTP headers and HTML meta tags, and then query their state using `is()` with specific user agents and time overrides. It also shows the usage of the static `isBot()` helper.","tag":null,"tag_description":null,"last_tested":null,"results":[]},"compatibility":null}