{"id":17635,"library":"express-robots-txt","title":"Express Robots.txt Middleware","description":"express-robots-txt is an Express.js middleware designed to either serve a static `robots.txt` file or dynamically generate one based on a JavaScript object or array configuration. The package is currently at version 1.0.0, with its last known publication approximately four years ago, indicating a stable but less actively developed state, likely in maintenance mode. It provides a straightforward API to define standard `robots.txt` directives such as `User-agent`, `Disallow`, `Allow`, `Crawl-delay`, `Sitemap` (supporting multiple entries), and `Host`. This flexibility allows developers to easily control how web crawlers interact with their Express applications, preventing unwanted indexing or specifying crawl behaviors, without needing to manage a static file manually for complex scenarios.","status":"maintenance","version":"1.0.0","language":"javascript","source_language":"en","source_url":"https://github.com/modosc/express-robots-txt","tags":["javascript","express","robots","robots.txt"],"install":[{"cmd":"npm install express-robots-txt","lang":"bash","label":"npm"},{"cmd":"yarn add express-robots-txt","lang":"bash","label":"yarn"},{"cmd":"pnpm add express-robots-txt","lang":"bash","label":"pnpm"}],"dependencies":[{"reason":"Peer dependency as it's an Express middleware.","package":"express","optional":false}],"imports":[{"note":"The primary export is a default export, typically imported without curly braces. The `robots` variable then becomes the middleware function.","wrong":"import { robots } from 'express-robots-txt'","symbol":"robots","correct":"import robots from 'express-robots-txt'"},{"note":"For CommonJS environments, the module is imported directly into a variable, which then holds the middleware function.","symbol":"robots (CommonJS)","correct":"const robots = require('express-robots-txt')"}],"quickstart":{"code":"import express from 'express';\nimport robots from 'express-robots-txt';\nimport path from 'path';\n\nconst app = express();\nconst port = process.env.PORT || 3000;\n\n// Option 1: Generate robots.txt from an object configuration\napp.use(robots({\n  UserAgent: '*',\n  Disallow: ['/admin', '/private'],\n  CrawlDelay: '10',\n  Sitemap: [\n    'https://www.example.com/sitemap.xml',\n    'https://www.example.com/sitemap-news.xml'\n  ],\n  Host: 'www.example.com'\n}));\n\n// Option 2 (alternative, comment out Option 1 to use): Serve a static robots.txt file\n// app.use(robots(path.join(__dirname, 'public/robots.txt')));\n\napp.get('/', (req, res) => {\n  res.send('Hello World! Check /robots.txt');\n});\n\napp.listen(port, () => {\n  console.log(`Server listening on port ${port}`);\n  console.log(`Open http://localhost:${port} and http://localhost:${port}/robots.txt`);\n});","lang":"javascript","description":"This quickstart demonstrates how to set up `express-robots-txt` middleware to dynamically generate a `robots.txt` file with common directives or serve a static file."},"warnings":[{"fix":"Upgrade to a recent Node.js version (v12.17.0+ for stable ESM support) or use CommonJS `require()` syntax if ESM issues persist.","message":"For ES Module (ESM) usage, ensure your Node.js version supports `esm_conditional_exports` or higher. Older Node.js versions might not correctly resolve the `import` statement, defaulting to CommonJS behavior or throwing errors.","severity":"gotcha","affected_versions":"<=1.0.0"},{"fix":"Use alternative methods for specifying canonical URLs, such as `<link rel=\"canonical\" href=\"...\">` in HTML headers, in addition to or instead of the `Host` directive.","message":"The `Host` directive in `robots.txt` is not universally supported by all web crawlers and is considered deprecated by Google. Relying solely on `Host` for canonicalization is not recommended.","severity":"gotcha","affected_versions":">=1.0.0"}],"env_vars":null,"last_verified":"2026-04-23T00:00:00.000Z","next_check":"2026-07-22T00:00:00.000Z","problems":[{"fix":"Ensure `app.use(robots(...))` is called with either a file path string or a configuration object/array, e.g., `app.use(robots({ UserAgent: '*', Disallow: '/' }))`.","cause":"The `robots` middleware was called without parentheses or a valid configuration object, thus not returning a function for Express to use.","error":"TypeError: app.use() requires a middleware function but got a Object"},{"fix":"Verify that the object structure follows the documented `robots.txt` directives (e.g., `UserAgent`, `Disallow`). If using a file, ensure the path is correct and the file is readable by the Node.js process.","cause":"The configuration object or array passed to `robots()` is invalid, or the specified file path does not exist/is inaccessible.","error":"My robots.txt output is empty or malformed."}],"ecosystem":"npm","meta_description":null,"install_score":null,"install_tag":null,"quickstart_score":null,"quickstart_tag":null}