{"id":17933,"library":"robots.txt","title":"Robots.txt Express Middleware","description":"The `robots.txt` package provides simple Connect/Express middleware to serve a static `robots.txt` file from the filesystem. Currently at version 1.1.0, this package was last published over a decade ago, indicating it is no longer actively maintained. Its core functionality involves a single synchronous read of the `robots.txt` file upon application startup, with subsequent requests served from memory. It automatically sets appropriate cache headers. Unlike many other `robots.txt` related packages on npm, this library is strictly for serving the file content at the `/robots.txt` endpoint, not for parsing or evaluating robots exclusion rules for crawlers. Its simplicity and 'fire-and-forget' setup were its primary differentiators in its active period, but it lacks modern features like asynchronous file handling, ESM support, or TypeScript typings.","status":"abandoned","version":"1.1.0","language":"javascript","source_language":"en","source_url":null,"tags":["javascript"],"install":[{"cmd":"npm install robots.txt","lang":"bash","label":"npm"},{"cmd":"yarn add robots.txt","lang":"bash","label":"yarn"},{"cmd":"pnpm add robots.txt","lang":"bash","label":"pnpm"}],"dependencies":[{"reason":"This package is designed as middleware for Express.js applications.","package":"express","optional":true},{"reason":"This package is also compatible with Connect-based applications.","package":"connect","optional":true}],"imports":[{"note":"This package exclusively uses CommonJS `require()` syntax as it was developed before widespread ESM adoption in Node.js.","wrong":"import robots from 'robots.txt'","symbol":"robots","correct":"const robots = require('robots.txt')"},{"note":"The middleware should be registered without a path argument to `app.use()` as it internally handles the `/robots.txt` route.","wrong":"app.use('/robots.txt', robots(__dirname + '/robots.txt'))","symbol":"robotsMiddleware","correct":"app.use(robots(__dirname + '/robots.txt'))"}],"quickstart":{"code":"const express = require('express');\nconst path = require('path');\nconst robots = require('robots.txt');\nconst app = express();\n\n// Create a dummy robots.txt file for demonstration\nconst robotsTxtContent = `User-agent: *\nDisallow: /admin/\nAllow: /\nSitemap: http://localhost:3000/sitemap.xml`;\nconst robotsTxtPath = path.join(__dirname, 'robots.txt');\nrequire('fs').writeFileSync(robotsTxtPath, robotsTxtContent);\n\n// Use the robots.txt middleware\n// The middleware will read robots.txt from the specified path once at startup.\napp.use(robots(robotsTxtPath));\n\n// Example route for a sitemap (often linked from robots.txt)\napp.get('/sitemap.xml', (req, res) => {\n  res.type('application/xml');\n  res.send('<urlset><url><loc>http://localhost:3000/</loc></url></urlset>');\n});\n\n// Example protected route\napp.get('/admin', (req, res) => {\n  res.send('Admin area - disallowed by robots.txt');\n});\n\n// Other routes\napp.get('/', (req, res) => {\n  res.send('Hello World! Check http://localhost:3000/robots.txt');\n});\n\nconst PORT = 3000;\napp.listen(PORT, () => {\n  console.log(`Server running on http://localhost:${PORT}`);\n  console.log(`View robots.txt at http://localhost:${PORT}/robots.txt`);\n});","lang":"javascript","description":"This quickstart demonstrates setting up an Express application to serve a `robots.txt` file using the middleware. It creates a dummy `robots.txt` on the fly and shows how the middleware integrates into an Express app, serving the file at the `/robots.txt` endpoint."},"warnings":[{"fix":"No direct fix within the package. For highly performant applications, consider serving static robots.txt via a web server (e.g., Nginx) or using custom asynchronous middleware.","message":"This package relies on synchronous file reading at startup. While it ensures the file exists, it can block the event loop during application initialization if the file is large or on a slow filesystem.","severity":"breaking","affected_versions":">=1.0.0"},{"fix":"Consider migrating to a manually served static file using `app.get('/robots.txt', ...)` or a more modern, actively maintained alternative like `express-robots.txt`.","message":"The package is effectively unmaintained, with its last publish being over 12 years ago. It may have compatibility issues with modern Node.js versions, Express versions (beyond 3.x/4.x), or lack crucial security updates.","severity":"breaking","affected_versions":">=1.1.0"},{"fix":"If dynamic `robots.txt` generation or parsing is required, use a different library explicitly designed for those purposes.","message":"This package only serves a static `robots.txt` file. It does not provide any parsing, validation, or dynamic generation capabilities. For parsing, other packages like `robots-txt-parser` or `@chronocide/robots-txt` are available.","severity":"gotcha","affected_versions":">=1.0.0"},{"fix":"Create a `d.ts` declaration file (e.g., `declare module 'robots.txt';`) or use a different, actively maintained solution that provides TypeScript support.","message":"The package does not support TypeScript out-of-the-box and does not have official `@types/robots.txt` definitions. Modern projects using TypeScript will need to declare their own module or resort to `require` with `@ts-ignore`.","severity":"gotcha","affected_versions":">=1.0.0"}],"env_vars":null,"last_verified":"2026-04-23T00:00:00.000Z","next_check":"2026-07-22T00:00:00.000Z","problems":[{"fix":"Ensure the path passed to `robots()` is an absolute path to a valid `robots.txt` file. Use `path.join(__dirname, 'robots.txt')` for relative paths within your project.","cause":"The path provided to the `robots()` middleware function does not point to an existing `robots.txt` file.","error":"Error: ENOENT: no such file or directory, open 'path/to/your/robots.txt'"},{"fix":"Verify that `robots.txt` is correctly installed and that the path passed to the `robots()` function is a non-empty string and resolves to a readable file.","cause":"This error typically occurs if `robots.txt` fails to load or if `robots()` is called without a valid path, causing it to return `undefined`.","error":"TypeError: app.use() requires middleware functions but got a [object Undefined]"},{"fix":"Ensure `app.use(robots(pathToRobotsTxt))` is placed early in your middleware stack, before any general static file servers or route handlers that might capture `/robots.txt`.","cause":"The `robots.txt` middleware is either not mounted correctly, or another middleware is intercepting the `/robots.txt` request before this middleware can handle it.","error":"Cannot GET /robots.txt"}],"ecosystem":"npm","meta_description":null,"install_score":null,"install_tag":null,"quickstart_score":null,"quickstart_tag":null}