{"library":"robots.txt","title":"Robots.txt Express Middleware","description":"The `robots.txt` package provides simple Connect/Express middleware to serve a static `robots.txt` file from the filesystem. Currently at version 1.1.0, this package was last published over a decade ago, indicating it is no longer actively maintained. Its core functionality involves a single synchronous read of the `robots.txt` file upon application startup, with subsequent requests served from memory. It automatically sets appropriate cache headers. Unlike many other `robots.txt` related packages on npm, this library is strictly for serving the file content at the `/robots.txt` endpoint, not for parsing or evaluating robots exclusion rules for crawlers. Its simplicity and 'fire-and-forget' setup were its primary differentiators in its active period, but it lacks modern features like asynchronous file handling, ESM support, or TypeScript typings.","language":"javascript","status":"abandoned","last_verified":"Thu Apr 23","install":{"commands":["npm install robots.txt"],"cli":null},"imports":["const robots = require('robots.txt')","app.use(robots(__dirname + '/robots.txt'))"],"auth":{"required":false,"env_vars":[]},"quickstart":{"code":"const express = require('express');\nconst path = require('path');\nconst robots = require('robots.txt');\nconst app = express();\n\n// Create a dummy robots.txt file for demonstration\nconst robotsTxtContent = `User-agent: *\nDisallow: /admin/\nAllow: /\nSitemap: http://localhost:3000/sitemap.xml`;\nconst robotsTxtPath = path.join(__dirname, 'robots.txt');\nrequire('fs').writeFileSync(robotsTxtPath, robotsTxtContent);\n\n// Use the robots.txt middleware\n// The middleware will read robots.txt from the specified path once at startup.\napp.use(robots(robotsTxtPath));\n\n// Example route for a sitemap (often linked from robots.txt)\napp.get('/sitemap.xml', (req, res) => {\n  res.type('application/xml');\n  res.send('<urlset><url><loc>http://localhost:3000/</loc></url></urlset>');\n});\n\n// Example protected route\napp.get('/admin', (req, res) => {\n  res.send('Admin area - disallowed by robots.txt');\n});\n\n// Other routes\napp.get('/', (req, res) => {\n  res.send('Hello World! Check http://localhost:3000/robots.txt');\n});\n\nconst PORT = 3000;\napp.listen(PORT, () => {\n  console.log(`Server running on http://localhost:${PORT}`);\n  console.log(`View robots.txt at http://localhost:${PORT}/robots.txt`);\n});","lang":"javascript","description":"This quickstart demonstrates setting up an Express application to serve a `robots.txt` file using the middleware. It creates a dummy `robots.txt` on the fly and shows how the middleware integrates into an Express app, serving the file at the `/robots.txt` endpoint.","tag":null,"tag_description":null,"last_tested":null,"results":[]},"compatibility":null}