Robots.txt Express Middleware

1.1.0 verified Thu Apr 23 auth: no javascript abandoned

The `robots.txt` package provides simple Connect/Express middleware to serve a static `robots.txt` file from the filesystem. Currently at version 1.1.0, this package was last published over a decade ago, indicating it is no longer actively maintained. Its core functionality involves a single synchronous read of the `robots.txt` file upon application startup, with subsequent requests served from memory. It automatically sets appropriate cache headers. Unlike many other `robots.txt` related packages on npm, this library is strictly for serving the file content at the `/robots.txt` endpoint, not for parsing or evaluating robots exclusion rules for crawlers. Its simplicity and 'fire-and-forget' setup were its primary differentiators in its active period, but it lacks modern features like asynchronous file handling, ESM support, or TypeScript typings.

Common errors

error Error: ENOENT: no such file or directory, open 'path/to/your/robots.txt' ↓

cause The path provided to the `robots()` middleware function does not point to an existing `robots.txt` file.

fix

Ensure the path passed to robots() is an absolute path to a valid robots.txt file. Use path.join(__dirname, 'robots.txt') for relative paths within your project.

error TypeError: app.use() requires middleware functions but got a [object Undefined] ↓

cause This error typically occurs if `robots.txt` fails to load or if `robots()` is called without a valid path, causing it to return `undefined`.

fix

Verify that robots.txt is correctly installed and that the path passed to the robots() function is a non-empty string and resolves to a readable file.

error Cannot GET /robots.txt ↓

cause The `robots.txt` middleware is either not mounted correctly, or another middleware is intercepting the `/robots.txt` request before this middleware can handle it.

fix

Ensure app.use(robots(pathToRobotsTxt)) is placed early in your middleware stack, before any general static file servers or route handlers that might capture /robots.txt.

Warnings

breaking This package relies on synchronous file reading at startup. While it ensures the file exists, it can block the event loop during application initialization if the file is large or on a slow filesystem. ↓

fix No direct fix within the package. For highly performant applications, consider serving static robots.txt via a web server (e.g., Nginx) or using custom asynchronous middleware.

breaking The package is effectively unmaintained, with its last publish being over 12 years ago. It may have compatibility issues with modern Node.js versions, Express versions (beyond 3.x/4.x), or lack crucial security updates. ↓

fix Consider migrating to a manually served static file using `app.get('/robots.txt', ...)` or a more modern, actively maintained alternative like `express-robots.txt`.

gotcha This package only serves a static `robots.txt` file. It does not provide any parsing, validation, or dynamic generation capabilities. For parsing, other packages like `robots-txt-parser` or `@chronocide/robots-txt` are available. ↓

fix If dynamic `robots.txt` generation or parsing is required, use a different library explicitly designed for those purposes.

gotcha The package does not support TypeScript out-of-the-box and does not have official `@types/robots.txt` definitions. Modern projects using TypeScript will need to declare their own module or resort to `require` with `@ts-ignore`. ↓

fix Create a `d.ts` declaration file (e.g., `declare module 'robots.txt';`) or use a different, actively maintained solution that provides TypeScript support.

Install

npm install robots.txt

yarn add robots.txt

pnpm add robots.txt

Imports

robots
wrong
```
import robots from 'robots.txt'
```
correct
```
const robots = require('robots.txt')
```
This package exclusively uses CommonJS `require()` syntax as it was developed before widespread ESM adoption in Node.js.
robotsMiddleware
wrong
```
app.use('/robots.txt', robots(__dirname + '/robots.txt'))
```
correct
```
app.use(robots(__dirname + '/robots.txt'))
```
The middleware should be registered without a path argument to `app.use()` as it internally handles the `/robots.txt` route.

Quickstart

This quickstart demonstrates setting up an Express application to serve a `robots.txt` file using the middleware. It creates a dummy `robots.txt` on the fly and shows how the middleware integrates into an Express app, serving the file at the `/robots.txt` endpoint.

const express = require('express');
const path = require('path');
const robots = require('robots.txt');
const app = express();

// Create a dummy robots.txt file for demonstration
const robotsTxtContent = `User-agent: *
Disallow: /admin/
Allow: /
Sitemap: http://localhost:3000/sitemap.xml`;
const robotsTxtPath = path.join(__dirname, 'robots.txt');
require('fs').writeFileSync(robotsTxtPath, robotsTxtContent);

// Use the robots.txt middleware
// The middleware will read robots.txt from the specified path once at startup.
app.use(robots(robotsTxtPath));

// Example route for a sitemap (often linked from robots.txt)
app.get('/sitemap.xml', (req, res) => {
  res.type('application/xml');
  res.send('<urlset><url><loc>http://localhost:3000/</loc></url></urlset>');
});

// Example protected route
app.get('/admin', (req, res) => {
  res.send('Admin area - disallowed by robots.txt');
});

// Other routes
app.get('/', (req, res) => {
  res.send('Hello World! Check http://localhost:3000/robots.txt');
});

const PORT = 3000;
app.listen(PORT, () => {
  console.log(`Server running on http://localhost:${PORT}`);
  console.log(`View robots.txt at http://localhost:${PORT}/robots.txt`);
});