Protego

0.6.0 · active · verified Wed Apr 01

Protego is a pure-Python robots.txt parser with support for modern conventions like those defined by Google. As of version 0.6.0, it actively supports Python 3.10 and newer, with regular updates aligning with new Python releases. It is widely used for web scraping and compliance checking.

Warnings

breaking Version 0.6.0 dropped official support for Python 3.9 and PyPy 3.10. Users on these Python versions should use Protego 0.5.x or upgrade their Python environment.
Fix: Upgrade to Python 3.10+ or pin Protego to a version <0.6.0 (e.g., `protego<0.6.0`).
breaking Version 0.4.0 dropped official support for Python 3.8.
Fix: Upgrade to Python 3.9+ or pin Protego to a version <0.4.0.
breaking Version 0.3.0 dropped support for Python 2.7, 3.5, 3.6, and 3.7. The `six` dependency was also removed in this version, making it Python 3 only.
Fix: Upgrade to Python 3.8+ and remove `six` if it was only a Protego dependency. Pin Protego to a version <0.3.0 for older Python environments.
gotcha In Protego 0.3.0 and later, `Protego.parse()` will raise a `ValueError` if the `robotstxt_body` argument is not a string.
Fix: Ensure that the `robotstxt_body` passed to `Protego.parse()` is always a string.
gotcha Version 0.1.16 fixed an issue where absolute URLs in `Allow` and `Disallow` directives were incorrectly parsed, ignoring their protocol and netloc. Older versions might misinterpret these directives, leading to incorrect access decisions.
Fix: Upgrade to Protego 0.1.16 or newer to ensure correct interpretation of absolute URLs in `robots.txt` directives.
gotcha Version 0.5.0 restructured the internal code from a single `protego.py` file into multiple modules. While the public API `from protego import Protego` remains stable, direct imports of internal modules (if any were used) would have broken.
Fix: Avoid importing internal modules; rely only on the documented public API (e.g., `from protego import Protego`).

Install

pip install protego Install stable version

Imports

Protego
```
from protego import Protego
```

Quickstart

Initialize the parser with robots.txt content and check URL access permissions for a specific user agent, retrieve crawl delay, and sitemaps.

from protego import Protego

robotstxt_content = """
User-agent: *
Disallow: /admin/
Allow: /admin/login
Crawl-delay: 5
Sitemap: http://example.com/sitemap.xml
"""

rp = Protego.parse(robotstxt_content)

# Check if a URL can be fetched by a user agent
can_fetch_admin = rp.can_fetch("http://example.com/admin/settings", "mybot")
can_fetch_login = rp.can_fetch("http://example.com/admin/login", "mybot")

print(f"Can 'mybot' fetch /admin/settings? {can_fetch_admin}")
print(f"Can 'mybot' fetch /admin/login? {can_fetch_login}")
print(f"Crawl delay for 'mybot': {rp.crawl_delay('mybot')} seconds")
print(f"Sitemaps: {list(rp.sitemaps)}")

view raw JSON →