Robotstxt
JSON →Robotstxt Parser allows you to check the accessibility of URLs and get other data. Full support for the robots.txt RFC, wildcards and Sitemap rules.
Robotstxt Parser allows you to check the accessibility of URLs and get other data. Full support for the robots.txt RFC, wildcards and Sitemap rules.