robots.txt tester
Check which URL paths a robots.txt file allows for a crawler, and which rule decided, following RFC 9309.
Input
Result
Fill in the input and press Run.
Your input is processed in memory and then discarded. It is not stored or logged. Even so, don't paste live passwords or keys into any online tool.
Use it from code or an agent
curl -s https://setget.net/v1/tools/robots-test \
-H 'Content-Type: application/json' \
-d '{"input":{"robots":"User-agent: *\nDisallow: *.gif$\nDisallow: /example/\nAllow: /publications/\n\nUser-agent: foobot\nDisallow: /\nAllow: /example/page.html\nAllow: /example/allowed.gif\n\nSitemap: https://example.com/sitemap.xml","user_agent":"foobot","paths":["/example/page.html","/example/allowed.gif","/","/publications/"]}}'
import json, urllib.request
body = {'input': {'robots': 'User-agent: *\nDisallow: *.gif$\nDisallow: /example/\nAllow: /publications/\n\nUser-agent: foobot\nDisallow: /\nAllow: /example/page.html\nAllow: /example/allowed.gif\n\nSitemap: https://example.com/sitemap.xml', 'user_agent': 'foobot', 'paths': ['/example/page.html', '/example/allowed.gif', '/', '/publications/']}}
request = urllib.request.Request("https://setget.net/v1/tools/robots-test", json.dumps(body).encode(),
{"Content-Type": "application/json"})
print(json.load(urllib.request.urlopen(request))["result"])
const response = await fetch("https://setget.net/v1/tools/robots-test", {
method: "POST",
headers: {"Content-Type": "application/json"},
body: JSON.stringify({"input":{"robots":"User-agent: *\nDisallow: *.gif$\nDisallow: /example/\nAllow: /publications/\n\nUser-agent: foobot\nDisallow: /\nAllow: /example/page.html\nAllow: /example/allowed.gif\n\nSitemap: https://example.com/sitemap.xml","user_agent":"foobot","paths":["/example/page.html","/example/allowed.gif","/","/publications/"]}}),
});
console.log((await response.json()).result);
Tool: run_tool
Arguments:
{
"slug": "robots-test",
"input": {
"robots": "User-agent: *\nDisallow: *.gif$\nDisallow: /example/\nAllow: /publications/\n\nUser-agent: foobot\nDisallow: /\nAllow: /example/page.html\nAllow: /example/allowed.gif\n\nSitemap: https://example.com/sitemap.xml",
"user_agent": "foobot",
"paths": [
"/example/page.html",
"/example/allowed.gif",
"/",
"/publications/"
]
}
}
{
"input": {
"robots": "User-agent: *\nDisallow: *.gif$\nDisallow: /example/\nAllow: /publications/\n\nUser-agent: foobot\nDisallow: /\nAllow: /example/page.html\nAllow: /example/allowed.gif\n\nSitemap: https://example.com/sitemap.xml",
"user_agent": "foobot",
"paths": [
"/example/page.html",
"/example/allowed.gif",
"/",
"/publications/"
]
}
}
Endpoint: POST /v1/tools/robots-test. Over MCP: run_tool with slug: "robots-test".
Machine-readable: input schema, Markdown. Connect an agent.
Inputs
| Name | Type | Required | What it does |
|---|---|---|---|
robots |
string | yes | The contents of the robots.txt file. |
user_agent |
string | yes | The crawler to test as: a product name such as Googlebot, or a full User-Agent string. |
paths |
array | yes | URL paths (or full URLs) to test. |
Examples
The example from RFC 9309, as foobot
Input
{
"robots": "User-agent: *\nDisallow: *.gif$\nDisallow: /example/\nAllow: /publications/\n\nUser-agent: foobot\nDisallow: /\nAllow: /example/page.html\nAllow: /example/allowed.gif\n\nSitemap: https://example.com/sitemap.xml",
"user_agent": "foobot",
"paths": [
"/example/page.html",
"/example/allowed.gif",
"/",
"/publications/"
]
}Result
{
"user_agent": "foobot",
"matched_group": "foobot",
"results": [
{
"path": "/example/page.html",
"allowed": true,
"rule": "Allow: /example/page.html",
"line": 8
},
{
"path": "/example/allowed.gif",
"allowed": true,
"rule": "Allow: /example/allowed.gif",
"line": 9
},
{
"path": "/",
"allowed": false,
"rule": "Disallow: /",
"line": 7
},
{
"path": "/publications/",
"allowed": false,
"rule": "Disallow: /",
"line": 7
}
],
"crawl_delay": null,
"sitemaps": [
"https://example.com/sitemap.xml"
]
}Wildcards for any other crawler
Input
{
"robots": "User-agent: *\nDisallow: *.gif$\nDisallow: /example/\nAllow: /publications/",
"user_agent": "Mozilla/5.0 (compatible; OtherBot/1.0)",
"paths": [
"/example/page.html",
"/images/logo.gif",
"/images/logo.gif?v=2",
"/about"
]
}Result
{
"user_agent": "Mozilla/5.0 (compatible; OtherBot/1.0)",
"matched_group": "*",
"results": [
{
"path": "/example/page.html",
"allowed": false,
"rule": "Disallow: /example/",
"line": 3
},
{
"path": "/images/logo.gif",
"allowed": false,
"rule": "Disallow: *.gif$",
"line": 2
},
{
"path": "/images/logo.gif?v=2",
"allowed": true,
"rule": null,
"line": null
},
{
"path": "/about",
"allowed": true,
"rule": null,
"line": null
}
],
"crawl_delay": null,
"sitemaps": []
}Questions
- Which rule wins when several match?
- The one with the longest path pattern. If an Allow and a Disallow are equally long, Allow wins.
- Which group applies to my crawler?
- The group whose User-agent name appears in the name you test, the longest such name if there are several. Only if none matches does the * group apply. Groups are never combined with *.
- Is the file fetched from my site?
- No. Paste its contents. Nothing is downloaded.
- Does robots.txt block access?
- No. It is a request that well-behaved crawlers follow. It is not access control.
Related tools
- Sitemap validatorValidate sitemap XML: structure, absolute URLs, dates, changefreq, priority, duplicates and the 50,000 URL limit.
sitemap-validate - User agent parserRead a User-Agent string: browser, engine, operating system, device type and known bots or AI agents.
user-agent-parse - Glob pattern testerTest file paths against a glob pattern with *, ?, **, {a,b} alternatives and [a-z] character classes.
glob-test - .gitignore testerCheck which paths a .gitignore file ignores and which line decided each one, using real Git matching rules.
gitignore-test - Cache-Control explainerExplain a Cache-Control header directive by directive and how browsers and CDNs will treat the response.
cache-control-explain - Card number (Luhn) and IBAN validatorCheck a card number with the Luhn checksum and name its network, or check an IBAN by length and mod-97.
luhn-iban-validate