robots.txt tester

Check which URL paths a robots.txt file allows for a crawler, and which rule decided, following RFC 9309.

Input

The contents of the robots.txt file.
The crawler to test as: a product name such as Googlebot, or a full User-Agent string.
URL paths (or full URLs) to test.

Result

Fill in the input and press Run.

Your input is processed in memory and then discarded. It is not stored or logged. Even so, don't paste live passwords or keys into any online tool.

Use it from code or an agent

curl -s https://setget.net/v1/tools/robots-test \
  -H 'Content-Type: application/json' \
  -d '{"input":{"robots":"User-agent: *\nDisallow: *.gif$\nDisallow: /example/\nAllow: /publications/\n\nUser-agent: foobot\nDisallow: /\nAllow: /example/page.html\nAllow: /example/allowed.gif\n\nSitemap: https://example.com/sitemap.xml","user_agent":"foobot","paths":["/example/page.html","/example/allowed.gif","/","/publications/"]}}'
import json, urllib.request

body = {'input': {'robots': 'User-agent: *\nDisallow: *.gif$\nDisallow: /example/\nAllow: /publications/\n\nUser-agent: foobot\nDisallow: /\nAllow: /example/page.html\nAllow: /example/allowed.gif\n\nSitemap: https://example.com/sitemap.xml', 'user_agent': 'foobot', 'paths': ['/example/page.html', '/example/allowed.gif', '/', '/publications/']}}
request = urllib.request.Request("https://setget.net/v1/tools/robots-test", json.dumps(body).encode(),
                                 {"Content-Type": "application/json"})
print(json.load(urllib.request.urlopen(request))["result"])
const response = await fetch("https://setget.net/v1/tools/robots-test", {
  method: "POST",
  headers: {"Content-Type": "application/json"},
  body: JSON.stringify({"input":{"robots":"User-agent: *\nDisallow: *.gif$\nDisallow: /example/\nAllow: /publications/\n\nUser-agent: foobot\nDisallow: /\nAllow: /example/page.html\nAllow: /example/allowed.gif\n\nSitemap: https://example.com/sitemap.xml","user_agent":"foobot","paths":["/example/page.html","/example/allowed.gif","/","/publications/"]}}),
});
console.log((await response.json()).result);
Tool: run_tool
Arguments:
{
  "slug": "robots-test",
  "input": {
    "robots": "User-agent: *\nDisallow: *.gif$\nDisallow: /example/\nAllow: /publications/\n\nUser-agent: foobot\nDisallow: /\nAllow: /example/page.html\nAllow: /example/allowed.gif\n\nSitemap: https://example.com/sitemap.xml",
    "user_agent": "foobot",
    "paths": [
      "/example/page.html",
      "/example/allowed.gif",
      "/",
      "/publications/"
    ]
  }
}
{
  "input": {
    "robots": "User-agent: *\nDisallow: *.gif$\nDisallow: /example/\nAllow: /publications/\n\nUser-agent: foobot\nDisallow: /\nAllow: /example/page.html\nAllow: /example/allowed.gif\n\nSitemap: https://example.com/sitemap.xml",
    "user_agent": "foobot",
    "paths": [
      "/example/page.html",
      "/example/allowed.gif",
      "/",
      "/publications/"
    ]
  }
}

Endpoint: POST /v1/tools/robots-test. Over MCP: run_tool with slug: "robots-test". Machine-readable: input schema, Markdown. Connect an agent.

Inputs

NameTypeRequiredWhat it does
robots string yes The contents of the robots.txt file.
user_agent string yes The crawler to test as: a product name such as Googlebot, or a full User-Agent string.
paths array yes URL paths (or full URLs) to test.

Examples

The example from RFC 9309, as foobot

Input
{
  "robots": "User-agent: *\nDisallow: *.gif$\nDisallow: /example/\nAllow: /publications/\n\nUser-agent: foobot\nDisallow: /\nAllow: /example/page.html\nAllow: /example/allowed.gif\n\nSitemap: https://example.com/sitemap.xml",
  "user_agent": "foobot",
  "paths": [
    "/example/page.html",
    "/example/allowed.gif",
    "/",
    "/publications/"
  ]
}
Result
{
  "user_agent": "foobot",
  "matched_group": "foobot",
  "results": [
    {
      "path": "/example/page.html",
      "allowed": true,
      "rule": "Allow: /example/page.html",
      "line": 8
    },
    {
      "path": "/example/allowed.gif",
      "allowed": true,
      "rule": "Allow: /example/allowed.gif",
      "line": 9
    },
    {
      "path": "/",
      "allowed": false,
      "rule": "Disallow: /",
      "line": 7
    },
    {
      "path": "/publications/",
      "allowed": false,
      "rule": "Disallow: /",
      "line": 7
    }
  ],
  "crawl_delay": null,
  "sitemaps": [
    "https://example.com/sitemap.xml"
  ]
}

Wildcards for any other crawler

Input
{
  "robots": "User-agent: *\nDisallow: *.gif$\nDisallow: /example/\nAllow: /publications/",
  "user_agent": "Mozilla/5.0 (compatible; OtherBot/1.0)",
  "paths": [
    "/example/page.html",
    "/images/logo.gif",
    "/images/logo.gif?v=2",
    "/about"
  ]
}
Result
{
  "user_agent": "Mozilla/5.0 (compatible; OtherBot/1.0)",
  "matched_group": "*",
  "results": [
    {
      "path": "/example/page.html",
      "allowed": false,
      "rule": "Disallow: /example/",
      "line": 3
    },
    {
      "path": "/images/logo.gif",
      "allowed": false,
      "rule": "Disallow: *.gif$",
      "line": 2
    },
    {
      "path": "/images/logo.gif?v=2",
      "allowed": true,
      "rule": null,
      "line": null
    },
    {
      "path": "/about",
      "allowed": true,
      "rule": null,
      "line": null
    }
  ],
  "crawl_delay": null,
  "sitemaps": []
}

Questions

Which rule wins when several match?
The one with the longest path pattern. If an Allow and a Disallow are equally long, Allow wins.
Which group applies to my crawler?
The group whose User-agent name appears in the name you test, the longest such name if there are several. Only if none matches does the * group apply. Groups are never combined with *.
Is the file fetched from my site?
No. Paste its contents. Nothing is downloaded.
Does robots.txt block access?
No. It is a request that well-behaved crawlers follow. It is not access control.