robots.txt generator

Build a robots.txt: allow or block all crawlers, add your own rules, block AI crawlers and list sitemaps.

Input

allow_all lets crawlers visit everything, block_all shuts them out, custom uses your rules.
For custom mode: a list of {"user_agent": "*", "disallow": ["/admin/"], "allow": ["/admin/help"], "crawl_delay": 5}.
Add a rule that blocks common AI crawlers: GPTBot, ClaudeBot, CCBot, Google-Extended, PerplexityBot.
Full addresses of your sitemap files.
Seconds crawlers should wait between requests, for the general rule. Google ignores this.

Result

Fill in the input and press Run.

Your input is processed in memory and then discarded. It is not stored or logged. Even so, don't paste live passwords or keys into any online tool.

Use it from code or an agent

curl -s https://setget.net/v1/tools/robots-txt-generate \
  -H 'Content-Type: application/json' \
  -d '{"input":{"mode":"allow_all","block_ai":true,"sitemaps":["https://example.com/sitemap.xml"]}}'
import json, urllib.request

body = {'input': {'mode': 'allow_all', 'block_ai': True, 'sitemaps': ['https://example.com/sitemap.xml']}}
request = urllib.request.Request("https://setget.net/v1/tools/robots-txt-generate", json.dumps(body).encode(),
                                 {"Content-Type": "application/json"})
print(json.load(urllib.request.urlopen(request))["result"])
const response = await fetch("https://setget.net/v1/tools/robots-txt-generate", {
  method: "POST",
  headers: {"Content-Type": "application/json"},
  body: JSON.stringify({"input":{"mode":"allow_all","block_ai":true,"sitemaps":["https://example.com/sitemap.xml"]}}),
});
console.log((await response.json()).result);
Tool: run_tool
Arguments:
{
  "slug": "robots-txt-generate",
  "input": {
    "mode": "allow_all",
    "block_ai": true,
    "sitemaps": [
      "https://example.com/sitemap.xml"
    ]
  }
}
{
  "input": {
    "mode": "allow_all",
    "block_ai": true,
    "sitemaps": [
      "https://example.com/sitemap.xml"
    ]
  }
}

Endpoint: POST /v1/tools/robots-txt-generate. Over MCP: run_tool with slug: "robots-txt-generate". Machine-readable: input schema, Markdown. Connect an agent.

Inputs

NameTypeRequiredWhat it does
mode string no allow_all lets crawlers visit everything, block_all shuts them out, custom uses your rules. One of: allow_all, block_all, custom. Default: "allow_all".
rules json no For custom mode: a list of {"user_agent": "*", "disallow": ["/admin/"], "allow": ["/admin/help"], "crawl_delay": 5}.
block_ai boolean no Add a rule that blocks common AI crawlers: GPTBot, ClaudeBot, CCBot, Google-Extended, PerplexityBot. Default: false.
sitemaps array no Full addresses of your sitemap files.
crawl_delay number no Seconds crawlers should wait between requests, for the general rule. Google ignores this. From 0 to 3600.

Examples

Input
{
  "mode": "allow_all",
  "block_ai": true,
  "sitemaps": [
    "https://example.com/sitemap.xml"
  ]
}
Result
{
  "text": "User-agent: *\nDisallow:\n\nUser-agent: GPTBot\nUser-agent: ClaudeBot\nUser-agent: CCBot\nUser-agent: Google-Extended\nUser-agent: PerplexityBot\nDisallow: /\n\nSitemap: https://example.com/sitemap.xml\n",
  "groups": 2
}

Custom rules for two crawlers

Input
{
  "mode": "custom",
  "rules": [
    {
      "user_agent": "*",
      "disallow": [
        "/admin/",
        "/cart"
      ],
      "allow": [
        "/admin/help"
      ]
    },
    {
      "user_agent": "Bingbot",
      "disallow": [
        "/"
      ],
      "crawl_delay": 10
    }
  ]
}
Result
{
  "text": "User-agent: *\nAllow: /admin/help\nDisallow: /admin/\nDisallow: /cart\n\nUser-agent: Bingbot\nDisallow: /\nCrawl-delay: 10\n",
  "groups": 2
}

Questions

Does robots.txt hide pages from search results?
No. It asks crawlers not to fetch pages. A blocked page can still be listed if other sites link to it. Use a noindex tag or a login to keep a page out of results.
Do AI crawlers obey it?
The named crawlers say they do, but robots.txt is a request, not a lock.
Where does the file go?
At the root of the site, for example https://example.com/robots.txt.