robots.txt generator
Build a robots.txt: allow or block all crawlers, add your own rules, block AI crawlers and list sitemaps.
Input
Result
Fill in the input and press Run.
Your input is processed in memory and then discarded. It is not stored or logged. Even so, don't paste live passwords or keys into any online tool.
Use it from code or an agent
curl -s https://setget.net/v1/tools/robots-txt-generate \
-H 'Content-Type: application/json' \
-d '{"input":{"mode":"allow_all","block_ai":true,"sitemaps":["https://example.com/sitemap.xml"]}}'
import json, urllib.request
body = {'input': {'mode': 'allow_all', 'block_ai': True, 'sitemaps': ['https://example.com/sitemap.xml']}}
request = urllib.request.Request("https://setget.net/v1/tools/robots-txt-generate", json.dumps(body).encode(),
{"Content-Type": "application/json"})
print(json.load(urllib.request.urlopen(request))["result"])
const response = await fetch("https://setget.net/v1/tools/robots-txt-generate", {
method: "POST",
headers: {"Content-Type": "application/json"},
body: JSON.stringify({"input":{"mode":"allow_all","block_ai":true,"sitemaps":["https://example.com/sitemap.xml"]}}),
});
console.log((await response.json()).result);
Tool: run_tool
Arguments:
{
"slug": "robots-txt-generate",
"input": {
"mode": "allow_all",
"block_ai": true,
"sitemaps": [
"https://example.com/sitemap.xml"
]
}
}
{
"input": {
"mode": "allow_all",
"block_ai": true,
"sitemaps": [
"https://example.com/sitemap.xml"
]
}
}
Endpoint: POST /v1/tools/robots-txt-generate. Over MCP: run_tool with slug: "robots-txt-generate".
Machine-readable: input schema, Markdown. Connect an agent.
Inputs
| Name | Type | Required | What it does |
|---|---|---|---|
mode |
string | no | allow_all lets crawlers visit everything, block_all shuts them out, custom uses your rules. One of: allow_all, block_all, custom. Default: "allow_all". |
rules |
json | no | For custom mode: a list of {"user_agent": "*", "disallow": ["/admin/"], "allow": ["/admin/help"], "crawl_delay": 5}. |
block_ai |
boolean | no | Add a rule that blocks common AI crawlers: GPTBot, ClaudeBot, CCBot, Google-Extended, PerplexityBot. Default: false. |
sitemaps |
array | no | Full addresses of your sitemap files. |
crawl_delay |
number | no | Seconds crawlers should wait between requests, for the general rule. Google ignores this. From 0 to 3600. |
Examples
Input
{
"mode": "allow_all",
"block_ai": true,
"sitemaps": [
"https://example.com/sitemap.xml"
]
}Result
{
"text": "User-agent: *\nDisallow:\n\nUser-agent: GPTBot\nUser-agent: ClaudeBot\nUser-agent: CCBot\nUser-agent: Google-Extended\nUser-agent: PerplexityBot\nDisallow: /\n\nSitemap: https://example.com/sitemap.xml\n",
"groups": 2
}Custom rules for two crawlers
Input
{
"mode": "custom",
"rules": [
{
"user_agent": "*",
"disallow": [
"/admin/",
"/cart"
],
"allow": [
"/admin/help"
]
},
{
"user_agent": "Bingbot",
"disallow": [
"/"
],
"crawl_delay": 10
}
]
}Result
{
"text": "User-agent: *\nAllow: /admin/help\nDisallow: /admin/\nDisallow: /cart\n\nUser-agent: Bingbot\nDisallow: /\nCrawl-delay: 10\n",
"groups": 2
}Questions
- Does robots.txt hide pages from search results?
- No. It asks crawlers not to fetch pages. A blocked page can still be listed if other sites link to it. Use a noindex tag or a login to keep a page out of results.
- Do AI crawlers obey it?
- The named crawlers say they do, but robots.txt is a request, not a lock.
- Where does the file go?
- At the root of the site, for example https://example.com/robots.txt.
Related tools
- XML sitemap generatorBuild a valid XML sitemap from a list of page addresses, with optional last-modified date, frequency and priority.
sitemap-generate - Redirect rules generatorTurn a list of old and new addresses into redirect rules for Apache .htaccess, Nginx, or a Netlify _redirects file.
htaccess-redirects - .gitignore generatorBuild a .gitignore for the languages, editors and operating systems you use, merged without duplicates.
gitignore-generate - ASCII banner text generatorTurn short text into large five-line banner letters for terminals, READMEs, logs and code comments.
ascii-banner - ASCII table generatorRender headers and rows as a box-drawing or plain ASCII table for terminals, logs and code comments.
ascii-table - Background pattern generatorMake seamless SVG background patterns (dots, grid, stripes, checker or waves) with your colours and scale.
pattern-image