# robots.txt generator

Build a robots.txt: allow or block all crawlers, add your own rules, block AI crawlers and list sitemaps.

- Endpoint: `POST https://setget.net/v1/tools/robots-txt-generate` with JSON `{"input": {...}}`
- MCP: `run_tool` with `slug` = `robots-txt-generate`
- Page: https://setget.net/tools/robots-txt-generate

## Inputs

- `mode` (optional, string, one of `allow_all`, `block_all`, `custom`, default `"allow_all"`): allow_all lets crawlers visit everything, block_all shuts them out, custom uses your rules.
- `rules` (optional, json): For custom mode: a list of {"user_agent": "*", "disallow": ["/admin/"], "allow": ["/admin/help"], "crawl_delay": 5}.
- `block_ai` (optional, boolean, default `false`): Add a rule that blocks common AI crawlers: GPTBot, ClaudeBot, CCBot, Google-Extended, PerplexityBot.
- `sitemaps` (optional, array): Full addresses of your sitemap files.
- `crawl_delay` (optional, number, range 0..3600): Seconds crawlers should wait between requests, for the general rule. Google ignores this.

## Example 1

```json
{"input":{"mode":"allow_all","block_ai":true,"sitemaps":["https://example.com/sitemap.xml"]}}
```

Result:

```json
{"text":"User-agent: *\nDisallow:\n\nUser-agent: GPTBot\nUser-agent: ClaudeBot\nUser-agent: CCBot\nUser-agent: Google-Extended\nUser-agent: PerplexityBot\nDisallow: /\n\nSitemap: https://example.com/sitemap.xml\n","groups":2}
```

## Example 2: Custom rules for two crawlers

```json
{"input":{"mode":"custom","rules":[{"user_agent":"*","disallow":["/admin/","/cart"],"allow":["/admin/help"]},{"user_agent":"Bingbot","disallow":["/"],"crawl_delay":10}]}}
```

Result:

```json
{"text":"User-agent: *\nAllow: /admin/help\nDisallow: /admin/\nDisallow: /cart\n\nUser-agent: Bingbot\nDisallow: /\nCrawl-delay: 10\n","groups":2}
```

**Does robots.txt hide pages from search results?** No. It asks crawlers not to fetch pages. A blocked page can still be listed if other sites link to it. Use a noindex tag or a login to keep a page out of results.

**Do AI crawlers obey it?** The named crawlers say they do, but robots.txt is a request, not a lock.

**Where does the file go?** At the root of the site, for example https://example.com/robots.txt.
