# robots.txt tester

Check which URL paths a robots.txt file allows for a crawler, and which rule decided, following RFC 9309.

- Endpoint: `POST https://setget.net/v1/tools/robots-test` with JSON `{"input": {...}}`
- MCP: `run_tool` with `slug` = `robots-test`
- Page: https://setget.net/tools/robots-test

## Inputs

- `robots` (required, string): The contents of the robots.txt file.
- `user_agent` (required, string): The crawler to test as: a product name such as Googlebot, or a full User-Agent string.
- `paths` (required, array): URL paths (or full URLs) to test.

## Example 1: The example from RFC 9309, as foobot

```json
{"input":{"robots":"User-agent: *\nDisallow: *.gif$\nDisallow: /example/\nAllow: /publications/\n\nUser-agent: foobot\nDisallow: /\nAllow: /example/page.html\nAllow: /example/allowed.gif\n\nSitemap: https://example.com/sitemap.xml","user_agent":"foobot","paths":["/example/page.html","/example/allowed.gif","/","/publications/"]}}
```

Result:

```json
{"user_agent":"foobot","matched_group":"foobot","results":[{"path":"/example/page.html","allowed":true,"rule":"Allow: /example/page.html","line":8},{"path":"/example/allowed.gif","allowed":true,"rule":"Allow: /example/allowed.gif","line":9},{"path":"/","allowed":false,"rule":"Disallow: /","line":7},{"path":"/publications/","allowed":false,"rule":"Disallow: /","line":7}],"crawl_delay":null,"sitemaps":["https://example.com/sitemap.xml"]}
```

## Example 2: Wildcards for any other crawler

```json
{"input":{"robots":"User-agent: *\nDisallow: *.gif$\nDisallow: /example/\nAllow: /publications/","user_agent":"Mozilla/5.0 (compatible; OtherBot/1.0)","paths":["/example/page.html","/images/logo.gif","/images/logo.gif?v=2","/about"]}}
```

Result:

```json
{"user_agent":"Mozilla/5.0 (compatible; OtherBot/1.0)","matched_group":"*","results":[{"path":"/example/page.html","allowed":false,"rule":"Disallow: /example/","line":3},{"path":"/images/logo.gif","allowed":false,"rule":"Disallow: *.gif$","line":2},{"path":"/images/logo.gif?v=2","allowed":true,"rule":null,"line":null},{"path":"/about","allowed":true,"rule":null,"line":null}],"crawl_delay":null,"sitemaps":[]}
```

**Which rule wins when several match?** The one with the longest path pattern. If an Allow and a Disallow are equally long, Allow wins.

**Which group applies to my crawler?** The group whose User-agent name appears in the name you test, the longest such name if there are several. Only if none matches does the * group apply. Groups are never combined with *.

**Is the file fetched from my site?** No. Paste its contents. Nothing is downloaded.

**Does robots.txt block access?** No. It is a request that well-behaved crawlers follow. It is not access control.
