Robots.txt Generator

Create custom robots.txt directives to instruct search engine crawlers which pages to index or ignore.

Crawl Directives & Rules

Default User-Agent

Default Crawl Action

XML Sitemap URL

Crawl Delay (Seconds)

Disallowed Directories (One per line) Paths bots shouldn't crawl

Allowed Directories (One per line) Exceptions to disallow

Generated robots.txt

Live Output

Quick Presets

Where Should I Place the robots.txt File?

Your robots.txt file must always be uploaded to the root directory of your web server (e.g. https://example.com/robots.txt). Search engine spiders check this exact location before indexing any pages on your domain.

Robots.txt Generator

Crawl Directives & Rules

Generated robots.txt

Where Should I Place the robots.txt File?

Related Tools