Logo

Robots.txt Generator

Craft SEO-friendly robots.txt files with presets, validation, and live previews.

User-agent Groups

Allow rules
Disallow rules

No disallow rules. All paths are currently crawlable.

Custom directives

Add "Noindex: /path" or other vendor-specific directives.

Global settings

Optional and mostly used by Yandex. Leave blank for most sites.

Sitemaps

Add absolute URLs for each sitemap document.

Useful for vendor-specific fields like Clean-param or custom comments.

No global directives. Add one to append raw lines at the end.

Presets & automation

robots.txt preview

# robots.txt
# Generated with the Utekar.com 's Robots.txt Generator Tool. Update contact and sitemap URLs.
# Generated: 5/11/2025, 1:26:09 pm

# Default rules for all user agents
User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

Diagnostics

No issues detected. Your robots.txt looks healthy.

Groups

1

Allow rules

1

Disallow rules

0

Sitemaps

1

Custom directives

0

Total lines

7

Building an Effective robots.txt

Use this generator to produce high quality robots.txt files that respect search engine guidelines, protect private areas, and document your crawl strategy.

Core Concepts

  • User-agent: Target specific crawlers with tailored rules. The wildcard * applies to all bots.
  • Allow/Disallow: Grant or deny access to URL patterns. Keep rules specific to avoid unwanted blocking.
  • Crawl-delay: Slow aggressive crawlers if your infrastructure needs breathing room.
  • Sitemap: Always expose sitemap URLs so bots can discover content you want indexed.

Best Practices

  • Keep critical resources such as CSS and JS accessible so search engines can render pages accurately.
  • Pair robots.txt rules with on-page meta directives or HTTP headers for precise indexing control.
  • Document every rule with comments so future maintainers understand the rationale.
  • Stage changes and verify the server actually returns HTTP 200 with text/plain content.

How search engines interpret robots.txt

  • The file is cached aggressively; announce changes by updating the modified timestamp or pinging search engines.
  • Rules are evaluated top-to-bottom within a group, but the most specific match (longest path) wins.
  • Allow rules can reopen URLs blocked by broader Disallow directives.
  • Google respects Crawl-delay only for select crawlers; Bing and Yandex use it more reliably.

Advanced Tips

  • Create dedicated groups for specialised bots like Googlebot-News or AdsBot-Google so they can crawl at the right cadence.
  • Use the global custom directives section for vendor-specific lines like Clean-param or Request-rate.
  • Export and version-control your robots.txt so you can track historical changes.
  • Remember that robots.txt is publicly accessible—do not place secrets here. Use authentication for true protection.

Deployment checklist

  • Serve the file at the root: https://example.com/robots.txt.
  • Ensure line endings are UNIX (LF) to avoid stray carriage returns.
  • Validate absolute URLs for sitemaps and confirm they return 200 responses.
  • Retest with Google Search Console → Robots Testing Tool before going live.

Common mistakes to avoid

  • Blocking / or /* unintentionally, which prevents all crawling.
  • Trying to hide sensitive files—robots.txt is public and only offers crawl guidance.
  • Using relative sitemap URLs; most crawlers require absolute paths.
  • Forgetting to update rules after migrating or renaming directories.