Logo

Robots.txt Generator

Craft SEO-friendly robots.txt files with presets, validation, and live previews.

User-agent Groups

Allow rules
Disallow rules

No disallow rules. All paths are currently crawlable.

Custom directives

Add "Noindex: /path" or other vendor-specific directives.

Global settings

Optional and mostly used by Yandex. Leave blank for most sites.

Sitemaps

Add absolute URLs for each sitemap document.

Useful for vendor-specific fields like Clean-param or custom comments.

No global directives. Add one to append raw lines at the end.

Presets & automation

robots.txt preview

# robots.txt
# Generated with the Utekar.com 's Robots.txt Generator Tool. Update contact and sitemap URLs.
# Generated: 24/11/2025, 7:53:59 pm

# Default rules for all user agents
User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

Diagnostics

No issues detected. Your robots.txt looks healthy.

Groups

1

Allow rules

1

Disallow rules

0

Sitemaps

1

Custom directives

0

Total lines

7

Related Tools

Open Graph Generator

Generate Open Graph meta tags for optimal social media sharing on Facebook, Twitter, LinkedIn, and other platforms

Keyword Density Checker

Analyze keyword and phrase frequencies with n-grams, stopword filtering, highlighting, and export options

Click Through Rate (CTR) Calculator

Calculate CTR with Wilson confidence intervals, A/B significance, sample size, and bulk campaign analysis

Meta Tag Generator

Generate SEO, Open Graph, Twitter, verification, and structured data meta tags with live previews and exports

Building an Effective robots.txt

Use this generator to produce high quality robots.txt files that respect search engine guidelines, protect private areas, and document your crawl strategy.

Core Concepts

  • User-agent: Target specific crawlers with tailored rules. The wildcard * applies to all bots.
  • Allow/Disallow: Grant or deny access to URL patterns. Keep rules specific to avoid unwanted blocking.
  • Crawl-delay: Slow aggressive crawlers if your infrastructure needs breathing room.
  • Sitemap: Always expose sitemap URLs so bots can discover content you want indexed.

Best Practices

  • Keep critical resources such as CSS and JS accessible so search engines can render pages accurately.
  • Pair robots.txt rules with on-page meta directives or HTTP headers for precise indexing control.
  • Document every rule with comments so future maintainers understand the rationale.
  • Stage changes and verify the server actually returns HTTP 200 with text/plain content.

How search engines interpret robots.txt

  • The file is cached aggressively; announce changes by updating the modified timestamp or pinging search engines.
  • Rules are evaluated top-to-bottom within a group, but the most specific match (longest path) wins.
  • Allow rules can reopen URLs blocked by broader Disallow directives.
  • Google respects Crawl-delay only for select crawlers; Bing and Yandex use it more reliably.

Advanced Tips

  • Create dedicated groups for specialised bots like Googlebot-News or AdsBot-Google so they can crawl at the right cadence.
  • Use the global custom directives section for vendor-specific lines like Clean-param or Request-rate.
  • Export and version-control your robots.txt so you can track historical changes.
  • Remember that robots.txt is publicly accessible—do not place secrets here. Use authentication for true protection.

Deployment checklist

  • Serve the file at the root: https://example.com/robots.txt.
  • Ensure line endings are UNIX (LF) to avoid stray carriage returns.
  • Validate absolute URLs for sitemaps and confirm they return 200 responses.
  • Retest with Google Search Console → Robots Testing Tool before going live.

Common mistakes to avoid

  • Blocking / or /* unintentionally, which prevents all crawling.
  • Trying to hide sensitive files—robots.txt is public and only offers crawl guidance.
  • Using relative sitemap URLs; most crawlers require absolute paths.
  • Forgetting to update rules after migrating or renaming directories.