Free tool

Robots.txt Generator

Build a clean robots.txt in a minute. Start from a WordPress, WooCommerce or Shopify preset, add your sitemap, decide about AI crawlers and copy the file.

  • 100% free
  • No sign-up
  • Runs in your browser

A note for the bots at your door

robots.txt is a small text file at the root of your website. Search engines and other crawlers read it before they look around, to learn which areas you would rather they skip, such as your admin area or cart. Well-behaved crawlers follow it. It is a polite request, not a lock.

What it cannot do

  • It does not hide private pages. Anyone can still open them, and the file itself is public. Use a password or a login for anything private.
  • It does not remove a page from Google. A blocked page can still appear in results if other sites link to it. To keep a page out of Google, use a noindex setting in your SEO plugin instead.
  • It does not stop bad bots. Spam and scraper bots usually ignore it.

AI crawlers: block or allow?

AI companies run their own crawlers to read the web. Allowing them means your content can be used and mentioned in AI answers, which can bring visibility. Blocking them keeps your content out of their training, but may also keep your business out of their answers. There is no single right choice. If people already ask AI assistants for businesses like yours, allowing them is usually the better trade.

FAQ

Questions people ask

Where do I put the file?

At the root of your domain, so it opens at yourdomain.com/robots.txt. On WordPress, many SEO plugins let you edit it from the dashboard without uploading anything.

What is the most common mistake?

A leftover "Disallow: /" from when the site was being built. That one line asks every search engine to skip the whole site.

Should I block CSS or JavaScript files?

No. Google needs them to see your page the way visitors do. Blocking them can hurt how your pages are understood.

Does every website need a robots.txt?

No, a site without one is crawled normally. Having one is still useful, mainly to point search engines to your sitemap and to skip areas like the cart or admin.

What does "User-agent: *" mean?

The star means "every crawler". Rules under it apply to all bots, unless a later section names a specific bot with its own rules.

What is the difference between Disallow and Allow?

Disallow asks bots to skip a folder or page. Allow makes an exception inside a blocked folder, for example one file that should still be read.

Should I add my sitemap to robots.txt?

Yes. A sitemap line helps every search engine find all your pages, not just the ones you submitted in Google Search Console.

Does Shopify let me edit robots.txt?

Yes, through a special template file in your theme. Shopify's own default is sensible, so only change it when you have a clear reason.

How do I test my robots.txt?

Open yourdomain.com/robots.txt to check it is live, then use the robots.txt report in Google Search Console to see how Google reads it.

Can you check my robots.txt and sitemap for me?

Yes. They are part of my free website audit, along with speed, SEO basics and how clear your pages are.

Free website audit

Is your website costing you customers?

Get an instant score for design, SEO, content and speed, plus a PDF report with clear fixes. It takes about 60 seconds.

  • 100% free
  • No sign up
  • Results in 60 seconds

Rather talk to a human? Message me on WhatsApp

Chat with me on Google Chat

Copy my chat address below, then start a new chat with it in Google Chat. I usually reply within minutes.

devnabiakhan@gmail.com
Copy and open chat
  1. Open Google Chat
  2. Tap "New chat"
  3. Paste my address and say hi
Ask us