A note for the bots at your door
robots.txt is a small text file at the root of your website. Search engines and other crawlers read it before they look around, to learn which areas you would rather they skip, such as your admin area or cart. Well-behaved crawlers follow it. It is a polite request, not a lock.
What it cannot do
- It does not hide private pages. Anyone can still open them, and the file itself is public. Use a password or a login for anything private.
- It does not remove a page from Google. A blocked page can still appear in results if other sites link to it. To keep a page out of Google, use a noindex setting in your SEO plugin instead.
- It does not stop bad bots. Spam and scraper bots usually ignore it.
AI crawlers: block or allow?
AI companies run their own crawlers to read the web. Allowing them means your content can be used and mentioned in AI answers, which can bring visibility. Blocking them keeps your content out of their training, but may also keep your business out of their answers. There is no single right choice. If people already ask AI assistants for businesses like yours, allowing them is usually the better trade.