Robots.txt File Settings
Squirrly SEO serves your robots.txt from Squirrly > Technical SEO > Tweaks & Sitemaps > Robots File, with your sitemaps listed at the top and the crawl rules you set below them. No file is written to the server, so it also works on WordPress Multisite.
The screenshots come from demo.squirrly.co, a test store for cat food, with Squirrly SEO 14.2.8 and the Advanced Pack 2.5.11.
Turn on the Robots File and find it
- Go to Squirrly > All Features.
- On the Robots.txt File card, click the switch if it reads click to activate.
- Go to Squirrly > Technical SEO > Tweaks & Sitemaps and click Robots File in the left column of the panel.
The card is off on a new install. The setup wizard turns it on when you pick SEO Recommended Mode or SEO Expert Mode. With the card off, WordPress serves its own robots.txt.
Default robots.txt rules
With the Edit the Robots.txt data box empty, Squirrly serves these rules:
User-agent: *
Disallow: */trackback/
Disallow: */xmlrpc.php
Disallow: /wp-*.php
Disallow: /cgi-bin/
Disallow: /wp-admin/
Allow: */wp-content/uploads/Above the rules, Squirrly adds one “Sitemap:” line for every active sitemap from Sitemap XML. On the demo site that is seven lines, from https://demo.squirrly.co/sitemap.xml to sitemap-archives.xml.
If Discourage search engines from indexing this site is ticked in Settings > Reading, the sitemap lines are replaced by “# Your blog is not public. Please see Site Visibility on Settings > Reading.”
Write your own robots.txt rules
The red note under the box says “Edit the Robots.txt only if you know what you’re doing. Adding wrong rules in Robots can lead to SEO ranking errors or block your posts in Google.”
- Go to Squirrly > Technical SEO > Tweaks & Sitemaps > Robots File.
- Type your rules in Edit the Robots.txt data, one per line. To start from the defaults, copy the block above.
- Click Save Settings.
- Open the robots.txt link under the title and check the result.
What you save replaces the default rules, and the “Sitemap:” lines stay on top. Empty the box and save to go back to the defaults.
The plugin readme names GPTBot, ClaudeBot, PerplexityBot and Google-Extended as AI bots you can control here. This example rule, added under your other rules, asks GPTBot to stay off the whole site:
User-agent: GPTBot
Disallow: /For one page or one post type, noindex in the Automation or the SEO Snippet is the cleaner tool. A page blocked in robots.txt can’t be crawled, so search engines never see its noindex tag.
If a real robots.txt file is on the server
A robots.txt file in the WordPress root folder is sent by the web server before WordPress loads. When you click Save Settings on the Robots File tab, Squirrly renames that file to robots_ren followed by a timestamp, so its own version starts to show. The old file stays on the server under the new name.
Troubleshooting
The Robots File tab is missing
The Robots.txt File card is off on All Features. Switch it on and reload Tweaks & Sitemaps.
robots.txt doesn’t show my changes
Purge the page cache and CDN cache for /robots.txt. If a physical file came back, save the Robots File tab again.
No “Sitemap:” lines in robots.txt
Turn on the Sitemap XML card, or untick Discourage search engines from indexing this site in Settings > Reading.
Frequently asked questions
Where is my robots.txt?
At https://yourdomain.com/robots.txt, linked under the Robots File title.
Does Squirrly add my sitemap to robots.txt?
Yes, every active Squirrly sitemap gets a “Sitemap:” line automatically.
Is robots.txt the same as llms.txt?
No, llms.txt is a separate file for AI crawlers, set on the LLMs File tab next to this one.
Related tutorials
- Sitemap XML Settings: the sitemaps listed in robots.txt.
- LLMs.txt File Settings: rules for AI crawlers and language models.
- Automation: noindex and nofollow per post type.
- Add Custom Noindex Meta: noindex for a single page.
