How to Create a robots.txt file for SEO Success

When a website is in staging or development, it should be restricted from being crawled and indexed by the search engines. This avoids prematurely ranking in the search results, and potentially having duplicate content competing and cannibalising with a live website’s traffic and rankings. (1)

Staging servers typically lack the performance of the production (live) site , and websites still in development are generally more fragile as well.

Various methods are used to block search engines from crawling a staging site, or to avoid content from being indexed, including these 3 methods:

1. Putting it behind a password-protected login – the most straightforward effective way to block all search engine crawlers and external viewers.
2. Adding a noindex tag – adding a no index meta tag to the code of a webpage to block specific pages from being indexed, hiding the content from users searching on the web. (1)

This is done by adding a meta tag within the  section of the HTML code on the page, typically looking like this:

    

   <meta name=”robots” content=“noindex”>
   
   The downside is that while “noindex” prevents indexing, it does not completely block crawlers from accessing the page. This will still allow Google to crawl the page, but it will not appear in the index. To stop Google from crawling the page completely, you should disallow it through using robots.txt

3. Using robots.txt – a plain text file that instructs web crawlers on how to interact with a website. At it’s most basic format, to block all search engines from crawling a site, the file would include these lines of code: (2)

However, you’ll find that using the above in a robots.txt file will also block staging sites from SEO audit tools. It’s essential to be able to audit sites in staging environments before they go live, which is where creating and optimising a robots.txt file can help. (3)

The robots.txt file, also known as the robots exclusion protocol or standard, is a text file that tells web robots (most often search engines) which pages web robots can and cannot to crawl on your site.

This stops the outside world from viewing site that’s incomplete or reveals improvements and plans that your competitors might gain insights from.

Whenever a search engine is about to visit a site it will check the robots.txt for instructions.
before it visits the target page. (4)

The robots.txt file is usually located in the root file directory of a website’s domain. For example www.yourdomain.com/robots.txt

Important Note: your staging site robots.txt will likely be in its own root directory to the live website’s domain which is what we’ll be editing in context of this article.

You still want to use site audit tools like Screaming Frog, Chrome Lighthouse and others
to be able to crawl and audit a site before going live.

You can create a custom robots.txt file specifying which search engines to block with handy tools like this Robots txt Generator

Here is a comprehensive list from Chat GPT of the codes of most search engine crawlers if you want to block all or specific search engines from crawling:

The above ensures all major search engines and modern crawlers are blocked. The User-agent: * line at the end ensures any other unspecified crawler not explicitly listed is also blocked.

If you want your staging site to be crawlable only by SEO audit tools (e.g., Screaming Frog or Semrush), while blocking all other search engines from indexing it, we need to explicitly allow those tools and disallow all others.

The above ensures Screaming Frog SEO Spider and SemrushBot are explicitly allowed with the Allow: / directive. If you have other SEO tools you want to allow, add them similarly (e.g., AhrefsBot or Moz’s crawler).

Optional Sitemap for Auditors:
If your SEO tools rely on the sitemap for a full crawl, you can include the sitemap URL

Sitemap: https://yourwebsite.com/sitemap_index.xml
If not necessary, leave it out to avoid accidentally signaling crawlers to index the site.

Chrome’s Lighthouse (part of DevTools) and similar tools don’t explicitly identify themselves with unique User-agent like “Screaming Frog” or “SemrushBot.” Instead, Lighthouse typically uses Chrome’s own user-agent, which often mimics a standard browser or a generic Googlebot-like crawler. In other words, Lighthouse mimics a real Chrome browser. The Lighthouse User agent may vary slightly depending on the version of Chrome or the environment in which Lighthouse is running. The standard browser User agent for Chrome Lighthouse in robots.txt may look like this:

A User agent is a snippet of text that uniquely identifies the software or browser (the “Agent”) that is making the request to a website. (5) Using Network Tab in DevTools:

1. Open Chrome DevTools (Ctrl + Shift + I or Cmd + Option + I on Mac).
2. Go to the Network tab.
3. Run Lighthouse.
4. Look for a network request made to your server (e.g., robots.txt or the homepage) and inspect its headers.
5. In the headers, check for the User-Agent field.

If your SEO audit tool is still blocked: If you can’t pinpoint the exact User-agent string for your SEO tools, consider temporarily relaxing the rules while auditing by modifying the User-agent: * section to: User-agent: * Allow: / This would allow unrestricted crawling during your audits. After the audit, revert back to stricter rules. It is highly recommended as an extra precautionary measure to enable a password login when you’re not running SEO audit tools or working on the staging site. Understanding the User agents for search engines and tools is critical for configuring a robots.txt file effectively.

Below is a list of the most widely used search engines and their user agents:

Below is a list of the most widely used SEO audit tools and their user agents:

• The whole point of your robots.txt setup is to block Googlebot and other search engines from indexing the site.
• In a staging environment, you’re blocking Googlebot intentionally. When your website is in staging or development, it must be hidden from search engines to prevent accidental indexing. This avoids duplicate content competing with your live site, protecting your traffic and rankings.
• An effective way to block search engines from staging sites is the use a robots.txt file – a file that instructs crawlers not to visit specific web pages or an entire site.
• Blocking crawlers using robots.txt also prevents SEO tools (like Screaming Frog or Lighthouse) from accessing the site.
  To allow SEO audits while keeping search engines out, configure robots.txt to allow trusted tools like Screaming Frog and Semrush and block others.
• Example robots.txt for Staging Sites:

• In conjunction to using robots.txt, it is recommended to enable site password protection, which is the simplest and most effective method to block all crawlers and users, whenever the site is not being audited or modified.
• By effectively managing your staging environment, you ensure privacy, prevent leaks, and prepare for a seamless site launch.

A tailored SEO strategy boosts your visibility, drives qualified traffic, builds credibility, and helps you outpace competitors—all while improving user experience and conversion rates. Take the first step with our custom SEO packages, designed to enhance your site’s visibility and performance. Let’s work together to create a website that attracts, engages, and converts.

1. How to keep your staging or development site out of the index, Search Engine Land
2. What Robots.Txt Is & Why It Matters for SEO, Semrush Blog
3. How To Crawl A Staging Website , Screaming Frog
4. How to Create the Perfect Robots.txt File for SEO , Neil Patel
5. What is a User Agent?,What is my browser?