What Is Robots.txt?

What it applies to

Every website. The robots.txt file is one of the first things Googlebot checks when visiting your site — a misconfiguration can prevent the entire site from being indexed.

How robots.txt works

• Located at the root domain: yourdomain.com/robots.txt
• Uses User-agent to specify which crawler the rules apply to (* = all crawlers, Googlebot = Google only).
• Uses Disallow to specify paths crawlers should not access.
• Uses Allow to explicitly permit paths within a disallowed directory.
• Includes a Sitemap directive pointing crawlers to the XML sitemap.

A safe, standard robots.txt for most websites

User-agent: *
Disallow: /admin/
Disallow: /private/
Allow: /

Sitemap: https://yourdomain.com/sitemap.xml

Critical rules

• Never disallow /css/, /js/, or /_next/ — blocking stylesheets and scripts prevents Google from rendering pages correctly, suppressing rankings.
• Never disallow the entire site (Disallow: /) unless the site is genuinely not meant to be indexed (staging environments, intranets).
• Test your robots.txt in Google Search Console → Settings → robots.txt Tester before and after any changes.

In Webflow

Webflow automatically generates a robots.txt file. Customise it via Project Settings → SEO → robots.txt. For staging sites (webflow.io subdomains), Webflow automatically adds a noindex header to prevent staging content from appearing in Google.