{ "@context": "https://schema.org", "@type": "FAQPage", "mainEntity": [ { "@type": "Question", "name": "Can robots.txt block a page from ranking?", "acceptedAnswer": { "@type": "Answer", "text": "Disallow prevents crawling but not indexing — use a noindex meta tag to prevent pages from appearing in results." } }, { "@type": "Question", "name": "What happens if robots.txt blocks all crawlers?", "acceptedAnswer": { "@type": "Answer", "text": "The entire site becomes invisible to search engines — a serious SEO mistake." } }, { "@type": "Question", "name": "Where is robots.txt located?", "acceptedAnswer": { "@type": "Answer", "text": "Always at the root: https://yourdomain.com/robots.txt" } } ] }
Definition

What Is Robots.txt?

A text file placed at the root of a domain that instructs search engine crawlers which pages or sections of a site they are allowed or disallowed from crawling.

Related Terms

SEO

Sitemap

Technical SEO

Above the Fold

Accessibility (A11y)

Example

A Webflow site's robots.txt disallows crawling of /staging-preview/ paths to prevent internal drafts from being indexed by Google.

Synonyms:

Robots exclusion protocol, Crawler instructions

Acronyms:

-