Is web scraping legal? What 67 top sites ask for in robots.txt
We asked 100 top-ranked domains for robots.txt on 2026-11-16. Of the 67 that served one, 13 hold Disallow: / for unnamed crawlers and 29 name an AI crawler.
On November 16, 2026 we asked 100 of the highest-ranked domains on the public Tranco list for their robots.txt, the one file where a site writes down what it asks of crawlers. 67 served one. Of those 67, 13 hold Disallow: / for any crawler they do not name, 29 name at least one AI crawler, 9 set a crawl delay and 48 point to a sitemap.
If you came here asking whether web scraping is legal, this page does one thing for you. It shows what sites ask for in the one file a script can read, the same count that our longer explainer on scraping sets beside fetching and cost. The legal answer depends on the jurisdiction and on facts this guide does not assess. This is not legal advice, and no number below says what is lawful for you.
What the 67 files say
| What the file does | Sites | n |
|---|---|---|
| Declares a sitemap | 48 | 67 |
| Names at least one AI crawler from our list of 40 | 29 | 67 |
Holds Disallow: / for at least one AI crawler | 17 | 67 |
Holds Disallow: / in its User-agent: * group | 13 | 67 |
The same, with no Allow line in that group | 10 | 67 |
| Sets a crawl delay for any agent | 9 | 67 |
| Sets a crawl delay for all agents | 6 | 67 |
Counted from files fetched between 02:28 and 02:56 UTC on 2026-11-16. The denominator is the 67 domains, of 100 asked, that served a robots file at the bare name or at www.
The median file is 2,005 bytes. The smallest is discord.gg at 23 bytes, which is two lines, User-agent: * and Allow: /. The largest is linkedin.com at 120,235 bytes. 23 of the 67 have a single group of rules, and gandi.net has 212.
13 files tell unnamed crawlers to stay out, and 3 of them add exceptions
The User-agent: * group is the one a crawler falls back to when no group names it. In 13 of the 67 files that group holds the line Disallow: /. The 13 are facebook.com, instagram.com, twitter.com, x.com, linkedin.com, netflix.com, pinterest.com, baidu.com, reddit.com, mail.ru, chatgpt.com, myfritz.net and registrar-servers.com. twitter.com and x.com returned the same bytes, so they are two domains with one file.
Ten of the 13 have no Allow line in that group. The other three do. mail.ru has 6, myfritz.net has 8 and chatgpt.com has 178, and in all three one of them is Allow: /$, which opens the homepage and nothing under it.
We did not trust our own parser on this, so we checked it against Python’s urllib.robotparser. We asked it whether an agent that no file names may fetch / and one made-up deep path. It refused both on the 10 files and refused only the deep path on the other three. On all 67 files it gave the same two answers as our own rule matcher.
Only reddit.com and registrar-servers.com stop at that one group. The other 11 also give named crawlers their own groups, and we did not work out what each of those allows. “Disallows everything” here describes the fallback group. It does not describe the whole site.
29 files name an AI crawler, and 12 of those hold no full disallow for one
Before the run we wrote down 40 user agent names that say they collect for an AI model or fetch on behalf of one. 29 of the 67 files name at least one. msn.com and snapchat.com each name 36.
| AI crawler name | Files that name it | Files that hold Disallow: / for it | n |
|---|---|---|---|
| Google-Extended | 23 | 13 | 67 |
| PerplexityBot | 22 | 12 | 67 |
| GPTBot | 20 | 12 | 67 |
| ClaudeBot | 19 | 13 | 67 |
| CCBot | 16 | 12 | 67 |
| ChatGPT-User | 15 | 7 | 67 |
| OAI-SearchBot | 15 | 4 | 67 |
| Bytespider | 13 | 12 | 67 |
17 of the 67 files hold Disallow: / for at least one name on the list. That leaves 12 files that name an AI crawler and hold Disallow: / for none of them. Naming a crawler is not the same as turning it away. cloudflare.com is the clearest case. It gives eight of these crawlers a group each, and each group reads Allow: /.
Crawl delays are rare and sitemaps are common
Nine files set a Crawl-delay for some agent, and six set one for all agents: twitter.com, x.com, vimeo.com, tumblr.com, apache.org and europa.eu. The values run from 0.1 seconds to 10. europa.eu asks for the 10. Crawl-delay is not part of the standard described below. It is an older extension, and a crawler may or may not read it.
Sitemaps go the other way. 48 of the 67 files declare at least one, so for most of these sites the file that says where not to go also says where the pages are.
The file that surprised us
We expected a file that holds Disallow: / for unnamed crawlers to be short. chatgpt.com’s is 4,573 bytes, and its fallback group is built the other way round. It lists 178 Allow lines first and puts Disallow: / after them, with a comment explaining the order:
# General rules for all other bots
User-agent: *
# Allow Googlebot to render the indexable homepage; production CSS and JavaScript live here.
Allow: /unauth-mweb/assets/
# Place allows first to avoid bots skipping after Disallow: /
# Allow exactly the homepage
Allow: /$
# Allow the homepage with any query parameters
Allow: /?*
[175 more Allow lines]
# Now block everything else
Disallow: /
[5 more Disallow lines]This is the content field of the response, trimmed where the brackets say so. A script that greps for Disallow: / files this site with reddit.com. A parser that applies the rules does not. That gap is why the table above has two rows for the same line.
The standard says the file is a request
A robots.txt is a published request. The site owner writes rules, puts them at a known address, and crawlers that follow the protocol read them before they crawl.
The protocol dates to 1994 and was published as RFC 9309 in September 2022. A few of its rules explain the counts above:
- A crawler obeys the group that names it. The
*group applies only when no other group matches. - When several rules match a path, the longest one wins. If an allow and a disallow are the same length, the crawler should use the allow.
- The robots.txt address itself is always allowed, which is how a survey like this one can read a file that disallows everything.
- On caching it says, “Crawlers SHOULD NOT use the cached version for more than 24 hours, unless the robots.txt file is unreachable.”
The RFC is also plain about what the file is not. Its introduction says, “These rules are not a form of access authorization.” The file states what the owner asks for.
Whether following a file, or ignoring one, carries legal weight is a different question. The answer depends on the jurisdiction and on facts this guide does not assess, such as a site’s terms, the kind of data and how it is used. Our earlier post on web scraping for AI training data covers the law on that one use.
What we would do with a file before the first request
Read the file before the first page, and read the group that applies to your agent. 29 of these 67 files name an AI crawler, so the * group may not be yours.
Apply the rules, do not grep them. Three of the 13 files that hold Disallow: / in the fallback group also allow the homepage.
If the file sets a crawl delay, pace to it. On Spider that is the delay option, which the API reference describes as “Milliseconds to wait between requests, up to 60000. Turns off concurrency.”
If the file declares a sitemap, start there. The same reference lists a sitemap option, “Add pages from the sitemap to the crawl. Default false.”
Leave respect_robots alone. The reference describes it as “Follow robots.txt. Default true.” None of the requests in this survey set it.
Then read the site’s terms and the licence on the content. robots.txt is one statement of what a site asks for, and it is the only one a script can count. When we built a fine-tuning dataset from one documentation site, robots.txt was one of three checks.
Repeat it
One request per domain, sent to the same endpoint as the keyless scraper tool, with the key in an environment variable:
curl -s https://api.spider.cloud/scrape \
-H "Authorization: Bearer $SPIDER_API_KEY" \
-H "Content-Type: application/json" \
-d '{"url": "https://github.com/robots.txt", "request": "http", "return_format": "raw", "metadata": true}'The file comes back as text in the content field.
The rule that picked the sites was written before we saw the list. Take Tranco list JZWZY in rank order. Skip a name when it matches a pattern for adult sites, for names that serve DNS, CDN, ad or API traffic, or for sign-in addresses. Stop at 100 kept names. That skipped 59, and the 100th kept name sits at rank 159. Each kept name got one request for https://<domain>/robots.txt. Where the answer was not a robots file, it got one more at https://www.<domain>/robots.txt, which added two sites.
A site is in the 67 when either address returned status 200 with at least one User-agent or Sitemap line and no HTML page in its place. The other 33 names are in no count. Some answered that address with a web page. Some are names the patterns should have skipped, because nobody visits them in a browser.
What the survey does not show
It does not show what is legal, what any site’s terms say, or whether a site enforces its file.
It does not describe the web. It is 67 files from one ranking on one day. Tranco ranks domain names by popularity, and a popular name is not always a site people read.
The AI list was fixed before the run, so a name we did not think of is not counted. Six of the 67 files have a group for __usage_ai-training, which was not on our list.
Each file was fetched once. A www address can redirect to another name, and we counted the file that came back. Files change, so a count from next month will differ.
Run the request against the ten sites you crawl most and read what comes back before your next job. Create an API key to start.
Frequently asked questions
Is web scraping legal?
It depends on the jurisdiction and on facts this guide does not assess, such as a site's terms and the kind of data involved. What can be counted is what sites ask for. On 2026-11-16, 67 of 100 top-ranked domains served a robots.txt, and 13 of those 67 hold Disallow: / for every crawler they do not name.
Is robots.txt legally binding?
This guide does not assess that. It depends on jurisdiction and facts. RFC 9309 defines the file and says nothing about law.
How many large sites disallow AI crawlers in robots.txt?
In this survey, run on 2026-11-16, 29 of 67 files name at least one AI crawler from a list of 40 names fixed before the run, and 17 of 67 hold Disallow: / for at least one of them. Google-Extended is the name that appears most, in 23 files.
Does Spider follow robots.txt?
The API reference lists respect_robots as a boolean described as "Follow robots.txt. Default true." The requests in this survey did not set it.
Sources
- Measured run, 2026-11-16, 02:28 to 02:56 UTC: one request for robots.txt to each of 100 domains, and one more at www where the bare name served no robots file, through the Spider scrape endpoint
- Tranco list JZWZY, the ranking the 100 domains came from
- RFC 9309, Robots Exclusion Protocol, September 2022
- Spider API reference, the respect_robots, delay and sitemap options
- Python urllib.robotparser, the second parser used to check our reading
Keep reading
How web scraping works: what a plain fetch gets and misses
How web scraping works, measured on November 16, 2026. A plain HTTP fetch returned 90% or more of a browser's words on 26 of 29 docs, news and product pages.
Web scraping vs web crawling, timed on one documentation site
Web scraping vs web crawling, measured on November 16, 2026. One crawl call returned 74 pages in 5.1 s. A scrape call per URL returned the same markdown.
Scraping 1 million pages: what actually happens
An engineering log of crawling 1 million pages across 10,000 domains with Spider's API. Throughput curves, failure modes, cost breakdown, and lessons learned.
Run this on a page you care about
The playground sends the request this page describes and shows you the response. Keyless runs work without an account and are rate limited.