# Learn web scraping: a library and a short course | Spider

Short pages that each answer one web scraping question, and a course to read in order. Every page lists its sources and the day we read them.

Canonical: https://spider.cloud/learn/

Each page here answers one question about fetching and reading web pages with a program. The answer comes first. Under it you get a request to paste, and the sources we read with the day we read them.

For the whole subject on one page, read the [web scraping guide](/web-scraping/).

One visit, line by line

1. Before the first page

   1. Client sends: `GET /robots.txt` [Read the site's rules before any page](/learn/robots-txt/)
   2. Server answers: `Disallow: /search` No server enforces this rule, so keeping it is up to you

2. The first request

   1. Client sends: `GET /products/42` [Send one request and read the answer](/learn/course/your-first-request/)
   2. Client sends: `User-Agent: examplebot/1.0` [Say who is calling](/learn/user-agent/)
   3. Server answers: `503 Service Unavailable` [Read the status code before the body](/learn/http-status-codes/)
   4. Server answers: `Retry-After: 120` Wait as long as the server asks

3. 120 seconds later

   1. Client sends: `GET /products/42` [Ask again once the wait is over](/learn/request-rate/)
   2. Server answers: `200 OK` Check that the body is the page you wanted
   3. Server answers: `ETag: "v7"` Keep the tag for the next visit

4. The next day

   1. Client sends: `If-None-Match: "v7"` Send the tag back to ask what changed
   2. Server answers: `304 Not Modified` Nothing changed, so there is nothing to download

Each underlined note opens the page for its line. The exchange is an illustration, not a recorded run. Its field names and status codes come from RFC 9110, and the robots.txt rule from RFC 9309.

## Web scraping, step by step

A short course on fetching web pages with care. Each lesson has one request you can paste and run.

1 of 16 lessons published

1. [01 Your first request: fetch one page and read the answer A scraper starts with one HTTP request. What goes out, what comes back, and how to read the status code before you trust the page. 5 min](https://spider.cloud/learn/course/your-first-request/)

[Start with lesson 1](https://spider.cloud/learn/course/your-first-request/) [Course outline](https://spider.cloud/learn/course/)

## What a site asks of a client

The rules a site publishes for programs, and how a well-behaved client follows them.

1. [What robots.txt is and how a well-behaved client reads it robots.txt is the file where a site says which paths crawlers may fetch. How to find your group, which rule wins, and what to do when the file fails to load. Concept · 6 min](https://spider.cloud/learn/robots-txt/)

## How to ask

What a request says about the client that sent it, and how often to send one.

1. [How many requests per second to send when you crawl a site No one rate fits every site. Start at one request per second to one host, then pick a delay from a small crawl we timed at three settings. How-to · 7 min](https://spider.cloud/learn/request-rate/)
2. [How to write a scraper user agent that names you What a server sees in your scraper's User-Agent header, how to build one with a name, a version and a contact, and one dated request that shows it arriving. How-to · 7 min](https://spider.cloud/learn/user-agent/)

## How to read the answer

What the status code and the headers tell a client to do next.

1. [HTTP status codes list, and what a client does next The HTTP status codes list in five classes, the codes a crawler meets most, and the next step a well-behaved client takes for each, checked against one dated crawl. Concept · 8 min](https://spider.cloud/learn/http-status-codes/)

## How to find the pages

Where a site lists its own addresses, and how to tell what changed since the last visit.

1. [How to scrape a paginated site and know when to stop Follow a listing from its first page to its last, and stop on the right signal: no next link, the page limit you set, or a page with no items. How-to · 8 min](https://spider.cloud/learn/pagination/)
2. [What a sitemap XML file is and what one real file holds A sitemap XML file lists the pages a site wants crawlers to find. The tags, the index file, and one real sitemap counted for URLs and last modified dates. Concept · 6 min](https://spider.cloud/learn/sitemaps/)

## Terms

The words the other pages use, each with a short definition.

1. [Web scraping glossary: the terms, each in a sentence or two Plain definitions of the words you meet when you scrape or crawl: crawler, depth, sitemap, user agent, rate limit and more, with one real run to read them against. Concept · 7 min](https://spider.cloud/learn/glossary/)
