Discover every URL on a website without downloading full page content. The Links endpoint is faster and cheaper than a full crawl because it only collects URLs. Use it for sitemap generation, broken link detection, site structure analysis, or as a first pass before targeted scraping.
Key capabilities
- ✓ URL discovery without full content extraction
- ✓ Lower cost per page than Crawl or Scrape
- ✓ Faster response times for large sites
- ✓ Recursive link following with depth control
- ✓ Great for building sitemaps and link graphs
- ✓ Streaming output as URLs are discovered