Settings
Account Settings
Adjust account settings to apply site wide configurations.
Request
Determine what type of request to use for crawling. Possible values are http, headless, and smart. Smart runs request using HTTP first and uses Chrome headless when needed. The default is HTTP for request.
Return Format
The format to return the data in. Possible values are markdown, commonmark, raw, text, and bytes. Use raw to return the default format of the page like HTML etc.
Crawl Limit
Set a limit to prevent the crawling of extra resources.
Metadata
Store meta-data for the website like title, description, and etc to help improve AI features.
Proxy
Set a premium proxy to be used to prevent pages from being blocked.
Full Resources
Crawl and store all resources like css, js, and etc.
Subdomains
Crawl all the subdomains found for the page.
TLD's
Crawl all the TLD's found for the web page.
Webhooks
Set up webhooks across your websites to deliver the desired information anywhere you need.
Crawl Budget
Set a crawl budget to limit the amount of pages per domain. Use a wild card * to determine all routes.