googlebot.com, as data

# GooglebotHTTP `user-agent` request headerin the request. However, both crawler types obey the same product token (user agent token) inrobots.txt, and so you cannot selectively target either Googlebot Smartphone or GooglebotFor most sites Google Search primarilyof the content. As such the majority of Googlebot crawl requests will be made using the mobilecrawler, and a minority using the desktop crawler.## How Googlebot accesses your siteFor most sites, Googlebot shouldn't access your site more than once every few seconds onaverage. However, due to delays it's possible that the rate will appear to be slightly higherover short periods. If your site is having trouble keeping up with Google's crawling requests, youWhen crawling for Google Search, Googlebot crawls the first 2MB of athe first 64MB of a PDF file. From a rendering perspective, each resource referenced in the HTML(such as CSS and JavaScript) is fetched separately, and each resource fetch is bound by the samefile size limit that applies to other files (except PDF files).Once the cutoff limit is reached, Googlebot stops the fetch and only sends the already downloadedpart of the file for indexing consideration. The file size limit is applied on the uncompressedOther Google crawlers, for example Googlebot Video and Googlebot Image, may haveWhen crawling from IP addresses in the US, the timezone of Googlebot istechnical properties of Googlebotare described in the overview of Google's crawlers.## Blocking Googlebot from visiting your siteGooglebot discovers new URLs to crawl primarily from links embedded in previously crawled pages.It's almost impossible to keep a site secret by not publishing links to it. For example, as soonas someone clicks a link from your "secret" site to another site, your "secret" site URL mayappear in the referrer tag and can be stored and published by the other site in its referrer log.