.Google.com's Gary Illyes reveals Googlebot gathers web links as opposed to "observing" all of them, negating main documentation.Googlebot collects links instead of observing all of them in real-time.Gary Illyes discussed this information in Google's 'Explore Off The File' podcast.Google's documentation negates the genuine spider habits.