.Google.com's John Mueller addressed an inquiry regarding why Google indexes web pages that are refused coming from creeping by robots.txt and why the it is actually risk-free to dismiss the similar Search Console documents about those creeps.Crawler Web Traffic To Concern Guideline URLs.The individual inquiring the question chronicled that crawlers were actually making links to non-existent question parameter Links (? q= xyz) to pages with noindex meta tags that are also blocked out in robots.txt. What urged the concern is that Google.com is crawling the web links to those web pages, receiving shut out through robots.txt (without seeing a noindex robotics meta tag) at that point receiving reported in Google.com Browse Console as "Indexed, though obstructed through robots.txt.".The individual asked the complying with inquiry:." However listed here's the large inquiry: why would Google mark pages when they can not also see the web content? What is actually the benefit in that?".Google.com's John Mueller validated that if they can not creep the web page they can't see the noindex meta tag. He likewise helps make an exciting reference of the internet site: hunt operator, suggesting to disregard the end results since the "average" users will not find those outcomes.He wrote:." Yes, you're proper: if our experts can not creep the web page, our team can not observe the noindex. That said, if our experts can't creep the pages, at that point there is actually not a great deal for us to index. Thus while you may observe several of those webpages with a targeted site:- query, the typical consumer won't view all of them, so I would not bother it. Noindex is likewise alright (without robots.txt disallow), it merely means the URLs will certainly find yourself being crept (as well as find yourself in the Browse Console report for crawled/not listed-- neither of these conditions result in problems to the remainder of the site). The vital part is that you do not produce them crawlable + indexable.".Takeaways:.1. Mueller's answer affirms the limitations being used the Website: hunt evolved search operator for diagnostic factors. One of those main reasons is actually given that it's certainly not hooked up to the normal search index, it is actually a distinct thing altogether.Google's John Mueller talked about the site hunt operator in 2021:." The short response is actually that a website: question is certainly not meant to become total, nor utilized for diagnostics reasons.An internet site inquiry is actually a details kind of search that confines the outcomes to a particular internet site. It is actually generally simply words internet site, a bowel, and afterwards the internet site's domain.This query confines the end results to a certain site. It's certainly not indicated to become an extensive selection of all the pages from that site.".2. Noindex tag without utilizing a robots.txt is great for these kinds of situations where a bot is connecting to non-existent web pages that are obtaining found out by Googlebot.3. URLs with the noindex tag are going to create a "crawled/not catalogued" entry in Look Console and also those will not possess an adverse effect on the remainder of the website.Read through the concern and also respond to on LinkedIn:.Why would certainly Google.com mark web pages when they can't also observe the content?Featured Graphic by Shutterstock/Krakenimages. com.