.Google.com's John Mueller answered an inquiry regarding why Google indexes web pages that are refused coming from crawling through robots.txt as well as why the it is actually secure to neglect the similar Search Console reports regarding those creeps.Bot Traffic To Question Guideline URLs.The person inquiring the question chronicled that crawlers were producing links to non-existent query criterion Links (? q= xyz) to webpages along with noindex meta tags that are actually likewise shut out in robots.txt. What caused the inquiry is that Google.com is actually crawling the web links to those web pages, receiving blocked out by robots.txt (without noticing a noindex robots meta tag) then obtaining shown up in Google.com Browse Console as "Indexed, though shut out by robots.txt.".The individual asked the adhering to question:." However below's the major inquiry: why would Google.com index pages when they can't even find the material? What is actually the perk during that?".Google.com's John Mueller verified that if they can't creep the webpage they can't view the noindex meta tag. He additionally makes an appealing mention of the website: hunt driver, suggesting to neglect the results because the "normal" users will not view those results.He composed:." Yes, you are actually right: if our team can not crawl the page, our company can not view the noindex. That claimed, if our team can't creep the web pages, then there's certainly not a lot for our team to mark. Thus while you might find some of those web pages along with a targeted website:- query, the normal user will not see all of them, so I would not bother it. Noindex is actually likewise fine (without robots.txt disallow), it only indicates the URLs will wind up being crawled (and wind up in the Browse Console file for crawled/not recorded-- neither of these standings trigger problems to the rest of the website). The fundamental part is that you do not make all of them crawlable + indexable.".Takeaways:.1. Mueller's response validates the constraints in operation the Website: hunt evolved hunt driver for analysis explanations. Among those causes is since it is actually not attached to the normal hunt mark, it is actually a separate thing completely.Google's John Mueller discussed the internet site hunt driver in 2021:." The short answer is actually that a site: query is actually not implied to be complete, nor utilized for diagnostics objectives.An internet site inquiry is a specific sort of search that restricts the results to a specific website. It is actually generally simply the word website, a digestive tract, and after that the website's domain.This inquiry restricts the outcomes to a specific site. It's not implied to be a detailed selection of all the pages coming from that internet site.".2. Noindex tag without making use of a robots.txt is great for these type of situations where a bot is actually linking to non-existent web pages that are receiving uncovered by Googlebot.3. URLs with the noindex tag will definitely produce a "crawled/not catalogued" item in Look Console and also those will not possess a negative impact on the rest of the site.Read through the question and answer on LinkedIn:.Why will Google index pages when they can not also find the content?Included Graphic by Shutterstock/Krakenimages. com.