Google Had Discussed Allowing Noindex In Robots.txt

March 19, 2024

Noindex is a part of the meta robots standard but not the robots.txt standard. John Mueller curious to know about noindex: in robots.txt”Why Noindex Directive In Robots.txt Is Unsupported By GoogleGoogle’s John Mueller answered that it was unsupported. Right now it’s possible to block crawling in robots.txt or to block indexing with the meta robots noindex directive. Mueller explained why Google decided to not move ahead with the idea of honoring the noindex directive within the robots.txt. The revelation about those internal discussions is interesting but it’s also deepens the sense of weirdness about Mueller’s robots.txt.

The source of this news is from Search Engine Journal