Comment by tzury

Seems like Google got a pass, since their search engine activity is not included. Once they cached the page, their AI can "crawl" internally.

(Same applies to Bing most likely)