Comment by wredcoll
3 days ago
I don't know if you've noticed, but there's a few websites these days that use javascript as part of their display logic.
3 days ago
I don't know if you've noticed, but there's a few websites these days that use javascript as part of their display logic.
Yes, and those sites take way more effort to crawl than other sites. They may still get crawled, but likely less often than the ones that don't use JavaScript for rendering (which is the main purpose of Anubis - saving bandwidth from crawlers who crawl sites way too often).
(Also, note the difference between using JavaScript for display logic and requiring JavaScript to load any content at all. Most websites do the first, the second isn't quite as common.)