Comment by krior

> After a "bad" page percentage threshold is exceeded just stop wasting your time crawling that domain entirely.

In the words of Bush jr.: Mission accomplished!