Comment by aorth

4 months ago

There was an interesting comment in the Lobsters thread about this article https://lobste.rs/s/gig2wt/you_don_t_need_anubis. Basically, Sec-Fetch-* headers are widely available on browsers https://caniuse.com/?search=sec-fetch-dest, so you can detect if a client that says they are Chrome, Firefox, or Safari are really Chrome, Firefox, or Safari.

This seems to work in Caddy, using a CEL expression:

    @unrealistic-browsers <<CEL
    {header.User-Agent}.matches("(Chrome|Firefox|Safari)")
        && ! ({header.Sec-Fetch-Dest}.matches("^.+$")
                && {header.Sec-Fetch-Mode}.matches("^.+$")
                && {header.Sec-Fetch-Site}.matches("^.+$"))
        CEL

    handle @unrealistic-browsers {
            abort
    }

Maybe there is a better way. And maybe this stops working when all low-effort bots add these headers to their crawlers.