Comment by 1dom
11 days ago
I really like the idea that crawlers who are profiting should have to pay content owners/creators per crawl.
On principal though, I think Cloudflare doing this is just one more thing to create the perception that you can't put something on the internet unless it's through Cloudflare. This harms a transparent and decentralised web and makes selfhosting seem even less appealing to those who don't know any better.
This should be implemented as a web protocol with crypto though so anyone can charge bots without having to be Cloudflare fronted. Not really a fanboi of 99% of crypto stuff, but IMO, a purely technical, open and decentralised solution to this sort of problem was the crypto dream.
We can all guess the people who will make the most money off this, and one of them is Cloudflare. A bunch of the other winners probably also run some of the more aggressive crawlers.
Yes, right, it should be an open protocol so any CDN or content provider can use it the same way. Hopefully it becomes a part of popular web servers so little guys can play along without a CDN.
It needn't be crypto, but would be convenient. Lacking that, it would need some unforgeable presentation of identity that could be connected to a bank account.
I shed not one tear for the crawlers - they had their chance to respect robots.txt on the honor system. Now we force them.
I can't work out how an open protocol implementation of this could work without crypto: ultimately if it's just fiat, a business entity needs to be the payment processor who aggregates microtransactions and pays them out to content owners, this is the role cloudflare is playing.
The problem is microtransactions are not feasible in fiat, and to remove the aggregator role like Cloudflare means a huge amount of microtransactions from each potential crawler to each content owner. That's just too much expensive work compared to the current position.
I agree though, I shed no tears for crawlers, but hopefully we're beyond the naivety of honour systems - again, the thing crypto was supposed to be solving.
Forcing big evil crawler entities to bend the knee by hiding behind big evil CDN entities feels silly though.
Brave has been trying and failing to get micropayments using crypto for years.
> This should be implemented as a web protocol with crypto though so anyone can charge bots without having to be Cloudflare fronted. Not really a fanboi of 99% of crypto stuff, but IMO, a purely technical, open and decentralised solution to this sort of problem was the crypto dream.
It's not just about payment. It's about refusing to serve content to bots, unless they paid. It might be hard to implement without Cloudflare, when bot developers specifically target your website.
The whole point of Cloudflare is to let them decide whether it's bot or user that hits your website. It is complicated task.
Unless you want to force all users to pay, both humans and bots.
I almost guarantee Cloudflare will give this feature away for free for all tiers (Free plan included). I doubt they will make a penny off this. If you knew the company history and culture, youd foresee that too.
I do know the company history, I've been a customer for well over a decade. As far as big tech goes, they've got fewer skeletons than most, but you're naive if you think they're not making a penny off this in any way.