Comment by akersten
19 hours ago
I'm old enough to remember discussions around the meaning of `User-Agent` and why it was important that we include it in HTTP headers. Back before it was locked to `Chromium (Gecko; Mozilla 4.0/NetScape; 147.01 ...)`. We talked about a magical future where your PDA, car, or autonomous toaster could be browsing the web on your behalf, and consuming (or not consuming) the delivered HTML as necessary. Back when we named it "user agent" on purpose. AI tooling can finally realize this for the Web, but it's a shame that so many companies who built their empires on the shoulders of those visionaries think the only valid way to browse is with a human-eyeball-to-server chain of trust.
Me too but it died when ads became the currency of the web. If the reason the site exists is to use ads, they’re not going to let you use an user agent that doesn’t display the ads.
> If the reason the site exists is to use ads, they’re not going to let you use an user agent that doesn’t display the ads.
They've been giving it the old college try for the better part of two decades and the only website I've had to train myself not to visit is Twitch, whose ads have invaded my sightline one time too many, and I conceded that particular adblocking battle. I don't get the sense that it's high on the priority list for most sites out there (knock on wood).
People who block ads are a minority. Sites that serve heavy content like video would care if someone wastes their resources but blocks ads, but why would a site that serves a few KBs of text spend the resources on blocking such users or making the ads beat the ad blocker in a tiresome cat and mouse game?
Those users could even share or recommend the site to someone else who doesn't use ad blockers, so it actually makes sense to not try to battle ad blockers if you want to make your site more popular.
This makes sense for sites that rely on network effects, like forums or classified ad sites and so on. Unless they have a near monopoly or some really valuable content, they would benefit financially if they let people block their ads.
I can't back that up with data or anything, but it makes sense to me.
1 reply →
Same, I just don't use Twitch when possible. Most streamers rehost their VODs on Youtube which has a better player anyway.
Adblocker is only few clicks away and a surprisingly large amount of users running one. So they might not like it, but they already letting plenty of users to use agent that doesn't display the ads.
Not only ads. Primary anti-scraping use today is obfuscation either as anti-competitive practice or hiding unlawful behavior like IP infringement etc.
> AI tooling can finally realize this for the Web
There was a concept named Web 3.0 a while ago, aka the 'Semantic Web'. It wasn't the crypto/blockchain scam that we call Web3 today. The idea was to create a web of machine readable data based on shared ontologies. That would have effectively turned the web into a giant database of sorts, that the 'agents' could browse autonomously and derive conclusions from. This is sort of like how we browse the web to do research on any topic.
Since the data was already in a structured form in Web 3.0 instead of natural language, the agent would have been nowhere near the energy hogs that LLMs are today. Even the final conversion of conclusions into natural language would have been much more energy-efficient than the LLMs, since the conclusions were also structured. Combine that with the sorts of technology we have today, even a mediocre AI (by today's standards) would have performed splendidly.
Opponents called it impractical. But there already were smaller systems around from various scientific fields, operating on the same principle. And the proponents had already made a lot of headway. It was going to revolutionize information sharing. But what I think ultimately doomed it is the same reason you mentioned. The powers that be, didn't want smarter people. They wanted people who earned them money. That means those who spend their attention on dead scrolling feeds, trash ads and slop.
> but it's a shame that so many companies who built their empires on the shoulders of those visionaries think the only valid way to browse is with a human-eyeball-to-server chain of trust.
Yes, this! But only when your eyeball and attention earns them profit. Otherwise they are perfectly content with operating behind your backs and locking you out of decisions about how you want to operate the devices you paid for in full. This is why we can't have good things. No matter which way you look, the ruins of all the dreams lead to the same culprit - the insatiable greed of a minority. That makes me question exactly how much wealth one needs to live comfortably or even lavishly till their death.
Just like then we were naive about folks not abusing these things to the point of making everyone need to block them to oblivion. I think we are relearning these lessons 30 years later.