Comment by aurareturn
3 days ago
then declined as sponsored results and SEO degraded things
It didn't decline because of this. It declined because of a general decade long trend of websites becoming paywalled and hidden behind a login. The best and most useful data is often inaccessible to crawlers.
In the 2000s, everything was open because of the ad driven model. Then ad blockers, mobile subscription model, and the dominance of a few apps such as Instagram and Youtube sucking up all the ad revenue made having an open web unsustainable.
How many Hacker News style open forums are left? Most open forums are dead because discussions happen on login platforms like Reddit, Facebook, Instagram, X, Discord, etc. The only reason HN is alive is because HN doesn't make need to make money. It's an ad for Y Combinator.
SEO only became an issue when all there is for crawlers is SEO content instead of true genuine content.
> The best and most useful data is often inaccessible to crawlers.
Interesting point.
> ost open forums are dead because discussions happen on login platforms like Reddit, Facebook, Instagram, X, Discord, etc
Ironically isn't one of the reasons some of those platforms started to use logins was so they could track users and better sell their information to ad people?
Obviously now there are other reasons as well - regulation, age verification etc.
Does this suggest that the AI/ad platforms need to tweak their economic model to share more of the revenue with content creators?
You can still use Reddit without logging in. In fact it's completely unlike Discord. Lots of Reddit discussions still show up in web search results.
Reddit does not expose all comments and posts to crawlers.
I seem to remember very few ads on the early web. Most sites I frequented were run by volunteers who paid out of their own pockets for webspace.
What year? 90s?
I remember ads everywhere in 2000s.