← Back to context

Comment by jillesvangurp

5 hours ago

I've been toying with LLM based agents for staying on top of feeds lately. Staying on top of the fire hose was always the challenge with RSS readers, a lot of interesting low frequency stuff from people you follow gets buried under the non stop feed of stuff from high volume news that is actively trying to drown everyone out with many updates per day. Back in the day I solved it by simply not using RSS feeds for high volume stuff. But I was following a lot of interesting people. I miss that.

This is what lead to algorithm based filtering. Hacker News uses a simplistic algorithm but it is definitely using one and it works well enough. It's why I come here. We all collectively vote things up and what remains is nominally interesting enough to skim from the front page. With a bit of editorializing.

Social networks tried to game the algorithms for ad revenue. Which is why they are a lot less popular these days. Sites like Medium, Substack, Tumblr, etc. took over from simple blogs and immediately started raising walled gardens around them to become discovery platforms, have recommendations, etc.

But at least they support RSS. A lot of websites still do. If you run any kind of website publishing regular news or article content and you don't support a feed, you are being an idiot. It's easy, doesn't really cost anything, and you might actually get people using your feed once in a while. Your site might actually have one without you realizing. Most news papers have feeds. They are everywhere. The main issue isn't finding them but sifting through them. It always was.

With agent based approaches, you control the algorithm. That wasn't possible in the past. LLMs can summarize, aggregate, categorize, group, filter, etc.

4 comments

jillesvangurp

Reply
SlinkyOnStairs  2 hours ago

> a lot of interesting low frequency stuff from people you follow gets buried under the non stop feed of stuff from high volume news that is actively trying to drown everyone out with many updates per day.

Every major RSS reader supports folders. Your problem is that you engage with RSS as if it were a social media feed, with it's single monolithic reverse-chronological feed.

Just don't do that. Stick all the high volume news feeds in a folder, and you can skim read the headlines & hit "mark all as read" once you're done or for whatever other reason don't want to look at the news anymore.

Stick the low volume things you care about in their own folder, and those will remain unread, in their own ordering for you to read at your own leasure.

Even for sites that don't offer granular feeds, every major feed reader offers filtering options, a lot of them offer fairly complex regex filtering.

> This is what lead to algorithm based filtering.

Feed aggregators (and most social media) exist because of discoverability, finding new stuff from new people you hadn't heard about before.

> With agent based approaches, you control the algorithm. That wasn't possible in the past. LLMs can summarize, aggregate, categorize, group, filter, etc.

You'd be spending tens of dollars of compute on something that every major RSS client was doing back in 2006 with the equivalent of less than a single penny worth of current day compute.

creamyhorror  5 hours ago

I'm building Subweb.net (not ready yet, it's just a few test feeds without the LLM pipeline turned on yet) to LLM-tag RSS feed items with topic, relevance/interest, location, and translations, and present them as feeds. I'm thinking I could maybe let users specify their preferred custom prompts and ranking params or similar, though the standard prompt is already fine.

I think the open web needs to come back, but in a fair way for everyone, giving readers control over their feeds while also sending traffic and comments back to the original sources. Not quite sure how to do that yet.