Comment by yakhinvadim

What you're describing is super close to the first version I had!

In the beginning I had 3 parameters: scale (number of people), magnitude (degree of change for those impacted) and additionally potential (how likely is this event to trigger downstream significant events).

The point behind including potential was to separate these two events:

1) A 80 year old dies from cancer 2) An 80 year old dies from a new virus called COVID

This worked roughly well but I kept adding parameters to improve the system: novelty, credibility, etc... The current system works on 7 parameters.

---

I never attempted to give LLM all options and rank them against each other.

1) as you said, for me 20k articles is just too much to fit into context window. Maybe some modern LLMs can handle it, but it wasn't the case for a long time, and I settled on current approach.

2) I don't want the "neighbors" to affect individual article ratings. With the current system I am able to compare news spread over months, because they were all rated using the same prompt.

3) I intentionally avoided giving AI examples, like "evaluate event X given that event Y is 7/10". I want it to give scores with a "clear mind" and not be "primed" to my arbitrary examples.