Comment by framapotari

> How do you evaluate the quality of a summary of a paper you do not have the knowledge to read and understand?

Tough question. I think the straightforward answer is that you can't.

That said, there is some confidence gained in an LLM's abilities based on its performance on papers in domains that I do understand. Yes, it's not going to be the same across all domains, but the frontier labs do publish capability scores across different domains, and that helps scrutinize the answers it provides, and how much salt to take with those.