Comment by niemandhier

You can use existing guardrails software to implement this efficiently:

NVIDIA NeMo offers a nice bundle of tools for this, among others an interface to Cleanlabs API to check for thruthfullness in RAG apps.