Comment by jmathai
10 hours ago
I’ve been using v4 flash for an app I’m building [1] and it’s amazing how cost effective and good it is coming from having always used gpt, opus and sonnet models.
It’s so cost effective I can offer a generous free tier since my goal isn’t to make money with it.
I get where you're coming from, and the intent to make it easier for people to find examples and verses, but there's a fine line with LLMs giving you answers, is that it's interpreting it in some form. Doesn't that run counter to prevailing ideology, that you're meant to either struggle with the materials / seek understanding yourself, or have your religious leaders interpret/receive those insights?
You’re correct. The hard line for me is ensuring that any scripture presented to the user is verified correct. LLMs can’t be trusted in this regard.
One feature of the app is that all scripture is verified and what’s show to the user doesn’t come from the LLM at all and instead a trusted source.
I think exploring scripture this way does not alleviate you from struggling to learn and apply it. It hasn’t for me.
I don't necessarily mean reguritating it, but choosing which part of the scripture to surface to the user is already some interpretation/choice. Even the devil can quote scripture (I'm playing the devil's advocate here).
3 replies →
> Doesn't that run counter to prevailing ideology, that you're meant to either struggle with the materials / seek understanding yourself, or have your religious leaders interpret/receive those insights?
I think it depends, Catholics wouldn't be able to use this because the Magisterium is the ultimate authority on interpreting Scripture, so the personal interpretation isn't really needed. This is not to say that Catholics don't read the Bible, they are encouraged to do so since it deepens their faith
On the other hand, for Protestant it varies, the High Church denominations are closer to Catholics (though none of them accept the Magisterium) in terms of scripture interpretation, but the Low Church ones (like Baptists or Non-Denominational ) are more open to personal interpenetration.
Disclaimer: I'm a Catholic, so if I made a mistake here fellow Protestants, please correct me.
It doesn't prevent you from going to the source and struggle with the text, nor seek expert commentary.
What's difficult and doesn't have to be with philosophy/ spirituality is to find relevant bits off situation, theme etc.
This app does that very well, LLMs are good at entity recognition.
Yes it can point you to the right place, but choosing which part of scripture to point you to is already a choice/interpretation
Just let them have their AI pacifier.
Is the difference between this and a frontier model that the scripture is guaranteed to be real?
I'm on a team that develops a Bible study app, and we're all relatively content with how the basic models converse regarding scripture. Even as far back as GPT-4 was excellent. They occasionally have minor hallucinations (a dealbreaker for a production app), but they do an excellent job with theology and Bible scholarship, given reasonable guardrails.
I'll admit I'm coming from the perspective of "should we be implementing this?" It seems, on the surface, that a strong embedding-based verse retrieval covers the bases at a microfraction of the cost.
If you're interested, check out the development server where we're working on this. You navigate to the search (magnifying glass) and then hit "Meaning". Sorry for the confusing route; we're still deciding on back-end details and haven't focused on the front yet.
[1] https://ai.stepbible.org
I hope the irony of being concerned about hallucinations in a bible study app is not lost on you or others reading this!
Philosophically, I don’t believe we should outsource the accuracy of scripture to any single entity (let alone a for profit secular one).
So it’s less about model choice and more about governance of scripture.
I will check out the link you sent for sure!
I was under the impression you were getting BOOK.CHAPTER.VERSE references from the model and then sourcing them from a ground truth db.
Anyways, impressive app! We haven't tackled such an ambitious project just for it being daunting.
2 replies →
All models hallucinate. Everything in production with a model will too. It is not a "deal breaker"
> Is the difference between this and a frontier model that the scripture is guaranteed to be real?
flash is suitable only for a toy apps, not for production environments :)