Comment by NotHowItWorks

It literally is token prediction with vector search.

Yes there’s an app layer in the chat product for convenience and parsing but the model is exposing methods like ‘complete’ that predict out word sequences.

When model temperature settings are not added in, you get the same exact response every time, just like ngram.

What you’re seeing with Gemini and ChatGPT is context caching to prevent getting sidetracked and response boilerplates with multi-modality so they can call out to image generation, a code completion service, etc. to assemble a complete response.

That’s why the mainstream providers seem so much better (or at least consistent in replies) - each prompt is actually requesting multiple models and doing a lot of application level work to assemble the response you see. That also why they take sooo long to respond.

The language model itself is quite literally a text completion machine, with settings.