Comment by aetherspawn
11 hours ago
I hope these get small and good enough to create “pet like” AIs for games. You know, like scream “follow me” at an NPC, STT stack translates it and feeds it to a local Jev-like model that then picks a number of things for the NPC to do.
I tried to use jev for this. I'll share what I learned for the interested.
--
The setup was a simple map with different rooms. Each room had 1-3 doors.
For the inputs: The AI had an array of "known places" empty at the start, the current position, the current doors with no information about where they lead to, and the list of past actions
The goal / task for the AI was to explore all rooms and save them to known places. The AI needed to decide if to move or save the place at every turn.
--
So I wasn't able to make the AI explore all of the rooms. The AI kinda always wanted to move to the first option when moving. Out of 6 rooms it was able to save 3. My hypothesis is that jev as it is now is really bad at making connections and understanding it's input. So for example, even if it had a list of previous actions, it wasn't able to reason about it and know where to go. For this to work I'd need to explicitly tell it where it did not go. So you could say that the model is also not good with uncertainty / ambiguous scenarios.
edit: one last thing is that i replaced jev with an standard llm and it finished the goal no problem with the same information given
edit 2: it also felt like the same tradeoffs between small model vs large model. With small models you need to be very conscious and careful with the input while large models are more forgiving. Maybe jev is a small model, and we just need a larger one.