Comment by lubesGordi
3 days ago
Does the Unreal API change a bit over versions? I've noticed when asking to do a simple telnet server in Rust it was hallucinating like crazy but when I went to the documentation it was clear the api was changing a lot from version to version. I don't think they do well with API churn. That's my hypothesis anyway.
I think the big thing with Unreal is the vast majority of games are closed source. It's already only used for games, as opposed to asking questions about general-purpose programming, but there is also less training data.
You see this dynamic even with swift which has a corpus of OSS source code out there, but not nearly as much as js or python and so has always been behind those languages.
There would be significant changes from 4 to 5, but sadly I haven’t had any improvement if clarifying version.
Clarifying can help but ultimately it was trained on older versions. When you are working with a changing api, it's really important that the llm can see examples of the new api and new api docs. Adding context7 as a tool is hugely helpful here. Include in your rules or prompt to consult context7 for docs. https://github.com/upstash/context7