Comment by fjfaase
5 days ago
When I ask chatGTP to generate the code for a M4F MCU that implements the VirtIO code for accessing the GPIO through RPMSG using FreeRTOS it produces two answers that are both incomplete and incorrect.
5 days ago
When I ask chatGTP to generate the code for a M4F MCU that implements the VirtIO code for accessing the GPIO through RPMSG using FreeRTOS it produces two answers that are both incomplete and incorrect.
This is very outside my domain :P I asked ChatGPT to explain the acronyms in your comment and still do not understand it. But I think a reason LLMs might struggle is that there are too many up front complex requirements.
Giving it only 1 or 2 complex requirements at a time, and then having it iterate, I've found to be more effective.
Most LLMs don't "think", so when asking an LLM something, I generally try to think "would I be able to do this without thinking, if I had all the knowledge, but just had to start typing and go?".
You could break down your prompt into separate prompts like this maybe: https://chatgpt.com/share/683eb7d7-e7ec-8012-8b3b-e34d523dc9...
I think it broke things down in a weird way, but I definitely can't analyse the correctness of anything it outputs in this domain :P
Coding specific agents like Copilot might be better able to handle a complex initial prompt, since they take the initial prompt, and use LLMs to break it down into smaller steps, which ChatGPT doesn't do. They can sort of "think". Deep research AIs also have a sort of thinking too, so might do better.
I did follow your suggestion to break it down in separate prompts, but that to did not result in any useful code. I did not try to compile it, but I get the impression that it does not even compile. There is nothing remotely related to the VirtIO protocol in the code. See Section 5.18 of https://docs.oasis-open.org/virtio/virtio/v1.3/csd01/virtio-...
When I added the prompt: 'Could you use the specification given in Section 5.18 of https://docs.oasis-open.org/virtio/virtio/v1.3/csd01/virtio-...' it produced almost the same code preceded with some babbling from the document, but not using anything from the specification, not even the code fragments mentioned in the section.
Ah darn. Yeah that is a tricky space, I'm curious to find if there's an incantation to make it actually do the right thing... But knowing next to nothing of your domain makes that impossible for me :P I will note ChatGPT isn't great at very long documents as input; Gemini Pro performs better there. Pulling out the specific section _might_ help, but that's getting into a-lot-of-work territory.
Alas, good luck!
I was just giving it as an example of how there are domains where AI assistance have little or no benefit. Even if you used them for searching for resources or instructions how to achieve something, they come up with nothing useful, or lead you in the direction you do not want to go. It is just like searching with Google: If you search for the thing that 90% of the people are searching for, your answer will be in the first page. If you search for something outside that, you have to skip many pages, or even not being able to find it at all.
One could also conclude that a large portion of software engineering is rather is mostly implementation of things that have been implemented many times before and that only a small portion consist of real software engineering where you have to develop code for a problem that nobody ever wrote before or that require a deep understanding of the problem domain.
> One could also conclude that a large portion of software engineering is rather is mostly implementation of things that have been implemented many times before
Strong +1. That's one of the reasons I enjoy using LLMs, cause I'm often using it for the pieces of code which I don't even find rewarding to write; it's just glue code which I know has been written 1000 times over, which _I've_ already written 1000 times over across different projects.
I wonder if "lack of data" problem is solvable within LLM. Humans can apply general programming language knowledge when using a new language, but LLMs have to have lots of concrete source code to work well.
Today these models fall apart very quickly once you step out of js/python/c land to rust/zig/ocaml.
With today LLM tech, feeding it language documentation does not work great.