Comment by kgeist
1 year ago
If it was possible to hook into token selection process (kind of like JSON restricted grammar but using custom scripts), then it would be possible to detect that GPT-4 is about to add "# impement code here" and then we could force it select a different set of tokens which would make GPT4 generate a proper method body.
That's called guidance and the problem is that it has to be done carefully or else you'll just get rephrasings that work around the block.
I think a better approach is multi-pass coding along with fine-tuning or prompting to use a particular form of TODO comment. Aider can already do a form of fake "fill in the middle" by making it emit diffs. If it notices that some code has been filled out lazily, it could go back and ask it to do the next chunk of work. Given that large tasks are normally split up into small tasks by programmers anyway, this seems like a natural approach that is required for scaling up regardless.