Comment by Tadpole9181

I disagree, this is all because it's understood.

You need a system promot because an LLM is fundamentally a token predictor. It needs to be primed for the work it's going to do, otherwise it's next token prediction has too little to go off in the beginning and goes nuts.

You use a separate AI to review the code, because it's a token predictor trained mostly on accomplishing tasks in a cost effective way - we haven't made AGI here. The first shot that actually makes the code will have rationalizations in its memory as well as prior research, which bias token prediction to accepting that as true (remember how they had to train our sycophancy?). Then because of the desire to be cost effective, they're slightly lazy, and so it won't always do the research to find new edge cases and problems and missing tests the initial research didn't find.

So spin up a separate, clean slate, and ask for it to review from scratch. Or have that one spin up multiple smaller ones to have them specialize in specific concerns or domains (security, data model corruption, code quality), then have the orchestrator validate those concerns and stitch them into a cohesive response.

I can't help you on your last question. Saying they are degrading in quality is not even close to my experience. 5.6 and Opus/Fable 5 are have been a huge step up. Though I do need to adjust memory rules as these new models come out, since ground-up retrained models often come with their own quirks that replace old ones - causing old, specific memories to have unintended side effects.