Comment by paimapi

I see it as a problem of context which maps to my theoretical understanding of LLMs. models trained on large data sets will probabilistically veer towards the median in all aspects - reasoning, assumptions, environments, etc. specialized context about your specific codebase's solutions don't exist unless you add them in, either in the prompt, as a skill or rule, or more generally in the harness via memories, tests, etc (though ideally a combination of all of the above). without that the LLM will suggest the median solution for the median codebase according to some ephemeral, unqualifiably trained understanding of best practices

it makes me wonder if the solution that businesses/users need to implement is just the same solution to everything since the beginning of time ie standardization. skills/harnesses/agents.md/etc maintained by codeowning teams that must be invoked for AI-assisted code changes on ABC part of the codebase, these existing as replacement for the bevy of other documentation required for the days of hand-written code. a company-wide orchestration skill knows how to search and pull down the relevant .mds, cleans it as cruft at the end of a session, every merge with a short changelog saved to a corpus somewhere with a TOC + appendix that an LLM can navigate to and read for context, major changes in the logic documented in the working skill doc, all of it generally automated but requiring HITL vetting

this wouldn't fully solve the problem of subject matter expertise but it seems like it would remove a lot of the friction for new employees and other teams with dependencies on your work or with whom you have dependencies