Comment by ndriscoll

I suspect that the quality is ironically correlated with the expertise of the user (i.e. it is knowledgeable if you are knowledgeable), which puts you in a conundrum (I can report that with a couple decades of experience, LLMs are giving me high quality, correct results, but I can already see that it somehow doesn't work as well for some of my less experienced colleagues. A lot of what I've been doing over the last couple months is trying to find how to make it "just work" for them.).

As a general principle, take advantage of the fact that it can easily generate stuff. If you don't know whether something is true, have it prove it. Make a PoC/test/benchmark to demonstrate what it's saying. Have it pull metrics that you have access to. Add more observability. Create feedback loops (or rather, ask it to create feedback loops). They're very good at reasoning given access to the ground truth, so give them more ability to ground themselves.

They also have fantastic knowledge of public things, but no knowledge of your company, so your instructions should mostly be documentation of what's unique to your company. If it can write an instruction on its own (e.g. how to use git or kubernetes), it is a useless instruction; it already knows that. What it doesn't know is e.g. where your git server is. It also doesn't know what matters to your company: are you a startup trying to find product market fit? Are you an established company that is not allowed to break customer setups? etc. You might even be able to ask it what kinds of questions a senior might ask about how a company/team works when coming into a new job, and then see if you can answer those questions (or find someone who can). In fact, go ask chatgpt:

> What are some questions a senior engineer might ask when coming into a new role to make themselves more effective?

> What are some questions a principle engineer might ask when coming into a new role to make themselves more effective?

> What are some questions an engineering manager might ask when coming into a new role to make themselves more effective?

> What are some questions an engineering director might ask when coming into a new role to make themselves more effective?