Comment by justrealist
2 years ago
The space of "real" options in IAM is small enough that hallucination is not a real problem.
Anecdotally I've used copilot to help write a lot of IAM polities in Terraform and the accuracy is basically 100% already.
Same could be said for ECS container definitions yet ChatGPT will happily give you a set of parameters which don't exist.
human in the loop, during the prompt->gen phase, makes a huge difference. You can hit backspace and try different things
With an API that has a hidden / predefined prompt, you'll run into hallucinations that are harder or impossible to handle