Comment by psadri
11 hours ago
I was in a similar situation and tried a different approach.
I had start with asking for a contact with some details I provided. But then I ask the model to be an experienced corporate lawyer and ask me a series of questions to gather the details it needed and then write the contract. The result covered a lot of details that were highly relevant but were absent from the original attempt.
The key insight is that you can lean on the model to cover your unknown unknowns.
Your comment is a great example of the phenomenon where people think AI is an expert in areas that are not because they have no way to determine if the seemingly good looking result is genius, gibberish, or somewhere in between.
The point was that you can go further than "write me a contact that says x,y and z". I actually believe the process I went through with the AI was a more custom, tailored experience than any interaction I have had with actual lawyers, who'd usually apply a template. The AI asked me a bunch of questions, many of which were relevant, and led to follow up questions. I could ask my own questions, make corrections, etc. and all along, they made sense to me. This part of the experience was great.
There could still be more unknowns that I did not encounter and by definition don't know about.
And the final output could be wrong, wording, exact legal terminology etc.
Can we say working with an actual lawyer is guaranteed to be better, more correct? Lawyers are people and they also make mistakes, wake up on the wrong side of the bed, hate their job etc, just like the rest of us.
Here's another techniqe that gets you (even) more out of LLMs: run each prompt several times and compare the various outputs. It is not uncommon for models to contradict their own advice, and also you will get additional insights not included in previous runs.
This is due to the fact that LLMs are statistical processes that rely on pseudo random numbers in chosing what to say and how to say it to a substantial degree.
> The result covered a lot of details that were highly relevant but were absent from the original attempt
Another one suckered by the plausibility engines.
Did you validate the output with a lawyer?
No admittedly not. I can try it and report back.