Comment by simonw
4 hours ago
This is a neat project, but the description made me realize that I don't actually know what the term "guardrails" means.
... which lead me to realize that it's one of those terms with multiple meanings - like "agent" or even "AI" itself - but where people who use it may not be aware of how many different definitions are floating around.
In this project it refers to validating tool calls - fixing invalid tool responses, making sure certain required tool calls have been made, maintaining an error budget after which the task is abandoned with an error.
Other projects might use "guardrails" to mean protecting against unsafe content (Llama Gaurd), refusing off-topic queries (NVIDIA NeMo Guardrails "topical rails", filtering PII, detecting jailbreaks, or human-in-the-loop checks of specific actions.
I've even seen people talk about running a coding agent in a sandbox (Docker, Firecracker etc) as a form of guardrail.
That's a fair point, and frankly something that might not age well in my docs one day. I genuinely don't know what the industry will standardize on when it comes to the use of the term "guardrails". I've seen the sec definitions as well.
You're 100% right about how I meant it and what it means within Forge though, but it's something that might lead to doc changes as things evolve.
I'm thinking of it like a guardrail that keeps your car from driving off the edge of a road, but in this case, it keeps your tool calls from driving off a cliff.