LLMs are powerful, but enterprises are deterministic by nature

This resonates a lot, and I think your example actually captures the core failure mode really well.

What your PM asked for isn’t an “agentic pipeline” problem - it’s an organizational knowledge and accountability problem. LLMs are being used as a substitute for missing context, missing ownership, and missing validation paths.

In a system like that (30+ years, COBOL, interdependent routines), the hardest parts are not parsing code — they are understanding why things exist, which constraints were intentional, and which tradeoffs are still valid. None of that lives in the code, and no model can infer it reliably without human anchors.

This is where I have seen LLMs work better as assistive tools rather than autonomous agents: helping summarize, cluster, or surface patterns — but not being expected to produce “the” design document, especially when there is no stakeholder capable of validating it.

Without determinism around inputs, review, and ownership, the output might look impressive but it’s effectively unverifiable. That’s a risky place to be, especially for early-career engineers being asked to carry responsibility without authority.

I don’t think the problem is that LLMs are not powerful enough — it is that they are often being dropped into systems where the surrounding structure (governance, validation, incentives) simply isn’t there.

Your task is certainly doable though.

You can ask AI to focus on the functional aspects and create a design-only document. It can do that in chunks. You don't need to know about COBOL best practices now, that's an implementation detail. Is the plan to modernize the COBOL codebase or to rewrite in a different language?

See what this skill does in Claude Code, you want something similar: https://github.com/glittercowboy/get-shit-done/blob/main/get...

That sounds very aligned. I like the way you phrased it - deterministic policy that agents can not bypass is exactly the right boundary, especially once you assume prompt injection and misalignment are not edge cases but normal operating conditions.

On the use case side, what we have been seeing (and discussing internally) isn’t one narrow workflow so much as a recurring pattern across domains: anywhere an LLM starts influencing actions that have irreversible or accountable consequences.

That shows up in security, but also in ops, infra, finance, and internal tooling - places where “suggesting” is fine, but executing without a gate is not. In those environments, the blocker usually isn’t model capability; it is the lack of a deterministic layer that can enforce constraints, log decisions, and give people confidence about why something was allowed or stopped.

Security tends to surface this problem first because the blast radius is obvious, but we are starting to see similar concerns come up once agents touch production systems, money, or compliance-sensitive workflows.

I am curious from your side — are you finding that security teams are more receptive to this model than other parts of the org, or are you still having to convince people that “agent autonomy” needs hard boundaries?

I don’t disagree with the underlying concern. In practice, “probabilistic” often does translate to unreliable when you put these systems in environments that expect reproducibility and accountability.

Where I think the framing matters is in how we respond architecturally. Treating LLMs as “just another unreliable program” is reasonable — but enterprises already have patterns for dealing with unreliable components: isolation, validation, gates, and clear ownership of side effects.

The problem we’re seeing is that LLMs are often dropped past those boundaries — allowed to directly author decisions or actions — which is why the downstream damage you mention (journals, courts, OSS) feels so chaotic.

The “suggestion engine” framing isn’t meant to excuse that behavior; it’s meant to reassert a familiar control model. Suggestions are cheap. Execution and publication are not. Once you draw that line explicitly, you can start asking the same questions enterprises always ask: who approves, what’s logged, and what happens when this is wrong?

Without that separation, I agree — you’re effectively wiring an unreliable component straight into systems that assume trust, and the failure modes shouldn’t surprise anyone.

I doubt ISO-9000 gets “replaced” so much as interpreted more strictly in the presence of LLMs. ISO-9000 isn’t about how work is done — it’s about whether processes are defined, repeatable, auditable, and improvable.

From that lens, LLMs actually create tension rather than an escape hatch. A system whose outputs can’t be reproduced, explained, or bounded makes it harder to demonstrate compliance, not easier. Saying “we use AI” doesn’t satisfy requirements around traceability, corrective action, or process control.

My guess is that ISO-style frameworks will push organizations toward explicitly classifying where LLMs are allowed to operate: as advisory inputs, as drafting aids, or as automation under defined controls — with clear ownership and validation steps around them.

In other words, the pressure probably won’t be to loosen standards, but to reassert them: define where probabilistic components sit, what checks exist before outputs become authoritative, and how failures are detected and corrected. Without that structure, it’s hard to see how certification survives unchanged.

I mostly agree — using LLMs to help author deterministic code is a good fit. The distinction I’m trying to draw is that once that code exists, the determinism has to live outside the model.

LLMs can assist in creating the rules, but they shouldn’t be the place where those rules are enforced or bypassed at runtime.

Thanks for sharing — that article points at the same core tension. Determinism isn’t about rejecting probabilistic systems, it’s about deciding where uncertainty is allowed to live.

What keeps breaking in practice is when probabilistic reasoning leaks into places that expect reproducibility and accountability.