Comment by manofmanysmiles

I wonder if you could train an LLM while holding the weights implementing fixed-function regions constant, so the training would produce a model that natively knows how to use these tools inside it's own reasoning steps.

I suppose it doesn't need to necessarily be in the weights, a hybrid LLM + internal fixed function blocks that aren't encoded in the weights could be quite interesting.

Combine this with latent reasoning and it gets pretty wild.

A very useful fixed function region in my mind is a symbolic execution engine, capable of only following a well-defined grammar. Perhaps a lisp would be good too?