Comment by svnt

They cannot be expected to produce useful new ideas because the ideas exist in lacunae in their probabilities: despite the extant possible novel combination of ideas (which isn’t the only option for new ideas: neologisms exist), the LLM has never seen it and so will (probabilistically) never produce it because it is equivalent to nonsense.

The exception to this is if the new ideas are somehow present in the structure of language and are internalized and/or presented in an emergent form.