Comment by alansaber
13 hours ago
I guess it's more performant to stuff in a bigger system prompt now that models can support larger input sizes
13 hours ago
I guess it's more performant to stuff in a bigger system prompt now that models can support larger input sizes
I would expect this only to be true for linear architectures like Mamba or Gated DeltaNet. Transformers and hybrid architectures do not have constant compute cost per token.
Performant could certainly mean “higher performing” and not “quicker”.