Comment by meatmanek

At least in the self-hosted LLM inference engines, you have to pre-allocate space for the maximum amount of context you want to allow for each parallel session. By using a lower maximum, you don't have to allocate as much VRAM for each session, allowing more usage for the same amount of hardware. Thus, cheaper.