The optimal inference configuration varies according to the model, the specific traffic (which changes in time), and the specific hardware it's running to. The "knowledge" for it doesn't exist yet.
Usually in similar scenarios an agent will try to experiment with various configurations and get to a good result. However, you can't have an agent test out different inference serving configurations/architectures, since every experiment might involves 10-60 minutes of spinning up a model (on very expensive GPUs)...
It's not that easy.
The optimal inference configuration varies according to the model, the specific traffic (which changes in time), and the specific hardware it's running to. The "knowledge" for it doesn't exist yet.
Usually in similar scenarios an agent will try to experiment with various configurations and get to a good result. However, you can't have an agent test out different inference serving configurations/architectures, since every experiment might involves 10-60 minutes of spinning up a model (on very expensive GPUs)...