Comment by mnahkies

I believe they batch requests so the same weights in vram are shared across many requests https://www.baseten.co/blog/continuous-vs-dynamic-batching-f...