Comment by sdan 8 hours ago I tried running this on a H100 and got 190ms compared to Jev's 170ms. Maybe I set it up wrong? 1 comment sdan Reply kevmo314 42 minutes ago It seems like caching has a huge contribution towards the low latency numbers they report. With a single request and no cache it appears to be quite slow, roughly in line with what you observed.
kevmo314 42 minutes ago It seems like caching has a huge contribution towards the low latency numbers they report. With a single request and no cache it appears to be quite slow, roughly in line with what you observed.
It seems like caching has a huge contribution towards the low latency numbers they report. With a single request and no cache it appears to be quite slow, roughly in line with what you observed.