Comment by Bluestein
2 days ago
This is the takeaway here: That's how they have been serving it at scale as Ox-Alpha. This is a definitional moment.-
Further quote:
"Compared with our initial baseline on the same hardware, we achieved a 3× improvement in end-to-end serving performance, reaching hardware efficiency and per-token cost comparable to mainstream NVIDIA GPUs. This demonstrates that Chinese chips can support frontier-model inference efficiently and economically at scale."
Anyone knows what are those Chinese chips? Can they be bought? (Assuming im not i the US, And actually im in a 3rd world country).
Perhaps these may be Huawei Ascend chips.
https://en.wikipedia.org/wiki/HiSilicon#Ascend_910
https://medium.com/@huaweiclouddevelper/a-brief-introduction...
They are high end really expensive Huawei ascend GPUs. It is kinda bruteforcing the performance on a older semiconductor processing tech, so total production is pretty low.
> comparable to mainstream NVIDIA GPUs
By this they probably mean RTX series GPUs? If so, then they are not comparing the hardware efficiency with the A100 / H100, etc. that are commonly used for training models