This means that you can fit larger and larger models into a single node, without having to go out over the network. The memory bandwidth on AMD is also quite good.
"driver" is such a generic word. tinygrad works on mi300x. If you want to use it, you can. Negates your point.
Additionally, ROCm is a giant collection of a whole bunch of libraries. Certainly there are issues, as with any large collection of software, but the critical thing is whether or not AMD is responsive towards getting things fixed.
In the past, it was a huge issue, AMD would routinely ignore developers and bugs would never get fixed. But, after that SA article, Lisa lit a fire under Anush's butt and he's taking ownership. It is a major shift in the entire culture at the company. They are extremely responsive and getting things fixed. You can literally tweet your GH issue to him and someone will respond.
What is true a year ago isn't today. If you're paying attention like I am, and experiencing it first hand, things are changing, fast.
So the MI300x has 8 different memory domains, and although you can treat it as one flat memory space, if you want to reach their advertised peak memory bandwidth you have to work with it like an 8-socket board.
AMD is consistently stacking more HBM.
This means that you can fit larger and larger models into a single node, without having to go out over the network. The memory bandwidth on AMD is also quite good.
It really does not matter how much memory AMD has if the drivers and firmware are unstable. To give one example from last year:
https://www.tomshardware.com/pc-components/gpus/amds-lisa-su...
They are currently developing their own drivers for AMD hardware because of the headaches that they had with ROCm.
"driver" is such a generic word. tinygrad works on mi300x. If you want to use it, you can. Negates your point.
Additionally, ROCm is a giant collection of a whole bunch of libraries. Certainly there are issues, as with any large collection of software, but the critical thing is whether or not AMD is responsive towards getting things fixed.
In the past, it was a huge issue, AMD would routinely ignore developers and bugs would never get fixed. But, after that SA article, Lisa lit a fire under Anush's butt and he's taking ownership. It is a major shift in the entire culture at the company. They are extremely responsive and getting things fixed. You can literally tweet your GH issue to him and someone will respond.
What is true a year ago isn't today. If you're paying attention like I am, and experiencing it first hand, things are changing, fast.
17 replies →
That was last year Mi300x firmware and software have gotten much better since then
7 replies →
So the MI300x has 8 different memory domains, and although you can treat it as one flat memory space, if you want to reach their advertised peak memory bandwidth you have to work with it like an 8-socket board.
Here is a good article on it:
https://rocm.blogs.amd.com/software-tools-optimization/compu...
MI355X isn't out yet, and the upcoming B300 also has 288GB HBM3e
June 12th.
B300 is Q4 2025.
Yes, they keep leapfrogging each other. AMD is still ahead in vram.