Stupid question: Is there any chance that I, as an engineer, can get away from learning the Math side of AI but still drill deeper into the lower level of CUDA or even GPU architecture? If so, how do I start? I guess I should learn about optimization and why we chose to use GPU for certain computations.
Parallel question: I work as a Data Engineer and always wonder if it's possible to get into MLE or AI Data Engineering without knowing AI/ML. I thought I only need to know what the data looks like, but so far I see every job description of an MLE requires background in AI.
Yes. They are largely unrelated. Just go to Nvidia's site and find the docs. Or there are several books (look at amazon).
A "background in AI" is a bit silly in most cases these days. Everyone is basically talking about LLMs or multimodal models which in practice haven't been around long. Sebastian Raschka has a good book about building an LLM from scratch, Simon Prince has a good book on deep learning, Chip Huyen has a good book on "AI engineering". Make a few toys. There you have a "background".
Now if you want to really move the needle... get really strong at all of it, including PTX (nvidia gpu assembly, sort of). Then you can blow people away like the deep seek people did...
Lets say you already have deep knowledge of GPU architecture and experience optimizing GPU code to saves 0.5ms runtime for a kernel. But you got that experience from writing graphics code for rendering, and have little knowledge of AI stuff beyond surface level stuff of how neural networks work.
How can I leverage that experience into earning the huge amounts of money that AI companies seem to be paying? Most job listings I've looked at require a PhD in specifically AI/math stuff and 15 years of experience (I have a masters in CS, and no where close to 15 years of experience).
Thank you! This really helps. I'll concentrate on Computer Architecture and lower level optimization then. I'll also pick one of the books just to get some ideas.
Neural networks are basically just linear algebra (i.e matrix multiplication) plus an activation function (ReLu, sigmoid, etc.) to generate non-linearities.
Thats first year undergrad in most engineering programs - a fair amount even took it in high school.
I'd like to re-enforce this viewpoint. The math is non-trivial, but if you're a software engineer, you have the skills required to learn _enough_ of it to be useful in the domain. It's a subject which demands an enormous amount of rote learning - exactly the same as software engineering.
hot take: i don't think you even need to understand much linear algebra/calculus to understand what a transformer does. like the math for that could probably be learned within a week of focused effort.
Not a stupid question at all! Imo, you can definitely dive deep into CUDA and GPU architecture without needing to be a math whiz. Think of it like this: you can be a great car mechanic without being the engineer who designed the engine.
Start with understanding parallel computing concepts and how GPUs are structured for it. Optimization is key - learn about memory access patterns, thread management, and how to profile your code to find bottlenecks. There are tons of great resources online, and NVIDIA's own documentation is surprisingly good.
As for the data engineering side, tbh, it's tougher to get into MLE without ML knowledge. However, focusing on the data pipeline, feature engineering, and data quality aspects for ML projects might be
> As for the data engineering side, tbh, it's tougher to get into MLE without ML knowledge. However, focusing on the data pipeline, feature engineering, and data quality aspects for ML projects might be
I have a feeling that companies usually expect MLE to do both ML/AI and Data Engineering, so this might indeed be a dead end. Somehow I'm just not very interested in the MLE part of ML so I'll dormant that thought for the meanwhile.
> Start with understanding parallel computing concepts and how GPUs are structured for it. Optimization is key - learn about memory access patterns, thread management, and how to profile your code to find bottlenecks. There are tons of great resources online, and NVIDIA's own documentation is surprisingly good.
Thanks a lot! I'll take these points in mind when learning. I need to go through more basic CompArch materials first I think. I'm not a good programmer :D
It's definitely possible to focus on the CUDA/GPU side without diving deep into the math. Understanding parallel computing principles and memory optimization is key. I've found that focusing on specific use cases, like optimizing inference, can be a good way to learn. On that note, you might find https://github.com/codelion/optillm useful – it optimizes LLM inference and could give you practical experience with GPU utilization. What kind of AI applications are you most interested in optimizing?
They have excellent resources to get you started with Cuda/Triton on top of torch. It also has a good community around it so you get to listen to some amazing people :)
> Math side of AI but still drill deeper into the lower level of CUDA or even GPU architecture
CUDA requires clear understanding of mathematics related to graphics processing and algebra. Using CUDA like you would use traditional CPU would yield abysmal performance.
> MLE or AI Data Engineering without knowing AI/ML
It's impossible to do so, considering that you need to know exactly how the data is used in the models. At the very least you need to understand the basics of the systems that use your data.
Like 90% of the time spent in creating ML based applications is preparing the data to be useful for a particular use case. And if you take Google ML Crash Course, you'll understand why you need to know what and why.
I will provide general advice that applies here, and elsewhere: Start with a project, and implement it, using CUDA. The key will be identifying a problem that is SIMD in nature. Choose something you would normally use a loop for, but that has many (e.g. tens of thousands or more) iterations, which do not depend on the output of the other iterations.
Some basic areas to focus on:
- Setting up the architecture and config
- Learning how to write the kernels, and what makes sense for a kernel
- Learning how the IO and synchronization between CPU and GPU work.
This will be as learning any new programming skill.
Thanks! I'll follow the link and see what happens. And thanks for recommending Andrew Ng's course too, hopefully it gives enough background to know how the users (AI scientists) want us to prepare the data.
If you want to dive into CUDA specifically then I recommend following some of the graphics tutorials. Then mess around with it yourself, trying to implement any cool graphic/visualization ideas or remixes on the tutorial material.
Yes, but the problems that need GPU programming also tend to require you to have some understanding of maths. Not exclusively - but it needs to be a problem that's divisible into many small pieces that can be recombined at the end, and you need to have enough data to work through that the compute cost + data transfer cost is much lower than just doing it on CPU.
From an infrastructure perspective, If you have access to the hardware, a fun starting point is running NCCL tests across the infrastructure. Start with a single GPU, then 8 GPUs on a host, then 24 GPU multi hosts over IB or RoCE. You will get a feel for MPI and plenty of knobs to turn on the Kubernetes side.
I mean yes, but without knowing the maths then knowing how to optimize the maths is a bit useless?
At the very least you should know enough linear algebra that you understand scalar, vector and matrix operations against each of the others. You don't need to be able to derive back prop from first principles, but you should know what happens when you multiply a matrix by a vector and apply a non-linear function to the result.
Thanks! Yeah I do know some Math. I'm not sure how much I need to know. I guess the more the merrier, but it would be nice to know a line that I don't need to cross to properly do my job.
Very nice-write up. The in-line quiz, which i think is AI generated(QnA) is very useful to test understanding. Wish all tutorials incorporated that feature.
I have a slightly tangential question: Do you have any insights into what exactly DeepSeek did by bypassing CUDA that made their run more efficient?
I always found it surprising that a core library like Cuda, developed over such a long time, still had room for improvement—especially to the extent that a seemingly new team of developers could bridge the gap on their own.
They didn’t. They used PTX, which is what CUDA C++ compiles down to, but which is part of the CUDA toolchain. All major players have needed to do this because the intrinsics for the latest accelerators are not actually exposed in the C++ API, which means using them requires inline PTX at the very minimum.
They basically ditched CUDA and went straight to writing in PTX, which is like GPU assembly, letting them repurposing some cores for communication to squeeze out extra performance. I believe that with better AI models and tools like Cursor, we will move to a world where you can mold code ever more specific to your use case to make it more performant.
Are you sure they ditched CUDA? I keep hearing this, but it seems odd because that would be a ton of extra work to entirely ditch it vs selectively employing some ptx in CUDA kernels which is fairly straightforward.
Their paper [1] only mentions using PTX in a few areas to optimize data transfer operations so they don't blow up the L2 cache. This makes intuitive sense to me, since the main limitation of the H800 vs H100 is reduced nvlink bandwidth, which would necessitate doing stuff like this that may not be a common thing for others who have access to H100s.
> with better AI models and tools like Cursor, we will move to a world where you can mold code ever more specific to your use case to make it more performant
what do you think the value of having the right abstraction will be in such a world?
Are all the CUDA tutorials geared towards AI or are there some, for example, like regular scientific computing? Airflow over wings and things that you used to see for high-performance computing would be fun to try.
Since this is on PySpur's website, does anyone have experience with these UI tools for AI agents like PySpur and n8n? I am looking for something to help me prototype a few ideas for fun. I would have to self-host it ($), so I would prefer something relatively easy to configure like Open Hands.
1) More AI-native features eg. Evals, RAG, or even UI decisions like seeing outputs directly on the canvas when running on the agent
2) Truly open-source Apache license
3) Python-based (in the sense that you can run and extend it via python)
On the other hand, n8n is
1) more mature for traditional workflows
2) offering overall more integrations (probably every single integration you can think of)
3) TypeScript based and runs on Node.js
Thanks for replying. Do you know when your docs will be a bit more comprehensive? Right now, there is very little information and some links don't work, e.g., Next Steps on this page: https://docs.pyspur.dev/quickstart
Any idea what changed recently and we can have end to end simulations (with branches) in the gpu (eg isaac sim) vs in the past where simulations were a cpu thing ?
Always been possible, but now the time cost of moving data between the GPU and CPU memory is too high to ignore. Branching may be slower on the GPU but it's still faster than moving data to the CPU for a time then back. The maturation of direct GPU-GPU transfers over the network also helped enable GPU-only MPI codes.
Triton sits between CUDA and PyTorch and is built to work smoothly within the PyTorch ecosystem. In CUDA, on the other hand, you can directly manipulate warp-level primitives and fine-tune memory prefetching to reduce latency in eg. attention algorithms, a level of control that Triton and PyTorch don't offer AFAIK.
Stupid question: Is there any chance that I, as an engineer, can get away from learning the Math side of AI but still drill deeper into the lower level of CUDA or even GPU architecture? If so, how do I start? I guess I should learn about optimization and why we chose to use GPU for certain computations.
Parallel question: I work as a Data Engineer and always wonder if it's possible to get into MLE or AI Data Engineering without knowing AI/ML. I thought I only need to know what the data looks like, but so far I see every job description of an MLE requires background in AI.
Yes. They are largely unrelated. Just go to Nvidia's site and find the docs. Or there are several books (look at amazon).
A "background in AI" is a bit silly in most cases these days. Everyone is basically talking about LLMs or multimodal models which in practice haven't been around long. Sebastian Raschka has a good book about building an LLM from scratch, Simon Prince has a good book on deep learning, Chip Huyen has a good book on "AI engineering". Make a few toys. There you have a "background".
Now if you want to really move the needle... get really strong at all of it, including PTX (nvidia gpu assembly, sort of). Then you can blow people away like the deep seek people did...
Lets say you already have deep knowledge of GPU architecture and experience optimizing GPU code to saves 0.5ms runtime for a kernel. But you got that experience from writing graphics code for rendering, and have little knowledge of AI stuff beyond surface level stuff of how neural networks work.
How can I leverage that experience into earning the huge amounts of money that AI companies seem to be paying? Most job listings I've looked at require a PhD in specifically AI/math stuff and 15 years of experience (I have a masters in CS, and no where close to 15 years of experience).
4 replies →
Thank you! This really helps. I'll concentrate on Computer Architecture and lower level optimization then. I'll also pick one of the books just to get some ideas.
Agreed, Rashka's book is amazing and will probably become the seminal book on LLMs
3 replies →
The math isn't that difficult. The transformers paper (https://proceedings.neurips.cc/paper_files/paper/2017/file/3...) was remarkably readable for such a high impact paper. Beyond the AI/ML specific terminology (attention) that were thrown out
Neural networks are basically just linear algebra (i.e matrix multiplication) plus an activation function (ReLu, sigmoid, etc.) to generate non-linearities.
Thats first year undergrad in most engineering programs - a fair amount even took it in high school.
I'd like to re-enforce this viewpoint. The math is non-trivial, but if you're a software engineer, you have the skills required to learn _enough_ of it to be useful in the domain. It's a subject which demands an enormous amount of rote learning - exactly the same as software engineering.
hot take: i don't think you even need to understand much linear algebra/calculus to understand what a transformer does. like the math for that could probably be learned within a week of focused effort.
1 reply →
May I plug-in with ClojureCUDA, a high-level library that lets you write CUDA with almost no overhead, but write it in the interactive Clojure REPL.
https://github.com/uncomplicate/clojurecuda
There's also tons of free tutorials at https://dragan.rocks And a few books! (not free) at https://aiprobook.com
Everything from scratch, interactive, line-by-line, and each line is executed in the live REPL.
Not a stupid question at all! Imo, you can definitely dive deep into CUDA and GPU architecture without needing to be a math whiz. Think of it like this: you can be a great car mechanic without being the engineer who designed the engine.
Start with understanding parallel computing concepts and how GPUs are structured for it. Optimization is key - learn about memory access patterns, thread management, and how to profile your code to find bottlenecks. There are tons of great resources online, and NVIDIA's own documentation is surprisingly good.
As for the data engineering side, tbh, it's tougher to get into MLE without ML knowledge. However, focusing on the data pipeline, feature engineering, and data quality aspects for ML projects might be
Thanks for the help!
> As for the data engineering side, tbh, it's tougher to get into MLE without ML knowledge. However, focusing on the data pipeline, feature engineering, and data quality aspects for ML projects might be
I have a feeling that companies usually expect MLE to do both ML/AI and Data Engineering, so this might indeed be a dead end. Somehow I'm just not very interested in the MLE part of ML so I'll dormant that thought for the meanwhile.
> Start with understanding parallel computing concepts and how GPUs are structured for it. Optimization is key - learn about memory access patterns, thread management, and how to profile your code to find bottlenecks. There are tons of great resources online, and NVIDIA's own documentation is surprisingly good.
Thanks a lot! I'll take these points in mind when learning. I need to go through more basic CompArch materials first I think. I'm not a good programmer :D
Agreed, not sure how much math is really needed.
It's definitely possible to focus on the CUDA/GPU side without diving deep into the math. Understanding parallel computing principles and memory optimization is key. I've found that focusing on specific use cases, like optimizing inference, can be a good way to learn. On that note, you might find https://github.com/codelion/optillm useful – it optimizes LLM inference and could give you practical experience with GPU utilization. What kind of AI applications are you most interested in optimizing?
I suggest having a look at https://m.youtube.com/@GPUMODE
They have excellent resources to get you started with Cuda/Triton on top of torch. It also has a good community around it so you get to listen to some amazing people :)
> Math side of AI but still drill deeper into the lower level of CUDA or even GPU architecture
CUDA requires clear understanding of mathematics related to graphics processing and algebra. Using CUDA like you would use traditional CPU would yield abysmal performance.
> MLE or AI Data Engineering without knowing AI/ML
It's impossible to do so, considering that you need to know exactly how the data is used in the models. At the very least you need to understand the basics of the systems that use your data.
Like 90% of the time spent in creating ML based applications is preparing the data to be useful for a particular use case. And if you take Google ML Crash Course, you'll understand why you need to know what and why.
I will provide general advice that applies here, and elsewhere: Start with a project, and implement it, using CUDA. The key will be identifying a problem that is SIMD in nature. Choose something you would normally use a loop for, but that has many (e.g. tens of thousands or more) iterations, which do not depend on the output of the other iterations.
Some basic areas to focus on:
This will be as learning any new programming skill.
IMO absolutely yes. I would start with the linked introduction and then ask myself if I enjoyed it.
for a deeper dive, check out the sth like Georgia Tech’s CS 8803 O21: GPU Hardware and Software.
To get into MLE/AI Data Engineering, I would start with a brief introductory ML course like Andrew Ng’s on Coursera
Thanks! I'll follow the link and see what happens. And thanks for recommending Andrew Ng's course too, hopefully it gives enough background to know how the users (AI scientists) want us to prepare the data.
If you want to dive into CUDA specifically then I recommend following some of the graphics tutorials. Then mess around with it yourself, trying to implement any cool graphic/visualization ideas or remixes on the tutorial material.
You could also try to recreate or modify a shader you like from https://www.shadertoy.com/playlist/featured
You'll inevitably pick up some of the math along the way and probably have fun doing it along the way.
Yes, but the problems that need GPU programming also tend to require you to have some understanding of maths. Not exclusively - but it needs to be a problem that's divisible into many small pieces that can be recombined at the end, and you need to have enough data to work through that the compute cost + data transfer cost is much lower than just doing it on CPU.
From an infrastructure perspective, If you have access to the hardware, a fun starting point is running NCCL tests across the infrastructure. Start with a single GPU, then 8 GPUs on a host, then 24 GPU multi hosts over IB or RoCE. You will get a feel for MPI and plenty of knobs to turn on the Kubernetes side.
I mean yes, but without knowing the maths then knowing how to optimize the maths is a bit useless?
At the very least you should know enough linear algebra that you understand scalar, vector and matrix operations against each of the others. You don't need to be able to derive back prop from first principles, but you should know what happens when you multiply a matrix by a vector and apply a non-linear function to the result.
Thanks! Yeah I do know some Math. I'm not sure how much I need to know. I guess the more the merrier, but it would be nice to know a line that I don't need to cross to properly do my job.
1 reply →
I found the gpumode lectures, videos and code right on the money. check them out.
You will probably have fewer job opportunities than the people working higher up, but be safer from AI automation for now :)
Try dipping your toes into graphics programming, you can still use GPUs for that as well.
Very nice-write up. The in-line quiz, which i think is AI generated(QnA) is very useful to test understanding. Wish all tutorials incorporated that feature.
thank you!
Thanks for sharing, enjoyed reading it!
I have a slightly tangential question: Do you have any insights into what exactly DeepSeek did by bypassing CUDA that made their run more efficient?
I always found it surprising that a core library like Cuda, developed over such a long time, still had room for improvement—especially to the extent that a seemingly new team of developers could bridge the gap on their own.
They didn’t. They used PTX, which is what CUDA C++ compiles down to, but which is part of the CUDA toolchain. All major players have needed to do this because the intrinsics for the latest accelerators are not actually exposed in the C++ API, which means using them requires inline PTX at the very minimum.
They basically ditched CUDA and went straight to writing in PTX, which is like GPU assembly, letting them repurposing some cores for communication to squeeze out extra performance. I believe that with better AI models and tools like Cursor, we will move to a world where you can mold code ever more specific to your use case to make it more performant.
Are you sure they ditched CUDA? I keep hearing this, but it seems odd because that would be a ton of extra work to entirely ditch it vs selectively employing some ptx in CUDA kernels which is fairly straightforward.
Their paper [1] only mentions using PTX in a few areas to optimize data transfer operations so they don't blow up the L2 cache. This makes intuitive sense to me, since the main limitation of the H800 vs H100 is reduced nvlink bandwidth, which would necessitate doing stuff like this that may not be a common thing for others who have access to H100s.
1. https://arxiv.org/abs/2412.19437
1 reply →
Targeting directly PTX is perfectly regular CUDA, and used by many toolchains that target the ecosystem.
CUDA is not only C++, as many mistake it for.
got it, thanks for explaining.
> with better AI models and tools like Cursor, we will move to a world where you can mold code ever more specific to your use case to make it more performant
what do you think the value of having the right abstraction will be in such a world?
6 replies →
What Jensen giveth, Guido taketh away.
lol. i guess this tutorial is about cutting out guido ;)
this book:
seems to be tailor mode for folks transitioning from cpu -> gpu arch.
Yes, it is great for key concepts but a bit outdated. Hence we added an LLM/FA section in the linked post!
Also check out https://github.com/rust-gpu/rust-gpu and https://github.com/rust-gpu/rust-cuda
Rust-Cuda is broken and has been for years.`cudarc` is the [only?] working one.
I am in the process of rebooting it: https://rust-gpu.github.io/blog/2025/01/27/rust-cuda-reboot/
this looks really cool and i love rust. just a matter of time until everything runs on rust.
Are all the CUDA tutorials geared towards AI or are there some, for example, like regular scientific computing? Airflow over wings and things that you used to see for high-performance computing would be fun to try.
Related: https://sakana.ai/ai-cuda-engineer/
https://www.reddit.com/r/MachineLearning/comments/1itqrgl/p_...
Wasn’t this a bunch of kernels that didn’t work?
What do you mean?
5 replies →
[dead]
Since this is on PySpur's website, does anyone have experience with these UI tools for AI agents like PySpur and n8n? I am looking for something to help me prototype a few ideas for fun. I would have to self-host it ($), so I would prefer something relatively easy to configure like Open Hands.
Disclaimer: I work on pyspur
I'd recommend pyspur if you seek
1) More AI-native features eg. Evals, RAG, or even UI decisions like seeing outputs directly on the canvas when running on the agent 2) Truly open-source Apache license 3) Python-based (in the sense that you can run and extend it via python)
On the other hand, n8n is 1) more mature for traditional workflows 2) offering overall more integrations (probably every single integration you can think of) 3) TypeScript based and runs on Node.js
Thanks for replying. Do you know when your docs will be a bit more comprehensive? Right now, there is very little information and some links don't work, e.g., Next Steps on this page: https://docs.pyspur.dev/quickstart
1 reply →
pyspur is apache 2. it is free to self-host.
Any idea what changed recently and we can have end to end simulations (with branches) in the gpu (eg isaac sim) vs in the past where simulations were a cpu thing ?
Always been possible, but now the time cost of moving data between the GPU and CPU memory is too high to ignore. Branching may be slower on the GPU but it's still faster than moving data to the CPU for a time then back. The maturation of direct GPU-GPU transfers over the network also helped enable GPU-only MPI codes.
Interestingly, the CUDA implementations are more readable than the pytorch ones.
interesting, you mean they are less obscure?
[dead]
If you are a Python dev, why not just use Triton?
Triton sits between CUDA and PyTorch and is built to work smoothly within the PyTorch ecosystem. In CUDA, on the other hand, you can directly manipulate warp-level primitives and fine-tune memory prefetching to reduce latency in eg. attention algorithms, a level of control that Triton and PyTorch don't offer AFAIK.
MLIR extensions for Python do though, as far as I could tell from LLVM developer meeting.
2 replies →
Triton is somewhat limited in what it supports, and it’s not really Python either.
or use Hidet compiler (open source)
never heard of Hidet before; for when/what would I use it over CUDA/Triton/Pytorch?
1 reply →
pyspur graph is cool, is there a startup building this kind of product but in typescript?
Thanks for unraveling this!
you're welcome!
I needed this
Hehe glad you did!
[dead]