That's just semantics around the word "performance".
We're messaging each other right now inside a heavily sandboxed virtual environment that interprets XML-like markup and runs a text-based JITted scripting language from the 90s. It doesn't really matter that it's extremely inefficient, since it's still effectively instant.
Actually not. Watt per task is a metric bothers more than LLM operators and users. We care about it a lot, and working a lot to measure and improve it, and we're not doing it in context of AI or LLMs.
Effectively instant means nothing, since everything is fast for small n. I love to give this example from a codebase I have worked on during my Ph.D. The problem which took half an hour under Matlab was done in ~40 seconds with mine, however, a thousand-step version of the same problem would take 11 hours. Saving a single second from that 40 second loop would shave my work by ~20 minutes, and this will compound for bigger problems still.
Just because we see the results instantly, doesn't mean it's fast or efficient by any means. It means we might forego optimizing it for sufficiently small n, if we don't care about that part of the code. That's all.
Nope. It'll be more efficient and faster.
That's just semantics around the word "performance".
We're messaging each other right now inside a heavily sandboxed virtual environment that interprets XML-like markup and runs a text-based JITted scripting language from the 90s. It doesn't really matter that it's extremely inefficient, since it's still effectively instant.
Actually not. Watt per task is a metric bothers more than LLM operators and users. We care about it a lot, and working a lot to measure and improve it, and we're not doing it in context of AI or LLMs.
Effectively instant means nothing, since everything is fast for small n. I love to give this example from a codebase I have worked on during my Ph.D. The problem which took half an hour under Matlab was done in ~40 seconds with mine, however, a thousand-step version of the same problem would take 11 hours. Saving a single second from that 40 second loop would shave my work by ~20 minutes, and this will compound for bigger problems still.
Just because we see the results instantly, doesn't mean it's fast or efficient by any means. It means we might forego optimizing it for sufficiently small n, if we don't care about that part of the code. That's all.
2 replies →
Imagine performing literally trillions of matrix multiplications in order to figure out the answer to 1 + 2.
Your computer/phone just performed millions of them in order to display the contents of my comment to you.
Pretty much what our brain does.
Sure, but what about performance per joule?
If performance per joule is important in this loosely-defined hypothetical scenario, then it's important.
If it's not, then it's not.