Comment by jcul
5 hours ago
It's crazy. Are they doing any precomputing as you type, I wonder if you paste a block of text is it the same speed.
5 hours ago
It's crazy. Are they doing any precomputing as you type, I wonder if you paste a block of text is it the same speed.
The magic is in the fact that they essentially have an ASIC llm device. There is no other trickery. The problem they will face is that it is actually locked in silicon, so upgrading models will be difficult, and likely require new hardware each time.
I pasted and instantly hit enter on this prompt: "I generated a filter set using REW v5.31.3 using real-world sweep tone measurements from the room I'm listening in . How can I use it as my MacOS output equalizer so that my spotify music is adjusted for this room and speakers"
and it gave a very reasonable answer in non-perceptible time.
No, it really does take ~0.03s to generate the answer. Try your browser's developer tools and watch the requests.
Isn't it crazy that we can send a message, across the world near instantaneously and have a coherent reply, generated by a computer, sent back to your screen, in under 500ms in most circumstances.
I find myself getting caught up in the sheer speed of modern computing and networking. The fact I can play an online game with 10 other people is just insane.
I got into Rust development via LLM last year, and being able to do things budgeted in nanoseconds is a heady feeling indeed. Real time video and audio analysis? Totally doable, plenty of time budget. 16.6ms is a long time, it turns out.