Comment by femto
21 hours ago
Most professional engineers, especially those in electrical or communications, have the tools to understand AI.
I recently started trying to understand how AI works. It turns out that I wrote an "AI" in the late 1990s without realising it. The "AI" was an MMSE (minimum mean square error) radio receiver for CDMA.
This receiver consisted of an FIR filter (mathematically a linear combination) with trainable coefficients (weights). The coefficients started with random values. A sequence of known training symbols (tokens) was fed in and a gradient search was used to adjust the coefficients to achieve MMSE. The unknown data to be received (classified) followed the training sequence, at which point the training was turned off and the system ran as an FIR filter. Once trained, the FIR was a matched filter.
In AI terminology the above is now called a "layer". Current AI cascades multiple layers, in which case you need to pass the output of each FIR/layer though a non-linear function. If you don't have the non-linear functions at the end of each layer the multi-layer system is mathematically equivalent to a linear combination, which could be implemented with a single layer, losing degrees of freedom.
I write this because many electrical/communications engineers probably know how AI works, but don't realise it.
My still incomplete understanding is that LLMs are "just" a graph of connected layers. More complex that the above MMSE receiver, but trained and operated in the same way. The smarts in AI are:
1) Arranging the layers and the connections between them.
2) Having access to the right training data
3) Having the computing power to perform the gradient search (aka training)
My point is that most professional engineers already have the knowledge to understand AI/LLMs, and potentially already do know how they work but don't realise it due to a difference in terminology.
This is like saying with chemistry, amino acids and human DNA you now have the basis to understand human culture.
The problem is one of complexity, the whole being greater than the sum of its parts. Electrical engineering creates a thinking that is opposite to what you need for understanding biology or human culture.
This is relevant because LLMs have basically swallowed all written human culture. Engineering is about creating parts that interact in a clean, well-defined way, like a MOSFET transistor or a REST API. The goal is that that when you combine the parts, and they all operate within specs, you can still predict what the whole system will do.
So saying LLMs are just a step away from the Pereceptron (or an equivalent filter) is mathematically true, but almost useless. It's like saying that a modern 3D game is basically just transistors, logic, CPUs and memory, nevermind the pattern in that memory or how it came to be. You really can't ignore all the cultural inputs and feedback loops if you want any kind of useful understanding what this thing is.
I would agree that a decent EE major is equipped with knowledge sufficient to grasp both Transformer architecture and gradient descent training. However, these elements do not fully cover the key parts of modern AI systems and also are not enough to predict the AI capabilities. You have to account for reinforcement learning from human feedback (RLHF) and with verifiable rewards (RLVR) which are crucial elements of the wider model creation/training process. These do not have obvious analogues in other engineering disciplines, but are still within your grasp if you are decent CS/EE/adjacent major. The issue is that you still can't deduce end result AI capabilities from first principles. You may have prefect understand of y = softmax(QK^T/sqrt(d_k))*V and it will give you no clue of the effectiveness of a given AI model. The range and complexity of training RLVR tasks may hint at the model capabilities, but the final results are very much emergent.
At best trying to scale this understanding from what you have stated might (gratuitously) get you to a multi-layer percepetron. Add a few ideas and you would hit a convolutional neural net which represented the frontier a decade ago. However this is still so far divorced from the modern frontier of AI, transformers, because the leaps are more focused on attention mechanisms than anything else. And even just understanding a transformer is merely a building block.
This is basically understanding a building block (A) of a building block (B) of a building block (C) that builds D and purporting that you can understand D from the building blocks of A. The equivalent is saying that knowing Newton's laws is enough to understand how we get to the moon. It might be fine for a hand-wavy idea, but its just not enough.
Dropping in to mention belief propagation networks, those things are likely being used right now by the modem on whatever device you use (WiFi or Ethernet, both use it!), and are surprisingly AI-like.
As used in LDPC decoders?
My gut feeling is that the AI field is eventually going to discover how much overlap it has with information and communications theory.
Question to any AI researchers out there: Can a layer in a trained AI be treated as a form of "matched filter"? Is that a concept that is used in the understanding AI: to view it as a cascade of matched filters? The next question is "what is it matching?"
"My gut feeling is that the AI field is eventually going to discover how much overlap it has with information and communications theory."
I think researchers in the field are very much aware of information and communications theory, it's a bit naive to think they don't.
There is a lot of terminological confusion in the field. The fundamental difference is about how the "AI" learns. And who, if at all, supervises the learning.
Some systems are fully formed AI. Often based on heuristics. Humans have encoded the rules into the system. Others do limited learning based on pattern recognition. Still others learn based on tagged data. And LLMs don't need even that. In a stream of N tokens, the trainer has access to K+1 when it is training a model that knows 1...K and can check if the prediction was correct.
AI literally refers to any artificial thing that feels intelligent to us: does some thing a simple bash script will not. Your rules based chess engine is AI. The bot you compete against in single player Age of Empires 2 is AI. DeepBlue was AI.
ML is where things become interesting because we teach machines to learn.
NN is where we borrow terminology from neuroscience and create "neurons" that learn.
DL is where we stack neurons deeply in layers.
I am trying to make sense of all this myself.[1] So I am building a curriculum for it as I learn. So much of the teaching material in the wild revolves around torch. Makes it useless if you want to peek inside the actual machinery.
[1] https://bhashika.org.in/logs/2026/an-ml-curriculum
> we teach machines to learn
I feel like this statement is too anthropomorphizing. We don't teach a computer anything. We write normal computer programs that can process data(training), turning it into what is essentially a configuration file(weights/parameters) for another normal computer program(inference engine). Upon receiving a request, the inference engine is invoked and passed the input, then it processes that input and its configuration(weights), producing an output. "Thinking" is just feeding the output back in as an input, allowing the model to revise it and refine it before it's returned to the user.
There is no teaching, nor learning. There is just data processing. There is nothing happening between requests to the inference engine. It's a relatively simple system altogether, but when the training data set is large enough it produces unbelievably good outputs.
With this in mind I really don't understand how people think these programs can be conscious. They're not doing anything fundamentally different than any other computer program. It's just crunching numbers. You might as well argue that a computer game or 7zip or a chess bot or whatever is conscious.
I know you specifically didn't claim they're conscious, I just thought it was worth mentioning because I think it's ridiculous how many people will seriously argue this as if there is any difference between "ai" and other programs. There isn't. It's just a set of instructions for doing computation, as computers have always done.
3D rendering is actually quite similar. There is a bunch of files (input/configuration) describing 3D meshes, textures, lighting etc which are fed into a rendering engine which processes all that data and turns it into a 2D image. Similarly to AI, once the input files are sophisticated enough, this can produce mindblowingly realistic and beautiful images, and you can run it in a loop to make video or make it interactive, moving the camera around in real time etc. But at the core of it all is just a bunch of files containing a bunch of numbers, and a program that reads those files and does computation with them.
The current leading materialistic model of consciousness is essentially that human consciousness is a computation in itself, and that the brain is basically a very very complex computer. So it's not at all illogical for someone who holds this view and who fully understands that AI is just a regular computer program to also believe that it could be conscious.
Of course, I'm not saying that you have to believe any of this - even defining what consciousness is is famously elusive. And there are a lot of people who think consciousness is not materialistic to begin with, or that we have not yet discovered the scientific nature of it.
5 replies →
We humans have the habit of falling into the trap of anthropocentric views and explanations of the universe and everything within it. The history of AI is the history of moving goalposts. I don't particularly care if they are conscious or not. Humans have a history of treating even actual, conscious, intelligent beings as chattel or food. So I don't really see why our attitudes would change if we somehow realized that LLMs are conscious.
I have been thinking about this for a while in the context of the quest for AGI. If all we want is something to take over some parts of our intellectual labor, then LLMs are already there. They can communicate with you in a language you understand and do stuff that you command them to. Often, on their own.
AGI will have its own telos. An LLM does not (at least I refuse to believe that it does).
So, yeah, you could think of LLMs as programs processing data. But there is definitely some emergent behavior you notice that emerges when the LLM runs. How you interpret all this depends on your view on consciousness and intelligence.
1 reply →
> "There is nothing happening between requests to the inference engine." This is true for vLLM/SGLang/etc but LayerScale is making use of idle GPUs. A new novel inference engine for agentic workloads. Our advanced scheduler enables double the workload on the same number of GPUs (e.g. prioritization of on-demand queries and background processing of batch workloads simultaneously). Let me know if you want to try it out!
> I feel like this statement is too anthropomorphizing.
That's number one mistake preventing people from getting a good grasp of how LLMs work and how to use them effectively.
That is, not anthropomorphizing is the mistake. It gives people bad mental models.
LLMs are not people in general, but behaviorally and in terms of system design, they may as well be. They have same failure modes and cognitive characteristics humans have, and are very much unlike regular, non-ML software tools.
1 reply →
Even if the model is simple, it is happening at a scale that is producing alien results. You understand the basic principles, but that doesn't matter: you, well rather no one, understands the final product.
This is a case of https://xkcd.com/2933/ . Your two paths are "Oh no. Simple layers can combine to produce complex behaviours!" and "Oh no. Modern layers are not this simple!"
This is interesting way to look at FIR filter and tuning parameters. never thought of it like this.
The difference with your FIR matrix and a deep net matrix is the latter generalized to trillions of parameters i.e. a trillion dimensional space to approximate general, computable functions. The machine learning ideas are fundamentally the same at a high level, just the architecture (nonlinearity whereas in engineering we prefer analyzing linear systems) specific details and tricks and such.
In fact at least at Berkeley a current professor teaching deep learning is a signals expert (Prof Sahai), his lectures are on YouTube.
This is like saying you understand the basic principles of internal combustion engines.
Sure, that gets you some level of understanding, but in no way prepares you to be even a mechanic, far less an engineer, let alone a researcher/engine innovator.
Tbf OP said they have the tools to understand what was going on, not that they could build a model to compete with OpenAI tomorrow. EE (especifically DSP and communications) uses a lot of the same building blocks as ML so yes it does prepare you.
You are right in that the emergent abilities come from humongous amounts of training data and compute which only a handful of companies (and the people working in them) deal with today. I'm sure there's a lot of secret gotchas which only arise at a certain scale so one cannot self-study for them. Companies also guard these wherever possible because of economic incentives. But that applies to most other engineering disciplines. You can take a course in civil engineering and have enough knowledge to reason about why a skyscraper will not fall apart in the wind; doesn't mean you have the ability to build one.
But it’s certainly enough to know that it’s not “conscious”
Sounds interesting. How did you back-propagate the coefficient gradients? Just tweak them one at a time, or was there a global algorithm that worked on the whole filter at once?
From memory the entire filter was updated in one symbol period (ie. global)
The symbol rate was 1Msymbol/s and there were 66 taps. The circuit (in an FPGA) ran at a clock rate of 66MHz. In each symbol period the error was calculated based on the output symbol, then an iteration was done over all 66 coefficients, using the error to update them.
Edit: I also remember that to make most efficient use of the FPGA I figured out the minimum precision required in the maths whilst preserving convergence. It was a 4-bit mantissa plus an exponent (3-bit or 4-bit exponent, I can't remember). It's interesting that today's AIs seem to have also figured out that things still work if 4-bit floating point is used, though that is 4-bits including the exponent?
> things still work if 4-bit floating point is used, though that is 4-bits including the exponent?
It is generally 4 bits amortised across an entire block. This works more like block-based mage compression schemes - you put a bunch of statistically-similar floats into the same block, and use your handful of bits to index into a codebook containing the deltas from a 16/32 bit reference value for the whole block