The IF statement is the root creator of software programming. It has the ability to compare two values against each other and branch out to blocks of instructions. So it is perceiving (reading), decision making and routing - all that which differentiate life from inanimate objects. The AI agents perform the exact same loop, by delegating the first two steps to a model.
Going further backwards, the transistor (or a PNP junction) is the hardware level enabler of the IF statement. The action (switching) driven by the current which in turn controls other switches, is the first manifestation of "observe and act" by inanimate things at the speed of electricity.
Mechanical equivalents existed ofcourse - speed of a governer which controls the flow of fuel which in turn controls the speed of the governer.
So, what if, we build a stack/set of transistors in same shape as a trained model? It would eliminate most of the software stack too and should run very fast. No memory/gpu required, the chip acts as both storage and processing device, purpose built to be physical model of a trained model.
The alternative IF expression or statement, in the form introduced by John McCarthy in 1958 (which he used in early 1959 to define his version of AND, OR and NOT), is one of the possible primitives for computation.
There exist several equivalent sets of primitive operations. While the sets containing only NAND or only NOR, or both AND and NOT or both OR and NOT are more notorious, these logical operations are more abstract and they do not indicate precisely a hardware implementation, i.e. there are many distinct hardware methods to make such logical gates.
Other sets of primitive operations map directly to hardware devices, e.g. the sets of primitive operations composed of maximum and complement or of minimum and complement map directly to a hardware implementation using rectifier diodes and inverting amplifiers (which can be made with either semiconductor devices or with vacuum tubes, or also with pneumatic or hydraulic devices).
Other sets of primitive operations are obtained by replacing the maximum or minimum circuits with series or parallel connections of switches, like in the CMOS logic that is nowadays dominant.
The alternative IF expression corresponds in hardware to a 2-way multiplexer, which, together with the 2 constant functions "0" and "1" (a.k.a. "false" and "true" or "low voltage" and "high voltage"), is sufficient for a complete set of computational primitives.
Besides those mentioned above, the main remaining variant for a complete set of computational primitives consists of an analog (possibly weighted) adder and an analog comparator, which had been used in the so-called RTL circuits (resistor-transistor logic) and which also corresponds to perceptrons. RTL had been used in some early integrated circuits, before being replaced by DTL and TTL circuits (which are based on minimum and complement functions).
In hardware, e.g. in RTL circuits, a combined analog adder+comparator can be made with a single high-gain amplifying device, together with a set of weighting resistors and a bias resistor. RTL circuits can implement complex logic with fewer devices (e.g. they can implement a neural network in the analog domain), but they were replaced during the sixties of the past century with DTL, then TTL, because those were faster (in RTL, the resistors limit the charging currents for input capacitors and parasitic capacitors, which slows down the logical transitions) and the fact that they needed more devices was not important, due to the quick increase in circuit density.
This reminds me of a book, whose title I forget, about creating a compact set of knowledge that will enable us to "rebuild civilization from scratch" in case of a future/fictional post-collapse scenario. I sometimes wonder, given the immense complexity and global supply-chain dependencies of our computing stack, whether we could bootstrap from first principles something of equivalent power and expressivity, but orders of magnitude simpler.
There's a wide variety of computational primitives, including lambda calculus, combinators, cellular automata, rewriting systems. Perhaps some are more practical to implement in hardware, particularly the kind of DIY electronics or analog machines that can also be put together from scratch. It might look like a whole building of mechanical switches, powered by a water wheel ("watermill"), for example.
In the early days of machine learning (before the first AI winter), networks like this were often implemented and trained in hardware: https://en.wikipedia.org/wiki/ADALINE
That was the first thing that came to mind when I read "the smallest brain you can build". Nowadays, that "small brain" would likely be built on a breadboard using op-amps instead.
If you want to learn the fundamentals of ML I recommend a book, such as Deep Learning: Foundations and Concepts by Chris Bishop. If you insist on staying online, one option is https://course.fast.ai/
If you don't know ML I don't think you're going to learn much through ad hoc demos.
>I wish I was smart enough to hold it all together.
I used to have a wife, but they took her in the divorce!
The human mind isn't very good at correlating its contents[0]. You can "know" something for years without realizing its implications.
The human mind traverses its knowledge like a man with a small flashlight in total darkness. Our beam of attention is small and narrow, so you need to put the right things in it, or the magic doesn't happen.
This has important implications for learning. I don't know what they are though.
Probably something like, "you can know something without knowing what it means." You haven't connected it to the things it's supposed to be connected to yet. I don't know how to fix that though. (Something involving the Feynman technique, maybe?)
I didn't know Bishop had released a new textbook. I will have to take a look at it. I wasn't the biggest fan of his Pattern Recognition book as I found it overly dense. I much preferred the Murphy and Alpaydin books.
I still find his pattern recognition book useful and informative. It may be dense, but some of us consider that a positive for 'reference' literature. That book was one of very few that still holds up well fr when it was published - truly in on of the last "dark ages" of ML.
I think those down voting you are perhaps overly eager. I upvoted. Grab "Deep Learning" - you'll find it useful, imteresting, and likely less 'dense' in the negative sense!
I have still so many questions left, but regardless of that it was a great read. Thanks for sharing!
The IF statement is the root creator of software programming. It has the ability to compare two values against each other and branch out to blocks of instructions. So it is perceiving (reading), decision making and routing - all that which differentiate life from inanimate objects. The AI agents perform the exact same loop, by delegating the first two steps to a model.
Going further backwards, the transistor (or a PNP junction) is the hardware level enabler of the IF statement. The action (switching) driven by the current which in turn controls other switches, is the first manifestation of "observe and act" by inanimate things at the speed of electricity.
Mechanical equivalents existed ofcourse - speed of a governer which controls the flow of fuel which in turn controls the speed of the governer.
So, what if, we build a stack/set of transistors in same shape as a trained model? It would eliminate most of the software stack too and should run very fast. No memory/gpu required, the chip acts as both storage and processing device, purpose built to be physical model of a trained model.
The alternative IF expression or statement, in the form introduced by John McCarthy in 1958 (which he used in early 1959 to define his version of AND, OR and NOT), is one of the possible primitives for computation.
There exist several equivalent sets of primitive operations. While the sets containing only NAND or only NOR, or both AND and NOT or both OR and NOT are more notorious, these logical operations are more abstract and they do not indicate precisely a hardware implementation, i.e. there are many distinct hardware methods to make such logical gates.
Other sets of primitive operations map directly to hardware devices, e.g. the sets of primitive operations composed of maximum and complement or of minimum and complement map directly to a hardware implementation using rectifier diodes and inverting amplifiers (which can be made with either semiconductor devices or with vacuum tubes, or also with pneumatic or hydraulic devices).
Other sets of primitive operations are obtained by replacing the maximum or minimum circuits with series or parallel connections of switches, like in the CMOS logic that is nowadays dominant.
The alternative IF expression corresponds in hardware to a 2-way multiplexer, which, together with the 2 constant functions "0" and "1" (a.k.a. "false" and "true" or "low voltage" and "high voltage"), is sufficient for a complete set of computational primitives.
Besides those mentioned above, the main remaining variant for a complete set of computational primitives consists of an analog (possibly weighted) adder and an analog comparator, which had been used in the so-called RTL circuits (resistor-transistor logic) and which also corresponds to perceptrons. RTL had been used in some early integrated circuits, before being replaced by DTL and TTL circuits (which are based on minimum and complement functions).
In hardware, e.g. in RTL circuits, a combined analog adder+comparator can be made with a single high-gain amplifying device, together with a set of weighting resistors and a bias resistor. RTL circuits can implement complex logic with fewer devices (e.g. they can implement a neural network in the analog domain), but they were replaced during the sixties of the past century with DTL, then TTL, because those were faster (in RTL, the resistors limit the charging currents for input capacitors and parasitic capacitors, which slows down the logical transitions) and the fact that they needed more devices was not important, due to the quick increase in circuit density.
This reminds me of a book, whose title I forget, about creating a compact set of knowledge that will enable us to "rebuild civilization from scratch" in case of a future/fictional post-collapse scenario. I sometimes wonder, given the immense complexity and global supply-chain dependencies of our computing stack, whether we could bootstrap from first principles something of equivalent power and expressivity, but orders of magnitude simpler.
There's a wide variety of computational primitives, including lambda calculus, combinators, cellular automata, rewriting systems. Perhaps some are more practical to implement in hardware, particularly the kind of DIY electronics or analog machines that can also be put together from scratch. It might look like a whole building of mechanical switches, powered by a water wheel ("watermill"), for example.
it not really an if statement here in a perceptron though. It is more akin a logic gate.
A transistor (driven to saturation) is a much better model.
You might enjoy playing with Turing Tumble.
In the early days of machine learning (before the first AI winter), networks like this were often implemented and trained in hardware: https://en.wikipedia.org/wiki/ADALINE
That was the first thing that came to mind when I read "the smallest brain you can build". Nowadays, that "small brain" would likely be built on a breadboard using op-amps instead.
I learnt a lot today from the interactive demo. You have the best clarity and right skill to educate
Thank you, I will try to make more demo on other concepts.
This is amazing insight, thanks for sharing!
Nice and minimalistic
I played with similar approach in JavaScript and built a NanoNeuron https://github.com/trekhleb/nano-neuron (it is more verbose than Python though)
> A perceptron *is* the smallest brain you can build.
> In 1958, a researcher named Frank Rosenblatt built a machine *he called* the perceptron.
> It was *inspired* by a single brain cell, a neuron.
Yes . But at least the post seems to be written by OP himself!
and its an a great learning resource - which is arguably more important :-)
If you want to learn the fundamentals of ML I recommend a book, such as Deep Learning: Foundations and Concepts by Chris Bishop. If you insist on staying online, one option is https://course.fast.ai/
If you don't know ML I don't think you're going to learn much through ad hoc demos.
Checked out the book on your recommendation, and they even have a free online option on their site! Very generous: https://www.bishopbook.com/
This book equipped me with the right intuition and tools to visualize machine learning. I wish I was smart enough to hold it all together.
>I wish I was smart enough to hold it all together.
I used to have a wife, but they took her in the divorce!
The human mind isn't very good at correlating its contents[0]. You can "know" something for years without realizing its implications.
The human mind traverses its knowledge like a man with a small flashlight in total darkness. Our beam of attention is small and narrow, so you need to put the right things in it, or the magic doesn't happen.
This has important implications for learning. I don't know what they are though.
Probably something like, "you can know something without knowing what it means." You haven't connected it to the things it's supposed to be connected to yet. I don't know how to fix that though. (Something involving the Feynman technique, maybe?)
[0] H.P. Lovecraft quote - https://www.goodreads.com/quotes/193944-the-most-merciful-th...
I didn't know Bishop had released a new textbook. I will have to take a look at it. I wasn't the biggest fan of his Pattern Recognition book as I found it overly dense. I much preferred the Murphy and Alpaydin books.
EDIT: His son is co-author?
I still find his pattern recognition book useful and informative. It may be dense, but some of us consider that a positive for 'reference' literature. That book was one of very few that still holds up well fr when it was published - truly in on of the last "dark ages" of ML.
I think those down voting you are perhaps overly eager. I upvoted. Grab "Deep Learning" - you'll find it useful, imteresting, and likely less 'dense' in the negative sense!
1 reply →
This fast AI course looks soo good man! Definitely I will start learning soon. Thank you!
I can build a smaller brain.
f(x) = 0.
This brain is interesting. Basically you get a no for everything you ask, right?
I take your brain, add a couple more rules, and presto! It can perform any computation.
That's great, now make it learn something :)
It's the simplest AI nihilist!
Yeah, this is small brain time.
[flagged]
[flagged]
[dead]
[flagged]
[flagged]
[dead]
[dead]
Okay, it’s conscious. But can it run doom? I rest my case.
Can you run Doom? Let’s find out!
I can speed run Doom. Appeal denied!
Probably.
https://www.dandwiki.com/wiki/Doom_(5e_Campaign_Setting)