Comment by sweezyjeezy
6 years ago
This post oversimplifies the story by putting all the emphasis on compute power. Deep Blue using brute force to solve chess obviously fits this pattern, but the others?
Let's take computer vision. Alex Krizhevsky et al destroyed the ImageNet competition with a neural network in 2012, kicking off the current AI hype cycle. Essentially everything in their model had been known about since the late 80s. But we also didn't know how to train deep networks much before this (it turned out how you initialise the neural network was important), and we also didn't have a big enough dataset to train such a deep model on until ImageNet. Since then, we have built models that perform another order of magnitude better than the 2012 model, mainly because of improvements to the architectures (a combination of ingenuity and a lot of trial and error).
So compute is necessary, but it isn't enough, I don't buy that we've 'brute forced' image recognition in the same way as chess.
Likewise, the search in Go is a Monte Carlo search, very different from the kind of search used in chess. And the neutral nets in alpha go are guiding where to run the search, which is very very different from brute force search.
Many of these things have required the giant leaks in compute, but still wouldn't work at all without the concurrent improvements in algorithms.
Along these lines, here's a classic blog post: https://www.johndcook.com/blog/2015/12/08/algorithms-vs-moor...
"Grötschel, an expert in optimization, observes that a benchmark production planning model solved using linear programming would have taken 82 years to solve in 1988, using the computers and the linear programming algorithms of the day. Fifteen years later — in 2003 — this same model could be solved in roughly 1 minute, an improvement by a factor of roughly 43 million. Of this, a factor of roughly 1,000 was due to increased processor speed, whereas a factor of roughly 43,000 was due to improvements in algorithms! Grötschel also cites an algorithmic improvement of roughly 30,000 for mixed integer programming between 1991 and 2008."
I think the authors point though is that all our effort into the algorithms where algorithms to do just one thing - search - and that we used that in conjunction with more compute power.
I'll agree that the author emphasizes compute power, but his real point still holds. Monte Carlo search may not be classic brute force, and neural networks guiding it may also not be standard, but the two just let you effectively search on a massive scale.
I don't think what you are saying contradicts the text. What he's saying is that we need to put our efforts into how to design and use the tools that tackle the problem space, rather than reasoning about the problem space itself, e.g. how to use neural nets, monte carlo search, etc. That doesn't mean we just throw a for-loop at the data.
But this doesn't work either - convolutional layers in neural networks have a very specific structure, which encodes strong prior knowledge that we have about the problem space (translation invariance). If we just had multilayer perceptions, we wouldn't be talking about this right now.
>convolutional layers in neural networks have a very specific structure, which encodes strong prior knowledge that we have about the problem space
Yes. The point of the author is that it doesn't do this symbolically.
Don't get confused with the terms "brute force", "neural net", etc.
The main idea of the author is that AI that uses brute force, simpler statistical methods, NN, etc, wins over AI that tries to implement some deeper reasoning about the problem domain the way humans do (when thinking about it consciously).
2 replies →
Your observations don't disprove anything in the article. The author doesn't say that brute force is the way to go, and one has to cease all tries to optimize just waiting for computing power to increase. He says that essentially all attempts to build AI by modeling human thinking lead to nowhere, because their are inherently too complex, with simpler statistical, or search-based methods constantly winning. What you wrote here is absolutely in line with his words. "Neural network" contrary to its name doesn't work by emulating human.