"If you do not work on an important problem, it's unlikely you'll do important work. It's perfectly obvious. Great scientists have thought through, in a careful way, a number of important problems in their field, and they keep an eye on wondering how to attack them. Let me warn you, `important problem' must be phrased carefully. The three outstanding problems in physics, in a certain sense, were never worked on while I was at Bell Labs. By important I mean guaranteed a Nobel Prize and any sum of money you want to mention. We didn't work on (1) time travel, (2) teleportation, and (3) antigravity. They are not important problems because we do not have an attack. It's not the consequence that makes a problem important, it is that you have a reasonable attack. That is what makes a problem important."
For most of this problems that they mention you need AGI, we do not know how to attack this problem also.
Solving any coding challenge is NP-hard problem. You need to understand not only what you need to do but also how language in which you need to do it works. For example in game of Go you have huge amount of possible states but you have only couple of moves to transition to this states. In programming language every line of code has large space of states and it grows exponentially with every new line of code. Sure you can brute force "hello world" program but good luck with writing 1000 lines of code that work together to solve complex problem. Pattern matching will not help you, training based on github will not help you.
I think you meant to say "NP-hard" . And indeed, there are many program synthesis problems which are NP-hard, 2EXP-complete, or undecidable. But there are also many program synthesis algorithms which are polynomial time. If you use Windows, there might even be one running on your computer.
We've been studying this problem for close to 50 years. I think you see the basic problems, but we know a lot about what to do about them.
To prove that statement you'd have to carefully define your terms.
If "Develop a polynomial time algorithm to determine whether any given 3SAT problem is satisfiable" is a valid coding challenge then the statement is obviously true.
On the other hand I doubt that solving the types of coding challenges that actually appear in competitions can be proven to be NP-hard because that would prove that humans are much better at solving NP-hard problems than any currently known algorithm. I don't think there is any such proof (though there may be some weak evidence for the proposition).
Is that really true? There's a long list of important science which was discovered by accident while working on other things (serendipitous discoveries). In astronomy, you have the discovery of gamma ray bursts and the cosmic microwave background. Viagra is another well known accidental discovery.
If you do not work on an important problem, it's unlikely you'll do important work.
An anecdotal counterpoint would be that Einstein was working as a patent examiner in 1905, his "annus mirabilis"... (Of course it's not very useful to generalise Einstein's career.)
Be careful to differentiate working on important problems and being paid to work on important problems. I've heard (though I can't find any sources) that one of the reasons Einstein took the job at the patent office was because it was such a low effort job that he had time to work on his more important ideas.
Also note that some of the major breakthroughs in the past decade or two have been by mathematicians and scientists that are outside of the "system" (Grigori Perelman, Yitang Zhang (though maybe he's not as much of an outsider)). Maybe a bit outdated but Riemann and Galois also fall into that category.
I thought that the point the grandparent is highlighting is that "important work" is not defined by what other people think is important, but by the fact that you have a unique angle of attack that suddenly makes a previously-unsolved problem tractable. Einstein had a unique angle of attack for all 4 of his annus mirabilis papers, and the photoelectric effect, Michelson-Morley, and Brownian motion were all known to be important unsolved paradoxes in the physics of the day.
> 2. Build an agent to win online programming competitions. A program that can write other programs would be, for obvious reasons, very powerful.
The people on top desperately want us gone, and when it happens it'll happen so quickly you won't know what hit you. Software engineers need to recognize this common threat and organize (labor) sooner than later. Even more important is for all engineers to plan for an imminent future where developers are not paid like they are now, if at all. We have it good, but we will be automated away like everybody else, just a little later.
For simple data import / export tasks, simple CRUD, generic admin dashboards, I agree that those tasks likely will be solvable by AI in the not too distant future.
As a matter of fact, I think it's somewhat bewildering we still need human developers for that kind of task because the tasks themselves don't really require a lot of intelligence. Problem is, the technologies and systems we use to implement them today still require a lot of human interaction and specialist knowledge.
Other than that, implementing an AI that is capable of understanding business problems and solving them via code probably is nothing short of artificial general intelligence. In that case we won't have to worry about not being needed anymore anyway because by then we'll either very rapidly have a post-scarcity economy or you know ... Skynet ...
My feeling is well. If standard CRUD coding gets automated nicely, my feeling is that many (at least some?) CRUD type coders could simply move into a sort of AI business analyst type role. To be honest, business analysis is already a fair bit of the CRUD coder type job. Current AI requires training data in order to learn what the rules are, which means that most likely they will need a team to formulate the rules in a manner that the AI understands.
If AI gets to that level where the programmer is completely replaced, well, regarding "the top" that "wants us gone"... many of them would also be gone. There's no reason a general AI of that power could not automate (to give typical "the top" occupations) much of finance, law, health care, or even management / executive work.
We already have a post-scarcity economy. There is enough of everything to go around for everybody. Kropotkin talked about this in "The Conquest of Bread" in the 19th century. If history is a lesson, no degree of abundance will result in collective plenty.
They took a pretty good swing at solving CRUD with CASE tools. It was an interesting approach, but never had the flexibility to get by without customization.
This is a cynical view of the world. Maybe the people "on top" simply want to improve their company's efficiency in order to return value to investors, etc. My point being its economics, not some evil intent.
Software engineers need to recognize this common threat and organize (labor) sooner than later
And the point of that would be what? To protect our jobs by demanding that industry ignore and no longer pursue innovation? That seems like heresy for anyone in Tech.
If the day does come that AI starts writing code, the lost of our fat salaries and stock options will likely be the least of our problems; or perhaps the world will be void of problems altogether.
When automation comes and replaces us, we need to be owners of robots and AI, in other words to have some kind of capital and enjoy a revenue stream from it. If not robots, then at least we need to have land to cultivate for food. When nobody gives you a job any more, you either cultivate your own food, or associate with others to become investors in the new tech.
Those who already have capital and can invest it smartly might fare much better than those who only rely on BHI, which is at the whims of politicians. A person need not be super rich, if she can associate with other to buy land for agriculture or robots for manufacturing. Robots are analogous to land. You gotta have one or the other :-)
Why should we fight that? That's no different from taxi companies fighting Uber, or how Intuit lobbies congress to avoid automatic filing of taxes. Hampering progress for the greater good just because it harms you in the short term is never a good idea.
This wouldn't harm me in the short term alone, it would be a lost career and a life catastrophe. A taxi driver can work as an Uber driver. I cannot become an automated programmer. So I don't give a damn about the "greater good" in this context, and question that premise also: Sibling mentions getting used to everybody being retirees, but what I see is collective poverty with token conveniences. Time will tell, etc.
> Hampering progress for the greater good just because it harms you in the short term is never a good idea.
I'd love to hear your thesis for this, in a way that doesn't whitewash away the negative effects on the individual. If you're going to claim we should all be throwing ourselves on the burning pyre of progress no matter what I'd love to hear more of a compelling argument than "never a good iea".
I think the better way to avoid our replacement is to think about our non-coding contributions to our companies.
I'm a full-time software engineer, but I spend maybe 50% of my day implementing code. I spend a lot of time figuring out trade-offs, exploring edge cases, and verifying that the business/marketing people know what the side-effects of a given feature will be. There's also a lot of flag-waving to make sure that small problems don't become big problems.
I've run into the "that should be able to be automated" hand-wave of a thought before. I think it's always done by people who haven't actually spent time thinking about the edge-cases and all the little decisions that are put into a final software project. Sure, you can hand-wave and get a CMS, but you're going to be unpleasantly surprised when the defaults don't match your subconscious expectations.
Here's how software engineers could retain their position in a world where our code is automatically generated:
- Understand and communicate the trade-offs of different solutions
- Embrace product design-- a lot of us unwittingly become novice visual designers and product designers during our work, and we should embrace that domain knowledge that we learn, rather than being frustrated that we're being taken away from our primary task, coding.
- Explore edge cases, and explore new ways of doing things
- Understand and get involved in the business process!
The capital holders in all industries would love expensive human labor to be replaced by cheap automated labor. If programming AI is created, no labor organization could save software engineers, just as it would be ridiculous to think horse and buggy drivers or telegraph operators could have stopped the march of progress.
The bigger issue is what has we've already started to witness: automation leads to accelerated accumulation of wealth and resources among those who already have the most capital. Historically, corporate profit and labor demand went up or down together. Now they're diverging: corporate profit grows while demand for labor shrinks. The spoils of corporate success goes to those who own the tools of automation, and others lose their job and income.
In any case, OpenAI's vision here is much bigger than making software developers obsolete: they want to study (or create) AI that can improve itself (i.e., a singularity scenario).
Almost every reply to my comment talks about saving jobs, but I never even brought that up. Organization is important because you want to be left with something more than a seat at the unemployment office when your job does disappear.
The solution to everyone being automated away isn't to halt the advances of automation and inflict inefficient make-work on the economy, it's to get used to the idea that we will all become effectively retirees.
If we can be automated away, we will be. No amount of organizing is going to stop that. And even in some alternate universe where it could, why would you want it to? Automation is what we do.
I see this coming from a much smaller perspective as well.
If one would reflect back on the applications that they've built in their career I suspect a number of those applications are now available generally as packaged software with customizable components.
E.g. The dawn of the web saw many publishing, & catalog systems being built. Now you don't need to, you can simply grab a commercial or OSS package and customize it for your needs. Same is true for document management, workflow.
This hasn't impacted jobs yet because we are still early in this industry, and growth is still outpacing this commoditization. But, yeh, I see the day where software developers become quite rare and software customizers are common.
The tricky challenge is organizing our labour efforts. It's already easy for IT organizations to outsource development to cheaper markets, I fear labour organization would accelerate that movement.
Imagine the potential of a world where anything can be built nearly instantly. If a computer can program itself, we can concentrate on trying to come up with useful things to make, instead of worrying about what we're capable of making. I have a really cool idea for a video game, but it's not going to be made because I don't have time to build it... but if a computer could make it. Wow. I become an imagineer, not a software engineer.
Step it up a level further, if computers could think of programs to build themselves (which they probably would have to if they are going to be anywhere close to practical), that's when the "singularity" truely starts to take off. At the speed that computers operate, programs would very quickly go beyond what humans are capable of.
I actually think that there has been a long-term awareness of this fact in our industry, for decades now, and its one of the driving, motivating factors behind the entropy of the field. The OS-vendor guys know if they improve their OS'es, the systems-programming guys will get automated away, and if they get automated away, the applications-programming guys will be next .. so, subtle imperfections and flaws in the way things operate at a system level are allowed to persist, such that further downstream there are still opportunities to perpetuate the need for hands-on engineering. I can think of a few examples of where it feels like OS vendors/systems developers intentionally hobbled the functionality of their environment in order to make the long-term viability of the industry more resilient ..
So, I think it'll be a long time until 'the people at the top' get what they want. Until they start coding a replacement for Dropbox or SublimeText that will be the end-all of filesharing/file-editing, we'll always need another IPFS, another Atom, etc., etc.
The people on top? How about the software engineers themselves? As a software engineer myself I look forward to the day I'm forced to change careers due to obsolescence if my skills. I can't wait to see what people can achieve when anyone with an idea can make it happen with such ease.
Just in case, since I think I'm up against Poe's law here: I am, in fact, 100% serious.
I'm going to venture a guess that programming will be among the last jobs to be automated away. It's not clear that there will be a career to change into.
>>The people on top desperately want us gone, and when it happens it'll happen so quickly you won't know what hit you.
Isn't, given a problem 'X', produce a solution 'Y' kind of automation, called AGI?
Well then you have little to worry about. If AGI is indeed invented you have bigger things to deal with than worrying about your job.
In any such eventuality 'people at the top' are likely to be eliminated rather more quickly than us. Because an AGI with huge resources can make far better decisions than any CEO ever can.
If everybody else will be automated, who will buy all of these products and services? I dont suppose that AI will need anything else beside electricity and cpu time.
You aren't thinking at the right scale. If a computer program can do any complex task humans can do, worrying about unemployment is the least of our problems. Such a world would be so different than our own. It may have an entirely different economy, and it may not have people at all. I don't know, but I think it's equivalent to a caveman predicting what the effects of steam power would be.
If we end up being automated fully, I think that least important thing anyone can think of is that we will be paid less. I think that something like this is unavoidable but quite dangerous, singularity, etc.
The upside is that I think it would be really accessible for anyone to have it's own personal army of programmers. Although that's also quite dangerous.
Personally, I think you overestimate how difficult it is to translate an arbitrary set of human business rules to an API/website. And how arbitrarily the business side wants to change things without a concrete understanding of what they want the result to look like.
Programming jobs are protected by the halting problem. It has been proven that no program cannot detect if other programs halt or not (i.e. goes into an infinite loop or completes). I paraphrase, but this is the general idea. As long as this is true computers will not be able to program themselves except in very limited ways.
The only way around the halting problem is to use something other than computation as we know it. I.e. an entirely new kind of machine based on different principals. And no, quantum computers do not solve the halting problem since they are still Turing-complete machines.
The halting problem is often misunderstand as "computers can never detect if a program halts". It actually states that "computers can never detect if an arbitrary program halts".
There is a fairly large subset of useful programs that can be proven to halt. Anything that uses straight-line linear control flow. So can anything with that plus conditionals. So can that plus foreach loops, as long as iterators do not reflect updates to their underlying collections. Add a "forever { ... }" construct and you can prove that the program will not halt; in combination with the other constructs, you can prove liveness on each request handler while also guaranteeing that the server itself will never go down.
The two constructs you have to watch out for are loops that mutate state used in the conditional and unbounded recursion. Even for these, there are techniques to increase the set of programs that can be reasoned about, eg. using dataflow analysis to identify which state is mutable and preventing it from being used in conditionals or tracking data & codata through the typesystem.
Such a language would not be Turing-complete; you won't be able to write an interpreter for a Turing-complete programming language in it. But the majority of common business problems don't require an interpreter for another programming language; most of them focus on storing data, triggering events, or computing functions of data.
def collatz(n):
while n > 1:
n = n/2 if n%2==0 else 3*n+1
Is the above program guaranteed to halt for all integer inputs?
In practice, programming doesn't require solving halting problems. We write programs that are on average easy to analyze, especially if you're calibrating the scale with busy beavers. There's no fundamental reason that a computer program can't collect requirements, collect clarifications, and translate those specs into executable code. Clearly it's hard (How do you do the translation? Optimizing Prolog isn't easy! And how do you avoid asking for millions of things that humans take for granted as obvious?), but I don't see anything that makes it impossible.
If we take the simple case of genetic algorithms, we already know that it is possible to brute-force the problem of programming.
The halting problem doesn't really matter that much in this context. Just spawn up a bunch of threads that churn away at the problem, using random mutations, and any that go on too long can just be considered flawed, regardless of whether they have any redeeming qualities.
Then you select the winners, based on the selection criteria, and churn away on some more mutations of those new variants.
There exist equivalent problems for human minds, though. E.g. the Riemann Hypothesis - people keep trying to find the proof, but noone knows if the proof exists, so it's not a given that the search will ever finish.
I'm acutely curious how we can attack the problem of detecting undisclosed AI breakthroughs, particularly those made by organizations uninterested in ever revealing them. Commercially, yes, many organizations want to boast about their AI, and we may find those AIs via games or news. But many organizations involved in national security and financial markets will want to hide their advantage. I would love to hear more about how Open AI thinks this problem can be approached.
By analyzing sequences of amazingly brilliant developments/products/inventions/movements performed by single company or conglomerate of linked companies, and which are few levels above what competitors can deliver.
If you came up with AI algorithms that gave you a massive edge on financial markets, presumably you would also be intelligent enough to execute the strategies discreetly, through highly distributed trading entities, just below levels that raise eyebrows, so as not to be found out.
Similar problem to bot detection on online poker networks, except much harder on real financial markets, and probably not something you can regulate.
Try to understand how exactly Renaissance or Two Sigma are making money with their algorithmic trading, from the outside, I don't think you'll have much luck.
By that measure, startups are powered by AI and large corps are not. By which I mean, so many variables go into successful product development. Google arguably has the strongest AI in the world, but it has been remarkably bad at introducing new products recently, so those two phenomena seem decoupled to me...
This is the biggest problem IMO. Just imagine a couple of AIs that systematically discover a few exploitable loopholes or trading algos all at once; it would be like black & scholes exponentiated. I doubt outside research would be able to infer anything from available data.
For #3, wouldn't the AI defending against hacks need to be a hacking AI itself? It would need to find vulnerabilities in the systems it protects in order to plug them before other AI's find them.
The moment I saw this project start, the first thing I thought was, "Isn't this how all terrible things are created, by trying to avoid having them be created?".
I wonder if the long-term strategy will be called Mutually-Assured Obsolescence™.
Yes and no. The analogy of 'attack' and 'defense' in hacking comes from the military. So to extend that analogy, defensive missiles are still missiles, but they are unquestionably different from offensive missiles.
The point is that in this analogy, no missiles exist yet. Nobody's gone to the effort to figure that out. And then someone says "somebody might figure this out an weaponize it! we gotta figure out how to neutralize it!" and then they do the fundamental research to figure out how to make a missile at all. Now you "just" have to swap out the guidance package and instead of a defensive missile, you've got an offensive on.
Making an AI that finds vulnerabilities is the hard part. Doing an offensive thing or a defensive thing is much easier if you've got an AI that's just spewing out vulns. They don't need to be tightly coupled at all.
It's funny to see grown adults in tech taking the simulation idea seriously just because Elon Musk said it. I was interested in science fiction in high school and thought about these things, but I concluded they were worthless.
Why? It is ontologically no different from 1) solipsism, a philosophical idea that even most philosophers considered worthless centuries ago, or more loosely 2) belief in a higher deity who created the world.
It's completely unfalsifiable. The answer to the question clarifies nothing about our understanding of the world [1] and furthermore has absolutely no impact whatsoever on how you ought to live your life.
[1] Even if you conclude we live in a simulation, we will never know for sure anything about the nature of the simulation or the nature of the world outside the simulation.
You can expand your reasoning to include all religion, while you're at it. And anything anyone does that evidences a belief in something irrational or unfalsifiable.
The thing is, you're not wrong. You're just kind of mean and arrogant to reduce a multiplicity of nuanced worldviews that many people care about very deeply to simple meaninglessness.
> Why? It is ontologically no different from 1) solipsism
Sure, but it is vastly interesting to think about, and imagine about and have meandering ideas about. There's probably also trappings of psychoanalysis in here somewhere.
I tried it for a course in college, "Simulation of Complex Systems". Our goal was to get genetic programs to evolve tribe or herd behavior. After fixing the energy conservation bug that made it evolutionarily successful to immediately eat your own children, I ended up intelligently designing some basic DNA on the night before our presentation.
If we're living in a simulation and our creators see that we're running our own simulations, they may terminate our process to avoid nested virtualization.
But by doing so they could invoke the wrath of the grandparent simulation. Logic leads to an implicit understanding that all recursive simulations must be allowed.
That we're living in a simulation is taken for granted. On the other hand the idea that we can comprehend the nature of the simulation or its creators should not be taken for granted.
Does a glider in Conway's Game of Life comprehend the hardware its running on? Does it know the mind of Conway?
Programs writing other programs is an interesting concept, but isn't it naive in most cases?
If program X can write program Y, program X is probably unnecessarily complex with regards to solving problem Y. Program X could simply execute program Y's operations at their abstract level, which would save it all of the outputted language understanding and translation. The only tangible difference between the two would be the ability to "save" the commands, at which point program X is a scripting language and the input commands are the script.
Wouldn't this solution be better in most cases? If program X at runtime requires significant configuration to produce program Y but those configuration inputs are not saved. Then if program Y were to be changed, program X must be rerun with all of the configuration input again. In the scripting language scenario, the script is modified.
For a programming language to receive something simpler than the language it is written in, the ideas must be abstracted to reduce the amount of code required to perform the task. For example simple addition of multiple elements can be reduced to a "sum" function. The "programmer" using this must still have knowledge of the sum function and its use. Abstractions require the system to have knowledge of that domain, so with each new domain a whole other set of abstractions and complications are introduced.
So if a machine were to write the entire program, how does it differ from machine learning? While machine learning might be often used for smaller algorithms, this case would apply ML to the entire problem as sets of smaller problems. If a human must still program in some abstracted sense, isn't it just a scripting/programming language? How would a component of this suggested project not fit into the "machine learning" or "scripting/programming language" categories?
#2 Is incredibly ambitious but I expect it should be doable by a lone grad student over the course of a summer. Something like this would range in complexity from completely redefining programming languages (drastically lowering the skill required for complex programming tasks and thus feeding into #1) all the way up to general intelligence.
#3 should also allow looking into the use of AI to break into systems, in addition to detecting and defending against AI breaking into systems. A prototype for #4 could create an environment where trilobyte fuzzers could co-evolve into fearsome Artificial intelligences. The AIs that break into systems need not be as smart as the systems being broken into. Just as viruses are much less complicated than eukaryotic cells and yet are capable of wreaking great havoc against mammals, might this be a possible mad deterrent against out of control AI, developed by #1's opponents who use #2's breakthroughs? No AI could plausibly be bug free.
See also: Schild's Ladder.
#4 It would also be cool if humans were allowed to visit and interact with this virtual world.
See also: The Lifecycle of Software Objects
These projects have the same feel as: A PROPOSAL FOR THE DARTMOUTH SUMMER RESEARCH PROJECT ON ARTIFICIAL INTELLIGENCE, whose #4 and #1[+] were the only ones to see much progress. It's difficult to say whether we are 10 or another 50 years away from making meaningful progress on OpenAI's list, but I'm glad they made it because it seems somewhere along the past 60 years, we forgot how to dream.
#2 Might be technically doable: I feel like there has got to be a way to "cheat" this. Maybe make something that Google searches similar problems and then automatically signs up for contests under several names and submits potential code snippet answers as solutions. Your program could run the snippets in sandboxes and ensure the inputs / outputs match the example code for the problem. You do this on a large enough scale you might eventually win a few just by dumb luck + your skill at programming something to detect how well random snippets of code fit a particular problem.
I think that sort of solution would work better as an educational tool or as part of an augmented programming environment. But I don't think it fits within the spirit of the task. It also would hit a ceiling really early on, if the parallel solutions to the Allen AI Science challenge are to be looked at as hints of how things might turn out.
It was tongue in cheek. The original AI researchers posed the problem of General Intelligence to a single grad student and thought he would have it completed by the end of the summer... 50+ years later here we are with no solution.
#2 - Build an agent to win online programming competitions.
Given that online programming competitions tend to fall into distinct classes (dynamic programming, algorithmic challenges, graph problems, string problems), this seems maybe more "solvable" with a non-AI implementation?
Imagine you have a framework that can spit out sub-pieces of a solution that worked in a Unix pipe-like way (e.g, sort the graphs | find strongly connected components in each graph | spit answer of graph with lowest number of SCC).
Then you need to grind out some type of expert system to replicate the competitive programmer who currently chunks together those framework pieces.
Of course, this probably doesn't work at all for something like kaggle or stockfighter.io, where parsing the instructions and observing a dynamic system are key parts of the hacking. I am more thinking of SPOJ or similar . . .
There's been significant progress in data science competitions (Kaggle etc).
Not so much parsing of the problem itself, but building generic frameworks where - given data and a target variable - it will work out the type of problem (eg classification vs regression) and develop a predictive model without human intervention.
ICML has had workshops for the last two years on this problem[1][2]
The hardest part here is not the actual generation of code, but to analyze the problem first and to determine what data structures/algorithms are best suited to tackle it. Also, there are a lot of real world examples used in these problems, so the AI would need a general understanding of the modern-day world, which again, is hard.
Lastly, even if the AI figures out the implementation, it has to take care of all the edge cases too, so that the test cases pass. But I suppose this is a trivial problem compared to the other two I mentioned above...
The goals are an AI that can augment itself (project 2) and defend itself (project 3). I've found Musk's fear of AI to be a little overblown, but this is nuclear weapon type rationalizing: "AI could be end the world as we know it, so we'd better race to create it first."
It's interesting that Projects 2 and 3 would make great businesses even if they only got halfway to "existential threat" level AI. Then there's project 4, version 0.1 of Musk's "we're all living inside a simulation" claim. Musk as creator of worlds.
> Detect if someone is using a covert breakthrough AI system in the world.
I bet the real purpose of this is not
"It seems important to detect this", at least against superhuman AGI.
As that's asking you to win a game against a more intelligent game player, who is going to be able to out-think you and hide.
It seems more likely that:
The real purpose of this problem is to impose a 'secrecy tax' on entities who would like to recoup investment by covertly using such a system, thus reducing the incentive to build one in secret. I.e. if you know someone has invested in research that's going to limit your ability to profit from covertly building an AGI, this means you can only use it up to the limit of game theoretic detectability, reducing the profit motive of acting covertly in the first place.
Its a bit like once you break Enigma, you have to be careful how you use it. Having alternative explanations for how you got your edge then becomes valuable.
So, if you see a corporation building something that could provide an alternative explanations for huge returns, other than AGI - ideally for something that's very noisy (high variance) - and where it'll be hard to tell whether their success was by chance or on purpose, that's probably (very weak) evidence (in the bayesian sense) they are working on a secret AGI.
I suggest YCombinator is an ideal candidate... huge returns, very hard to tell if its luck or skill.
Theses "ideas" sound like either they were written by a 14 year old kid or someone very high. I was hoping for real science.
Yes I know my comment is probably against HN guidelines but there ought to be an exception for making fun of uber rich Silicon Valley types. They parody themselves. It reminds me a lot of the show Silicon Valley.
edit: Boneheaded passion got ahead of me and I forgot that the Gym was intended originally to be cross functional. TO be fair I don't think that aspect of the gym has been well publicized. Original comment retained for context:
How about instead of wasting time on this stuff, they build an actual benchmark test for AGI. Something everyone in the community agrees needs to be built, and basically nobody is working on. I suggest they start with the "anytime intelligence test" from Hernandez-Oralloa as a stepping stone.
"If you do not work on an important problem, it's unlikely you'll do important work. It's perfectly obvious. Great scientists have thought through, in a careful way, a number of important problems in their field, and they keep an eye on wondering how to attack them. Let me warn you, `important problem' must be phrased carefully. The three outstanding problems in physics, in a certain sense, were never worked on while I was at Bell Labs. By important I mean guaranteed a Nobel Prize and any sum of money you want to mention. We didn't work on (1) time travel, (2) teleportation, and (3) antigravity. They are not important problems because we do not have an attack. It's not the consequence that makes a problem important, it is that you have a reasonable attack. That is what makes a problem important."
Richard Hamming, "You and Your Resarch"
http://www.cs.virginia.edu/~robins/YouAndYourResearch.html
For most of this problems that they mention you need AGI, we do not know how to attack this problem also.
Solving any coding challenge is NP-hard problem. You need to understand not only what you need to do but also how language in which you need to do it works. For example in game of Go you have huge amount of possible states but you have only couple of moves to transition to this states. In programming language every line of code has large space of states and it grows exponentially with every new line of code. Sure you can brute force "hello world" program but good luck with writing 1000 lines of code that work together to solve complex problem. Pattern matching will not help you, training based on github will not help you.
Hi, I'm a researcher in program synthesis.
I think you meant to say "NP-hard" . And indeed, there are many program synthesis problems which are NP-hard, 2EXP-complete, or undecidable. But there are also many program synthesis algorithms which are polynomial time. If you use Windows, there might even be one running on your computer.
We've been studying this problem for close to 50 years. I think you see the basic problems, but we know a lot about what to do about them.
11 replies →
Solving any coding challenge is NP-hard problem.
To prove that statement you'd have to carefully define your terms.
If "Develop a polynomial time algorithm to determine whether any given 3SAT problem is satisfiable" is a valid coding challenge then the statement is obviously true.
On the other hand I doubt that solving the types of coding challenges that actually appear in competitions can be proven to be NP-hard because that would prove that humans are much better at solving NP-hard problems than any currently known algorithm. I don't think there is any such proof (though there may be some weak evidence for the proposition).
2 replies →
Is that really true? There's a long list of important science which was discovered by accident while working on other things (serendipitous discoveries). In astronomy, you have the discovery of gamma ray bursts and the cosmic microwave background. Viagra is another well known accidental discovery.
If you do not work on an important problem, it's unlikely you'll do important work.
An anecdotal counterpoint would be that Einstein was working as a patent examiner in 1905, his "annus mirabilis"... (Of course it's not very useful to generalise Einstein's career.)
Be careful to differentiate working on important problems and being paid to work on important problems. I've heard (though I can't find any sources) that one of the reasons Einstein took the job at the patent office was because it was such a low effort job that he had time to work on his more important ideas.
Also note that some of the major breakthroughs in the past decade or two have been by mathematicians and scientists that are outside of the "system" (Grigori Perelman, Yitang Zhang (though maybe he's not as much of an outsider)). Maybe a bit outdated but Riemann and Galois also fall into that category.
1 reply →
I thought that the point the grandparent is highlighting is that "important work" is not defined by what other people think is important, but by the fact that you have a unique angle of attack that suddenly makes a previously-unsolved problem tractable. Einstein had a unique angle of attack for all 4 of his annus mirabilis papers, and the photoelectric effect, Michelson-Morley, and Brownian motion were all known to be important unsolved paradoxes in the physics of the day.
> An anecdotal counterpoint would be that Einstein was working as a patent examiner in 1905, his "annus mirabilis"...
Presumably he worked on big physics problems while also performing his job as patent examiner.
Awesome. A quantitative version of Hamming's thought process seems to be 80,000 hours' Problem Framework.
https://80000hours.org/articles/problem-framework/
What about unexpected discoveries or breakthrough, surprising new uses of previous minor innovations?
Those who developed the minor innovation were "doing important work", as it turned out, without even realizing it.
My understanding is that these are common mechanisms through which important work is accomplished.
When you are uber rich and have no real technical skills you think that things get done by rich people demanding it and throwing money at it.
Well, that can help.
> 2. Build an agent to win online programming competitions. A program that can write other programs would be, for obvious reasons, very powerful.
The people on top desperately want us gone, and when it happens it'll happen so quickly you won't know what hit you. Software engineers need to recognize this common threat and organize (labor) sooner than later. Even more important is for all engineers to plan for an imminent future where developers are not paid like they are now, if at all. We have it good, but we will be automated away like everybody else, just a little later.
For simple data import / export tasks, simple CRUD, generic admin dashboards, I agree that those tasks likely will be solvable by AI in the not too distant future. As a matter of fact, I think it's somewhat bewildering we still need human developers for that kind of task because the tasks themselves don't really require a lot of intelligence. Problem is, the technologies and systems we use to implement them today still require a lot of human interaction and specialist knowledge.
Other than that, implementing an AI that is capable of understanding business problems and solving them via code probably is nothing short of artificial general intelligence. In that case we won't have to worry about not being needed anymore anyway because by then we'll either very rapidly have a post-scarcity economy or you know ... Skynet ...
My feeling is well. If standard CRUD coding gets automated nicely, my feeling is that many (at least some?) CRUD type coders could simply move into a sort of AI business analyst type role. To be honest, business analysis is already a fair bit of the CRUD coder type job. Current AI requires training data in order to learn what the rules are, which means that most likely they will need a team to formulate the rules in a manner that the AI understands.
If AI gets to that level where the programmer is completely replaced, well, regarding "the top" that "wants us gone"... many of them would also be gone. There's no reason a general AI of that power could not automate (to give typical "the top" occupations) much of finance, law, health care, or even management / executive work.
We already have a post-scarcity economy. There is enough of everything to go around for everybody. Kropotkin talked about this in "The Conquest of Bread" in the 19th century. If history is a lesson, no degree of abundance will result in collective plenty.
8 replies →
For simple data import / export tasks, simple CRUD, generic admin dashboards
Tell that to the people who insist that Real Programmers™ always write these things from scratch, never use a library to help them with it, etc.
I hear what you are saying about CRUD, but then I look at Kendo and weep...
2 replies →
They took a pretty good swing at solving CRUD with CASE tools. It was an interesting approach, but never had the flexibility to get by without customization.
1 reply →
>I agree that those tasks likely will be solvable by AI in the not too distant future.
What experience or understanding about "AI" do you base your opinion on?
The people on top desperately want us gone
This is a cynical view of the world. Maybe the people "on top" simply want to improve their company's efficiency in order to return value to investors, etc. My point being its economics, not some evil intent.
Software engineers need to recognize this common threat and organize (labor) sooner than later
And the point of that would be what? To protect our jobs by demanding that industry ignore and no longer pursue innovation? That seems like heresy for anyone in Tech.
If the day does come that AI starts writing code, the lost of our fat salaries and stock options will likely be the least of our problems; or perhaps the world will be void of problems altogether.
When automation comes and replaces us, we need to be owners of robots and AI, in other words to have some kind of capital and enjoy a revenue stream from it. If not robots, then at least we need to have land to cultivate for food. When nobody gives you a job any more, you either cultivate your own food, or associate with others to become investors in the new tech.
Those who already have capital and can invest it smartly might fare much better than those who only rely on BHI, which is at the whims of politicians. A person need not be super rich, if she can associate with other to buy land for agriculture or robots for manufacturing. Robots are analogous to land. You gotta have one or the other :-)
Why should we fight that? That's no different from taxi companies fighting Uber, or how Intuit lobbies congress to avoid automatic filing of taxes. Hampering progress for the greater good just because it harms you in the short term is never a good idea.
This wouldn't harm me in the short term alone, it would be a lost career and a life catastrophe. A taxi driver can work as an Uber driver. I cannot become an automated programmer. So I don't give a damn about the "greater good" in this context, and question that premise also: Sibling mentions getting used to everybody being retirees, but what I see is collective poverty with token conveniences. Time will tell, etc.
13 replies →
> Hampering progress for the greater good just because it harms you in the short term is never a good idea.
I'd love to hear your thesis for this, in a way that doesn't whitewash away the negative effects on the individual. If you're going to claim we should all be throwing ourselves on the burning pyre of progress no matter what I'd love to hear more of a compelling argument than "never a good iea".
9 replies →
I have no interest in organizing to artificially give myself a job that isn't necessary. In fact that seems insane.
I think the better way to avoid our replacement is to think about our non-coding contributions to our companies.
I'm a full-time software engineer, but I spend maybe 50% of my day implementing code. I spend a lot of time figuring out trade-offs, exploring edge cases, and verifying that the business/marketing people know what the side-effects of a given feature will be. There's also a lot of flag-waving to make sure that small problems don't become big problems.
I've run into the "that should be able to be automated" hand-wave of a thought before. I think it's always done by people who haven't actually spent time thinking about the edge-cases and all the little decisions that are put into a final software project. Sure, you can hand-wave and get a CMS, but you're going to be unpleasantly surprised when the defaults don't match your subconscious expectations.
Here's how software engineers could retain their position in a world where our code is automatically generated:
- Understand and communicate the trade-offs of different solutions
- Embrace product design-- a lot of us unwittingly become novice visual designers and product designers during our work, and we should embrace that domain knowledge that we learn, rather than being frustrated that we're being taken away from our primary task, coding.
- Explore edge cases, and explore new ways of doing things
- Understand and get involved in the business process!
The capital holders in all industries would love expensive human labor to be replaced by cheap automated labor. If programming AI is created, no labor organization could save software engineers, just as it would be ridiculous to think horse and buggy drivers or telegraph operators could have stopped the march of progress.
The bigger issue is what has we've already started to witness: automation leads to accelerated accumulation of wealth and resources among those who already have the most capital. Historically, corporate profit and labor demand went up or down together. Now they're diverging: corporate profit grows while demand for labor shrinks. The spoils of corporate success goes to those who own the tools of automation, and others lose their job and income.
In any case, OpenAI's vision here is much bigger than making software developers obsolete: they want to study (or create) AI that can improve itself (i.e., a singularity scenario).
Almost every reply to my comment talks about saving jobs, but I never even brought that up. Organization is important because you want to be left with something more than a seat at the unemployment office when your job does disappear.
3 replies →
> Software engineers need to recognize this common threat and organize (labor) sooner than later.
Has organized (labor) ever saved organized labor's job?
From obsolescence? Never. The only thing it does is anticipate it.
Organized labor is useful against a whole lot of problems, but not this one.
The solution to everyone being automated away isn't to halt the advances of automation and inflict inefficient make-work on the economy, it's to get used to the idea that we will all become effectively retirees.
If we can be automated away, we will be. No amount of organizing is going to stop that. And even in some alternate universe where it could, why would you want it to? Automation is what we do.
I see this coming from a much smaller perspective as well.
If one would reflect back on the applications that they've built in their career I suspect a number of those applications are now available generally as packaged software with customizable components.
E.g. The dawn of the web saw many publishing, & catalog systems being built. Now you don't need to, you can simply grab a commercial or OSS package and customize it for your needs. Same is true for document management, workflow.
This hasn't impacted jobs yet because we are still early in this industry, and growth is still outpacing this commoditization. But, yeh, I see the day where software developers become quite rare and software customizers are common.
The tricky challenge is organizing our labour efforts. It's already easy for IT organizations to outsource development to cheaper markets, I fear labour organization would accelerate that movement.
Imagine the potential of a world where anything can be built nearly instantly. If a computer can program itself, we can concentrate on trying to come up with useful things to make, instead of worrying about what we're capable of making. I have a really cool idea for a video game, but it's not going to be made because I don't have time to build it... but if a computer could make it. Wow. I become an imagineer, not a software engineer.
Step it up a level further, if computers could think of programs to build themselves (which they probably would have to if they are going to be anywhere close to practical), that's when the "singularity" truely starts to take off. At the speed that computers operate, programs would very quickly go beyond what humans are capable of.
Organize to do what, exactly? If software engineers became obsolete, why should anyone listen to an organization of them?
It's not in any danger of happening soon, but if/when it happens, you might as well join the union of street gas lamp lighters.
I actually think that there has been a long-term awareness of this fact in our industry, for decades now, and its one of the driving, motivating factors behind the entropy of the field. The OS-vendor guys know if they improve their OS'es, the systems-programming guys will get automated away, and if they get automated away, the applications-programming guys will be next .. so, subtle imperfections and flaws in the way things operate at a system level are allowed to persist, such that further downstream there are still opportunities to perpetuate the need for hands-on engineering. I can think of a few examples of where it feels like OS vendors/systems developers intentionally hobbled the functionality of their environment in order to make the long-term viability of the industry more resilient ..
So, I think it'll be a long time until 'the people at the top' get what they want. Until they start coding a replacement for Dropbox or SublimeText that will be the end-all of filesharing/file-editing, we'll always need another IPFS, another Atom, etc., etc.
The people on top? How about the software engineers themselves? As a software engineer myself I look forward to the day I'm forced to change careers due to obsolescence if my skills. I can't wait to see what people can achieve when anyone with an idea can make it happen with such ease.
Just in case, since I think I'm up against Poe's law here: I am, in fact, 100% serious.
I'm going to venture a guess that programming will be among the last jobs to be automated away. It's not clear that there will be a career to change into.
>>The people on top desperately want us gone, and when it happens it'll happen so quickly you won't know what hit you.
Isn't, given a problem 'X', produce a solution 'Y' kind of automation, called AGI?
Well then you have little to worry about. If AGI is indeed invented you have bigger things to deal with than worrying about your job.
In any such eventuality 'people at the top' are likely to be eliminated rather more quickly than us. Because an AGI with huge resources can make far better decisions than any CEO ever can.
If everybody else will be automated, who will buy all of these products and services? I dont suppose that AI will need anything else beside electricity and cpu time.
You aren't thinking at the right scale. If a computer program can do any complex task humans can do, worrying about unemployment is the least of our problems. Such a world would be so different than our own. It may have an entirely different economy, and it may not have people at all. I don't know, but I think it's equivalent to a caveman predicting what the effects of steam power would be.
If we end up being automated fully, I think that least important thing anyone can think of is that we will be paid less. I think that something like this is unavoidable but quite dangerous, singularity, etc.
The upside is that I think it would be really accessible for anyone to have it's own personal army of programmers. Although that's also quite dangerous.
Personally, I think you overestimate how difficult it is to translate an arbitrary set of human business rules to an API/website. And how arbitrarily the business side wants to change things without a concrete understanding of what they want the result to look like.
Programming jobs are protected by the halting problem. It has been proven that no program cannot detect if other programs halt or not (i.e. goes into an infinite loop or completes). I paraphrase, but this is the general idea. As long as this is true computers will not be able to program themselves except in very limited ways.
The only way around the halting problem is to use something other than computation as we know it. I.e. an entirely new kind of machine based on different principals. And no, quantum computers do not solve the halting problem since they are still Turing-complete machines.
The halting problem is often misunderstand as "computers can never detect if a program halts". It actually states that "computers can never detect if an arbitrary program halts".
There is a fairly large subset of useful programs that can be proven to halt. Anything that uses straight-line linear control flow. So can anything with that plus conditionals. So can that plus foreach loops, as long as iterators do not reflect updates to their underlying collections. Add a "forever { ... }" construct and you can prove that the program will not halt; in combination with the other constructs, you can prove liveness on each request handler while also guaranteeing that the server itself will never go down.
The two constructs you have to watch out for are loops that mutate state used in the conditional and unbounded recursion. Even for these, there are techniques to increase the set of programs that can be reasoned about, eg. using dataflow analysis to identify which state is mutable and preventing it from being used in conditionals or tracking data & codata through the typesystem.
http://blog.sigfpe.com/2007/07/data-and-codata.html
Such a language would not be Turing-complete; you won't be able to write an interpreter for a Turing-complete programming language in it. But the majority of common business problems don't require an interpreter for another programming language; most of them focus on storing data, triggering events, or computing functions of data.
9 replies →
Humans can't solve the halting problem either.
Here's a nice simple one for you:
Is the above program guaranteed to halt for all integer inputs?
In practice, programming doesn't require solving halting problems. We write programs that are on average easy to analyze, especially if you're calibrating the scale with busy beavers. There's no fundamental reason that a computer program can't collect requirements, collect clarifications, and translate those specs into executable code. Clearly it's hard (How do you do the translation? Optimizing Prolog isn't easy! And how do you avoid asking for millions of things that humans take for granted as obvious?), but I don't see anything that makes it impossible.
If we take the simple case of genetic algorithms, we already know that it is possible to brute-force the problem of programming.
The halting problem doesn't really matter that much in this context. Just spawn up a bunch of threads that churn away at the problem, using random mutations, and any that go on too long can just be considered flawed, regardless of whether they have any redeeming qualities.
Then you select the winners, based on the selection criteria, and churn away on some more mutations of those new variants.
1 reply →
There exist equivalent problems for human minds, though. E.g. the Riemann Hypothesis - people keep trying to find the proof, but noone knows if the proof exists, so it's not a given that the search will ever finish.
That's the only way to totally solve it. You can get pretty far with imperfect shortcuts. Maybe far enough that it doesn't matter.
This profession is probably the only one that automates itself. It has always been like that for developers, yet they 're still in high demand.
Seriously, why would you need programmers if the machine does it better (this needs to be assessed somehow first)?
I'm acutely curious how we can attack the problem of detecting undisclosed AI breakthroughs, particularly those made by organizations uninterested in ever revealing them. Commercially, yes, many organizations want to boast about their AI, and we may find those AIs via games or news. But many organizations involved in national security and financial markets will want to hide their advantage. I would love to hear more about how Open AI thinks this problem can be approached.
By analyzing sequences of amazingly brilliant developments/products/inventions/movements performed by single company or conglomerate of linked companies, and which are few levels above what competitors can deliver.
I am not from OpenAI though.
If you came up with AI algorithms that gave you a massive edge on financial markets, presumably you would also be intelligent enough to execute the strategies discreetly, through highly distributed trading entities, just below levels that raise eyebrows, so as not to be found out.
Similar problem to bot detection on online poker networks, except much harder on real financial markets, and probably not something you can regulate.
Try to understand how exactly Renaissance or Two Sigma are making money with their algorithmic trading, from the outside, I don't think you'll have much luck.
By that measure, startups are powered by AI and large corps are not. By which I mean, so many variables go into successful product development. Google arguably has the strongest AI in the world, but it has been remarkably bad at introducing new products recently, so those two phenomena seem decoupled to me...
1 reply →
This is the biggest problem IMO. Just imagine a couple of AIs that systematically discover a few exploitable loopholes or trading algos all at once; it would be like black & scholes exponentiated. I doubt outside research would be able to infer anything from available data.
Step one: Stop calling equations "AI".
Step zero: Define "Intelligence"consistently and unambiguously.
This reminds me of an excellent two post series by Tim Urban about the upcoming AI revolution: http://waitbutwhy.com/2015/01/artificial-intelligence-revolu...
For #3, wouldn't the AI defending against hacks need to be a hacking AI itself? It would need to find vulnerabilities in the systems it protects in order to plug them before other AI's find them.
Exactly[1].
The moment I saw this project start, the first thing I thought was, "Isn't this how all terrible things are created, by trying to avoid having them be created?".
I wonder if the long-term strategy will be called Mutually-Assured Obsolescence™.
1. https://en.wikipedia.org/wiki/Skynet_(Terminator)
Yes and no. The analogy of 'attack' and 'defense' in hacking comes from the military. So to extend that analogy, defensive missiles are still missiles, but they are unquestionably different from offensive missiles.
The point is that in this analogy, no missiles exist yet. Nobody's gone to the effort to figure that out. And then someone says "somebody might figure this out an weaponize it! we gotta figure out how to neutralize it!" and then they do the fundamental research to figure out how to make a missile at all. Now you "just" have to swap out the guidance package and instead of a defensive missile, you've got an offensive on.
Making an AI that finds vulnerabilities is the hard part. Doing an offensive thing or a defensive thing is much easier if you've got an AI that's just spewing out vulns. They don't need to be tightly coupled at all.
> 4. A complex simulation with many long-lived agents.
Maybe someday we'll discover that this was exactly how the ancestor simulation we're living in started.
It's funny to see grown adults in tech taking the simulation idea seriously just because Elon Musk said it. I was interested in science fiction in high school and thought about these things, but I concluded they were worthless.
Why? It is ontologically no different from 1) solipsism, a philosophical idea that even most philosophers considered worthless centuries ago, or more loosely 2) belief in a higher deity who created the world.
It's completely unfalsifiable. The answer to the question clarifies nothing about our understanding of the world [1] and furthermore has absolutely no impact whatsoever on how you ought to live your life.
[1] Even if you conclude we live in a simulation, we will never know for sure anything about the nature of the simulation or the nature of the world outside the simulation.
You can expand your reasoning to include all religion, while you're at it. And anything anyone does that evidences a belief in something irrational or unfalsifiable.
The thing is, you're not wrong. You're just kind of mean and arrogant to reduce a multiplicity of nuanced worldviews that many people care about very deeply to simple meaninglessness.
4 replies →
The negative ("we are not in a simulation") is the falsifiable version. It could be falsified by discovering a flaw in the simulator.
Bostrom's formulation has a properly constructed hypothesis: see http://simulation-argument.com/simulation.pdf
1 reply →
> Why? It is ontologically no different from 1) solipsism
Sure, but it is vastly interesting to think about, and imagine about and have meandering ideas about. There's probably also trappings of psychoanalysis in here somewhere.
"clarifies nothing about our understanding of the world" ... LOL
I tried it for a course in college, "Simulation of Complex Systems". Our goal was to get genetic programs to evolve tribe or herd behavior. After fixing the energy conservation bug that made it evolutionarily successful to immediately eat your own children, I ended up intelligently designing some basic DNA on the night before our presentation.
This particular subject interests me to no end.
Here are a few resources that might be helpful to anyone interested
https://www.complexityexplorer.org/
http://web.mit.edu/redingtn/www/netadv/Xcomplexit.html
http://tuvalu.santafe.edu/~simon/page6/page6.html
1 reply →
Evolutionarily beneficial to eat your own children? Could you explain how?
1 reply →
I played a similar game on the Tandy computer in the 80s.
If we're living in a simulation and our creators see that we're running our own simulations, they may terminate our process to avoid nested virtualization.
But by doing so they could invoke the wrath of the grandparent simulation. Logic leads to an implicit understanding that all recursive simulations must be allowed.
Why would our creators want to avoid nested simulation?
4 replies →
They probably already figured it out if they can build a simulation this huge.
That we're living in a simulation is taken for granted. On the other hand the idea that we can comprehend the nature of the simulation or its creators should not be taken for granted.
Does a glider in Conway's Game of Life comprehend the hardware its running on? Does it know the mind of Conway?
Programs writing other programs is an interesting concept, but isn't it naive in most cases?
If program X can write program Y, program X is probably unnecessarily complex with regards to solving problem Y. Program X could simply execute program Y's operations at their abstract level, which would save it all of the outputted language understanding and translation. The only tangible difference between the two would be the ability to "save" the commands, at which point program X is a scripting language and the input commands are the script.
Wouldn't this solution be better in most cases? If program X at runtime requires significant configuration to produce program Y but those configuration inputs are not saved. Then if program Y were to be changed, program X must be rerun with all of the configuration input again. In the scripting language scenario, the script is modified.
For a programming language to receive something simpler than the language it is written in, the ideas must be abstracted to reduce the amount of code required to perform the task. For example simple addition of multiple elements can be reduced to a "sum" function. The "programmer" using this must still have knowledge of the sum function and its use. Abstractions require the system to have knowledge of that domain, so with each new domain a whole other set of abstractions and complications are introduced.
So if a machine were to write the entire program, how does it differ from machine learning? While machine learning might be often used for smaller algorithms, this case would apply ML to the entire problem as sets of smaller problems. If a human must still program in some abstracted sense, isn't it just a scripting/programming language? How would a component of this suggested project not fit into the "machine learning" or "scripting/programming language" categories?
#2 Is incredibly ambitious but I expect it should be doable by a lone grad student over the course of a summer. Something like this would range in complexity from completely redefining programming languages (drastically lowering the skill required for complex programming tasks and thus feeding into #1) all the way up to general intelligence.
The tests referenced in [https://arxiv.org/ftp/arxiv/papers/1604/1604.04315.pdf], currently dominated by information retrieval techniques, seem more realistic and still feel hopelessly far away.
#3 should also allow looking into the use of AI to break into systems, in addition to detecting and defending against AI breaking into systems. A prototype for #4 could create an environment where trilobyte fuzzers could co-evolve into fearsome Artificial intelligences. The AIs that break into systems need not be as smart as the systems being broken into. Just as viruses are much less complicated than eukaryotic cells and yet are capable of wreaking great havoc against mammals, might this be a possible mad deterrent against out of control AI, developed by #1's opponents who use #2's breakthroughs? No AI could plausibly be bug free.
See also: Schild's Ladder.
#4 It would also be cool if humans were allowed to visit and interact with this virtual world.
See also: The Lifecycle of Software Objects
These projects have the same feel as: A PROPOSAL FOR THE DARTMOUTH SUMMER RESEARCH PROJECT ON ARTIFICIAL INTELLIGENCE, whose #4 and #1[+] were the only ones to see much progress. It's difficult to say whether we are 10 or another 50 years away from making meaningful progress on OpenAI's list, but I'm glad they made it because it seems somewhere along the past 60 years, we forgot how to dream.
[+] #3 warrants a honorable mention.
#2 Might be technically doable: I feel like there has got to be a way to "cheat" this. Maybe make something that Google searches similar problems and then automatically signs up for contests under several names and submits potential code snippet answers as solutions. Your program could run the snippets in sandboxes and ensure the inputs / outputs match the example code for the problem. You do this on a large enough scale you might eventually win a few just by dumb luck + your skill at programming something to detect how well random snippets of code fit a particular problem.
I think that sort of solution would work better as an educational tool or as part of an augmented programming environment. But I don't think it fits within the spirit of the task. It also would hit a ceiling really early on, if the parallel solutions to the Allen AI Science challenge are to be looked at as hints of how things might turn out.
You seriously think #2 is so easy a lone grad student could casually do it over a summer? ......
It was tongue in cheek. The original AI researchers posed the problem of General Intelligence to a single grad student and thought he would have it completed by the end of the summer... 50+ years later here we are with no solution.
#2 - Build an agent to win online programming competitions.
Given that online programming competitions tend to fall into distinct classes (dynamic programming, algorithmic challenges, graph problems, string problems), this seems maybe more "solvable" with a non-AI implementation?
Imagine you have a framework that can spit out sub-pieces of a solution that worked in a Unix pipe-like way (e.g, sort the graphs | find strongly connected components in each graph | spit answer of graph with lowest number of SCC).
Then you need to grind out some type of expert system to replicate the competitive programmer who currently chunks together those framework pieces.
Of course, this probably doesn't work at all for something like kaggle or stockfighter.io, where parsing the instructions and observing a dynamic system are key parts of the hacking. I am more thinking of SPOJ or similar . . .
There's been significant progress in data science competitions (Kaggle etc).
Not so much parsing of the problem itself, but building generic frameworks where - given data and a target variable - it will work out the type of problem (eg classification vs regression) and develop a predictive model without human intervention.
ICML has had workshops for the last two years on this problem[1][2]
[1] https://sites.google.com/site/automl2016/
[2] https://sites.google.com/site/automlwsicml15/
The hardest part here is not the actual generation of code, but to analyze the problem first and to determine what data structures/algorithms are best suited to tackle it. Also, there are a lot of real world examples used in these problems, so the AI would need a general understanding of the modern-day world, which again, is hard.
Lastly, even if the AI figures out the implementation, it has to take care of all the edge cases too, so that the test cases pass. But I suppose this is a trivial problem compared to the other two I mentioned above...
> non-AI implementation
> a framework that can spit out sub-pieces of a solution
> some type of expert system
What you've described is an AI implementation.
I love how much these things just FEEL like Elon Musk fantasizing about an idea, and just really wanting someone to work on it.
lol fagget
The goals are an AI that can augment itself (project 2) and defend itself (project 3). I've found Musk's fear of AI to be a little overblown, but this is nuclear weapon type rationalizing: "AI could be end the world as we know it, so we'd better race to create it first."
It's interesting that Projects 2 and 3 would make great businesses even if they only got halfway to "existential threat" level AI. Then there's project 4, version 0.1 of Musk's "we're all living inside a simulation" claim. Musk as creator of worlds.
The security project reminded me of the DARPA Cyber Grand Challenge - https://www.cybergrandchallenge.com/
> Detect if someone is using a covert breakthrough AI system in the world.
I bet the real purpose of this is not "It seems important to detect this", at least against superhuman AGI.
As that's asking you to win a game against a more intelligent game player, who is going to be able to out-think you and hide.
It seems more likely that:
The real purpose of this problem is to impose a 'secrecy tax' on entities who would like to recoup investment by covertly using such a system, thus reducing the incentive to build one in secret. I.e. if you know someone has invested in research that's going to limit your ability to profit from covertly building an AGI, this means you can only use it up to the limit of game theoretic detectability, reducing the profit motive of acting covertly in the first place.
Its a bit like once you break Enigma, you have to be careful how you use it. Having alternative explanations for how you got your edge then becomes valuable.
So, if you see a corporation building something that could provide an alternative explanations for huge returns, other than AGI - ideally for something that's very noisy (high variance) - and where it'll be hard to tell whether their success was by chance or on purpose, that's probably (very weak) evidence (in the bayesian sense) they are working on a secret AGI.
I suggest YCombinator is an ideal candidate... huge returns, very hard to tell if its luck or skill.
I suggest YCombinator is an ideal candidate... huge returns, very hard to tell if its luck or skill.
Maybe pg is actually an artificial super-intelligence. I mean, has anybody actually seen him in real life???
5. Focus on how you would use AI to augment personal human thinking.
#1 - reminded me of "Endgame: Singularity" (https://en.wikipedia.org/wiki/Endgame%3A_Singularity)
That stuff sounds so generic, not sure what is special about it :-)
I christen number 1 "The Ghost in the Shell Problem", in reference to the project at the core of the first movie's plot.
#1 - talk about distributed, automated, statistical analysis. What an exciting project to get to work on.
> What an exciting project to get to work on.
Indeed :). Anyone who's excited, please apply! Feel free to ping me with any questions: gdb@openai.com.
Anyone who starts a project for #1 will get instantly targeted by Samaritan's agents, right?
Is this job application for humans only or AI can apply?
Riffing on project no. 1, here's our post on "How to Control AI": http://deeplearning4j.org/skynet
>A complex simulation with many long-lived agents.
No need to start a new one here, just download Dwarf Fortress.
Except that DF Dwarves are, by definition, not long-lived.
This will certainly backfire
Theses "ideas" sound like either they were written by a 14 year old kid or someone very high. I was hoping for real science.
Yes I know my comment is probably against HN guidelines but there ought to be an exception for making fun of uber rich Silicon Valley types. They parody themselves. It reminds me a lot of the show Silicon Valley.
care to elaborate why this isn't "real science"?
It's real science fiction. None of these problems are even close to being practical except in the fantasy world created by pop-sci news outlets.
I was hoping for a serious discussion about the problems on the forefront of machine learning.
12 replies →
Out there are a lot of announcements of really existent research results, even popular ones.
1 reply →
edit: Boneheaded passion got ahead of me and I forgot that the Gym was intended originally to be cross functional. TO be fair I don't think that aspect of the gym has been well publicized. Original comment retained for context:
How about instead of wasting time on this stuff, they build an actual benchmark test for AGI. Something everyone in the community agrees needs to be built, and basically nobody is working on. I suggest they start with the "anytime intelligence test" from Hernandez-Oralloa as a stepping stone.
Oh wait, benchmarks aren't sexy. Nevermind.
> they build an actual benchmark test for AGI
This is actually the project I'm currently leading (goal #1 here: https://openai.com/blog/openai-technical-goals/). We should have something interesting to share in a few months.
Duh, forgot about the gym. Sorry.
1 reply →