Comment by robotresearcher
1 day ago
Two different answers:
1. The discussion setup where you imagine a counterfactual is given by magic does not entail that magic can do any other thing you choose.
“Imagine I give you $100, what do you spend it on?”
“A McClaren, since you could have given me $300K”
2. Taking your question directly: because these are not closely related problems.
1. The discussion setup where you imagine a counterfactual is given by magic does not entail that magic can do any other thing you choose.
My question was a sincere ask to define, in this scenario, what is prohibiting that other magic?
The thought experiment was not assuming AGI, but something like an old school perfect expert at a given task. Maybe like C-3PO only useful as a “protocol droid”, but fairly useless at everything else
I've been so unconvinced by any of these arguments, that all seem to amount to "because I said so." If it can design software that is out of our reach, it can design hardware. It can look at manufacturing data and identify optimizations. "Fairly useless at everything else"? The denial is getting scary, and it's going to come back and bite us.
Then it’s answer number two.