Comment by pixl97

> it doesn't seem they would be able to think, but rather calculate

This may be a distinction without a difference. Just because a program has a 'goal' doesn't mean it will ever reach that goal (halting problem). There is a potentially unbounded, even infinite number of paths a significantly advanced program can take to attempt to reach a destination. Then there is things like ideals of a universal simulation theory that anything that can occur in our universe and also be simulated in binary. This would mean any 'machine' could perform a simulation of anything a human could do.

Hard to say at this point, we still have more to learn about reality at this point.