Comment by vidarh
5 hours ago
I agree re: subjective experience. It seems like something inherently unknowable.
What we could do is break down other facets of what we talk about under the umbrella of consciousness, and find measurable subsets.
E.g. a lot of people insists LLMs can't reason either. Coming up with a testable definition of what that means might be doable.
Overall also just separating the subjective experience from the rest leaves us in a position where proving the possibility of AGI "just" rests on whether or not human brains exceed the Turing computable.
If they don't, then subjective experience or not is irrelevant for the question of reasoning and intelligence, as in that case a subjective experience either can't affect the computation or must itself be at least possible to fully simulate by any Turing complete system.
The problem of subjective experience then would largely be down to faith and feelings but would also be entirely orthogonal to the rest.
Yes, our whole concept of "consciousness" is very anthropocentric. There's no particular reason to think subjective experience, moral standing, and general problem-solving ability are inherently connected. (self-nitpick: you could argue moral standing requires subjectivity, but it's still easy to imagine a being with subjectivity that's incapable of suffering and indifferent to its fate)