Comment by TheOtherHobbes
11 hours ago
Possibly, but our knowledge of physical laws is subjective and imperfect. By the time you get to Frauchiger-Renner naive realism starts to look quite threadbare, so I'd be a little more tentative about assuming we have much of a clue about what's going on.
It's just as likely - far more likely IMO - that we're physically incapable of understanding physical laws and physical systems on their own terms.
Evolved systems need no high order understanding of how they work. You have evolution take care of that hard work for you, hence why these models take exaflops of compute and gigawatts of power to train.
Understanding and accidentally creating are different things. If we already understood consciousness and weren't worried about having accidentally created one already, we won't be having this discussion. If we don't know what consciousness is, in my view its more reason to exercise some caution regarding a system that appears to exhibit intelligent thinking, talking, etc.