Comment by nilkn

I don't know that any of this really matters. I could just as well say that you don't know anything at all about the internal experience of your dog's mind. All you can do is read behavioral patterns and attempt to make inferences. At best you have an approximation, and approximations tend to be wrong. Math is correct knowledge, not approximations of reality. Correctness is not a property of the idea or inference itself, since ideas and inferences can be wrong; it's the result of a verification process which you cannot possibly carry out in its entirety relating to the internal conscious experience of your dog.

We can dig into the philosophy of these definitions, but I think the far more interesting point is that even if we grant the existence of this kind of knowledge in the minds of human mathematicians, we have passed the threshold where that "knowledge" can keep up with systems that do not have access to it. Moreover, to claim humans have a "vast amount of mathematical knowledge" that is apparently valuable and that AI systems don't have, you'd have to prove that this "knowledge" is not implied or cannot be reverse engineered from the entire corpus of mathematical writing on which AI systems are trained. You'd also have to demonstrate that this "knowledge" leads to actual results that AI systems cannot generate without it. Given the results AI systems are producing, which are far beyond human ability at this point, it is more likely that AI systems have already internalized the entirety of this so-called "tacit knowledge" and then went much further, much faster, without humans in the loop at all.