Comment by Buttons840

Sure, that's kind of my point.

There is an optimal set of weights that minimizes the loss function for a given set of training data, but we cannot find it.

Granted, even if we could, it might just be overfitting.