Comment by d_tr

"A Course in Modern Mathematical Physics: Groups, Hilbert Space and Differential Geometry" by Peter Szekeres has several chapters on linear algebra early on, starting with sets and groups and including tensors (which you absolutely need for QM).

I loved it.