Comment by bluegatty

This is a fine thought, I'm reluctant about it. It could work, I don't think it's obvious. It's very, very hard to know what to train for and not, and this still leaves the 'fact v. skill' problem - even LORA won't enable a model to remember your favourite lunch place.

This is kind of an existential problem with context I think. Maybe we need new architectures.