Comment by Lerc

I'm not sure what 17 year old me would have done with YouTube tutorials for everything under the sun available.

I'm much older and less wise now, but I still afforded myself the opportunity to follow karpathy's tutorials to build a LLM from scratch. Got to play with a few ideas. Seen similar ideas turn up in frontier model work, which is quite gratifying.

There are so many ideas to try.

Currently playing with autoencoders that takes A and B and produce latents A', B', and C'. Reconstruction of A is from A' and C', B is from B' and C'

The idea is if C' can be made to improve both outputs, it must store as much information as it can about what is common to both inputs.