Comment by mkirsten
7 hours ago
Interesting write up! I’ve thought about how university exams are done effectively nowadays. I took my degree in CS almost 20 years ago, and being a user of LLMS - I can’t really see how any of my old exams would work today if students would be allowed LLMs.
I graduated 15 years ago, and I think the exams in my degree were actually the most LLM-proof part of the student assessment. They were no-aid written exams with pencils and paper, whereas the assignments were online-submitted code only that an LLM could easily write.
Spoiler: they don't.
CS exercises that we can expect an average student to solve is trivially solved by LLMs. Even smaller local models.
This comes to not "smartness" of LLMs. But reality that we do not even want anything novel in these exercises or exams. And same areas are repeated multiple times so naturally there is lot of these in training data.
This is one area where LLMs really should excel at. And that doesn't really mean that students should not also learn it and be able to solve same issues. Which is real dilemma for the school system...
Except some times they aren't solved by LLMs, but appear to be. A CS student should be able to look at it and tell the difference.