Comment by knallfrosch
7 hours ago
I don't understand.
10 years ago, we wrote exams by hand with whatever we understood (in our heads.)
No colleagues, no laptops, no internet, no LLMs.
This approach still works, why do something else? Unless you're specifically testing a student's ability to Google, they don't need access to it.
I am returning to this model in my classes: pen in paper quizzes, no digital devices. I also do seven equally weighted quizzes to deescalate them individually. I have reduced project/programming weight from 60-80% of my grade to 50% because it is not possible to tell if the students actually did the work.
I am also doing the same. 50% for project work and 50% for individual work, including paper and pen exams with no digital devices allowed.
The days of take home exams and coding lab assignments are gone...
is "individual work" the pen and paper?
For a project I'm but so sure banning LLMs is actually the right approach.
Industry is full of people trying to use them to become more productive.
Why wouldn't you let students use the same tools?
Seems like you need to make the projects much harder.
But the problem is, students need to learn to do the easy things themselves before they can do the hard things with LLMs.
If you ask them to build a web browser when they can't do a hello world on their own, it's going to be a disaster. LLMs are like dumb juniors that you command, but students are less skilled than dumb juniors when they start programming classes..
It's like learning to factor polynomials even thought a computer algebra system on a graphic calculator can do that.
Do you think children should still be expected to be able to do arithmetic by hand?
I think the answer maybe comes down to figuring out exactly what the goal of school is. Are you trying to educate people or train them? There is for sure a lot of overlap, but I think there's a pretty clear distinction and I definitely favor the education side. On the job, a person with a solid education should be able to use whatever language or framework they need with very little training required.
We're trying to evaluate the student not the LLM. You need to tease apart their contributions. Isn't this obvious?
> This approach still works, why do something else?
One issue is that the time provided to mark each piece of work continues to decrease. Sometimes you are only getting 15 minutes for 20 pages, and management believe that you can mark back-to-back from 9-5 with a half hour lunch. The only thing keeping people sane is the students that fail to submit, or submit something obviously sub-par. So where possible, even for designing exams, you try to limit text altogether. Multiple choice, drawing lines, a basic diagram, a calculation, etc.
Some students have terrible handwriting. I wouldn't be against the use of a dumb terminal in an exam room/hall. Maybe in the background it could be syncing the text and backing it up.
> Unless you're specifically testing a student's ability to Google, they don't need access to it.
I've been the person testing students, and I don't always remember everything. Sometimes it is good enough for the students to demonstrate that they understand the topic enough to know where to find the correct information based on a good intuition.
I want to echo this.
Your blue book is being graded by a stressed out and very underpaid grad student with many better things to do. They're looking for keywords to count up, that's it. The PI gave them the list of keywords, the rubric. Any flourishes, turns of phrase, novel takes, those don't matter to your grader at 11 pm after the 20th blue book that night.
Yeah sure, that's not your school, but that is the reality of ~50% of US undergrads.
Very effective multiple choice tests can be given, that require work to be done before selecting an answer, so it can be machine graded. Not ideal in every case but a very quality test can be made multiple choice for hard science subjects
3 replies →
Stanford started doing 15 minute exams with ~12 questions to combat LLM use. OTOH I got a final project feedback from them that was clearly done by an LLM :shrug:
Open book exams are not a new thing and I've often had them for STEM disciplines (maths and biology). Depending on the subject, you will often fail those unless you had a good prior understanding of the material.
If you can pass an exam just by googling something, it means you're just testing rote-memorization rather, and maybe a better design is needed where synthesis and critical thinking skills are evaluated more actively.
Open book, sure. But you don't even need a computer for that.
I make a point of only using references that are either available for free online or through our university’s library subscriptions. These are all electronic. My open book exam became an open computer exam when I realized students were printing hundreds of pages just for a 3-hour exam. This semester I’m switching to no-computer, bring your own printed cheat-sheet for the exam.
Except that the physical book isn't the way people lookup facts these days.
The open book test is purposes is to not have to know all facts (formulas) but proving how to find them and how to apply them. (Finding is part of it as the more you look, the less time you got to use it, thus there is an optimisation problem which things to remember and which to look up)
In modern times you wouldn't look those up in a book, thus other research techniques are required to deal with real life (which advanced certifications should prove)
1 reply →
And even if you are allowed to use a computer, you cannot use internet (and should not be hard to prevent that).
5 replies →
I was in university around the same time. While there I saw a concerted effort to push online courses. Professors would survey students fishing for interest. It was unpopular. To me the motivation seemed clear: charge the same or more for tuition, but reduce opex. Maybe even admit more students to just have then be remote. It watered down the value of the degree while working towards a worse product. Why would a nonprofit public university be working on maximizing profit?
Online courses are also increases admin overhead.
Universities aren’t profit maximizing. They are admin maximizing. Admin are always looking to expand admins budget. Professors, classrooms, facilities all divert money away from admin and they don’t want to pay it unless they have to.
Also applies to hospitals in USA.
> Why would a nonprofit public university be working on maximizing profit?
Because 'nonprofit' is only in reference to the legal entity, not the profit-seeking people working there? There is still great incentive to increase profitability.
You're thinking of not-for-profit. Non-profits do not seek increased profitability in the same way since it's expected (mandated?) they don't have any.
1 reply →
So they can educate more students? Many university classes are lecture only with 200+ students in the class and no direct interactions with profs. Those courses might was well be online.
One potential answer is that this tests more heavily for the ability to memorise, as opposed to understanding. My last exams were over ten years ago and I was always good at them because I have a good medium-term memory for names and numbers. But it's not clearly useful to test for this, as most names and numbers can just be looked up.
When I was studying at university there was a rumour that one of the dons had scraped through their fourth-year exams despite barely attending lectures, because he had a photographic memory and just so happened to leaf through a book containing a required proof, the night before the exam. That gave him enough points despite not necessarily understanding what he was writing.
Obviously very few students have that sort of memory, but it's not necessarily fair to give advantage to those like me who can simply remember things more easily.
Have you ever seen a programmer who really understands C going to stackoverflow every time they have to use an fopen()? Memorization is part of understanding. You cannot understand something without it being readily available in your head
Right, and a lot of them probably got that understanding by going to stackoverflow every time they needed to use fopen() until they eventually didn’t need to anymore.
In the book days, I sometimes got to where I knew exactly where on a page I would find my answer without remembering what that answer was. Nowadays I remember the search query I used to find an answer without remembering what that answer was.
I wrote a long answer, but I realised that even advanced C users are unlikely to have memorised every possible value of errno and what they all mean when fopen errors. There's just no point as you can easily look it up. You can understand that there is a maximum allowable number of opened files without remembering what exact value errno will have in this case.
> because he had a photographic memory and just so happened to leaf through a book containing a required proof
It makes for good rumours and TV show plots, but this sort of "photographic memory" has never been shown to actually exist.
Huh, TIL [0]. Thanks. There are people who can perform extraordinary memory feats, but they're very rare and/or self-trained.
[0] https://skeptoid.com/episodes/542
>His photographic memory manifested itself early — he would amuse his parents’ friends by instantly memorizing pages of phone books on command.
https://medium.com/young-spurs/the-unsung-genius-of-john-von...
When I was in university, in my program, the most common format was that you were allowed to bring in a single page of notes (which you prepared ahead of time based on your understanding of what topics were likely to come up). That seemed to work fine for everyone.
I have a colleague who does that.
My students then often ask me to do the same, to permit them to bring one page of notes as he does.
Then I would say: just assume you're writing the exam with him and work on your one-pager of notes, optimize your notes by copying and re-writing them a few times. Now, the only difference between my exam and his exam is that the night before, you memorize your one-pager (if you re-wrote it a few times you should be able to recreate it purely from memory from that practice alone).
I believe having had all material in your memory at the same time, at least once for a short while, gives students higher self-confidence; they may forget stuff again, but they hopefully remember the feeling of mastering it.
I go to school right now, and most classes actually enforce paper and pencil tests despite how annoying it is to grade and code on.
I teach at MSc level. My students are scattered around the country and world. This makes hand-written exams tricky. Luckily, the nature of the questions they are asked to solve in the essay I give them following their coursework are that chatbots produce appalling bad submissions.
> 10 years ago, we wrote exams by hand with whatever we understood (in our heads.)
You did, but the best exam I had was open book bring anything. 25 and some change years ago even.
I've also had another professor do the "you can bring one A4 sheet with whatever notes you want to make on it."
I had some take home exams in Physics that you could use internet, books, anything except other people (but that was honor code based). Those were some of the hardest exams I ever took in my life. Pages and pages of mathematical derivations. An LLM with how they can do a pretty good job at constructing mathematics, would actually have solved that issue pretty well.
People really struggle to go back once a technology has been adopted. I think for the most part, people cannot really evaluate whether or not the technology is a net positive; the adoption is more social than it is rational, and so it'd be like asking people to change their social values or behaviors.
[dead]
It was the same when I graduated 6 years ago. We had projects to test our ability to use tools and such, and I guess in that context LLMs might be a concern. But exams were pencil and paper only.
I think the key difference is what you're trying to measure
Optics.