Comment by kzz102
4 hours ago
It's not so clear that AI is an amplifier.. The paper has some fascinating analysis on this topic:
"At the other end of the distribution, AI students who spend more than 65 minutes on their homework receive homework and exam scores similar to those of non-AI students, suggesting that these students do not use generative AI for homework assignments. However, this group consists entirely of students who adopted generative AI no more than Öve months. Six months after adoption, no AI student spends more than 65 minutes completing their homework (see Figure A5). This is consistent with the gradual process of learning how to use AI tools. It also suggests that AI crowds out the highest level of e§ort."
"Interestingly, in the range of 50-65 minutes, the median and the interquartile range of exam scores of AI and non-AI students are similar. This implies that, in the range where AI students and non-AI students have overlapping homework times, students who spend the same amount of time completing homework on average receive similar exam scores."
"This pattern shows that students who spend the same amount of time on homework learn similarly, with or without generative AI. In other words, generative AI reduces time spent learning for the majority of AI students but not learning efficiency for those who spend the same time studying as the non-AI students."
Right I was not clear enough about separating my opinion from the findings.
It would be good to see the effect of access to AI during preparation on those who previously achieved top 10% points in exams of similar topics before. I suspect they would benefit further.
My apologies for coming off as over-enthusiastic, I am currently obsessed with this study. Here is another quote:
"The negative learning effects are larger for students with higher initial achievement. The differences in the estimated full (6-10 month average) effects are substantial, with a 50% gap between the most negative effect (-24 percent) for the highest tercile and the least negative (-16 percent) for the lowest tercile. "
Not top 10% as you asked, but the closest to what you asked. My working hypothesis is that top performance is highly correlated with willingness to work hard, and AI decreases the motivation to work hard.
Interesting. Top academic performance is mostly correlated with conscientiousness (willingness to work hard and keeping track of things) and intelligence. And I'd add motivation and interest to that too.
I think if you take a physics class where the student is intelligent and intrinsically motivated through their own interest (I admit this is rare) then AI probably helps.
1 reply →
Depends who's using it. Like many tools the force multiplier depends on the operator.
I'm confident it's an amplifier for people who know how learning works and already do a lot of it, successfully. However the level of "learning fluency" I'm talking about isn't reached for many until late college or grad school, and sometimes not at all. So I'm not surprised by the quoted results for 12-18 year olds.
> Depends who's using it. Like many tools the force multiplier depends on the operator.
How would you prove/disprove this assumption without falling into a True Scotsman fallacy?
And people who want to learn. Most teenagers lack agency in their studies. They aren't in high school because they love it but because they have no choice.
Sometimes things are just common sense pure and simple.
> Sometimes things are just common sense pure and simple.
The discussion is about "AI", so common sense is out the window. These people's professional reputations depend on addict-level "AI" usage remaining socially acceptable.
well most university exams are designed to measure how much you study. so we didn't really need a study to tell us, "Exams continue to measure what they are designed to measure."
they're not designed to measure general aptitude, or function as admissions criteria, or screen for job applications, or any other numerous things they are used for.
there can be many questions of pedagogy. one of them is, what do our exams measure and how do we use them? professors who say, "My exam is designed to measure who studies, not be used for all these other purposes that they are actually used for" - I don't buy it. It's the same as late night comedians saying they are not responsible for solutions, even when spending 90% of their air time making political jokes.
THIS is the pedagogical issue, that pedagogy has NEVER caught up with the scope of responsibilities. This is acute in STEM - I mean, the humanities departments are generally pretty well run, all things considered, in this regard. Generative AI is accelerating that pre-existing crisis.
The fact that exam scores are correlated with how much you study is not the same as exams only reflect how much you study. Two students who study the same amount could have very different exam scores. The reason that there still is a strong correlation between exam score and time of study is because if all other things being equal, students who studies more have higher exam scores.
i'm not saying they reflect how much you study. they reflect a lot of things, including that. but you ask the people who write the tests, they're going to say, how much you know or how much you study, but nonetheless, they are limited. i agree with you. that's part of my point.
let's imagine a different study. we instead compare AI-users and non-users on a Wechsler (IQ-adjacent) test.
overall, it would be surprising if AI usage impacted your Wechsler scores. someone has done this study and the impact is quite quite small. BUT. do we care? We don't use Wechsler scores for admissions, we don't use them for jobs, we don't use them for... are you getting it now? A Wechsler family test is measuring something real, just like a university exam measures something. But what do we USE them for? Wechsler and a typical university exam are, in some senses, EQUALLY vague in terms of their fitness for purpose for answering a question like, "should we hire this guy?"
Like there is an association between IQ and earnings but it is actually surprisingly small! There is an association with math education and earnings and it is also surprisingly small. And consider how many people get by just fine without using a single piece of math education once they have finished school - like what if maximizing your earnings isn't all that it is about? Are you getting it now?
The issue isn't the AI usage. I can find tests that are immune to AI usage. The issue is using tests for things that they are not designed for. We pick and choose, for some subtle but nonetheless pervasive cultural reasons, which tests we use for which purpose, and very frequently, not because they are calibrated for the chosen purpose. This is coming from someone who scores very well on all these tests, and have kids, so I have a very strong incentive to buy into the status quo, and I'm telling you: academic testing has been fucked up for a long, long time.
1 reply →
I never studied in university and yet I acheived good exam scores. If you understand a topic and ave a reasonable memory and ability to apply my our understanding not much studying is required in my experience. If you don't understand the big picture then you got to laboriously keep track of and manipulate a bunch of disparate pieces.
> well most university exams are designed to measure how much you study.
Huh? They're designed to measure how much you know. They can't see how much you study, nor would they have reason to be interested.
at least in my experience in university - i didn't really ask this question, since it is obvious to me, but some students have asked it during lecture, or some instructors have volunteered the answer ahead of time - if you ask how to perform better on the exam, usually the instructors say, "here's what you should study." they never say, "know more." the thing i am talking about is consistent with the paper. really, your takeaway should be, exams can't see how much you know!
3 replies →