Comment by jameshart
10 months ago
I’ve come across this analogy that I think works well:
Using an LLM to do schoolwork is like taking a forklift to the gym.
If all we were interested in was moving the weights around, you’d be right to use a tool to help you. But we’re doing this work for the effect it will have on you. The reason a teacher asks you a question is not because they don’t know the answer.
If students went to college only to learn, colleges wouldn't bother giving diplomas.
Compare: My piano teacher doesn't give diplomas because none of her students would care, her students actually want to learn. When my piano teacher cancels class, I am disappointed because I wanted to learn. My piano teacher doesn't need to threaten me with bad grades to get me to practice outside of class (analogous to homework), because I actually want to learn.
There are many college students for whom none of these tests would pass. They would not attend if there was no diploma, they're relieved when their professors cancel class, and they need to be bullied into studying outside of class.
What made us think these students were ever interested in learning in the first place? Instead, it seems more likely that they just want a degree because they believe that a degree will give them an advantage in the job market. Many people will never use the information that they supposedly learn in college, and they're aware of this when they enroll.
Personally, the fact that they can now get a degree with even less wasted effort than before doesn't bother me one bit. People who want to learn still have every opportunity to.
Students want the diploma because it has value. It has value because a student can only get it by learning and problem-solving.
If students find a way to get a diploma without doing the work, it will soon be worth less than the paper on which it is printed.
Further, if a student can get a diploma without work, then the diploma does not have value anymore. If diplomas are no longer valuable, the signal they provide in the labor market will turn into noise.
If employers no longer look for the diploma-signal in an employee, what will be the reason an employer will hire an employee?
I think this story will become true, and society will radically shift into one where critical thinking skills will actually be the only skills employers look for in employees, since the grunt work can be automated.
What becomes the signal then? Will we shift back into apprenticeship based employment? How do potential laborers display their critical thinking skills apart from displaying them in person?
22 replies →
> If students find a way to get a diploma without doing the work, it will soon be worth less than the paper on which it is printed.
Amen.
I look forward to the era where we train professionals the old fashion way: apprenticeships. It sure worked for blacksmiths and artisans for hundreds of years.
15 replies →
Soon?
I hire people now and where they went to school means little to me. The first priority is “can they do the work?” which is a niche programming. After that is established, I barely take note of school.
I don’t personally count a CS degree as an indication the person is a good programmer, or thinks logically, or has good work ethic.
3 replies →
> It has value because a student can only get it by learning and problem-solving.
No, it has value because it gatekeeps class mobility. Degrees haven't signified learning or problem solving longer than I've been alive. The attitude is that if you pay for an education, you're entitled to a degree. The education aspect is optional.
> If students find a way to get a diploma without doing the work, it will soon be worth less than the paper on which it is printed.
I think this is already the case.
> It has value because a student can only get it by learning and problem-solving.
No. It has value, because companies value it. It sets the starting point for your first salary and then for every salary negotiation moving forward.
As someone who did not go to university, but has the same knowledge self-taught I can tell you that this piece of paper would have opened so many doors and made life so much easier. It took me 15 years to get a salary, that people get with the piece of paper after graduating. And not because it took me 15 years to reach that level of knowledge. I had that by the time the other graduated.
I had a friend who always cheated in school, and now he works for a big car company and earned a fuck ton of money.
Life is unfair and companies only care about the paper your diploma is printed on. If students would ask me for advice, I would tell them to cheat whenever possible.
9 replies →
> It has value because a student can only get it by learning and problem-solving.
No. That's how it should be, but the reality looks different: it has value because it shows that someone spent 3+ years doing what they were told to do, enduring all the absurdities they were subjected to in the meantime. Whatever means they used to cheat don't matter, since they still worked on what someone told them to and produced results satisfying the expectations.
There are, perhaps, institutions where learning and problem-solving are seen as the most important while "following orders" and "staying in line" are deemphasized. For the students of all the others, putting up with an utterly absurd environment is often one of the biggest barriers to learning. Yet, it's a requirement without fulfilling which you can forget about graduating. Hence my conclusion: the diploma from most learning institutions certifies you as a good corporate drone - and that's enough of a signal in many situations, so why bother trying to fix it?
3 replies →
And that won't be their problem, they'll be long gone having achieved what they wanted when they were cheating. It is a societal problem or institutional and industry problem. If the market and these universities want these degrees to mean anything as signaling mechanism they must stop these students from cheating some way somehow...the students are never going to stop themselves.
But it will always be worth more than not having it, so companies will still require it.
Good. It's about time.
You can also get it by cheating and bribing.
Consider that there is some in-between.
Some college students may be genuinely interested in one particular subject, but they're required to take a bunch of other courses, and consider those to just be hurdles.
I still think they're better off at least making an effort and trying to learn something, but I do think it's important to note that just because a student has no interest in one particular class, doesn't mean they have no interest in any class.
There is no in-between here. Consider:
The course I'm interested in gets kinda hard, and my "just pull up an LLM" muscle is very, very strong, (and besides, I'm not used to struggling! and why should I get used to it in the classes i like?! I can't afford a C in my major!) so ... I use LLMs on my "I'm interested in it" class too and... we're back to the original argument.
Most people don't go to college to learn for the sake of learning – that would be a very expensive luxury. They go for the opportunities that it opens up. The learning itself helps with that, but so does being able to prove that they've learned it. That what the diploma is meant to be for.
That's a US-centric view. A huge number of people in the world have access to high level education free of charge.
2 replies →
Yes, but piano lessons are not a music degree. Similarly, vocational programs or apprenticeships are not a formal education either.
I find a lot of these comments more disturbing than the concerns about AI.
> Similarly, vocational programs or apprenticeships are not a formal education either.
They are in some countries, you get at the vocational programs or apprenticeships alongside the highschool, and in the end you might get the opportunity to apply to the university or just carry on with your job.
That is how I did mine in the 1990's Portuguese education system, and how I was already coding and understanding the big boys computer world at 16y.
But if your goal is to be a musician, a music degree is basically useless. Whether you want to play in an orchestra, perform in a rock band, or compose video game soundtracks, nobody cares whether you have a degree or not - they want to hear you perform.
6 replies →
I don't think that's a good comparison. I went college to learn and was also relieved occasionally when professors canceled class and still had to force myself to study. I'm sure plenty of your piano teachers students don't enjoy practicing the same notes over and over but do so because they want to know how to play piano.
Absolutely. Also, the level of cheating in college, even pre-AI is often overlooked in these articles.
For the exact reasons you state, pre-AI homework was often copied and then individuals just crammed for the tests.
AI is not the problem for these students, it's that many students are only in it for the diploma.
If it wasn't AI it would just be copying the assignment from a classmate or previous grad.
And I imagine the students who really want to learn are still learning because they didn't cheat then, and they aren't letting AI do the thinking for them now.
I'm not sure this is entirely fair. When I was in college I genuinely enjoyed learning and now that I'm out of school I still spend time learning about the subjects of my major and minor in my free time, but this would have described me pretty well in school, "they're relieved when their professors cancel class, and they need to be bullied into studying outside of class." I love to learn, but something about being forced to do it makes me rebel against it.
In some ways offering the diploma and all the requirements that go with that take the joy out of the learning for me.
There are jobs that turns you down if you don't have that diploma. No amount of real world experience will get you past that bar. If it's not there, they wont even look at the rest of your application.
Sure, if we can agree that there exist courses for which we can agree that using an LLM to pass is a reasonable thing to do, would we also not agree that the course should be nuked from orbit?
I think several orthogonal concepts are touched in this.
1. Students given bad incentives to be thrown into a system with a completely different purpose than their main goal. Then those jobs turning face to suddenly say "schools teach you nothing" and even refuse to hire the newest generation.
2. Students in general not being stimulated by primary school and given direction and vision on what to do in life. Simply being pushed by parents to "be successful".
3. The crippling reality as of late that a job doesn't even guarantee keeling a roof over your head anymore. Leading to discouragement to even bother trying.
4. Connected to #2, the decline of various apprenticeships, internships (which are now a college recruiting pipeline), and other ways to invest in employees. Even if they complain about new grad output, they are still content outsourcing such training instead of investing in their employees for a career.
There's a lot of systems failing which can arguably cause an entire collapse in the country. Then no one will get an opportunity to properly learn.
> students went to college only to learn, colleges wouldn't bother giving diplomas.
You have this option with things like mits open courseware. Some colleges are OK with you just wanting to learn
This is, famously, why nobody gets a degree in fine arts. (/s)
Your piano teacher does not give a diploma because she is not offering a university education. If she worked with a few other experts and they designed a coordinated curriculum and shepherded students through it over the course of two to four years, and documented that process to the point where they could file with an accrediting agency, then she could issue a degree in piano.
It's worth noting, plenty of universities do this. You can get a degree "in piano".
3 replies →
The point of education isn't to actually learn though. It's to receive the credential.
This is much larger than a cultural problem with the students of today. They believe, rightfully and accurately, that the university degree is yet another part of the machine that they will become a cog in.
What should be alarming to everyone is that these students will graduate without having learned anything and then go into the workplace where they will continue to not use their atrophied critical thinking skills, to simply do yet more, as a cog in the machine.
This attitude is part of a more general cultural shift. Back in the 1960s, the majority of students said the primary motivation for going to college was to develop a philosophy of life, and a minority said the main goal was to be very financially successful. Somewhere around the 1980s this started to shift and the proportions are now inverted.
* according to the UCLA CIRP freshman survey
I would also say some of the attitude shift is also contradictory. The amount of people I interact with who have a lot of bad things to say about the education who tell me universities should focus on education in a general meanwhile also say that schools to should focus on student getting jobs. Probably one that has been heard before, something along the lines of, "why don't high schools teach plumbers courses." I mean they can. While also, "colleges are too focused on checking the boxes so students can get jobs."
8 replies →
And there were fewer students going to university and so you had a higher proportion of people doing it for the love of learning. It was easier to get a job without a degree. It's not just a cultural shift, it's a change of supply and demand.
Notably, the late 70’s and early 80’s is when large scale social changes started to happen due to high inflation and major problems in the US economy. Also a lot of social unrest, and a pretty unhinged president (Tricky Dick).
3 replies →
Arguably the cause of the cultural shift is economic. Wages began stagnating in the early 1970s, which caused increased demand for diplomas as a way to increase wages. This is most striking in the number of students seeking law degrees, which shoots off at around the same time.
Yeah because in the 60s you could support a wife and 4 kids, buy a house and 2 cars, without a high school diploma, all before turning 25. There was no reason to go to college unless you were interested in learning.
These days you need a college degree just to afford a 1 bedroom apartment by the time you are 40.
We (America) made university educations super expensive. Causality is complicated, but it probably started as an effort to destroy the anti-war movement in the 70's.
4 replies →
> Somewhere around the 1980s this started to shift and the proportions are now inverted.
Yeah, that's when the great "push for education" came, as well as neoliberalism which preached continuous hustling and individuality. And in the 90s, the ADA and other anti discrimination laws hit, and requiring a college degree was and still is a very useful pre-screening filter for HR to continue discrimination.
1 reply →
An anecdote to add to this:
Me and most of my peers in college had the choice between two courses. Course A was interesting, yet vastly more challenging and therefore time consuming, with the additional downside of lower grade expectation. Course B was boring, a gentle breeze in comparison, yet with an almost guaranteed perfect grade.
Imagine which course most students choose?
Even if a student wants to take on the more interesting course, incentives matter, and the incentive is: better grades qualify for better compensated positions and prestigious degrees. Only students who didn't care about this or were confident enough in their ability did choose Course A. In the end, barely a handful of students out of hundreds went with A.
Well, the credentials defintely help to GET the first job. As a cog in the machine though you are most often valued for skills more than for credentials, at least in the U.S.
> The reason a teacher asks you a question is not because they don’t know the answer.
A decent amount of my professors don't know the answers because they bought the course, test questions, and lectures from Cengage. During exam review, they just regurgitate the answer justification that Cengage provided. During the lectures, they struggle to explain certain concepts since they didn't make the slides.
Professors automate themselves out of the teaching process and are upset when students automate themselves out of the learning process.
I can tell when the faculty views teaching as a checkbox that they officially have to devote 40% of their time to. I can tell when we are given busywork to waste our time instead of something challenging.
To use your analogy, I'm being told to move 1000 plush reproductions of barbells from Point A to B by hand because accreditation wants to see students "working out" and the school doesn't want high failure rates.
We are all pulling out the forklift. Some of us are happy because we don't have to work as hard. Others are using the forklift so we can get in a real workout at home, as school is not a good use of our time. Either way, none of us see value moving paperweights all day.
edit:
My favourite course during my Computer Engineering degree was Science Fiction because that professor graded us on substance instead of form. It was considered a hard class because one would get good marks on the essays by focusing on building substantive points instead of strict adherence to the form of a five-paragraph hamburger essay.
The call to action is to make courses harder and stop giving students plush barbells.
For example, University of Toronto Engineering Science (hardest program in Canada) gives first-year students a "vibe coding" lab in which students learn how to solve a problem that AI cannot.
https://www.cs.toronto.edu/~guerzhoy/vibecoding/vibecoding.h...
There are many issues here. The lack of incentives is probably the most important one. For new professors (in research universities), good teaching is usually just a good thing to have, but it is not a deciding factor for their tenure. When they get their tenure, they probably have enough students, and they need to work hard to apply for funding and keep the students paid. Administrators care most about ranking, and teaching isn't really evaluated in the ranking. They just push the professors to do more research and apply for more funding.
It is also hard to evaluate university teaching because there are no benchmarks for that (compared with high school, for example), and it is hard to judge if teaching is good from student feedback. You can only know if someone fucked up or did really well, which are outliers.
There are other issues as well. Professor IMO is a ridiculous job, you are supposed to be an expert in the field, be a researcher, be a manager, be a teacher, be a salesman, all at the same time. There are people who can excel in all these, but these are probably just outliers. It doesn't help when PhD training doesn't train you to be a proper manager and teacher. While there are some teaching training, I think we are not really held to a high enough standard. E.g. One can pass the teaching course if they just show up and spend some time, even though their teaching is horrible.
Meanwhile at my uni, at Masters, I'm being taught how to create and delete rows in HTML - I wish I wasn't kidding. :(
But that requires professors to do work and give a damn!
These projections about bad professor experience is exactly why colleges became so hyperconpetitive. The difference between a competitive (and not even exclusive level like Ivies) and a "semi-competitive" is night and day in terms of rigor, staff talent, and overall environment.
But sure, you're always going to find a few meh or bad professors. And they will stick out as much ad thr great professors
That's a good one.
One analogy I use a lot: if I have a professor sitting next to me, what is the best way to learn a topic?
Struggle through it on my own and I won't be leveraging the professors knowledge.
Ask the professor to do everything for me and I won't be learning anything at all.
Now if the professor is an AI, the same trade-offs hold.
For example, I will back and forth conversations with AI to explain subjects to me. I ask questions, push back, ask for examples, and so on.
If I do ask the AI to answer something for me, I then ask it to break down the answer for me so I can make sure I understand it deeply.
And of course, none of this matters if I don't want to learn something :)
The fatal flaw here is that an AI is more like asking a politician. Except maybe it won't gaslight you. It often has no idea what it's talking about, and pushing back may even have it change it's tune.
>And of course, none of this matters if I don't want to learn something :)
Society makes people do a lot of things they don't want. I wonder if we're going to hit a breaking point this generation.
Using an LLM to do schoolwork is like taking a forklift to a gym where you're told the goal is to be healthy and strong, but they can't really stop you from using a forklift, and jobs and compensation are given out according to how much you lifted irrespective of forklift use.
Except part of the job involves lifting things in places where forklifts can't fit.
They fit in most places or will over the next few years with the next “forklift” version.
1 reply →
> If all we were interested in was moving the weights around, you’d be right to use a tool to help you.
Does the use of a quantifiable metric like a GPA not exacerbate this? In a world where people take a GPA seriously, you'd have to be irrational to not consider cheating a viable option.
You could say the same about credit score and dating apps. These institutions assist the most predatory and harm the most vulnerable.
Ooh ranked gaming, too. Also any kind of market activity.
Why do some countries cheat in the Olympics? Because it is no longer a contest of human achievement, it's just about the medals as a symbol of national glory. Of course: once all countries are doping, the medals will become meaningless. College degrees will suffer the same fate if everyone cheats to get them.
The problem is there is no reliable way to test for LLM usage unlike for doping. There’s no way to police this problem.
Sure there is - evaluate the candidates actual skill level without the ability to use automated tools.
The current and old school way is a proctored exam.
2 replies →
Diversity in evaluation methodology is a reliable way for many subject. Even short, minimal handwritten exams would help to assess student understanding at a few key checkpoints and should should be reintroduced.
> The reason a teacher asks you a question is not because they don’t know the answer.
I remember illustrating a point to a class by posing a question and then calling on a student I figured wasn't smart enough to answer correctly so that everyone could see her make the mistake.
The ethics of that still bother me.
If I could have a healthy and good looking physique by never going to the gym I would never go to the gym.
Right, but people using LLMs aren't actually getting the mental equivalent of a healthy and good looking physique if they let LLMs do stuff for them.
Well, you could. Most people prefer their pre-processed food over never going to the gym though.
The tragedy is not that some students are going to college to get a diploma while learning as little as possible. It is that the boards of many private universities see their students' cash as more important than their education, and force the professors to pass everybody who went to higher education to buy a diploma.
This has a negative feedback loop where universities have to lower standards to bring dumber and lazier students to compete with other diploma mills.
First, you will have excellent forklift skills in the end. A real profession!
Second, girls dig forklift operators or so I was told.
I use chatgpt in a socratic way from time to time because I don't want answers I want the joy of thinking and learning. I heard there were efforts to make educational LLMs (whatever that means). Maybe it will help multiply teachers leverage so that more kids get inspired without having the teacher spend 1-on-1 time with them.. I don't know.
I love this analogy because it's also not a waste of time to learn how to use a forklift!
But the gym isn't the best place to engage in forklift training. And you engage in forklift training at the gym, expect to learn how to use a forklift to lift gym weights. Don't expect to also get the benefits that the gym is designed to impart.
I think you're quoting the Sci Fi author - Ken Liu from his article in some major news outlet.
I related with that analogy too, infact that whole piece is worth reading. I can't seem to find it's link though!
It's because you've mixed him up with Ted Chiang - the article in question is Why A.I. Isn’t Going to Make Art [1]
[1] https://archive.is/kS8NY
I think LLMs, if used correctly, can be useful for BOTH the credentialing and the human resource development (*cough*)
Essentially, since they are a summary of "the" state of knowledge, the teacher should be able to ask them to put a number on how novel a piece of text is.
Once LLMs are able to evaluate, independently, the soundness of an argument... (Hopefully, this will be achieved AFTER $5 H100s reach the average consumer)
LLMs are sometimes wrong, and they don’t know they are wrong, nor do students.
They are the wrong tool for pedagogy.
But going to school to prepare for life is like going to gym to lift weights to prepare for a marathon.
It's an e-bike for the mind!
I think we have to hold off a bit on the whole thing here.
Look, we have no idea what the feedback is like that this grad student gives, what the class sizes are like, what the cadence is, what the grade percentages are, etc. All we know is that Clayton Ramsey is a grad student at Rice in the robotics department and that he wrote a hot take here.
For me, the most important thing is if this grader is bothering to really grade at all. I think we've all had a harried grad student just dash off a few red lines on the week one HW about a week before the final exam. That's not a 2 way street, and if the feedback isn't as in-depth as he wants the work to be, well, he shouldn't be surprised. He can't be expecting students to put in the time unilaterally. But, we don't know any of that really.
Personally, I think that before the decade is out, we're not going to be talking about this at all. Because the students will be adept enough at using the LLMs to make it look like their own writing anyways. This is a problem that experience will solve for them.
And also, I think that the days of the massive lectures and essays are pretty much cooked. That 'cheap' model of education can't survive this LLM revolution. We obviously have to change what the heck higher education is trying to do.
My take is that we're going to go to smaller class sizes like those at St. John's or Oxbridge. Under 10 people, you have to have done the reading or look like a fool, all with a PhD in the subject as a guide/teacher. Large classes weren't cutting it for decades (ask any Frat about their test banks), and now the veil is just ripped off.
We will never reduce all class sizes to under 10 people. Large R1 schools are not going to reduce their number of students by a factor of 10 or increase their hiring by a factor of 10.
I have classes like these, but at a small Masters programme and only on our courses directly related to specialisation :D
forklift gym gang gang
Dangers of Intelligence and Other Scientific Essays by Asimov predicted all this hullabaloo quite a while ago. So, yeah, seems like evidence to support your position. Welcome to the party. :)
> Using an LLM to do schoolwork is like taking a forklift to the gym.
I'm sure the time has come for college students to master using LLMs. It's just as important as grammar or basic math now. The software I build (and the entire tech industry) automates huge swaths of business processes with AI. Students need to be able to understand, work with, and manage swarms of AI agents doing work.
To stick to the analogy:
I need skilled forklift drivers, not big buff workers like I used to.
But the thing you are missing is that one needs a solid foundation of knowledge of the actual work to be able to manage it well.
Someone with years of coding experience is going to be able to laser guide an AI agent to the answer/result than someone who has muddled their way through comp sci 101 using an AI chatbot.
No one is saying they don't need a solid foundation of knowledge. The knowledge needed is different. What we're seeing now is lot like teaching people to care for a horse even though the automobile is now the dominant form of transportation
If you need forkloft drivers, don't recruit at the gym and be mad no one is forklift certified there.
This isn't even an opinion on LLMs, it's recruiting 101. You're free to convince the gym to train forklift drivers, but don't be surprised when you're laughed out the room.
It's the other way around... people will happily pay to be "forklift driver certified" and a piece of paper that gets them a bit higher in the hiring line. Especially when they'll pay way more for that than ethereal "thinking skills" that they can't even reason why they'd need that in the first place
If you took a forklift to the gym, you'd come out of the experience not only very good at "lifting weights", but having learned a whole lot more about the nature and physics of weightlifting from a very different angle.
Sure, you should lift them yourself too. But using an AI teaches you a shit-ton more about any field than your own tired brain was going to uncover. It's a very different but powerful educational experience.
> But using an AI teaches you a shit-ton more about any field than your own tired brain was going to uncover.
If you never learn to research, sure. Otherwise, you should be worried about accuracy, up to date information, opinionated takes, and outright lies/misinformation. The tool you use doesn't change these factors.
No but it increases the speed and ease at which you can check any of those - making a lot of those steps practical when they were a slog before. If people aren't double-checking LLM claims against sources then they were never on guard for those without an LLM either.
Besides, those are incredibly short-term concerns. Recent models are a whole lot more trustworthy and can search for and cite sources accurately.
1 reply →