Comment by baq
9 days ago
Replace ‘CTF’ with ‘high school’ or ‘university’ and you’ve described the total slow motion collapse of education; the only saving grace is that most of it requires in person presence.
We’ve figured out the human replacement pipeline it seems, but we haven’t figured out the eduction part. LLMs can be wonderful teachers, but the temptation to just tell it ‘do it for me’ is almost impossible to resist.
Everything we've learned in the last 10 years is telling us that computers do not help human education in the slightest. We remember better when we write with pen and paper. We learn better with whiteboards and paper books. The simple answer: Remove most computing from education entirely. Blue composition books, pencils, whiteboards is what trains humans. Calculators are helpful perhaps but it is quite possible that slide rules are better. We need humans that can critically think from first principles to counter the recycled information generated by AI.
> computers do not help human education in the slightest
I had no access to anyone who could teach me calculus as a kid except Khan Academy, so I think this is a gross exaggeration. But I agree in the end, that all my "real" learning did come from pen-and-paper practice, not watching videos.
The reality is that a human will learn, given any materials including LLMs, but only if they truly desire to learn. We've had MOOCs, gigantic libraries, all full of free information. You can obtain a PhD level understanding in any technical field of your choice today just by consistently going to the library and consistently applying yourself.
It's not unlike going to the gym, and we see how many people do that regularly. Except it's even funnier, because people serious about the gym but what? Tutors. They call them personal trainers. We've known for a millennium or more that 1-on-1 instruction is vastly better than anything else, but most people actually don't want to get into shape, and most people actually don't want to learn.
8 replies →
Yeah I agree. I grew up in a very blue-collar town, and anything I wanted to learn (outside of public schooling) either came from emaciated websites or whatever books I could find at the library. Having YouTube and Khan Academy and everything else would have made such a huge difference for me.
4 replies →
> except Khan Academy
But that's not using "computers" as a computer but as a video player. When evaluating whether computers are "good for learning", I don't think we should include using a computer as a video player, a book, or even flash cards. It should be things a computers uniquely offer which a books, paper, videos and a physical reference library cannot.
Based on the results of deploying hundreds of millions of computer to schools in the 80s and 90s, the evidence was mostly that computers are good for learning computer programming and "how to use a computer" but not notably better than cheaper analog alternatives for learning other things.
Interestingly, a properly trained and scaffolded LLM could be the first thing to meaningfully change that. It could do some things in ways only human teachers could previously since it is theoretically capable of observing learner progress and adapting to it in real-time.
5 replies →
Khan did not throw at you a 100-slide Powerpoint deck in 45'.
He really took the time to replicate the manual teaching process of writing on whiteboard. He improved upon it by using colors. But basically had the same pace as a teacher writing on a whiteboard.
When professors are given a projector, they just throw together some slides and add their narration.
This is not very efficient. To learn you need to suffer. Or you need to watch the suffering.
That's not really a computer helping you though, that's just a computer allowing a human that's far away to help you right?
I think what the author meant is that it does help not more than the same knowledge provided the old way.
Every child reads a book about solving problems, assumes they can now solve problems, and is disappointed when that is not true.
I think this overlooks the potency and scarcity of 1:1 time with the teacher. If you've only got maybe a few minutes of that in an average schoolday there's a huge difference between whether or not you've talked it through with an AI before trying the question out on the teacher.
They're wrong sometimes, but usually in verifiable ways. And they don't seem to know the difference between medicine and bioterrorism, so often they refuse. But these limitations are worth tolerating when the alternative is that our specialists in topic X are bogged down by questions about topic Y to the point where X isn't getting taught.
And now they'll have less time because they will be bombarded with slop to no end.
1 reply →
Nah, I wrote physics programs on my computer at home in high school and it absolutely helped with my schooling. Yeah, maybe iPad apps aren't the best things in schools but you're throwing the baby out with the bathwater. Computers bad is simply not true.
I learned calculus thanks to wolfram alpha step by step solving feature
> humans that can critically think from first principles
This has never been achieved by, nor is it the point of, education for the masses.
But it should be, right?
I'm not going to disagree with step by step videos ... those are a HUGE help. I'm really talking about solving problems using pen and paper, whether math or writing, is how my problem-solving patterns actually changed.
I would start saying that many people need presence in a real environment with people to learn. We don't use all our senses in a remote environment.
I disagree with that statement. There is nothing inherently wrong with using computer to learn and if your personal goal is to learn it in lot of cases makes it much easier, whether to search for or visualise a piece of knowledge you're' learning.
The problem is frankly computer and now computer with LLM makes it easy to cheat.
The kid doesn't want to learn, the kid wants good grades so parent is happy with them, and the young adult wants to get the paper coz they were told that is required for good life. It's misalignment of incentives.
I don't think computers automatically make us more educated, but if you want to make a point don't use reductive exaggerations. > We need humans that can critically think from first principles to counter the recycled information generated by AI.
I agree with this.
We are interviewing for a software dev role and we made the first round in person to prevent cheating. The gap between people who learned pre ai vs post is immense. I had a dev with supposedly 3 years experience and a degree in software who wouldn't have been able to write fizzbuzz without AI.
Can’t say you’re wrong but the last anecdote describes many I’ve had to review for jobs long before LLMs. Fizzbuzz is a classic thing that shockingly many devs genuinely cannot do, even at home.
Yeah, I've interviewed people like this 15 years ago. Degrees and experience mean nothing in this field. The best predictor I found was personal passion projects. Let them get as nerdy as possible, then you will see pretty quickly where their skills are at and what their limits are. And you will immediately filter out people who just studied CS because they heard you can make good money.
7 replies →
Something that is for sure new is the AI interview cheating tools which listen in on the call and provide answers in an overlay invisible to screen sharing. The only way to deal with it would either be invasive spyware on the applicants computer or asking them to do the interview face to face.
4 replies →
Why is it important that a dev can’t do fizzbuzz without ai?
If they can ship code that matches a spec, why does it matter if they’re using ai or not?
Genuinely curious.
34 replies →
> I had a dev with supposedly 3 years experience and a degree in software who wouldn't have been able to write fizzbuzz without AI.
If you remove the "without AI" and the end, I've been hearing similar anecdotes about fizzbuzz for years (isn't the whole point of fizzbuzz to filter out those candidates?)
Because "the next generation is ruined" is always a popular sentiment. It has been with us for at least two thousand years, and it surely won't go away in our lifetime.
When this AI era's devs grow older they'll complain the newer generation can't even vide code too.
13 replies →
While this is true, it seems undeniable that if you use AI to do everything for you, you will never learn the skills. I'm seeing a massive amount of developers submitting stuff for review and admitting they have no idea how it works and they just generated it.
Some percentage of developers before AI were unable to code fizzbuzz. Some significantly higher percentage of them are not able to do so now.
Saying there have always been bad developers doesn't change that there's a higher ratio of them now.
No stats to back this up. Just interviews I've done recently and historically.
That's actually the origin of FizzBuzz! A puzzle invented to weed out the perplexing multitude of CS graduates who apparently cannot program.
https://blog.codinghorror.com/why-cant-programmers-program/
I developed for 15 years. I don’t think I can do with AI anymore. Why would I even want to do that? It’s like telling a car driver to build an engine.
It's more like asking a driver the laws for when traffic lights are out. It's not something that comes up often, but it's not completely outside the scope of the task either (I arguably don't even drive a car that has an engine).
As a car driver, you should understand a little about how your car works. What if you get a flat tire? At the very least, you should know not to drive on that flat tire.
Software is full of leaky abstractions
I first did fizz buzz about 10 years ago fresh out of college. Now, after 10 years in full stack and fully vibe coding, I forgot basic python syntax. An interview like yours would have false positives if you are checking for syntax because well, its like looking up spelling, I just ask the AI for the syntax inline.
> I forgot basic python syntax
If you cannot write "basic syntax" for any language then you are not a programmer, and certainly not a software engineer? This is not a value judgement, it's ok (probably good tbh) to not be a programmer. But you are wasting everyone's time by interviewing for a programming position in this case.
5 replies →
Which part of the syntax for fizzbuzz can you not recall from memory? The for loop? Printing to std out? The modulus operator?
There’s almost nothing to forget? I’m just struggling to understand.
You would not have been a good fit for this position in that case.
1 reply →
I wonder if you’re filtering for the right things.
We usually hire for problem solving capabilities and not so much for technical know-how.
That’s at least how I read your comment.
Ultimately in a software development role you need both technical know how and problem solving capabilities.
This situation in particular was a React role so there is an expectation that when you list React as one of your skills on your resume then you know at least the basics of state, the common hooks, the difference between a reference to a value vs the value itself.
These days you can do a surprising amount with AI without knowing what you are doing, but if you don't have any clue how things work you'll very quickly run in to problems you can't prompt away.
Isn't wiring coding solving a problem? If the candidate can't do that then even if they use AI for coding how are they going to review the code properly?
Meh. Before AI I've had "senior" colleagues with 10 and 8 years experience each, doing pair programming for 2 days straight, and in that time they hadn't managed to checkout a new branch in git.
It's not even that they got distracted, they sat there trying, for 2 whole days, with concerned colleagues giving them hints like "have you tried checkout -b"... They didn't manage!
How the hell do you work for a decade in this business without learning even the most basic git commands? Or at least how to look them up? Or how to use a gui?
Incompetent devs is not a new thing.
It is ok to work somewhere that does not use git. But how do you not figure out how to do the basics given 30 mins and an Internet connection?
Don't worry, i never thought I would see someone unable to write fizzbuzz, but it happened 9 years ago.
Also how many people work with linux and can't tell you what 'ls -alh' is doing is staggering (lets ignore the h, even al people struggle hard).
People working with docker for YEARS and don't even understand how docker actually works (cgroups)...
Interviewing was always a bag of emotions in sense of "holy shit my job is save your years to come" and "srsly? how? How do you still have a job?"
Isn’t this like interviewing accountants but prohibiting use of calculators or spreadsheets?
I don’t care what someone can do without the tools of their trade, I care deeply about their quality of work when using tools.
We would still expect an accountant to know the formula to arrive at the expected result if they did not have a calculator at hand
You absolutely need to have some basic level of abilities if you are going to be operating AI coding tools for software that is going to have paying users.... I use these tools very very heavily I'm not against them at all and I don't scrutinize every single line of code that they write but it is very often that I catch it doing some brain dead stuff and if I didn't have a decade plus of experience I wouldn't know that it was brain dead.
8 replies →
Calculators and spreadsheets cannot autonomously create a double-entry bookkeeping system for a small business and prepare their taxes. AI can. Poorly, but it can.
Everybody knows calculators and spreadsheets are adjuncts to skill. Too many people believe AI is the skill itself, and that learning the skill is unnecessary.
> Replace ‘CTF’ with ‘high school’ or ‘university’ and you’ve described the total slow motion collapse of education; the only saving grace is that most of it requires in person presence.
So something like, "Frontier AI has broken the 'high school' or 'university' format"?
The hype surrounding AI is just pervasively exhausting: you've got the folks talking about an entire new age for humanity where we're shortly going to take over the entire universe. And you've got the folks talking about how our entire society is crumbling.
Education is one place folks seem to throw up their hands and say nothing can be done.
The fix is simple: students are to be evaluated on their performance in person. That's it.
Any other "collapse of education" isn't due to AI, it's something else.
I found this interview [0] on the subject of AI in CS education on the Oxide & Friends podcast very illuminating. Of course, Brown University CS != All education, but interesting angle nevertheless.
[0] Episode webpage: https://share.transistor.fm/s/31855e83
Wonderful teachers that give unreliable information with total confidence?
I had human teachers who did that in middle/high school. Took me many years to pick out all the hallucinated bits of "knowledge". I don't think the current models are any less reliable that what we currently have on average.
I'll always remember my middle school science teaching telling us that nuclear fusion violates conservation of mass because the 2 protons in a pair of hydrogen nuclei combine to make helium with 4 nucleons. It's not true, but that's not the point.
But he was a great teacher anyway. He was engaging and kept the kids in line and learning. I eventually learned the truth, and most of my classmates forgot about it. Teaching, like flying a plane or driving a train, might become more about keeping watch over a small group of people and ensuring that things don't go off the rails, and that's fine.
18 replies →
That's an American problem though. In most of Europe you need a masters degree to teach highschool and that involves at least an undergrad level of understanding the subjects you will teach.
E.g. in Hungary I had a university CS professor that originally wanted to be a highschool teacher and a highschool physics teacher that originally wanted to be researcher. Their choice of degree didn't determine which outcome they got. The researcher and teacher curriculum had an 80%+ overlap.
1 reply →
I think they are less reliable. For factually verifiable facts LLMs are doing worse than 90% for me. I've been told some incorrect things by educators, but at a much lower rate.
The problem is that people seem to trust whatever AI hallucinated way more than if they heard same thing from human
Off the top of my head: DOMS being little crystals in muscles, tongue having separate areas for each type of taste, food pyramid, blue blood in the veins, the appendix being useless, body temperature doesn't change disregarding whether it's exposed to cold or to heat, and a whole lot of stuff related to politics and history I'd rather just omit (I don't live in the US).
All things I learned in school which were wrong information.
Not to mention, the current state of education is far worse. I don't think most realize how low the bar is.
One of my teachers in elementary school told us that people in the Arabic world wore long garments because as Muslims, they believed the Messiah would be born by a male, and thus, it was important to have something to catch the baby as it unexpectedly popped out one day and would otherwise hit the ground.
She only really had two faults: She wasn't very bright, and she wasn't fond of children. I had her in about 80% of all my classes for six years. High school was a relief.
2 replies →
My biology teacher in school once tried to teach us that winds created by God. Not like spiritually or something but that God literally made the wind I guess.
My “earth sciences” teacher also once tried to argue with me against the universal law of gravitation. (no, she was not referring to Special/General Relativity. She didn’t agree two objects in a vacuum fall at the same speed regardless of mass.
To be fair, that was much of my actual experience with human professors in university.
Veritasium proved that in a difficult challenge.
A Physics Prof Bet Me $10,000 I'm Wrong
https://www.youtube.com/watch?v=yCsgoLc_fzI
Yeah one of my teachers was able to identify which high school I had come from due to something I had been mistaught.
They'll also encourage and praise you even when you're heading down the wrong path until you think you've uncovered the secret of the universe or proven that established science was wrong this whole time when really you've just been bullshitting with an engagement bot.
No, they don't really do that anymore, if you use the latest models with reasoning enabled.
Like almost everything else about LLMs, this unfortunate tendency has gotten a lot better recently, which you might not realize if you gave up after getting some lame answers or bogus glazing on the free ChatGPT page a couple of years ago.
Anti-intellectualism is at it again, hu?
The amount of bullshit and blatant lies I’ve heard from my human teachers dwarfs the hallucinations produced by today’s LLMs.
Like humans.
I think we should go a little deeper on this idea.
We can all agree that both human "experts" and LLMs can sometimes be right, and sometimes be confidently wrong.
But that doesn't imply that they're equally fit for purpose. It just means that we can't use that simple shortcut to conclude that one is inferior to the other.
So where do we go from here?
2 replies →
Education is also figured out. You just need to learn, do and practice for yourself. Telling the agent "to just do it for you" is tempting, but it's not learning. You need to be deliberate when you're trying to actually learn and internalize.
Also, you could spin up your own educational agent with very strict instructions on guiding the user instead of just doing the work. Of course you can always go around it but if you're making an effort to learn, this is a good middle ground.
They were a forcing function for skillz and they no longer are. We need new forcing functions for skillz or we will become WALL-E blobs.
Well, they were ostensibly forcing functions... ten years ago everyone was paying the exchange student to do their homework and assignments for them, and that guy was paying his cousin back in his home country, but the whole thing is a bit more efficient now.
You haven't explained why anyone should value education in the world we're building, other than as a hobby.
We've already had consolidation of education for a while now. Even before all the edutech courses, there were Youtubers educating better than many university professors. 10-15 years ago students were already skipping lectures and just showing up for tests.
The best frontier LLMs can't solve 4th grade math homework yet. Don't hold your breath on that collapse of education.
(Real mathematics problems, not American-style ""math"".)
Do you have an example of a 4th grade problem in mind that isn't "American-style"?
In my university education (2007-2011), 80% of the grade was based on exams at the end of each year, with no resits.
>LLMs can be wonderful teachers
Are they or aren't they
Mostly, no. They will explain things to you and you'll feel like you understand them. When you have to do it, though, you'll find you're not any better off than when you started.
I used to see this with students in calculus who abused the tutoring resources. They'd have tutors just work problems (often their homework...) in front of them. "Ah! Obviously that trig substitution integral worked that way. Oh, of course, that proof is very obvious in retrospect." And then they'd walk away from the exam with a 30% and no idea how their 20 hours of "study" for it didn't result in the same performance as their peers who worked problems, read the materials and asked questions, etc., got.
Most AI use is that same in my experience. "Show me how the fundamental theory of calculus works." The LLM puts together a very elaborate and flashy presentation that they skim. Great. That's no different than reading a text book. Even if you ask the LLM questions and have it elaborate on things, you've never once done one of the most important things a student can do: spend time confused trying to work hard at understanding something that's not obvious. The LLM will make it obvious at every point. Total lack of friction. Works about as well as a spotter who does the lifting for you.
A million times better than any human teacher I’ve ever had, for sure.
Now I’m certain that there exist those mythical human instructors who can do better, but that’s not worth much if 99.99% of people don’t have access to them. Just like a good human physician who takes their time with the patient is better than an LLM, but that’s not worth much either given that this doesn’t match most people’s experience with their own physicians.
Did an LLM teach you a topic you did not feel like learning?
For me the best human teachers were the ones that managed to make me interested on topics that I thought are boring/useless (many times my opinion being stupid, mostly due to lack of experience).
So far with LLM I learn about things I know something (at least that they exist) and I am interested in, which is a small subset of things that one should learn during lifetime.
5 replies →
>A million times better than any human teacher I’ve ever had, for sure.
Not really, not if you want to ask it deep questions. It won't have an answer that is deeper than something that you can find online, and if pressed it will just keep circling around the same response.
The reason is that this "thing" was never curious, never asked questions, and never really learned anything. It just has learned the Internet "by heart", and is as boring as a human teacher who is not really curious about the subject they are teaching, and has just got some degree by "by hearting" some text book. Of course it does it much better than a human, but it is fundamentally the same thing.
>Now I’m certain that there exist those mythical human instructors who can do better,
You're certain that mythical instructors exist (?) who "can" do better?
Are human instructors more competent as teachers than AI teachers, or are AI teachers more competent as teachers than human teachers? No "this or that can happen," just a definitive statement please.
AI is likely a million times better student than my dimwit cybersec meatbags...er, majors, for sure, as well! Don't have a reliable way to measure or experience why/how, tho, so I'm not out here claiming it. Even if I did, why would I argue for their replacement?
hammers are both a great tool and a deadly weapon at once
Not at once, surely
1 reply →
As usual it depends. When it does well it's because it can do well. When it does poorly it's because you're prompting it wrong.
>When it does well it's because it can do well.
Can't argue with that logic
They can be incredible. One on one teaching with an infinitely patient teacher who can generate interactive problems on the fly, for dollars a month? Wild. A year of paid ChatGPT would pay for about 9 hours of cheap tutoring here.
That's not going to work out the way you think it will when a student won't even know how to ask questions.
"Education is just a CTF for the valuable flag of a credential. In this essay I will --"
Smart people will use LLMs to learn things faster. Education will adapt by doing all assessments in person.
> We’ve figured out the human replacement pipeline it seems, but we haven’t figured out the eduction part.
No we have not.
I started teaching “how to build quality products using LLMs” full time recently, and most of what I teach is literally just the 101s of systems engineering, reliabily engineering, product development and project management:
Exceptional clarity on the problem you have
Know how to measure the problem you’re solving
Numerically define what “done” is
Make a deterministic and fully observable prototype
Iterate in production with the user
Expand user base as desired with user iteration in parallel forever
Etc…
Obviously a lot more in the details and these are all case by case, but these chatbots are basically perfect productivity machines for this process.
The massive caveat to all of this is this only works for people that can reliably and truthfully define those items above, are willing to structure organization to make those your priorities.
And actually most financial incentives demand the opposite of this process
If most organizations were honest about it, they would simply say “we’re here to make the most money possible and we’re gonna do whatever it takes to do that”
A lot of people don’t like that, so they don’t say it to come up with other bullshit.
Ultimately that’s why I felt like my only option right now is to teach people how to do this because I assumed it was obvious and it is not.
[flagged]