Comment by rdiddly

I’ve always had a similar feeling.

In classes where I knew the subject well, I generally did well on the tests. In classes where I had gaps in my understanding, I usually did poorly. In classes where the grades were posted publicly, my general subjective judgment of how well people knew the material matched up with their scores. Not perfectly, obviously, but the correlation has been high enough that I’ve never really been convinced that testing in general is “missing” some critical element of learning.

I guess, but, all the same, I see a world of difference between a topical test, designed by a teacher, to measure your understanding of what they just taught you, and these standardized tests that have become so common.

For one, the latter are often bullshit, as TFA points out. For two, they measure all sorts of stuff aside from actual proficiency. If nothing else, unless they took all the same classes - literally the same classes - throughout their school careers to date, no two kids got the same education. As anyone who's got the vaguest training in science can tell you, that kind of uncontrolled variability in your population will destroy any validity your measure might have. And lastly, so often these standardized tests really do have hacky questions put together with hack procedures. For my part, I distinctly remember completely stumping an IQ test proctor when I was a kid (yeah, I had helicopter parents). I had 5 cards, each showing a house, with the sun and shadows in different positions. I was supposed to put them in the correct order. So, naturally, my first move was to ask if the pictures showed the north or south side of the house.

There's no standardized test that measures the kinds of reasoning skills that really matter in life, such as the kind you'd use to make an educated guess that a question is being asked by the kind of people who would assume, unwittingly, that east is always on the right hand side.

It’s because tests aren’t just testing knowledge. They are testing the ability to express the entirety of that knowledge under completely arbitrary conditions (time limits, schedules, no references, etc.) with outsized consequences for mistakes.

Also good tests are hard to write. I’ve seen T/F questions that could go either way. Multiple choice questions with more than one correct answer. (Professors will tell you to choose the “best” answer. But that’s a matter of opinion in many, if not all, cases.)

I think what someone can DO with their knowledge is more important than what individual bits of knowledge they maintained. I’d rather hire or work with someone who can get things done and knows certain algorithms exist than hire or work with someone who can’t get anything done but can recite the same algorithms from memory. Tests favor the second person. (I was that person in HS calculus. Aced the class without understanding a thing just because I have a gift for remembering and applying rules. I had no idea what I was doing.)

My understanding is that your experience is correct, but incomplete - it doesn't account for people who don't know the subject, but do well on the test.

Particularly with multiple-choice tests, this is not only possible, but in my experience common. It became a kind of running joke among my classmates that we know our teachers, not the material - over time we just learned the quirks of whoever designed the test and eliminated the wrong answers based on that.

That's not what I meant. I didn't say "tests are hard". I said "It's hard to measure knowledge using tests", from the perspective of whoever needs to elaborate those.

In the end, tests end up approving a lot of people that learned next to nothing, as I myself succeeded in several by memorizing for the short term. I couldn't tell how many jingles I used to go through tests. In rarer cases, they also bombing someone with a reasonable understanding of the subject that maybe just was having a bad day or was sick.

It might be somewhat safe to take a poor test result as correlating with poor knowledge on a subject.

And it might be safe to assume a high test score is repeatable.

The challenge is to find a test that correlate high scores with good knowledge - not merely with being good at taking tests.

Because rarely do we care about how good someone is at taking tests; we'd like to measure how good their knowledge is. That's hard to do if we only can infer poor knowledge from a poor test result, but not good knowledge from a good test result.

What does it mean to do well on a test covering a subjective subject?

I've never understood how someone could master a subject and yet be unable to answer any questions about it.

How about spoken English? I know a great many people who are able to express themselves clearly and gramatically - surely mastery of spoken English - but ask them what the rule is for order of adjectives or some such and many of them would even need to double check what an adjective is.

They are a product of industrialization! But here is what's the nagging thought on my mind. This industrialized schooling has allowed basic human knowledge to be disseminated at a faster pace than ever before. The need to do so caused it to roll back on quality, sometimes greatly. But without this tradeoff, would knowledge be able to reach so many? Is there some sort of balance point to be reached?

I agree that currently, education systems are terrible, but is there any way to maintain them at scale that is not?