Comment by skookumchuck
6 years ago
> tests are one of the hardest ways of doing so
I've never understood how someone could master a subject and yet be unable to answer any questions about it.
In my experience, people who did well on tests tended to understand the topic, and the people who didn't do well made excuses.
I’ve always had a similar feeling.
In classes where I knew the subject well, I generally did well on the tests. In classes where I had gaps in my understanding, I usually did poorly. In classes where the grades were posted publicly, my general subjective judgment of how well people knew the material matched up with their scores. Not perfectly, obviously, but the correlation has been high enough that I’ve never really been convinced that testing in general is “missing” some critical element of learning.
There might be a really strong correlation, but the deviations from that correlation might not be totally random. That is, there might be people who consistently test above or below their skill level. This creates some weird imbalances that are far from fair.
It would be a lot more acceptable if deviation of test scores from skill were to be totally random for every test.
I can understand biases in a specific test (e.g. cultural biases), but those can (and should!) be corrected with better design and specific accommodations. Do you think that testing as a general method of evaluation is biased though? Because the objection I typically see isn't "we need better test design" it's "there's a component of student ability that cannot be measured by a test".
3 replies →
I guess, but, all the same, I see a world of difference between a topical test, designed by a teacher, to measure your understanding of what they just taught you, and these standardized tests that have become so common.
For one, the latter are often bullshit, as TFA points out. For two, they measure all sorts of stuff aside from actual proficiency. If nothing else, unless they took all the same classes - literally the same classes - throughout their school careers to date, no two kids got the same education. As anyone who's got the vaguest training in science can tell you, that kind of uncontrolled variability in your population will destroy any validity your measure might have. And lastly, so often these standardized tests really do have hacky questions put together with hack procedures. For my part, I distinctly remember completely stumping an IQ test proctor when I was a kid (yeah, I had helicopter parents). I had 5 cards, each showing a house, with the sun and shadows in different positions. I was supposed to put them in the correct order. So, naturally, my first move was to ask if the pictures showed the north or south side of the house.
There's no standardized test that measures the kinds of reasoning skills that really matter in life, such as the kind you'd use to make an educated guess that a question is being asked by the kind of people who would assume, unwittingly, that east is always on the right hand side.
Maybe the test was really asking you which was the normal frame of reference. Of course, the direction would also be reversed if you were in the southern hemisphere, or were on a planet that spun the other way, or maybe a planet in a binary star system, etc.
I would expect the intelligent test taker to understand the test was not a trick, and that unstated assumptions mean use the defaults.
As for math, there are many ways to teach math, but 2+2=4 in all of them. It is reasonable to assume it is base 10, not base 3, unless the test said "in base 3".
It’s because tests aren’t just testing knowledge. They are testing the ability to express the entirety of that knowledge under completely arbitrary conditions (time limits, schedules, no references, etc.) with outsized consequences for mistakes.
Also good tests are hard to write. I’ve seen T/F questions that could go either way. Multiple choice questions with more than one correct answer. (Professors will tell you to choose the “best” answer. But that’s a matter of opinion in many, if not all, cases.)
I think what someone can DO with their knowledge is more important than what individual bits of knowledge they maintained. I’d rather hire or work with someone who can get things done and knows certain algorithms exist than hire or work with someone who can’t get anything done but can recite the same algorithms from memory. Tests favor the second person. (I was that person in HS calculus. Aced the class without understanding a thing just because I have a gift for remembering and applying rules. I had no idea what I was doing.)
> Also good tests are hard to write. I’ve seen T/F questions that could go either way. Multiple choice questions with more than one correct answer. (Professors will tell you to choose the “best” answer. But that’s a matter of opinion in many, if not all, cases.)
This is why I am so glad that all of my university exams (and the _vast_ majority of exams before that, at least post-Y9/age 13) were open-ended questions[1], then marked by someone who will (likely) know the subject better than you even will. Even if you couldn't get the the answer, but could understand and articulate the starting points, or made a compelling argument but misread or misunderstood part of the question, you will at least get partial credit. The physics exams would also have a standardised formulae reference sheet.
[1] A typical paper would be three hours, answering 5-6 questions, and a typical subquestion can be as open-ended as "Write brief notes about a tree representation of functional arrays, subscripted by positive integers according to their representation in binary notation. How efficient are the lookup and update operations?"
It seems like you'd be surprised to learn that the ability to recall and utilize something under constraints is highly indicative of a person's understanding of that material. The kids who "memorize formulas" aren't the ones getting straight A's. The ones who understand and grasp the material are the ones who will have no trouble performing under pressure, because if they forget something, they can recall it using their knowledge structures.
E.g. Say I forgot the formula for Simpson's Rule on a test, but remember that it had to do with approximating integrals with trapezoids. Someone who thoroughly understands the course material could re-derive this formula in under 5 minutes if they had forgotten.
Good test questions are hard to write. That's why standardized tests usually have lots of them so deficiencies with individual questions become statistically irrelevant. At the end of the day, we aren't trying to probe exactly what the students know. We aren't trying to read minds here. We just want an accurate distribution curve. We don't need great test questions for that. We just need some correlation between the correction answers and academic abilities.
> They are testing the ability to express the entirety of that knowledge under completely arbitrary conditions (time limits, schedules, no references, etc.) with outsized consequences for mistakes.
Right. And if you know the material, this isn't a problem.
My understanding is that your experience is correct, but incomplete - it doesn't account for people who don't know the subject, but do well on the test.
Particularly with multiple-choice tests, this is not only possible, but in my experience common. It became a kind of running joke among my classmates that we know our teachers, not the material - over time we just learned the quirks of whoever designed the test and eliminated the wrong answers based on that.
> learned the quirks
Why waste your time learning that? Just learn the material. I'm often bemused by the effort people put into avoiding learning the material, much more than it would take to just learn the material.
Sometimes the material is absolutely unhelpful when doing a badly designed test, and you may even fail them. I had a Calculus teacher that taught 100% from the books but his tests were always 5 special case questions, that you might very well get wrong in under 45' even if you know the bread and butter.
Still learning how to do "Professor X's questions" was a guaranteed ace. Forget the subject, memorize the borderline cases, max grade. Do all exercises on the text book, also, get a second text book and do all of those too, and you might fail the test.
You completely misunderstood; no effort was put into it. There was no active learning, just recognized patterns from prior tests.
1 reply →
That's not what I meant. I didn't say "tests are hard". I said "It's hard to measure knowledge using tests", from the perspective of whoever needs to elaborate those.
In the end, tests end up approving a lot of people that learned next to nothing, as I myself succeeded in several by memorizing for the short term. I couldn't tell how many jingles I used to go through tests. In rarer cases, they also bombing someone with a reasonable understanding of the subject that maybe just was having a bad day or was sick.
It might be somewhat safe to take a poor test result as correlating with poor knowledge on a subject.
And it might be safe to assume a high test score is repeatable.
The challenge is to find a test that correlate high scores with good knowledge - not merely with being good at taking tests.
Because rarely do we care about how good someone is at taking tests; we'd like to measure how good their knowledge is. That's hard to do if we only can infer poor knowledge from a poor test result, but not good knowledge from a good test result.
A perfect system is hardly necessary. Just a good system. Tests have worked for a thousand years. People wouldn't use them if they were useless.
What does it mean to do well on a test covering a subjective subject?
If you have to ask, you don't know the subject :-)
I've never understood how someone could master a subject and yet be unable to answer any questions about it.
How about spoken English? I know a great many people who are able to express themselves clearly and gramatically - surely mastery of spoken English - but ask them what the rule is for order of adjectives or some such and many of them would even need to double check what an adjective is.
That's hardly being "unable to answer ANY questions"; try asking them which is more common: "the big red car" vs "the red big car", do you think they'd be unable to answer?
That's not a question about the English language. That's asking someone to use it.
The question of this nature about the English language would be "how do I know what order to put adjectives in?" and the answer would be something along the lines of "opinion-size-age-shape-colour-origin-material-purpose".
I do think they would be unable to answer. The rule of course then goes deeper, into ablaut reduplication and so on. People master spoken English without knowing anything [0] about it.
[0] Yes, I know, I said "anything" and now your literalist side is jumping in joy at being able to say "Aha, aha, they DO know at least one thing and therefore your statement is not literally correct!" I don't care.
1 reply →