Comment by csa

> Very effective multiple choice tests can be given, that require work to be done before selecting an answer, so it can be machine graded.

As someone who has been part of the production of quite a few high stakes MC tests, I agree with this.

That said, a professor would need to work with a professional test developer to make a MC that is consistently good, valid, and reliable.

Some universities have test dev folks as support, but many/most/all of them are not particularly good at developing high quality MC tests imho.

So, for anyone in a spot to do this, start test dev very early, ideally create an item bank that is constantly growing and being refined, and ideally have some problem types that can be varied from year-to-year with heuristics for keys and distractors that will allow for items to be iterated on over the years while still maintaining their validity. Also, consider removing outliers from the scoring pool, but also make sure to tell students to focus on answering all questions rather than spinning their wheels on one so that naturally persistent examinees are less likely to be punished by poor item writing.