Cut Score vs. Raw Score Evaluation (Grading) Scales
Definitions:
Raw Score: The actual score, represented as a decimal or percent showing how many items the student answered correctly divided by the total number of items on the examination.
Published Passing Score: Also known as a published cut score, is set by an institution and represents the minimum percentage a student must achieve to pass the examination (aka-make the cut).
Calculated Cut Score: Also known just as cut score, represents the minimum passing score of an examination based on the average of the cut score of each item included in the test.
Item Cut Score: A number assigned to an item using a modified Nedelsky scale based on the number of good distracters present in an item. The choices typically are 0.9, representing no good distracters, 0.6 representing one good distracter, 0.45 representing two good distracters, and 0.36, which would indicate all 3 of the distracters present in a 4-choice item are good distracters.
Good Distracter: At least 10% of the population taking an item selects this distracter. Data is obtained during the pilot phase and identified once the item has achieved an acceptable discrimination level and has been answered at least 100 times.
Why is there an issue?
Let’s look at two different examinations.
Exam 1
Item Number Calculated Cut Score
1 0.9
2 0.9
3 0.9
4 0.9
5 0.9
This exam represents a test with no good distracters, so a student would need to achieve a 90% to pass to demonstrate proficiency.
Exam 2 (Covering the same material as Exam 1)
Item Number Calculated Cut Score
1 0.6
2 0.6
3 0.6
4 0.6
5 0.6
This exam represents a test with one good distracter present in each item, so to demonstrate proficiency, a student would need to achieve a score of at least 60% to pass.
It is worth remembering that all multiple-choice exams can be thought of as multiple guess exams. If a student passes (or fails), an exam is based on a statistical probability a student comprehends (or fails to comprehend) the material within the examination.
Why do we not expect 100%? Shouldn’t the student know all the information?
Yes, we want the student to know 100% of the information. If you publish a passing score of 70% should not imply that it is acceptable for a student to fail to know 30% of the material. What it implies is your evaluation tools allow for inadvertent errors (I meant to select B but selected A by accident), interpreting the item differently than the author intended (often referred to as “reading into the question”), and where the incorrect answer may be correct answer under certain circumstances that the author of the item failed to recognize.
Now, let’s get back to our two test examples. The two examples are created for simplicity as most exams would contain many more items and be much less homogenous. Your program has chosen a published passing score of 70% (Raw score). Because this test only has five items, the student can only receive a 100, 80, 60, 40 20, or 0%. Let’s look at the case where a student misses only 1 question and achieves a raw score of 80%. The student would have passed this exam according to the published cut score. However, if the student took Exam 1, they really needed to achieve at least a 90% to demonstrate proficiency statistically, so, in fact, they should fail the exam. They would have clearly passed had they taken Exam 2. According to the published cut score, if another student missed 2 of the items and achieved a 60%, they would have failed. Still, had they taken Exam 2, the student would have statistically demonstrated proficiency.
When you choose to set your cohort up with grading by Raw Score, you need to adjust each of your examinations, so the raw score and the calculated cut score of every exam are identical.
Note: This takes a lot of time on your part, but it is an option when setting up your grade book within the Platinum EMS Testing program.
Our Recommendation:
What we recommend is selecting the grading by the Cut Score option. What occurs behind the scenes, in this case, is the program performs a conversion (like converting a temperature from Fahrenheit to Celsius) where in Exam 1, achieving a 90% would be converted to its equivalent in your grading scale of 70%. And similarly, in Exam 2, achieving a 60% would be converted to 70%. Any other grade would be converted using a pro-rated conversion formula to take into consideration of the raw score the student obtained, the calculated cut score of the examination, and your published cut score.