How do you standard set the OSCE?

The process used to set pass marks for assessments is known as standard setting. Different methods are used in Liverpool for the written and clinical exams. The pass marks for the written papers are set using the modified Angoff method, which is a test-based method and the clinical exams (OSCE and LOCAS) use borderline regression, which is an examinee-based method.

standard-settingThe borderline regression method uses information provided by examiners on the marks sheets (global rating). Examiners complete the mark sheet to derive a station score, they also provide a ‘global’ or “overall” rating of the student’s performance on a scale from Not yet competent to Outstanding.

Using all the data from a station, a regression line can be plotted relating station score to “global” or “overall” rating. This is used to determine the score which is equivalent to a borderline performance on the station.

The overall passing score for the OSCE is calculated by adding up all the passing scores for the stations. The borderline regression method is preferred for the clinical exams, as the modified Angoff method is very time consuming in these cases, and has the advantage that it utilises information that can be obtained during the assessment.

 

Why do we use absolute methods?

We use absolute methods (borderline regression) as these are fairer to the students and allow us to be more confident that everyone that passes the assessments is competent to progress to the next stage of the course. If we have a good cohort of students and we always failed the bottom 5% (which would be a relative method of standard setting) we might be failing students who are in fact competent. By using absolute methods that use expert judgement, we are able to adjust the pass marks to reflect the difficulty of the exams, so that if we set a difficult exam, we will not fail students who might have passed on an easier paper, and the different cohorts which might occur if we always used the same fixed pass mark.