Reliable Pre-interview Selection of Domestic & International PostGraduate Candidates: CASPer Kelly Dore, PhD, Brad Petrisor, Michelle Ghert, Moyez Ladhani, Harold Reiter, Sharyn Kreuger McMaster University
I do not have an affiliation (financial or otherwise) with a pharmaceutical, medical device or communications organization.
Je n’ai aucune affiliation (financière ou autre) avec une entreprise pharmaceutique, un fabricant d’appareils médicaux ou un cabinet de communication.
Author: Kelly Dore, PhD Date: Sept 26th, 2013
Dilemma • Large applicant pool (Domestic & International) How do bring in the “right” applicants? • Once admitted, it is difficult to remove a trainee • High costs of interviews to program and applicants
Discipline Knowledge
C2LEO, OSCE, PHELO
National Licensing Exam uGPA, MCAT
MMI
Pre-Interview Selection
Computer-based Assessment for Sampling PERsonal characteristics (CASPer)
CASPer Format • Online Computer Test – Computer System Pre-test
• 8 Video Vignettes (SJT format) – Applicant’s perspective – ≈ 90 seconds
• 4 Self-Descriptive Questions – Behavioural Descriptive
• 3 follow-up questions – 5 min open-ended response
You are Dr. Jones
Evaluation • 2 Independent Raters using online portal – Faculty, Community Members and Trainees – Access to resources
• 9-point Likert scale global rating – Red Flags – Comments – Benchmarking
Our Experience in Undergraduate Assessment • 2010 - n = 3510 • 2011 - n = 4209 • 2012 - n = 4558 • Completed CASPer on 2 days (one unadvertised backup day) • Reliability always above 0.7 for 90 minutes of testing
Pilot Application: Postgraduate 2012 CASPer piloted for CaRMS – IMG and Domestic Orthopedics – IMG Pediatrics
• Completed CASPer as part of preinterview selection – 162/178 (Ortho) – 213/281 (Peds)
• Cost absorbed by a research grant
CASPer Postgraduate Selection Overall Test Reliability • Ortho IMG (0.85) • Ortho Domestic (0.65) • Peds IMG (0.77)
Correlation with Interview (MMI) • Correlation was 0.539 for domestic applicants • IMG was underpowered – Only 3 were interviewed
Take Home Messages • CASPer has demonstrated high reliability at the UG applicants, pilot data supports continued review for PG esp with IMGs
• Preliminary correlation with MMI supports the use of CASPer pre-interview, predictive validity still needed • Wider-spread implementation data to follow in the next couple of years
Thank You
[email protected] http://fhs.mcmaster.ca/casper/
Help us improve.
Aidons-nous à nous améliorer.
Your input matters.
Votre opinion compte.
• Download the ICRE App,
• Téléchargez l’application de la CIFR
• Visit the evaluation area in the North Building, Upper Level, near registration, or
• Visitez la zone d’évaluation, au bâtiment nord, niveau supérieur, près du kiosque d’inscription, ou
• Go to: http://www.royalcollege.ca/ icreevaluations to complete the session evaluation.
• Visitez le http://www/collegeroyal .ca/evaluationscifr afin de remplir une évaluation de la séance.
You could be entered to win 1 of 3 $100 gift cards.
Vous courrez la chance de gagner l’un des trois chèques-cadeaux d’une valeur de 100 $.
CASPer Postgraduate Selection Mean scores – Ortho Domestic (3.85) – Ortho IMG (2.7) – Peds IMG (3.0)
You are Dr. Jones
UG Correlation with the MMI 1
0.9
0.8
0.7 0.6
0.6
Study 1 2006
0.51
Study 2 2008
0.48
0.5
2010 Admissi on 2011 Admissi on
0.4
0.36
0.3
0.2
0.1
0 1
UG CASPer Inter-Rater Reliability 1
0.95
0.9
0.84
0.81 0.8
0.7
0.6
Study 1 2006 Study 2 2008
0.5
2010 Admissi on 2011 Admissi on
0.4
0.3
0.2
0.1
0 1
UG CASPer Overall Reliability 1
0.9 0.83 0.8
0.76
0.78
0.72 0.7
0.6
Study 1 2006 Study 2 2008
0.5
2010 Admissi on 2011 Admissi on
0.4
0.3
0.2
0.1
0 1
Wong et al, McMaster – IMG’s cite loss of professional identity, practice in context
Zulla et al, U of Toronto – PDs cited concern about communication, collaboration
Hall et al, U of Ottawa – Multi-source cite communication, practice in context, culture
Reuveni et al, Schabort et al – greater use of screening predicts for better outcome
Implications: Cost • Currently charging students $30 to cover cost – 3 days of helpdesk + pretest helpdesk – 21 new videos – Programming Updates
• Rater Resources – 1 rater uses about 180 raters for 4500 applicants
October 2011 CASPer Administration
Professionalism, 92%
Knowledge, 8%
$50,000,000,000
Please rate the candidate’s performance on this station relative to the pool of all applicants. You may adjust your scores as necessary. Place marks inside the boxes. Score the applicant’s overall performance on this station.
Unacceptable
Marginal
Good
Comments:
Red flags: Yes••
No
If Yes please provide comment:
Excellent
Superior
