2023RSS0066 - A Situational Judgement Test (SJT) for use in Family Medicine (FM) Residency Selection

The University intends to award a 3-year service contract to Work Psychology Group to provide
the following services:

Facilitate a half-day virtual session with Subject Matter Experts (SMEs) to contextualize previously validated Situational Judgement Test (SJT) scenarios to a Canadian family medicine (FM) context.
Create a new SJT that contains new and previously contextualized scenarios for a Canadian FM setting.
Coordinate and validate the SJT scenario answer key with SMEs using a rigorous concordance methodology.
Develop additional practice materials for test candidates.
Review current documentation materials to ensure candidates are clear on the purpose and process of the SJT.
Coordinate in collaboration with the UCalgary project team the set-up, review, and delivery of the SJT via the test platform twice per year.
Coordinate with the test platform vendor to upload, review, and deliver 6 versions of the SJT and supporting materials in English and French: (i) Standard test, English; (ii) Standard test, French; (iii) 1.5X time, English; (iv) 1.5X time, French; (v) 2X time, English; (vi) 2X time, French.
Liaise with the test platform vendor to ensure candidates only have access to 1 of 6 test versions.
Liaise with the test platform vendor to test and implement a remote photo test security invigilation software application.
Liaise with test platform vendor regarding the provision of required data in usable format for scoring and analysis purposes.
Data cleaning and psychometric analysis of all results provided by test-platform vendor.
Provision of SJT scores to UCalgary project team and to the Canadian Residency Matching Service (CaRMS).
Quantitative and qualitative analysis of post-test evaluation surveys completed by candidates.
Quantitative analysis of candidate demographic data and SJT scores.
Comprehensive annual technical report of all quantitative, qualitative, and psychometric analyses related to the SJT delivery and research projects related to post-test evaluations and candidate demographics.
Provide technical and logistical recommendations to improve the usage of the SJT.
Participate in all joint planning and coordination meetings.