Our Research
This research program is a multisite collaboration initiative that commenced ten years ago, and grew out of a commitment from a small group of clinician researchers to address what was an important, but difficult and enduring problem: objective and valid assessment of practitioner competence in psychology. The project has grown rapidly, and today includes eleven partner institutions across Australia, New-Zealand and the U.K. The research has made major contributions to the international literature on competence assessment in psychology and other health disciplines, has won major national grant awards, and has been disseminated in multiple national and international conferences. The instruments generated by our group are currently in use by more than 24 universities worldwide, and the research and its impact continue to grow.
Supervised field placements are essential in the training and credentialing of psychologists, and other clinical practitioners. As supervisors directly observe trainee performance across a wide variety of real-life situations and over an extended period of time, their judgments of competence are credible and were assumed to accurately assess trainee competence. However, recent research indicates that supervisor assessments are vulnerable to a range of rating biases including halo and leniency effects. The lack of reliable and valid competence instruments contributes, in an important way, to compromised assessment outcomes.
The aim of the project is to design, develop, test, evaluate and refine competence assessment instruments.
To date, the focus has been on two instruments:
CΨPRS
The CΨPRS is a Likert-based assessment tool that can be used by supervisors to evaluate competence levels of trainees at both mid- and end-placement. The mid-placement assessment consists of ten items that evaluate trainee competencies on ten overall domains: Counselling, Clinical Assessment, Case Conceptualisation, Intervention, Ethical Attitude and Behaviour, Scientist-practitioner Approach, Professionalism, Psychological Testing, Reflective Practice, and Response to Supervision.
The 2017 end-placement version comprises 10 overall items and an additional 50 items that allow for a subtler and more comprehensive evaluation of skills and competencies. In addition, new sections allow for qualitative comments about progress during placement and satisfactory/unsatisfactory completion of the placement.
Competence ratings are made on a visual analogue scale that charts progress across four different stages: Stage 1- Beginner (1.0 to 1.9), Stage 2 (2.0 to 2.9), Stage 3 (3.0 to 3.9), and Stage 4 – Competent (4.0 to 4.9). Both the mid- and end-placement assessments can be completed in an easy-to-use online platform.
Institutions that would like to use the online version of the CΨPRS for trainees in their clinical psychology programs may do so by paying a modest annual subscription. Please contact us for details.
Development and Revisions
The CΨPRS has been systematically evaluated, revised and updated over the course of ten years. The content of the scale are updated regularly and are consistent with key developments in competency pedagogy and competence frameworks. Please contact the Project Officer should you like a hardcopy of the scale.
Normative Data
Our research project has collected over 1000 de-identified assessments from universities across Australia. The data has been used to assess the psychometric properties of the CΨPRS, chart developmental trajectories of competencies over time, and to establish the instrument’s reliability and validity. Thresholds that could suggest high and low achievement derived from normative data are available for the online version of the scale. Please contact us to learn more.
Research Impact and Acclaim
Currently, the CΨPRS or one of its variations is being used by twenty four or more universities in Australia and New Zealand. In addition, pilot initiatives to translate the CΨPRS in other languages (e.g., Portugese, Norwegian) are under consideration. Research on the CΨPRS has led to a number of local and international conferences, and publications.
New and Ongoing Research Initiatives
A new instrument that combines the V-MAT for the assessment of overall competence and discrete items from the CΨPRS to rate specific competencies within the domain is currently under trial by several institutions.
The development of accurate self-appraisal is an important competency for a practitioner and an essential element of reflective practice. Concordance between trainees’ and supervisors’ evaluations of trainee competence is an ongoing area of research.
Vignette Matching Assessment Task
Supervised field placements are essential in the training and credentialing of psychologists and other clinical practitioners. However, recent research indicates that supervisor assessments are influenced by leniency and halo rating biases, and multiple attempts to remedy this through better definition of the anchors used in the Likert scaling and by additional training to supervisors, have proven largely ineffective.
A Novel Approach to Trainee Assessment
The V-MAT takes an alternative approach to assessing trainee competence. Rather than using a Likert-scale, it presents detailed vignettes of student competencies that are expected at various training stages. A vignettes is a capsule summary of about 100 words (see below for an example) that profiles essential features of competencies a trainee possesses and does not possess in reference to a specific domain (e.g. counselling skills). Supervisors read the vignettes and match their trainees’ performance with the vignettes in each of the domains in order to determine if they have acquired the level of competence expected at their stage of training.
The detailed descriptions in the vignettes are designed to offer clear and comprehensive benchmarks of expected performance, and are intended to minimise the ambiguity that often arises from Likert-based scales. Consequently, the benefits of the V-MAT are proposed to lie in its capacity to elicit assessments of trainee performance that reduce leniency and halo rating biases and, consequently, are more accurate than those seen with the traditional Likert scales.
Collaborative Research
A large team from 11 different universities from Australia, New Zealand and the UK worked on the design, development, trial, standardisation and evaluation of the V-MAT. The research was funded by two national research grants funded by the Australian Learning and Teaching Council (ALTC; PP10- 1624) and by the Office for Learning and Teaching (ID14-3639), led by Professor Craig Gonsalvez .
Benefits of the V-MAT
The current version of the V-MAT comprises a catalogue of 45 vignettes across 10 competency domains, with typically, four vignettes per domain. Each of the vignettes was carefully crafted, revised and then calibrated – anchored to a specific point along the developmental trajectory from beginner to competent – by a team of 45-experts. Calibrated vignettes that satisfy a set of specific criteria are termed “standardised vignettes” that then serve as developmental milestones or stable anchor points against which the trainee's progress towards competence is measured. Given the importance of trainee placement, and of the need for accurate assessment, the aim of the V-MAT is to provide a measure of trainee competency that is realistic, credible, effective and efficient. The V-MAT has shown to be superior to the CΨPRS and, among other advantages also reduces halo-effects.
Research Acclaim
The research on the V-MAT has led to a number of publications and conference presentations.
Wide Applicability
The V-MAT has exciting cross-disciplinary applications in medicine and other health sciences that are being explored.
Cutting-Edge Research
A new instrument that combines the V-MAT for the assessment of overall competence and discrete items from the CΨPRS to rate specific competencies within the domain is currently under trial by several institutions.
If you are interested in participating in this project, or would like more information, please contact us.
This research program is a multisite collaboration initiative that commenced ten years ago, and grew out of a commitment from a small group of clinician researchers to address what was an important, but difficult and enduring problem: objective and valid assessment of practitioner competence in psychology. The project has grown rapidly, and today includes eleven partner institutions across Australia, New-Zealand and the U.K. The research has made major contributions to the international literature on competence assessment in psychology and other health disciplines, has won major national grant awards, and has been disseminated in multiple national and international conferences. The instruments generated by our group are currently in use by more than 24 universities worldwide, and the research and its impact continue to grow.
Supervised field placements are essential in the training and credentialing of psychologists, and other clinical practitioners. As supervisors directly observe trainee performance across a wide variety of real-life situations and over an extended period of time, their judgments of competence are credible and were assumed to accurately assess trainee competence. However, recent research indicates that supervisor assessments are vulnerable to a range of rating biases including halo and leniency effects. The lack of reliable and valid competence instruments contributes, in an important way, to compromised assessment outcomes.
The aim of the project is to design, develop, test, evaluate and refine competence assessment instruments.
To date, the focus has been on two instruments:
CΨPRS
The CΨPRS is a Likert-based assessment tool that can be used by supervisors to evaluate competence levels of trainees at both mid- and end-placement. The mid-placement assessment consists of ten items that evaluate trainee competencies on ten overall domains: Counselling, Clinical Assessment, Case Conceptualisation, Intervention, Ethical Attitude and Behaviour, Scientist-practitioner Approach, Professionalism, Psychological Testing, Reflective Practice, and Response to Supervision.
The 2017 end-placement version comprises 10 overall items and an additional 50 items that allow for a subtler and more comprehensive evaluation of skills and competencies. In addition, new sections allow for qualitative comments about progress during placement and satisfactory/unsatisfactory completion of the placement.
Competence ratings are made on a visual analogue scale that charts progress across four different stages: Stage 1- Beginner (1.0 to 1.9), Stage 2 (2.0 to 2.9), Stage 3 (3.0 to 3.9), and Stage 4 – Competent (4.0 to 4.9). Both the mid- and end-placement assessments can be completed in an easy-to-use online platform.
Institutions that would like to use the online version of the CΨPRS for trainees in their clinical psychology programs may do so by paying a modest annual subscription. Please contact us for details.
Development and Revisions
The CΨPRS has been systematically evaluated, revised and updated over the course of ten years. The content of the scale are updated regularly and are consistent with key developments in competency pedagogy and competence frameworks. Please contact the Project Officer should you like a hardcopy of the scale.
Normative Data
Our research project has collected over 1000 de-identified assessments from universities across Australia. The data has been used to assess the psychometric properties of the CΨPRS, chart developmental trajectories of competencies over time, and to establish the instrument’s reliability and validity. Thresholds that could suggest high and low achievement derived from normative data are available for the online version of the scale. Please contact us to learn more.
Research Impact and Acclaim
Currently, the CΨPRS or one of its variations is being used by twenty four or more universities in Australia and New Zealand. In addition, pilot initiatives to translate the CΨPRS in other languages (e.g., Portugese, Norwegian) are under consideration. Research on the CΨPRS has led to a number of local and international conferences, and publications.
New and Ongoing Research Initiatives
A new instrument that combines the V-MAT for the assessment of overall competence and discrete items from the CΨPRS to rate specific competencies within the domain is currently under trial by several institutions.
The development of accurate self-appraisal is an important competency for a practitioner and an essential element of reflective practice. Concordance between trainees’ and supervisors’ evaluations of trainee competence is an ongoing area of research.
Vignette Matching Assessment Task
Supervised field placements are essential in the training and credentialing of psychologists and other clinical practitioners. However, recent research indicates that supervisor assessments are influenced by leniency and halo rating biases, and multiple attempts to remedy this through better definition of the anchors used in the Likert scaling and by additional training to supervisors, have proven largely ineffective.
A Novel Approach to Trainee Assessment
The V-MAT takes an alternative approach to assessing trainee competence. Rather than using a Likert-scale, it presents detailed vignettes of student competencies that are expected at various training stages. A vignettes is a capsule summary of about 100 words (see below for an example) that profiles essential features of competencies a trainee possesses and does not possess in reference to a specific domain (e.g. counselling skills). Supervisors read the vignettes and match their trainees’ performance with the vignettes in each of the domains in order to determine if they have acquired the level of competence expected at their stage of training.
The detailed descriptions in the vignettes are designed to offer clear and comprehensive benchmarks of expected performance, and are intended to minimise the ambiguity that often arises from Likert-based scales. Consequently, the benefits of the V-MAT are proposed to lie in its capacity to elicit assessments of trainee performance that reduce leniency and halo rating biases and, consequently, are more accurate than those seen with the traditional Likert scales.
Collaborative Research
A large team from 11 different universities from Australia, New Zealand and the UK worked on the design, development, trial, standardisation and evaluation of the V-MAT. The research was funded by two national research grants funded by the Australian Learning and Teaching Council (ALTC; PP10- 1624) and by the Office for Learning and Teaching (ID14-3639), led by Professor Craig Gonsalvez .
Benefits of the V-MAT
The current version of the V-MAT comprises a catalogue of 45 vignettes across 10 competency domains, with typically, four vignettes per domain. Each of the vignettes was carefully crafted, revised and then calibrated – anchored to a specific point along the developmental trajectory from beginner to competent – by a team of 45-experts. Calibrated vignettes that satisfy a set of specific criteria are termed “standardised vignettes” that then serve as developmental milestones or stable anchor points against which the trainee's progress towards competence is measured. Given the importance of trainee placement, and of the need for accurate assessment, the aim of the V-MAT is to provide a measure of trainee competency that is realistic, credible, effective and efficient. The V-MAT has shown to be superior to the CΨPRS and, among other advantages also reduces halo-effects.
Research Acclaim
The research on the V-MAT has led to a number of publications and conference presentations.
Wide Applicability
The V-MAT has exciting cross-disciplinary applications in medicine and other health sciences that are being explored.
Cutting-Edge Research
A new instrument that combines the V-MAT for the assessment of overall competence and discrete items from the CΨPRS to rate specific competencies within the domain is currently under trial by several institutions.
If you are interested in participating in this project, or would like more information, please contact us.