Evaluating the Comprehensiveness/Effectiveness of Seating and Wheelchair Prescription: A Validity Test of a Clinical Rationale Measure

Laura Cohen PT, ATP
Human Engineering Research Laboratory
University of Pittsburgh

Slide 1
Evaluating the Comprehensiveness/Effectiveness of Seating and Wheelchair Prescription: A Validity Test of a Clinical Rationale Measure

slide with text only

Laura Cohen PT, ATP
Human Engineering Research Laboratories, VAMC
University of Pittsburgh

This is a report on the results of the validity study conducted during the International Seating Symposium 2001 in Orlando.

The purpose of this entire project was to develop a measure to quantify clinical rationale of wheelchair and seating prescription. This presentation will discuss the development process of the measure and report concurrent validity test results.

Slide 2
Background

slide with text only

Scarcity of research related to effective means of increasing competence and proficiency of therapists

Scope and depth of therapists and skills can vary widely affecting advice and prescription of equipment to consumers

A review of the literature reveals a dearth of research related to effective means of increasing competence and proficiency of professionals working in the field of seating and wheeled mobility.

Prescribing wheelchairs and seating systems is typically completed by a team consisting of the client, RTS, therapist and physician.

The therapist usually performs the physical, functional and environmental evaluation.

The scope and depth of evaluation skills of the therapist can vary widely resulting in varying quality of advice and equipment prescription for consumers.

Slide 3
Background

slide with text only

Professional training will improve the consumer/technology match

Specially trained therapists can be hard to find

Many feel that targeted professional training will maximize the consumer/technology match, unfortunately, experienced or specially trained PTs and OTs trained to provide seating and wheeled mobility prescription can be hard to find.

Slide 4
Background

slide with text only

The most widely accepted means to participate in upgrading professional competence is through continuing education (CE) activities

Literature suggests there is sparse data to support the effectiveness of CE in changing therapist behavior and influencing pt outcomes

Slide 5

slide with text only

A tool measuring clinical competency and proficiency is required

In order to determine the most effective way to improve competency and proficiency of professionals, a tool measuring clinical competency and proficiency is required. Currently, no tool exists.

Slide 6
Purpose

slide with text only

Describe the development of both the tool and the scoring method used to evaluate clinical rationale fundamental to seating and wheeled mobility prescription

Test concurrent validity of this measure

The purpose of this paper is to describe the development of both the tool and the scoring method used to evaluate clinical rationale fundamental to seating and wheeled mobility prescription and test concurrent validity of this measure.

Slide 7
Test Development

slide with text only

The development of a quantitative testing and scoring method was completed over two separate trials.

Trial One

  • 20 PT and OT students about to enter the field

Trial Two

  • 89 PT and OT professionals with varying amounts of experience

Slide 8
The Test

slide with text only

Subjects viewed a videotaped seating and wheeled mobility evaluation

Completed an assessment form identifying problems, goals and recommendations for the client presented

A videotape of an ATP performing a seating and wheeled mobility evaluation of a client was shown in order to standardize information presented to subjects.

Subjects were provided with a generic wheelchair prescription form consisting of three columns (problems, goals, recommendations).

Subjects were asked to identify client problems, translate them into goals and write a generic prescription/recommendation for a wheelchair and seating system.

Test duration was approximately 1 and a half hour (45 minute video, 45 minute documentation).

Slide 9
Scoring System

slide with text only

Experts were polled and a list of common seating and mobility problems, goals and equipment features was created

List was used as a checklist to transfer data from subject test sheets

6 expert clinicians with a mean of > 12 years of experience were polled to create a list of common seating and mobility problems, goals and equipment features

This list was used as a checklist to transfer data from subject test sheets

Slide 10
Scoring System

slide with text only

Checklist sheet for each subject was compared to a gold standard answer key of potential correct answers

A score for each subject was tallied

Two separate expert clinicians viewed the videotape independently and completed the checklist-grading sheet and came to consensus of what constituted a correct response. The agreed upon response became the gold standard key.

The subjects checklist was compared to the gold standard and a score tallied.

Slide 11
Reliability

slide with text only

Trial One scoring system tested for interrater and intrarater reliability and found to be poor

Due to low reliability, the checklist-grading sheet was revised and retested in trial two

Trial Two scoring system tested for interrater reliability and resulted in poor to moderate reliability

Slide 12
Results

slide with text only

Concurrent Validity Testing (classification method)
(N= 20)

  1. Spearman's rho analyses comparing average grand total (AVGT)
  • Hrs/wk of seating and mobility service provision
  • Yrs of clinical experience
  • Yrs of seating and mobility experience
  • Age

No significant relationships were found

Nonparametric tests were chosen due to the small sample size (n=20). The Spearman's rho correlation revealed a significant relationship (p = .000) between CGT scores for the two independent scorers. Because of this significant relationship the scores were averaged from the two scorers resulting in grand total scores used for further analyses.

Spearman's rho analyses were performed comparing average grand total (AVGT) to

  • hours per week of seating and mobility service provision,
  • years of clinical experience,
  • years of seating and mobility experience
  • age

No significant relationships were found.

Slide 13
Results

slide with text only

Concurrent Validity Testing (classification method)

2. A Kruskal-Wallis analysis comparing AVG

  • Profession
  • Gender

No significant relationships were found

A Kruskal-Wallis test was performed to investigate the relationship between AVGT and profession and AVGT and gender.

No significant results were found for AVGT and profession. AVGT and gender was significant (p=0.01), however after reviewing gender with other characteristics (i.e. years of experience, years of seating and mobility experience and profession) we determined that the significance was most likely due to the small sample size (N= 5 males, 15 females).

Slide 14
Discussion

slide with text only

Difficult to develop a measurement tool since evidence of competency and rationale are not tangible and must be inferred

Ability is dependent on contextual & situational factors

In test situations researchers are challenged to standardize patient examples to increase reliability often compromising external validity

It is difficult to develop a measurement tool that will provide evidence of competency and rationale since abilities are not tangible; and must be inferred.

Because ability is dependent on contextual or situational factors, it is difficult to develop a reliable and valid measurement tool that will include the factual knowledge, clinical skills and professional judgments a clinician must demonstrate to provide evidence of competency.

In test situations, researchers are challenged to standardize patient examples and protocols in order to increase reliability of clinical grading. This practice often compromises external validity.

Typically, competency tests are performed because we intend to draw conclusions about a clinician's ability in nonstandardized contexts.

Slide 15
Summary
Criterion Validity

slide with text only

  1. Concurrent Validity
  • Study group differences and AVGT score
  • Used when sample characteristics known

2. Predictive Validity

  • Evaluation of a test to determine how accurately a target test relates to the performance on a criterion test
  • Currently no criterion test available

To determine criterion validity, studies of group differences and studies of correlates of the test under study are most frequently performed.

Two different types of criterion validity include concurrent and predictive validity and differ in the criteria used as the standard of judgment.

The classification method of concurrent validity testing can be used when sample characteristics are known (i.e. years of experience, years of seating and mobility experience, profession, gender, etc.) Studies of group differences involve administering a new instrument to groups who are expected, in theory, to score at different levels on the instrument and to compare the obtained results with these expectations.

If the results concur with expectations, the evidence accumulated demonstrates the criterion validity of the new instrument.

Predictive validity is the evaluation of a test to determine how accurately it can be used to make a prediction using as evidence the relationship between a target test and a criterion test. Currently there is no criterion test available to measure clinical rationale for this purpose.

Slide 16
Conclusion

slide with text only

The Clinical Rationale Assessment of Prescription (CRAP) suggests poor concurrent validity

In summary, our measure The Clinical Rationale Assessment of Prescription is as the acronym suggests CRAP. Our results suggest poor concurrent validity.

Slide 17
Conclusion & Limitations

slide with text only

Decreased generalizability

Lacks concurrent validity causing us to question findings in previous research

Poor interrater / intrarater reliability

Due to the specificity of the one case example the findings from this tool can not be generalized to other nonstandardized contexts.

The negative findings indicate this test instrument lacks concurrent validity and causes us to question the findings in our previous research project entitled Teaching clinical rationale for seating and wheeled mobility prescription: A randomized controlled trial of four instructional methods.

The intrarater and interrater reliability results of this clinical rationale measure primarily show values below .75, which according to Portney and Watkins, are indicative of poor to moderate reliability. Portney and Watkins suggest guidelines for clinical measurements should exceed .90 to ensure reasonable validity.

Slide 18
Future Studies

slide with text only

Development and testing of a NEW clinical rationale measure underway entitled:

Seating and Mobility Script Concordance Test (SMSCT)

Future work will include the development and testing of a different clinical rationale measure based on what we have learned and applying established test development strategies towards its development.

Slide 19
Acknowledgements

slide with text only

Co Investigators:
Shirley Fitzgerald, PhD
Elaine Trefler, Med, OTR/L, FAOTA, ATP
Michael Boninger, MD
Michael McCue, PhD, CRC

Jessica Pederson staff of the Center for Assistive Technology at the University of Pittsburgh

Funded by the VA Center of Excellence in Wheelchair and Related Technology

The End

Return to Slide Series

Updated: March 22, 2002

" "

Return to:
WheelchairNet Home Page 

Please let us know if you find a link that doesn't work or have an idea about something to include!

Contact information:
  Department of Rehabilitation Science and Technology  Telephone: 412.624.6279

 © Copyright 2006 University of Pittsburgh. All rights reserved.
No quotes from the materials contained herein may be used in any media without attribution to WheelchairNet and the Department of Rehabilitation Science and Technology.


Please note: This information is provided a archival information from the Rehabilitation Engineering Research Center on Wheeled Mobility from 1993 to 2002.

" "