Instrument Validation
“Validation” generally refers to the process of establishing documented evidence that an assessment system produces measurements that are consistently associated with an outcome of interest, such as criminal recidivism. When this process is applied to offender assessment instruments, validation studies may strive to answer questions such as:
-
How faithfully are assessment practitioners administering the instrument?
-
What difference does additional practitioner training have on instrument validity and reliability results?
-
Is the instrument a reliable predictor of re-arrest, probation violation, or other recidivism measures such as termination status, time to recidivate, and self-report grade retention and employment stability?
-
How does instrument validity and reliability differ for subgroups such as females, youths, minorities, offenders from different regions, or sex, domestic violence, or English as a second language (ESL) offenders?
-
Are instrument subscales (when applicable) better predictors of various recidivism measures than the total risk score for the general population? For subgroups?
-
When more than one assessment instrument is administered, do the characteristics they measure overlap according to existing literature, and to a degree consistent with the intercorrelations produced by these instruments and outcomes?
-
When the instrument is used to reassess offenders over time, do changes in subscales (such as gains and losses in risk or protective scores) reflect expected changes in recidivism measures?
Once these questions are answered for an assessment instrument, agencies may decide:
-
Whether to continue use of the instrument,
-
How to interpret results differently for different sub-populations,
-
What kind of additional training or quality assurance measures practitioners might need to increase instrument validity, and
-
How to use the instruments for prioritizing the use of resources (for supervision, intervention, etc.)
This kind of information is essential for agencies desiring to follow Evidence-Based Practices for community corrections.
Our Validation Experience
Over the past 10 years, J-SAT has conducted validation studies and reliability testing for each of the following agencies:
-
Connecticut Judicial Branch CSSD Adult and Juvenile Probation and Provider Network (JAG, SUS, LSI-R, ASUS)
-
Cook County Juvenile Probation (JASP)
-
Hawaii Department of Public Safety and Judicial Probation (Proxy)
-
North Dakota Department of Corrections (Proxy)
-
Salt Lake County Division of Substance Abuse Services (LSI-R)
-
San Diego Juvenile Probation (RRC, with Surgeon General’s “Protective Scale”)
As we are also experienced in instrument design and in training staff in juvenile and adult assessment instrument implementation, we are able to anticipate many of the logical and logistical obstacles that validation studies typically encounter.
Setting Up a Validation Study
Before you contact us about setting up a validation study for your agency, you may want to think through some of the following questions, as we will need this kind of information from you to tailor our services to your needs:
-
What instrument(s) would you like validated?
-
What specifications or attributes about the instrument(s) should be the focus of the validation study?
-
What size of offender sample is available for the analysis? (Typically, 100 is the minimum sample size for reliable analysis.)
-
Which sub-populations would be important to consider within the overall sample?
-
How can you ensure that the sub-populations represent a significant enough proportion of the entire sample for reliable analysis?
-
Over what time period should instrument results be measured?
-
What logistical, financial, or political obstacles should be thought through in advance for successful project design, data collection, and completion of the analysis?
Services That Can Accompany Instrument Validation
Outcome Analysis—allows your agency to see which processes or treatment strategies show the greatest reductions in recidivism over time.
Quality Assurance Structure Analysis—assesses agency goals, documents current quality assurance procedures, and recommends revisions or additions to QA practices that optimize goal achievement.
Dashboard Agency Measures—an automated application tailored to your agency’s quality assurance goals, providing allowing for quick, objective measurement of current progress toward agency goal completion.
Automated Case Planning Software—streamlines the assessment-to-treatment-matching process for juvenile offenders, ensuring that case planning strategies follow Evidence-Based Practices.
Workload Analysis—identifies discrepancies in staff workload assignments and process inefficiencies; measures actual time spent on various functions and tasks against agency expectations or practices shown to reduce recidivism.
|
