Another Route to Control

Published in the Spring 2011 issue of SSA Magazine

Sidebar article for "The Science of Social Welfare"

In some cases, creating a randomized control group study simply isn't possible. For example, over more than two years, a total of 120 mothers participated in Dean Neil Guterman's study of the Parents Together program, in peer groups of five to 20. However, he knew that the agencies running the program wouldn't be able to bring in the sufficient numbers of mothers who were eligible for inclusion at any given time that would be necessary to also create a comparable control group.

However, with a tool called propensity score matching, researchers can still bring the power of statistical comparison to their work, shining a light on the impact a specific intervention had on a study group. With propensity score matching, a researcher can use observed predictor factors, usually obtained from a statistical strategy called logistic regression, to create a counterfactual group—essentially building a virtual control group and a virtual treatment group. "In this case, it was a modified compromise that still allowed us to find some very promising trends that are being tested with comparison groups," Guterman says.

For Assistant Professor Matthew Epperson's study of defendants with mental illnesses in the criminal justice system, propensity score matching will provide a way to approximate a quasi experimental design. His study will look at the impact of three court-based programs operating in Chicago: a mental health court, a specialized probation mental health unit that has staff trained in mental health, and standard probation.

"It is possible as a researcher to do randomized control studies in the criminal justice system, but it wasn't feasible in this case," he explains. "But with this method, I'll be able to compare the three programs to each other to see if they have different strengths, if they work better for different populations. I plan to conduct a retrospective analysis to build a longitudinal study next year. I'll examine arrest data from cohorts of people who graduated from these programs in 2007 and see the impact of each program on recidivism five years later. Evaluations of these specialized court-based programs for persons with mental illnesses are still in the infancy stages, and to my knowledge no study had done a comparative evaluation like this."

Building comparable groups is as crucial in propensity score matching as it is to create balanced cohorts in a randomized control study. "Indistinguishable is what we're aiming for when you're talking about creating balance between two study groups," says Assistant Professor Jennifer Bellamy, who has used the methodology in her research of the impact of outpatient mental health treatment for children in foster care. "You need to create statistically equivalent groups. The number of boys and girls, similar baseline symptoms, the history of the different homes they've lived in, their experience with child maltreatment—it all needs to be as much the same as possible. Not for each child, but in the aggregate for the group."

Bellamy mined the huge National Survey of Child and Adolescent Wellbeing to find the data she needed for the study. She found that the outpatient mental health services didn't make a difference in outcomes for the children in foster care, but she points out that the findings raise a new set of questions. "Many of the kids only received short-term services, and we don't know the quality of the services or the context in which they were delivered," she says. "To help these kids, we now know the next step is more research on the best way to deliver services and how to create quality services."