For more than two decades, the Bronx Defenders has provided a holistic form of legal aid to indigent residents in New York. In addition to pro-bono criminal defense, lawyers help clients with the gamut of social and legal challenges that often surface in the wake of an arrest. The work is expensive, and like many nonprofits, the program has relied on client testimonials, surveys, and tireless advocacy to make the case to legislators and deep-pocketed donors that it’s worth funding.
But in January 2019, fighting to keep the lights on got a little bit easier. Researchers from the RAND Corporation and the University of Pennsylvania performed an evaluation of the Bronx Defenders, and found that it has overwhelmingly succeeded: The program cut pretrial detention time by 9 percent; it reduced incarceration rates by 16 percent; and it shortened sentence lengths by 24 percent, an amount researchers concluded had saved taxpayers an estimated $165 million. Those findings, published in the prestigious Harvard Law Review, confirmed what Executive Director Justine Olderman already believed, and provided her with ironclad evidence for the organization’s future funding pitches.
So it shocked Olderman when, 13 months later, she saw that CrimeSolutions, a project of the National Institute of Justice that translates academic research for policymakers and practitioners, had come to the opposite conclusion. Using the RAND study as the basis of its evaluation, the federal rating system declared that the work of the Bronx Defenders had “no effects.”
“I cite [the RAND] study everywhere I go, we all do,” Olderman told The Trace, referring to colleagues in the criminal defense field. “And here they’re saying it’s proof the program doesn’t work.”
CrimeSolutions’ take on the Bronx Defenders is one in a series of questionable ratings that has undercut the platform’s reputation in academic circles and prompted debate about whether it needs reform. Some criminal justice researchers say the federal initiative is falling short of its mission and jeopardizing the support of promising crime interventions in the process. It’s particularly dangerous now, they say, as cities across the country scramble for solutions to a record spike in violent crime.
“It’s hard enough to get people outside of academia to look for the evidence,” said Jennifer Doleac, an economist at Texas A&M University who has been an outspoken critic of CrimeSolutions. “So when they do look for evidence and it steers them in the wrong direction it means A) [policy makers land on a policy] that is at best ineffective and at worst harmful and we’re wasting time and money implementing it or B) it means these policy makers might just throw up their hands and lose faith in the utility of science and research.”
Thomas Abt, a senior fellow at the Council on Criminal Justice who helped get CrimeSolutions off the ground, said that while he, too, had noticed questionable ratings on the site — particularly during the Trump administration — he believes it has continued to provide an important service to the policy world. “I would take the position that we should fix what’s wrong, not do away with it.”
CrimeSolutions launched in 2011 as part of a broader push by President Obama’s Department of Justice to integrate evidence into policy making at the federal, state, and local levels. A press release called it a “central, credible resource to inform practitioners and policymakers about what works in criminal justice, juvenile justice, and crime victim services.” The site detailed a rigorous procedure — led in part by a team of PhD researchers from venerated universities — for selecting programs to assess and for reviewing academic literature about those programs’ efficacy. At the conclusion of a review, the site would note that research was inconclusive, or award one of three evidence ratings: “effective,” “promising,” or “no effects.”
Since its inception, CrimeSolutions has doled out “effective” ratings sparingly. Of the 630 programs it has evaluated, only 91 met the threshold for a rating of “effective.” A little more than 150 were rated “no effects.” Researchers can appeal ratings they disagree with, but according to data from The National Institutes of Justice, less than 20 have been appealed.
The majority of the ratings reviewed for this story involved programs that showed mixed influence on their intended outcomes and were rated “no effects.” In one particularly stark example, University of Michigan economist Sara Heller conducted an evaluation of a Chicago youth employment program called One Summer Plus. Her study appeared in the peer-reviewed journal Science, bearing the title: “Summer Jobs Reduce Violence Among Disadvantaged Youth.” It found that — during a 16-month follow-up period — violent crime arrests among program participants decreased 43 percent relative to a control group of young people who did not participate in the program. CrimeSolutions, reviewing only Heller’s study in its determination, rated the program “no effects.” Heller is appealing the decision.
Representatives from the National Institute of Justice told The Trace that portions of Heller’s study failed to meet the threshold for statistical significance imposed by the platform, a point that is outlined on the One Summer Plus program review page.
Doleac and other researchers interviewed for this story agreed that CrimeSolutions’ rating system is too rigid to accommodate the level of nuance that policymakers and practitioners need to make informed decisions. “Sometimes you just get mixed evidence,” Doleac said. “Instead of calling something ineffective when there’s mixed or not enough evidence, rate it as ‘more research needed’ or something to that effect. That feels like an easy change to make.”
Representatives from the National Institute of Justice told The Trace that CrimeSolutions’ assessment procedure undergoes periodic review and said that the language used to describe evidence ratings “would be reviewed” as part of this standard quality improvement process. They added that moving forward, reviewers would notify study authors when a program was set to receive a “no effects” rating — formerly, only “effective” and “promising” ratings prompted a notification.
In the case of the Bronx Defenders, researchers in charge of the study had concluded that the program achieved a majority of its intended outcomes, but failed to produce any statistically significant reduction in the conviction or rearrest rates of its clients. The study’s authors appealed the rating in February of 2021, and CrimeSolutions informed them that this failure had justified the “no effects” rating. The appeal is ongoing.
Paul Heaton, a University of Pennsylvania economist who co-authored the RAND study, questioned the logic of handing out “no effects” ratings for programs that are at least partially effective. “If I told you I have some new program you can implement and we can release 25 percent more people from prison without any negative impact on public safety, most people would look at that and say, ‘Hey, that looks great,’” he explained.
CrimeSolutions breaks out individual program outcomes in a tab on the main evaluation page. On the page for the Bronx Defenders review, the site notes that the program successfully reduced the length of sentence and the likelihood of a jail sentence for its clients. But, Heaton said, the rating does not reflect this nuance.
In another instance from 2018, researchers from the University of California, Irvine, and George Mason University published a study evaluating a program at the Seattle Police Department designed to teach officers to act more fairly in civilian encounters. The program failed to meet a number of its intended outcomes — it did not reduce civilian complaints about officer interactions, or reduce the percentage of interactions that ended in an arrest. It did, however, reduce the likelihood that participating officers would be involved in use-of-force incidents. Still, CrimeSolutions rated it “no effects.” The authors of this study appealed the rating, but were unsuccessful.
Heller and Heaton noted that CrimeSolutions staff have been accommodating and responsive throughout their appeals processes, but do not yet have answers about whether the ratings of their programs will be adjusted.
Olderman, from the Bronx Defenders, said that ultimately, she supports the CrimeSolutions mission, and thinks the site is badly needed. But she worries that lumping truly ineffective interventions together with others that have seen moderate success risks more than just the survivability of a single program. “A lot of the jurisdictions where our model of holistic defense is being tried rely on the RAND study” to make appeals to for their own funding, she said. “So the implications of a rating that says this work is not effective are not just for us as an institution, but for public defense as a field.”
