Nicole P. Marwell, PhD

Diverse stakeholders argue that randomized controlled trials have the potential to revolutionize the effectiveness of U.S. human service nonprofits by building evidence for program effectiveness and focusing scarce resources. This book provides a counterpoint by examining the potential unintended consequences of this trend for nonprofits and the marginalized communities they serve. 

What makes a nonprofit organization effective? There are many potential answers to this question. Increasingly, however, funders of nonprofits, including government, foundations, and individual donors, seek quantitative measurements of program effectiveness as proof that nonprofits “work.” Such measurements are legible across a wide range of audiences, and carry with them the promise of certain knowledge in an uncertain world. In recent years human service nonprofits have been facing mounting pressure to produce a particular kind of quantitative measurement about this work: “net impact,” or the change in individuals that can be directly traced to a nonprofit’s efforts and would not have occurred without those efforts. There are a number of ways to assess net impact, but one approach is usually referred to as the “gold standard:” the randomized controlled trial (RCT). RCTs often are used to determine whether a specific program “works” at a small scale, with the goal of standardizing and replicating programs across multiple human service organizations.

High quality human service nonprofits, however, do not simply provide “programs that work.” They also engage in both community building work and individualized service provision, neither of which the RCT evaluation paradigm is well-suited to assessing. First, while participants in human service nonprofits can certainly benefit from standardized, evidence-based programs aimed at individual change, many nonprofits engage in community-oriented work that connects organizational participants to one another, aligns nonprofit endeavors with participant priorities, and contributes to the development and pursuit of collective aims. Activities that foster this goal include engaging volunteers, advocating for people who are socially excluded, serving as community hubs, building social connections, and developing innovative models of change. This work is hard to measure and requires a long-term orientation. Second, nonprofits often seek to provide services that are tailored to individuals and the unique communities in which they are embedded. Doing so successfully means building strong relationships with participants to understand the challenges posed by their economic and social conditions, being responsive to individualized and emerging needs, and tailoring services to participants’ cultural backgrounds. This stands in direct contrast to the standardized program model that RCT evaluation work promotes and is especially important for nonprofits serving marginalized populations or rural communities.

This book thus asks how the rise of RCT-based measures of effectiveness is shaping today’s nonprofit human services sector in the United States, and what the implications are for the future of the nation’s nonprofit organizations and the people and communities they serve. We argue that because the RCT evaluation paradigm focuses attention primarily on standardized, scalable programs that ameliorate individual need, it may lead to scrimping on support for the work required for nonprofits to build enduring personal relationships and foster community strength.

As society increasingly relies on the Internet for work, education, health care, recreation, and many other aspects of daily life, the prevalent and persistent inequity in people’s ability to access, adopt, and use the Internet is more evident than ever. In the wake of the COVID-19 pandemic, these inequities have become apparent at the global, national, municipal, and neighborhood scales. For example, at the start of the pandemic in the City of Chicago, more than 100,000 children under the age of 18 lacked access to broadband Internet connectivity, and in certain neighborhoods nearly half of children under 18 lacked Internet connectivity. These gaps become even more pronounced in times of crisis and natural disaster, when communities rely almost exclusively on Internet communications for many critical aspects of daily life. The issue has not gone unnoticed by policymakers, who have developed a variety of local efforts to provide schoolchildren with Internet service and Internet-connected devices, and recently passed federal legislation committing nearly $50 billion to improving Internet access, affordability, and literacy. 

Stakeholders who are making decisions about where and how to direct these investments need better information along the following three dimensions

1. Measurement techniques and datasets that directly address unknown questions and evaluate the effectiveness of different interventions;
2. Data-driven collaborations with communities that are under-served by current internet infrastructure to develop and test different options for infrastructure investments, whose effectiveness can depend critically on the specific characteristics and needs found in different communities;
3. Better data and analysis about how Internet connectivity relates to the social and individual conditions that contribute to whether and how the Internet actually improves people’s lived experience.

Answering these questions requires developing entirely new approaches. The IEI is developing measurement techniques and datasets that address unknown questions about Internet performance, collaborating with communities that are under-served by Internet infrastructure to test different options for infrastructure investments, and evaluating how Internet connectivity actually improves people's lived experience in key areas such as education, health, and work. This project has been supported by data.org, the Crown Family School, and other funders. The Internet Equity Initiative is a research initiative of the UChicago Data Science Institute (DSI).

Understanding how research evidence is used in real-world contexts requires attention to the ways that policymakers, managers, and frontline workers also leverage other kinds of knowledge (e.g., lived experience, professional knowledge, cultural practices) as they carry out their work. This project investigates the process of knowledge integration and its impact on equity and responsiveness through a longitudinal comparative case study of the implementation of the Family First Prevention Services Act (FFPSA). This law promises to remake child welfare by allowing federal matching funds for prevention services but restricting the use of these funds to specific evidence-based practices (EBPs). We ask three questions: 1) How do EBP mandates influence the perceived legitimacy of and commitment to different types of knowledge by child welfare stakeholders? 2) How is research evidence, embedded in EBPs, weighed against and integrated with other forms of knowledge? 3) To what degree do variations in knowledge legitimacy and knowledge integration result in differences in knowledge equity? This project has been supported by the W.T. Grant Foundation and the Crown Family School.

Today’s world is experiencing an explosion of new data and analytic methods whose consequences for reshaping social life are only beginning to be understood. As new forms of data and analysis affect practice in fields as diverse as criminal justice, agriculture, social services, health, education, disaster management, and city governance, scholars and thought leaders are examining a broad set of processes composing the “data-society interface.” This term aims to call attention to the ways in which the availability of new data, more data, and new analytic methods are changing the social practices of individuals, groups, organizations, and societies. Scholarship about the data-society interface has emerged in earnest over the last decade, grounded in diverse disciplinary traditions and addressing many different substantive topics. This report, supported by the Robert Wood Johnson Foundation and co-authored by Cameron Day, provides the reader with an introduction to key discourses and findings across a range of approaches, particularly as these issues touch on the goal of using data to solve social problems.