The following post was written to enhance the discussion that took place during the April 30 webinar, which explored the topic of using evaluation to scale for impact. “Evaluation: What an Organization Needs to Scale for Impact” is now available online.
With apologies to the Beatles, developing evidence-based programs that can be taken to scale does not happen in one hard day’s night. Rather, it truly is a long and winding road. At least for this organization.
OASIS, a national nonprofit organization and member of the S&I 100, is dedicated to promoting successful aging for adults age 50 plus. Our three-fold approach provides opportunities for people to participate in lifelong learning, active lifestyles and community involvement. Through its programs, OASIS puts into practice the findings of the landmark MacArthur Foundation study of Aging in America. Researchers Rowe and Kahn found that the key ingredients for a high quality of life are maintaining a low risk for disease, a high level of engagement with the community and high physical and cognitive function; “It is the combination of all three that represents the concept of successful aging fully.”
Defining Impact
To achieve maximum social impact, our board and executive leadership made a commitment in 2003 to develop and offer evidence-based programs that included the ingredients for successful aging. To that extent, we made the decision to take to scale only those programs that were evidence-based. In discussing our criteria for evidence-based programs, we chose this definition: A program must have research that demonstrates its efficacy (randomized controlled trial, experimental or quasi-experimental, longitudinal); it must have completed translation in a community site; published results in a peer-reviewed journal; and must be available to the public.
We are now able to offer several programs that meet this definition. I believe that our commitment to the efficacy and positive impact of our programs was not only instrumental in becoming part of the S&I 100 Index, but it was also crucial to our ability to scale, which I discussed during the Social Impact Exchange’s webinar last week. (For a more in-depth discussion on “Evaluation: What an Organization Needs to Scale for Impact,” you can view the webinar live here).
As a follow-up to the webinar, I wanted to focus on one recent challenge Oasis has had in scaling our Intergenerational Tutoring program. Unlike some of our other evidence-based programs such as CATCH Healthy Habits, which was developed and scaled years following one of the largest school-based controlled clinical trails for health, the lack of rigorous research surrounding the impact of our tutoring program presented a roadblock for its growth. The tutoring program has actually been active for 24 years in school districts throughout the country and it is well-loved, but it has not yet shown the results for children that would pass the definition for evidence-based.
OASIS Tutoring is an in-school literacy program that pairs trained older adult volunteer tutors with children in grades K-4 who have been identified by their teachers as academically at-risk because they are reading below grade level. The volunteers work one-on-one with students each week in the schools. Children are selected by their teachers and, with parental consent, are paired with OASIS tutors. OASIS Tutoring is viewed as a valued resource by school districts because its activities align with U.S. Common Core State Standards. Through their service, the OASIS Tutors are meeting a demonstrated need because significant numbers of third-grade children throughout the country are not reading at grade level. As states face continuing budget cuts, schools are asked to support the needs of students with fewer personnel. As a result, children do not receive the one-on-one attention they need during the formative years.
Pinpointing Results
When the tutoring program started, we asked principals, teachers, OASIS tutors and other school personnel to complete an annual survey. Results are consistently positive. For 2011-2012, results were compiled from the 3,322 responses and revealed that 100% of principals felt that OASIS tutors added a positive element to their schools; 96% of teachers reported improved confidence and self-esteem in their students who were tutored; and 90% reported improved academic performance among those students who were tutored.
As we began seeking funding to scale the program from corporate, government and private sources, it became apparent that our survey feedback, while encouraging and informative about the level of our partners’ satisfaction with the program, just wasn’t enough.
Over the past several years, OASIS applied for but did not receive federal funding through the U.S. Department of Education and the Justice Department to complete the needed research. As we continued to seek funding, we carried out a series of small research studies to better understand the program’s impact on students. Each one demonstrated value offered by the program from avoiding reading regression during the summer, to improvements in areas such as initial sound fluency, letter naming fluency, and oral reading fluency. While these results were encouraging, none of the studies used a control group or other key characteristics of an experimental or quasi-experimental design, and none demonstrated definitively that the program has a positive impact on children’s reading scores. This, cast against the reality of funders demanding greater emphasis on measurable outcomes combined with a growing need by schools for additional help, led us to where we are today.
Advice on Using Evaluation to Scale
This spring OASIS entered into a formal partnership with Maryville University in St. Louis to conduct a research study. In fact, partnering with an academic research department at a local university was one of the suggestions given to nonprofits during the webinar because academic institutions are often willing to do research work pro-bono. The third-party study of OASIS Tutoring will account for 375 students (125 tutored in once-a-week format; 125 tutored using the high-intensity format of multiple times per week; and 125 in a control group). The groups will be comparable based on demographics and academic achievement. Standardized tests will be administered at the beginning of the school year with post-test assessment of reading level at the end of the school year. The study will be designed to determine improvement in student literacy skills after participation in OASIS Tutoring.
The evaluation team will be led by Sam Hausfather, PhD, Dean of the School of Education at MaryvilleUniversity. OASIS will draw upon its network of 33 St. Louis area school districts that participate in the program to select schools for the two-year study.
The cost of the three-year study is approximately $600,000. OASIS has worked with private and corporate foundations in the St. Louis region for years. The leaders at several foundations understand the program through its long history in St. Louis and are willing to invest in developing the research. The plan is to complete the fundraising by the end of 2014.
Organizations seeking to establish an evidence-based program must be prepared to invest significant time and money to conduct strong research. The other ingredients are a high level of determination and commitment that the program has significant impact and the need is critical. In our case two generations are impacted positively. With the completion of the research, OASIS will have what it needs to take the program to scale.
Marcia Kerz is the president and CEO of Oasis Institute. If you would like to connect with Marcia, contact her at mkerz@oasisnet.org.