Welcome to The Community Guide’s new website. Comments and suggestions on the site’s look and feel are welcome: communityguide@cdc.gov.

Health Equity: Out-of-School-Time Academic Programs – General

Tabs

What the Task Force Found

About The Systematic Review

This Task Force finding is based on evidence from a meta-analysis published in 2006 (Lauer et al., 35 studies, search period 1985–2003) combined with more recent evidence (25 studies, search period 2003-2011). Of the 35 studies included in the Lauer et al. meta-analysis, this review excluded three studies which reported only school grades, a relatively subjective measure of reading and math achievement. The Task Force used evidence from an independent systematic review of summer school programs to confirm Community Guide review findings (Cooper et al., 2000).

The effectiveness evidence is based on a systematic review of all available studies, conducted on behalf of the Task Force by a team of specialists in systematic review methods, and in research, practice, and policy related to the use of educational interventions for the promotion of health equity.

Context

Overall, children from low-income and racial and ethnic minority populations in the United States have lower academic achievement levels than children from the higher-income and majority populations. As a long-term consequence, these children often grow to be adults with lower income levels and poorer health, perpetuating a "cycle of poverty" (Duncan et al., 1998). Out-of-school-time academic programs aim to interrupt this cycle by assisting children who are at risk for low academic achievement.

Summary of Results

  • Out-of-school-time academic programs led to modest improvements in academic achievement, as measured by standardized achievement tests. The degree of effectiveness was largely dependent on program focus.
    • General academic programs:
      • Reading achievement: 0.09 standard deviations (SD) (interquartile interval [IQI]: 0.00 to 0.26; 21 studies)
      • Math achievement: 0.06 SD (IQI: ‑0.01 to 0.24; 20 studies)
  • Summer programs were more effective than after-school programs, especially for general academic programs.

Summary of Economic Evidence

Fourteen studies were included in the economic review of out-of-school-time academic programs (reading-focused, math-focused, general, minimal academic content). Only program costs were reported. Monetary values are 2012 U.S. dollars.

  • Annual program costs ranged from $623 to $8705 per student and varied greatly by hours of operation (14 studies).
  • Hourly costs per student ranged from $3.06 to $13.17 (11 studies).
  • Major cost drivers included salaries for teachers and staff, costs for facilities and utilities, and transportation costs, with salaries being the largest expense.

Applicability

Based on the settings and populations from included studies used in the systematic reviews of out-of-school-time academic programs (reading-focused, math-focused, general, minimal academic content), results are applicable to:

  • Urban K-12 schools in the United States
  • Low-income populations
  • Summer and after-school programs

Evidence Gaps

Each Community Preventive Services Task Force (Task Force) review identifies critical evidence gaps—areas where information is lacking. Evidence gaps can exist whether or not a recommendation is made. In cases when the Task Force finds insufficient evidence to determine whether an intervention strategy works, evidence gaps encourage researchers and program evaluators to conduct more effectiveness studies. When the Task Force recommends an intervention, evidence gaps highlight missing information that would help users determine if the intervention could meet their particular needs. For example, evidence may be needed to determine where the intervention will work, with which populations, how much it will cost to implement, whether it will provide adequate return on investment, or how users should structure or deliver the intervention to ensure effectiveness. Finally, evidence may be missing for outcomes different from those on which the Task Force recommendation is based.

Identified Evidence Gaps

The following outlines evidence gaps for reviews of these out-of-school-time academic programs: reading-focused, math-focused, general, minimal academic content.

Effectiveness Review

  • More high-quality research is needed to identify which features of out-of-school-time academic programs contribute to their effectiveness (Beckett et al., 2009).
  • More evidence is needed to determine variability in effectiveness of interventions based on:
    • Student characteristics (e.g., race or ethnicity, socioeconomic status)
    • Program focus
    • Degree of student participation in combination with program duration (short programs with full participation or long programs with minimal participation are likely to be less effective)
  • In order to better understand program effects, more details are needed about activities in the intervention and control groups.
  • Research is needed to assess the long-term effects of OSTA programs.
  • Additional research is needed on ways to improve implementation of, and participation in, out-of-school-time academic programs.
  • Transportation may be a challenge for children in low-income families, and should be investigated as a barrier to participation.
  • • A re-evaluation of the 21st Century Community Learning Centers program is needed, now that it has an increased academic focus (James-Burdumy et al., 2007).

Economic Evidence

  • A standard methodology should be used to collect information about the actual costs of out-of-school-time academic programs.
  • More in-depth studies are needed to understand cost variations of different program types used in different locations.
  • A complete and accurate economic assessment of the cost-benefit of out-of-school-time academic programs would better inform decision-makers.

References

Beckett M, Borman G, Capizzano J, Parsley D, Ross S, Schirm A, Taylor J. Structuring out-of-school time to improve academic achievement: A practice guide (NCEE #2009-012). Washington(DC): National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education; 2009. Available from URL: http://ies.ed.gov/ncee/wwc/publications/practiceguides

James-Burdumy S, Dynarski M, Deke J. When elementary schools stay open late: results from the national evaluation of the 21st Century Community Learning Centers Program. Educational Evaluation and Policy Analysis 2007;29(4):296–318.

Study Characteristics

The following characteristics describe studies included in systematic reviews of these out-of-school-time academic programs: reading-focused, math-focused, general, minimal academic content.

  • All included studies were conducted in the U.S.
  • 37 (65%) of evaluated programs were implemented in urban areas
  • 28 (49%) of programs were for elementary school students (K-12), 8 (14%) for both elementary and middle school students, 7 (12%) for middle school students, 3 (5%) for middle and high school students, 7 (12%) for high school students and 4 (7%) for all students.
  • The studies evaluated similar numbers of summer and after-school programs (29 and 28, respectively).
  • Study populations were predominantly racial/ethnic minorities (i.e., Black and Hispanic). Specifically, 25 (43%) of studies evaluated majority Black students, 4 (7%) majority Hispanic, 2 (4%) majority non-White (unspecified), 7 (12%) majority White, 4 (7%) mixed, and 15 (26%) did not report race/ethnicity.
  • 42 (74%) of studies evaluated majority low SES populations.
  • Studies included similar numbers of male and female students.
  • 17 (30%) of programs employed tutoring or individualized instruction as the instructional method, 24 (42%) employed group instruction, 10 (18%) used both tutoring and group instruction, and 6 (11%) did not report didactic method.
  • 23 (40%) of programs were reading-focused, 7 (12%) were math-focused, 23 (40%) were general academic and 4 (7%) were minimal academic

Publications

Knopf JA, Hahn RA, Proia KK, et al. Out-of-school-time academic programs to improve school achievement: a Community Guide health equity review. Journal of Public Health Management & Practice. 2015;21(6):594–608.

Community Preventive Services Task Force. Out-of-school-time academic programs are recommended to improve academic achievement and health equity. Journal of Public Health Management & Practice. 2015;21(6):609–12.