Study of the Impact of
Statewide Systemic Initiatives

NAEP ANALYSIS

STATE ASSESSMENT ANALYSIS

NAEP STATE PROFILES

NAEP ITEM ANALYSIS

Overview The main goal of the proposed project is to study the outputs, outcomes, and impact of statewide systemic initiatives on student learning, curriculum, and policy. A series of studies is proposed, proceeding from investigating what are general relationships between student outcomes and state's participation in the National Science Foundation Statewide Systemic Initiatives (SSI) program to investigating very specific relationships between student achievement and an SSI as implemented by a state. Along with information on the relationship between SSIs and student achievement, information will be gleaned from a selected set of SSI states on what can be learned about large-scale systemic reform in mathematics and science.

Two key research questions will be addressed by the proposed study: What has been the impact of statewide systemic initiatives on student learning and other important variables? What lessons have been learned about designing, implementing, evaluating, and supporting statewide systemic reform?    Three studies will be conducted to answer the first research question. The first study will analyze National Assessment of Educational Progress (NAEP) data in mathematics at three grade levels for three years--1990, 1992, and 1996. The percent of students at or above proficiency will be determined for at least two groups of states, the 22 of the 26 SSI states with NAEP data and the 17 non-SSI states with NAEP data. This analysis will compare change in scores by other groupings based on the degree to which the state has engaged in efforts of statewide systemic reform. In addition to producing information on the relationship of SSI with student achievement, the first study will provide information that will be used in selecting states for the other studies. The second study will analyze existing student outcome data for five target states--three SSI states and two non-SSI states--over the time period of 1990 through 1999. Both NAEP data and state assessment data will be used. Change will be studied in both mathematics and science and for total score and scale scores. Multiple analyses will be conducted including raw gain scores, value added, and proficiency scale. The third study will analyze, in the same five states, the relationship among change in student outcomes, process indicators, and systemic reform attributes with the level that sites (districts/schools) within a state have engaged in a SSI. Existing assessment data will be used as outcome variables, with necessary linking studies to fill in gaps. Sampling techniques will be used to gather data on the process variables and systemic reform attributes. Multiple methods will be used to answer the second research question on what lessons can be learned about systemic reform. Existing information on systemic reform that has been produced by the National Institute for Science Education, Horizon Research, SRI, CPRE, SSI evaluators, and others will be mined and summarized through a series of small focus conferences of expert consultants. New research findings from the three studies described above will be used to refine this information or generate new interpretations. Case studies will be used, as appropriate, to gather historical information and update existing dababases on how systemic reform has been advanced in the target states. Generated theories and competing hypotheses about systemic reform in mathematics and science will then be tested with the five target and other states.