The Quality of Information addition to age level) make it possible to link NAEP results more closely to state and local assessments (and school practices) that are typically associated with grade levels. Although earlier assessments employed matrix sampling of assessment areas, the new NAEP design employs a variation of matrix sampling called "balanced incomplete block spiraling," which allows for assessments of relationships between content areas and background variables yet keeps assessment time per student to a minimum (thereby reducing the respondents' burden). This alteration in test administration, in combination with advances in measurement (for example, using item response theory), has improved the ability to compare and interpret the findings (through scaling procedures) over time and between groups. Although NAEP has generally been viewed in a positive light, department officials have expressed concern over the appropriateness of some of the interpretations of the data resulting from special analyses.3 In terms of some indicators of NAEP quality, school response rates for 1984 reading assessments uniformly high (ranging from 84 to 90 percent). The sample sizes have been enlarged for each age group, increasing NAEP's overall precision (although the assessment cycle was changed from 1 to 2 years). With regard to sampling variability, technical changes meant that the most recent reading assessment had to report estimated standard errors and provide extensive caveats regarding their level of accuracy. Nonsampling errors are reduced because the data collection process is standardized through the use of hired and trained staff. All information is collected by field staff, who maintain comparability. Further, unlike previous assessments that employed a paced audiotape to provide directions to the students while they took the test, in the most recent assessment (reading and writing) the students were given oral instructions. To assess the influence of this change in testing, special data collection and analysis documented differences attributable to the changes in testing procedures. What is commendable about this research is that it systematically examined the comparability of important changes in procedures. It also points up the importance of preserving some of the basic methodology of past assessments in order to maintain the usefulness of data on trends. 3In a 1986 letter to the NAEP contractor, the assistant secretary for OERI criticized the contractor's interpretation of NAEP data on bilingual education. The Quality of Information Impact The fourth indicator of quality refers to the use of the information that is gathered. In 1976, we concluded that NAEP's results had to be made more useful to state and local policymakers. We based our conclusion on responses to a survey in which 71 percent of the local education agency officials indicated that they had not used NAEP concepts, methods, or material. Further, while 90 percent of the state education agency officials said they had reviewed NAEP information, 44 percent of those who responded rated the concepts, methods, and material moderate to limited in their utility. In commenting on our report, officials within the department acknowledged the need to improve the usefulness of NAEP and provided a plan for implementing our recommendation. Sebring and Boruch conducted a more recent review of the uses of NAEP, finding numerous instances of state and local use. They reported that 12 states had replicated NAEP completely as part of their own statewide assessments and that 14 states had adapted the NAEP model. Seventy percent of Minnesota's districts participated in the "piggyback program,” in which local districts work under contract with the state to conduct local assessments. Similar practices were found in Connecticut. However, Sebring and Boruch noted that not all the uses to which NAEP or NAEP-like studies have been applied meet acceptable research practices. Overall, in ranking the usefulness of NAEP to different audiences, Sebring Summary and Conclusions Although NAEP has provided data on the nation's children since 1969, recent changes have helped enhance NAEP's role in understanding the condition of education in the United States. Our review of the available evidence suggests that as NAEP is currently structured, it ranks relatively high on all four indicators of quality (technical adequacy, timeliness, relevance, and impact). NAEP continues its tradition of providing a 4P. A. Sebring and R. F. Boruch, “How Is the National Assessment of Educational Progress Used?” Educational Measurement: Issues and Practice, Spring 1983, pp. 16-20. The Quality of Information national perspective on the condition of education and has stimulated better state and local assessments. NAEP has not escaped constraints associated with the shrinking fiscal support for educational information. It has been affected by budgetary declines in two ways. The number of target populations was reduced from five to three principal groups, and the assessment cycle was altered from annual to biennial, or longer, some content areas being assessed at 4- and 6-year intervals. Determining the optimum interval between test administrations goes beyond the issue of fiscal resources. The ability of NAEP to record changes in performance depends on maintaining short intervals between assessments. As an interval increases, the ability to signal changes becomes more limited. Further, NAEP does not assess many groups, including students younger than 9 years old. Expanding the substantive scope or the target populations will inevitably require more funding, further restrictions in the coverage of principal populations, or greater reliance on "other" funds (see table III.1). Funding reductions appear to be associated with some benefits to the overall quality of NAEP. For example, budget restrictions seem to have resulted in the use of more-efficient sampling designs and testing procedures. Altering the testing cycle from 1 to 2 years has allowed an increase in sample size and, therefore, greater precision. The trade-off here is clear-although successful adaptations to fiscal constraints seem to have resulted in increased technical adequacy, relevance and timeliness have declined somewhat. The Common Core of Purpose and Background Within NCES, data on elementary and secondary education are gathered from several distinct types of survey (NAEP among them) and administrative records. This case illustrates the quality of the Common Core of Data, which is based mostly on administrative records maintained by state education agencies. Several types of fiscal and nonfiscal data are obtained from state administrative records. CCD was initiated in 1974 CCD generally collects information annually or on some other schedule. The version of CCD during our review consisted of six surveys (see table III.4). Information on instructional staff, students, expenditures, revenues, and school characteristics are gathered from schools and local education agencies by a state education agency and transmitted to NCES. Planning for the system began with four grants to develop the requirements of state and local users of the Common Core of Data. The system was field-tested in 1976 and implementation began in the 1977-78 school year. It was intended to replace ELSEGIS. In its original form, CCD contained more program elements than we give in table III.4. In fact, our third case illustration-the Fast Response Survey System-was originally developed as part of the early CCD. Other surveys have been eliminated or scaled back (as discussed in chapter 2) in concert with a special task force of the Committee on Evaluation and Information Systems (CEIS). Beginning with an education statistics advisory council report in 1957, concerns have been raised about the quality of administrative data from intermediate sources. In 1985, the department undertook its own internal review of the elementary and secondary education data system, including CCD. Department officials asked the committee on national statistics of the National Academy of Sciences (NAS) to conduct an independent assessment of the center. Both internal and external reviewers relied heavily on over 50 letters, comments, and other papers that NCES officials solicited from professional organizations and users and producers of NCES data. NCES staff and outside consultants also synthesized the comments from these reviewers and commissioned analyses on how the system should be reconfigured to take into account its problems and deficiencies. NCES staff have issued a draft report on how to alter the system to improve its overall quality. 5 National Center for Education Statistics, Projects, Products, and Services (Washington, D.C.: U.S. Government Printing Office, 1976, p. 102). 6The NAS study, issued in September 1986, was quite broad in its charge and included four major aspects: to review, describe, and assess data quality and quality-assurance processes, program contents and services, and the timeliness of data collection and dissemination and to identify issues that obstruct or hinder NCES in accomplishing its mission. Only part of the study is described here. The Quality of Information The series of reports generated by these reviews substantially agree on the general problems of the technical adequacy and usefulness of CCD and other surveys in the current system. Many experts regarded data stemming from the elementary and secondary education program as deficient on one or more of our quality indicators. Evidence for their conclusions does not focus exclusively on CCD, however. Closer inspection of each quality dimension points out CCD's strengths and weaknesses. Relevance Several assessments raise questions about the relevance of NCES data collection efforts for elementary and secondary education. One review emphasized the lack of comprehensiveness of the entire elementary and secondary education information system. The reviewers noted that CCD was inadequate for answering questions on the relationships between student background, processes, and outcomes but that it did contain some relevant information on resources. In contrast, a similar, though not strictly comparable, analysis in a 1976 NAS report suggested that ELSEGIS, the forerunner to CCD, had been more responsive to the assessment of inputs, processes, and outcomes, suggesting that CCD had deteriorated. A 1986 internal department review of the comprehensiveness and availability of data across various programs, including CCD, revealed considerable gaps in the present statistical system. The chief source of dissatisfaction stemmed from the system's inability to answer nine fundamental questions about educational input, participation, process, content, cost, and outcomes. Looking across levels of education, the assessment showed that data were almost nonexistent for preschool children and completely absent for persons no longer in school. The only area for which CCD provided adequate data concerned the providers of educational services. One potential strength of CCD was that data were obtained from all state 7V. Plisko, A. Ginsburg, and S. Chaikind, "Assessing National Data on Education," U.S. Department of Education, Washington, D.C., August 1985. |