Introduction mechanisms that are employed, we describe each one in detail for each individual case. Overview of the Case Examples We examined aspects of quality associated with three mechanisms for In choosing these particular cases, we considered two factors. First, each case represents an established method for gathering information. There Introduction fore, it is possible to examine changes in quality over time and to identify factors that plausibly influence increases or decreases in quality. Second, the methods or procedures we examined represent three of the most common ways in which statistical information has been collected. Study Limitations and Three limitations to our report should be noted. First, the findings are based only on the information that was accessible in agency documents, which were limited. Second, the case examples do not represent all education information; they exclude, for example, information from evaluation and research. However, they include major initiatives that used or are using a good deal of the department's resources. In addition, they specifically represent particular kinds of data collection strategies and serve as examples of the kind of work that can be accomplished, given a particular set of conditions (see table 1.2). Third, given limited time and resources, we could not do our own full assessment of the technical quality of the data. Instead, we drew on the writings of reviewers of education information activities and on the help of outside experts. We supplemented these with our own judgments on such matters as sample selection and analytic techniques. Our report has strengths as well. First, we looked at three kinds of information-research, statistics, and evaluation-in three different organizations-NIE, NCES, and OPBE. It is rare that such an approach is taken; the focus is more typically on one kind of information (such as statistics) or one type of information activity (such as a particular data set). Our approach allowed us a more complete look at the status of federally sponsored education information. Second, our longitudinal approach allowed us to look at changes in information activities over time and to track factors associated with periods of high and low activity. Third, while case studies did not give us a representative sample of information activities, they did allow us to look in-depth at some important statistical series. We were able to look at the origins of these series, the request strategy, implementation factors associated with particular negative or positive outcomes, the different components of quality, and critiques of the work. Chapter 2 The Production of Information Determining what information is produced and how it has changed over We also found changes in priorities. For research, there was a shift away Awards for We found notable decreases in the number of awards across all three units. NCES reduced many of its activities. In NIE and OPBE, awards for activities stood at less than one third and one quarter of their 1980 levels, respectively. Research We examined the complete set of research activities in NIE from 1980 to 1985 only-all years for which adequate data were available. The activities are those listed in NCER'S annual reports. In this discussion, we separate awards made to the mandated regional laboratories and national centers from all other grants and contracts (including those made in the three program areas, in the unsolicited proposal program, and in other units such as the office of the director). The number of awards from 1980 to 1985, as shown in table 2.1, The Production of Information Table 2.1: Number of National Institute of Education Awards for Fiscal Years 198085 by Program Funding Area Overall, from 1980 to 1985, the number of awards for information activities declined 65 percent. Except for "other" awards (primarily those from the office of the director and the regional laboratories and national centers), all the program areas showed a marked decline in the number of awards from 1980 to 1985. The change from 1980 to 1985 was the most dramatic for the unsolicited proposal program because it was cut completely. The 84-percent decrease for educational policy and organization was nearly as deep. Teaching and learning and the dissemination and improvement of practice declined 54 and 70 percent, respectively. Area Teaching and learninga *Includes centers that were not part of the regional laboratory and center network. These were the bCovers miscellaneous awards not identified under any of the program funding areas listed, such as Source: National Council on Educational Research annual reports for fiscal years 1980 to 1985. Statistics To obtain an overview of basic statistical data-gathering within NCES, we relied on various reports describing NCES programs and plans.' In this discussion, we focus on the portfolio of information-gathering activities in the four fundamental education domains-elementary and secondary education, higher education, adult and vocational education, and library resources and specialized surveys (the National Longitudinal Survey and Fast Response Survey System). We consider other related activities in support of these efforts (such as quality control) separately. As seen in table 2.2, the total number of surveys NCES planned and conducted grew by 49 percent (from 37 surveys to 55) between 1974 and 1National Center for Education Statistics, The Condition of Education, Part 2, Programs and Plans The Production of Information Table 2.2: Number of National Center for 1980. New surveys in adult and vocational education and in library resources accounted for most of this increase. The amount of data collection (as measured by the number of ongoing and planned surveys) in elementary and secondary education and higher education remained roughly constant. Between 1980 and 1983, the last year for which data permitting this analysis were available, data collection activity declined by 31 percent, returning to its 1974 aggregate level. In three of the four major education areas, the number of planned and ongoing surveys declined by 31 to 42 percent in this same period. Excludes the National Assessment of Educational Progress. (NAEP was an activity of NCES in 1974 "Time did not permit our verifying information the department provided in response to a draft of this Source: Department of Education data for 1984 and 1985; National Center for Education Statistics, We also examined the frequency of data collection—that is, periodicity, |