Influences on Production and Quality, Agency The number of staff at NCES fell 20 percent from 1980 to 1986: from 123 (0 excepted service, 123 civil service) in 1980 to 98 (2 excepted service, 96 civil service) in 1986. Turnover has been significant since 1980; only 46 of the 123 staff in 1980 still remained at NCES in 1986. These 46 make up a little less than half the current number. An additional 19 of the 1980 staff are now in positions elsewhere in the department. For the Office of Planning, Budget, and Evaluation, our data are taken from our survey of federal program evaluation activities for 1980 and 1984 and are limited to the evaluation component. Within this unit, staffing declined by 16 percent from 1980 to 1984, from 32 to 27, and according to the staff, remained at about the 1984 level through 1986. There were no excepted-service employees until recently, when 3 were hired. Thus, there were many changes in upper management but there was little staff turnover at the lower levels, providing some stability to the unit. In summary, we found overall declines in the number of professionals available to carry out the information-producing functions. Further, although the Office of Planning, Budget, and Evaluation was reportedly stable, turnover within the other units suggested less consistency in carrying out information activities and potential problems with staff knowledge of department operations and legal procedures. The consequences of management changes can be found in the operations and priorities of the information-producing units. For example, one priority for identified research at NIE in 1978 was complex learning skills. To develop this new area, NIE commissioned papers to identify key questions for further research, and the papers were reviewed in 1980 in a conference bringing together researchers and practitioners. In 1981, when the first grant competition was held, more than 90 proposals were received and reviewed by panels of experts, which recommended 30 for funding. In 1981, however, the turnover of directors resulted in a hold on funding. The new director did not regard this area as a priority. No awards were made after the grant competition. In other words, the cycle for research funding from the initiation of a priority through the awards process to reporting the results may take many years, but the tenure of the director is typically less than a year-long enough to stop what was started but not long enough to see initiatives to completion, except where they are protected by congressional mandates, such as in the laboratories and centers and NAEP competitions. 10 10Some awards for research on complex learning were eventually made under later NIE directors. Influences on Production and Quality, Agency Paperwork Reduction Congressional actions other than mandates for certain types of information have influenced the production of information, as when FEDAC was established in the 1978 Education Amendments to eliminate excessive detail and unnecessary information requests. In 1979, FEDAC began its reviews of education data collection in the Department of Health, Education, and Welfare. Within FEDAC's first 9 months of operation, the data burden was reportedly reduced by almost 13 percent (a reduction of about one million burden hours). In 1980, the Congress passed the Paperwork Reduction Act (Public Law 96-511), whose key objective was to ensure that information requested by federal agencies was needed by an agency, unavailable elsewhere, and efficiently collected. The act appears to have had a substantial effect on the volume of paperwork required for federal operations. By fiscal year 1984, the department reported reductions of 35 percent from the 1980 base in reporting requirements and paperwork. As we illustrated in previous examples, strict interpretation of More direct reductions in information-gathering have been imposed in specific program legislation. For example, reporting and evaluation requirements for the major federally funded compensatory education program were made inapplicable with the passage of the Education Consolidation and Improvement Act of 1981 (Public Law 97-35). This act amended previous legislation in an effort to "eliminate burdensome, unnecessary, and unproductive paperwork and free the schools of unnecessary Federal supervision, direction, and control" (section 552). Further, the Deficit Reduction Act of 1984 (Public Law 98-369) set targets for savings in federal government operations. Areas in the act relevant to information-gathering include staff travel, the use of consultants, and publications. Agency Comments and The Department of Education agreed in general with the findings cited Our Response in a draft of this report. It believed our report will perform a valuable function by documenting a long-term decline in resources for research, statistics, and evaluation. (Its letter is reproduced in appendix IV.) However, the department raised three main concerns. First, while it Influences on Production and Quality, Agency acknowledged the validity of many of the points raised for the period covered by the report, it believed that in recent years the department has "taken clear and decisive action to address most of the problems cited in the report." In particular, it noted that since 1985 each of the three information-producing units has been reorganized or its lines of authority and responsibilities have been altered. According to the department, the report does not accurately reflect the current situation. Second, the department disagreed with our analyses of shifts in priorities. Specifically, it argued that increases in the proportion of funds allocated for dissemination are a positive step toward improving the utility of research, not a threat to new data collection, as we claimed; it asserted that there has been a significant degree of consistency and continuity in research priorities and that areas of study that we claimed were not being funded are currently supported under awards to laboratories and centers, "minicenters," an "urban superintendents' network," an intramural research project, and research grants. Third, the department disagreed with our assessment of the implications of shifts in who is producing educational information and how its production is funded. The department agreed that funding patterns have limited its flexibility for determining who is funded but pointed out that 10 new awards were issued as part of the fiscal year 1986 field-initiated grants competition and that since fiscal year 1986 seven OERI fellowships have been filled by scholars, researchers, and practitioners. The department also argued that its procurement methods are an attempt to foster, rather than constrain, creativity and "invite alternative strategies and fresh ideas." In its comments, the department also presented an extended discussion of the current situation, provided additional budget figures, and offered further documentation on the number and types of activities for years not covered in the draft report. With respect to the department that our findings do not represent the department's current situation, we acknowledge that changes have been initiated since 1985 in the structure and operation of information-producing units. With regard to OPBE and CES, we explicitly mentioned many of the topics raised in the department's comments. Several other actions the department referred to were initiated as recently as March 1987 and could not have been incorporated into our assessment, since our data collection extended up to fiscal year 1986. Since we were unable to verify all new data that the department provided in its letters, we have not Influences on Production and Quality, Agency altered our text or observations, except where it has been feasible to note updated information. A substantial proportion of the department's letter is devoted to describing current and planned activities. It is too early to determine whether the organizational and procedural changes that have been initiated will adequately address the problems we identified in this report or new ones that the changes might create. Our analyses were based on the most current information available at the time of our study, and, for the most part, new data still are not available to assess the effects of the actions the department reported. None of the problems we reported occurred overnight, and many of them have been longstanding. Thus, while it is useful to have the additional information on the department's recent efforts, sustained attention will be needed to improve the status of education information. Our report could serve as a partial baseline against which to assess the effects of departmental initiatives to improve the quality, relevance, and timeliness of education information. In reviewing the department's comments, however, we found no mention of any plans to assess progress empirically. It is too soon now to measure the effect of recent department efforts, but making formal plans for evaluation would be a most timely endeavor. The department's second concern involved several points about our analyses of shifts in priorities. The department seems to have misconstrued our central point about the shift in emphasis toward service-oriented activities—notably dissemination—at the expense of new data collection. We did not, as the department contends, distinguish dissemination from research. Rather, we distinguish service-oriented activities like dissemination from new data collection. Further, while we agree that dissemination is a fundamental part of the research process, in light of the dramatic reductions in fiscal resources, more for dissemination means that there can only be less for new data collection. Dissemination can usefully remain a critical part of the research process only if the data that are being disseminated are relevant and timely. If resources were to decline further, there would be less new information to disseminate. Herein lies the threat that increased emphasis on dissemination poses for new data collection. The department made two additional points about changes in priorities. It took issue with our example relating changes in leadership to changes in priorities. We stated in the report that the consequences can be felt in both the operation of and priorities for information-producing units. Our Influences on Production and Quality, Agency data show that the instance we cited was not unique. Changes in research priorities associated with shifts in top leadership are acknowledged in the seventh annual National Council on Educational Research report to the President and the Congress. The chairman of NCER stated that since 1981 NIE's leadership has attempted to move away from the efforts of its predecessors and, by 1983, changes in research priorities were perceptible from a review of new grants. Further, the department argued that funds were not available to permit significant changes in priorities, since the Congress restricted an increasing percentage of the NIE budget for institutional awards. We disagree. Our data in chapter 2 show that the resources that remained after mandated activities were funded were spent in very different areas in 1980 and 1985, our years of focus (see table 2.9). The department also objected to our analysis of changes in areas of investigation, citing activities directed at gathering information on "many of the most important questions and issues in education today" (emphasis added). Forming centers that address education reform issues is an important step in addressing these contemporary issues for educators and policymakers. However, this step does not make up for the years when these and other topics were emerging as critical issues but no work was being done. Some way for keeping in step with the areas that are emerging and currently important is needed in education. Planning efforts should include review by experts to reduce gaps between the information that exists and the issues being dealt with by educators and policymakers. In other words, the department needs to ensure that efforts will be made to identify new and emerging issues and to implement information-gathering on these issues. The department's third concern was with our assessment of the consequences of changes in who is producing information and how producing it is funded. Our analyses show an increase in the use of contracts and a decline in the number of awards issued to individual researchers. With regard to procurement practices, we acknowledge that NIE used many funding mechanisms to accomplish its work, including grants, contracts, purchase orders, and interagency agreements. However, the data we reported in chapter 2 indicate that the majority of the later work was funded by contracts. The department asserted that contracts are used when it wants to ensure that the substance of the work is clearly articulated and to provide for an appropriate level of accountability. However, we think that other consequences go along with using contracts, such as the type of review described. Furthermore, while it is admirable |