Influences on Production and Quality, Agency Expert Reviews of the high-quality information will be available when it is needed. For example, NVEDS was mandated in the Education Amendments of 1976. In 1983, OMB disapproved the department's request for data collection for the school year 1983-84, on the grounds of severe technical problems with the data. Here, the system was mandated with little consultation with the department, no resources were provided, and the time for reporting information back to the Congress proved to be unrealistic. The mission statements of the three units we examined are general, providing considerable latitude for the development of a portfolio of inforInformation-Gathering mation. To ensure that these functions are being properly carried out, Function different review methods have been tried out, varying the authority given to the reviewers (policymaking versus advisory) and the regularity of review (periodic versus ad hoc). For example, until recently, NIE was guided by NCER, a policymaking group that was given broad authority and met regularly. In addition to assessing the NIE portfolio of activities and reporting to the Congress, it served as a policymaking body, setting priorities on dissemination, for example, and on equity-related activities. From the reports that were issued, it is evident that NCER's review was "National Council on Educational Research, "Sixth Report of the National Council on Educational Research," Washington, D.C., fiscal year 1980. 8 National Council on Educational Research, "Research in Retrospect," seventh annual report, Washington, D.C., 1982. Influences on Production and Quality, Agency A variety of advisory groups have reviewed the statistical function within the department since the mid-1950's. In carrying out their advisory functions, these groups submitted reports that contained recommendations on ways of improving statistical activities. In fact, many of the problems the National Academy of Science's committee on national statistics identified in its 1986 report on NCES bear a striking resemblance to problems in a 1957 report issued by an advisory group. Given the persistence of the problems, the advisory panels do not appear to have been very influential in improving the quality of the data-gathering activities, despite the fact that early councils were composed of highly regarded individuals. NAEP has been reviewed through a complex process that was built into legislation. Since its inception, NAEP has been guided by a panel of experts who have meet regularly and have decisionmaking powers. In addition, periodic reviews by experts (at least once every 3 years) were mandated. Available evidence suggests these reviews have had several positive influences on the contents, operation, and management of NAEP. In particular, one aspect of the mandated review was a stipulation that the users of NAEP be given an opportunity to comment on its relevance and utility. Although many forces were at work in shaping NAEP, the department summarized several comments from the field in its report to the Congress that resulted in alterations in NAEP that have been regarded as improvements in relevance and technical adequacy. Similar improvements by the regional laboratories and national centers have resulted from reviews by one-time panels with advisory powers. For example, in the Education Amendments of 1976, the Congress established a 15-member panel of educators to review and report on the laboratories and centers. The panel members were appointed by the director of NIE from a field of 450 candidates nominated by organizations and associations in the education community. The panel reviewed each laboratory and center in terms of the potential value of the work that was proposed and its national significance, management, and track record. The Congress charged the panel with making recommendations for improving and continuing individual laboratories and centers. Relative to other reviews, the scholarly manner with which the panel executed its charge is readily seen in its systematic method. Each laboratory and center was examined on common criteria, and projects were judged on their relevance to the mission of the center or laboratory and on their technical merits. The assessment was balanced, in the sense Influences on Production and Quality, Agency Changes in Leadership and Staffing that the strengths and weaknesses of the proposed projects were identified. The panel members articulated rationales for the importance of given focal areas, their contribution to education practice, and expected accomplishments. A subsequent review initiated by NCER did not assess the level of technical adequacy in the laboratory and centers products but showed that, collectively, they have contributed information in many of the same areas identified by the national reform studies, suggesting that the laboratories and centers were dealing with issues of national importance. NAEP, national centers, and regional laboratories are major activities in the department. They consume a large share of the budgets allocated to NCES and NIE. Extensive review activities are therefore justified and appropriate. For activities such as FRSS, a different level of review may be warranted. With the exception of incomplete reporting practices, we judged the overall quality of FRSS as moderate to high, suggesting that elaborate review and external review are not always necessary. FRSS received little formal review besides the routine review of data quality by project monitors and through the recent competition of the contract. This seemed entirely satisfactory, given the level of funds allocated to this activity. While expert reviews have positively influenced quality in general, there are limits to what can be expected, and quality-control activities may have to compete for funds. In studies of the information-producing units, different directors and changes in senior staff were reported to have notably influenced the priorities and operations of the units. We were concerned in this report that changes in leadership in the education information units could similarly influence data production and quality. Therefore, we examined the changes and, where possible, attempted to identify their influences on education information. Each of the information units changed in top management positions dur- aRepresents a best estimate; one of the five, the associate director for field-initiated and internal stud- bOne position, assistant administrator for research and analysis, went unfilled from November 1980 to Source: Department of Education, Office of Personnel and executive offices for the Office of Education NIE had a total of 7 different directors from 1980 to 1986, 3 of the 7 serving as acting director. At least 16 persons served in the 5 other top management positions, 1 of which was created in 1984. In six cases, individuals served on an acting basis or as special assistants. The position of administrator for NCES was much more stable than the position of director of NIE. From 1980 to 1986, NCES had 2 top administrators and an acting director serving for 2 months in the transition between them. The turnover in the other statistical management positions, however, was similar to what occurred in NIE. There were 5 top management positions at NCES, 17 persons serving in them from 1980 to 1986. Two individuals served in an acting capacity. The Office of Planning, Budget, and Evaluation is headed by a deputy undersecretary. One of the 5 persons in this position from 1980 to 1986 served in an acting role. Seventeen individuals served in the 9 top management positions during this period. In summary, there have been many shifts in leadership in informationproducing units since 1980. NCES has been the most stable, but all three units have had multiple changes in the top position. All three have also had multiple turnover in other top management posts, some managers leaving in a matter of a few months. Other positions have been vacant for various periods. Staffing levels from roughly 1980 to 1986 are presented in table 4.3. 6 2 C Influences on Production and Quality, Agency professional staff: a decrease in the total number of positions and staff turnover. In NIE, 91 persons were in excepted service on May 15, 1979, and 191 were in civil service positions on September 30, 1981. In 1986, there were 200 professionals-39 in excepted service and 161 in civil service positions. These changes represent declines of 57 percent and 13 percent for the excepted service and civil service, respectively.9 Table 4.3: Changes in Education Information Professional Staff Between Fiscal Years 1980 and 1986 Excepted Civil CWithin the Office of Educational Research and Improvement, 29 new hires were in the office of eSince figures are from two different periods, calculating a total would be misleading. Source: Executive office of the Office of Educational Research and Improvement and GAO surveys of The turnover was high, especially in excepted-service positions. Of the 91 excepted-service employees in May 15, 1979, only 6 persons (7 percent) were still in the department in September 1986; none of these individuals had been reassigned to other units in the department. Of the 191 professionals in civil service positions, 47 remained and 9 of them (5 percent) were in positions elsewhere in the department or had been reassigned to the Center for Statistics. 9Excepted-service positions are unclassified civil service positions or those outside the competitive service, excepted from the requirements of competitive service by law, executive order, or commission regulation. 1 |