Influences on Production and Quality, Agency federal nondefense agencies.5 During this period, the office showed a 63percent decline in purchasing power for program evaluation activities. For research, statistical, and evaluative information, the patterns of decline in funding were consistent and precipitous. Further, the consistency of decline in resources across these three types of information suggests across-the-board reductions in information rather than a substitution of research for evaluative data or of statistics for either research or evaluation. Funding support for the production of all three general forms of information about the condition of education has decreased more than 60 percent in real terms since the early to middle 1970's. For all three forms of education information, major declines between 1980 and 1984 ranged from 28 to 63 percent in 1980 constant dollars. In all three cases, the declines exceeded those observed for agencies and departments with similar missions. The greatest discrepancy, however, was for the research function; while federally supported research received, in general, a small increase from 1980 to 1984, funds for NIE were reduced by nearly half. This change led to the decrease in awards reported in chapter 2. Fiscal Influences on The case studies show that fiscal resources influence quality. For example, in 1976, NAEP reduced the number of skill areas it routinely assessed from 10 to 5. When the assessment schedule was altered to accommodate budget cuts in 1980, increased demands were placed on NAEP Staff to assist state and local agencies in the use of NAEP material. NAEP budgets have been supplemented by assessments conducted for other units within the department (see table III.1). Reviews of NAEP have been critical of some of these changes, suggesting that skill areas should be expanded, assessments should be more frequent, and scope should be broadened to younger students, for example, or to higher-order cognitive skills. To accommodate these concerns, funding would have to be adjusted upward—that is, the proposed 5U.S. General Accounting Office, Federal Evaluation: Fewer Units, Reduced Resources, Different Studies From 1980, PEMD-87-9 (Washington, D.C.: January 23, 1987). "Estimates differ on OPBE's funding level. Figures from OPBE's review of a draft of this report indicate a 28-percent decline in current dollars, or a 45-percent decline in constant dollars. OPBE based its figures on data obtained from other sources in addition to the annual evaluation reports. Influences on Production and Quality, Agency Resources and Support changes are simply beyond the current capacity of the information-gathering activity. Our review suggests that other forms of support that are indirectly For data collected within CCD, the evidence on the utility of support services and technical assistance is mixed. In examining the components of the pre-CCD data system, we found that NCES had devoted 5 staff years annually to developing and updating a series of handbooks that provided common definitions for education-related terms and data requirements under ELSEGIS. This aspect of the NCES portfolio was eliminated when staff assigned to this function retired. The last revision of the financial accounting handbook was published in 1980. Another component of pre-CCD assistance from NCES was on-site visits and workshops to assist state and local personnel in completing information requests. As fiscal resources within NCES declined, field visits were curtailed. Although the early CCD experiences with technical assistance and other types of support have been unsatisfactory and have not appeared to rid CCD of persistent problems, we did find evidence in another department activity that federal assistance can have some positive effects on the quality of data reported from state and local education agencies— namely, the Title I Evaluation and Reporting System. At first, participation in TIERS was voluntary, and 5 years were devoted to implementing the system. Its use was mandated in the 1978 Education Amendments. TIERS ultimately produced data that were comparable in format across states, but the level of technical support, research and development, and state capacity-building was substantial, costing about $37 million between 1975 and 1981. That is, quality was improved but it took time and money. With regard to the direct effects of fiscal resources, one seemingly obvious corrective action would be to restore funding levels. Our three case Influences on Production and Quality, Agency studies suggest that reductions in funding can influence aspects of quality but that funding must be examined in relation to the purposes of the data collection activity. Quality seems to be related not to dollars alone but, rather, to the match between required activities and required funds. For example, the three case examples differ considerably in fiscal support. NAEP, a major activity within the center, commanded over $6 million in resources annually. CCD began in 1976 with $900,000 annually and was reduced to about $400,000 by fiscal year 1986; funds were withdrawn in fiscal year 1987. FRSS is operated at about $200,000 to $350,000 per year. Our review of quality shows that NAEP and FRSS, the most and least costly, received moderate to high marks for relevance, timeliness, and technical adequacy while CCD seems to have faltered under various levels of funding. Furthermore, our review suggests that despite losses in purchasing power, NAEP has increased in technical adequacy and improved one aspect of its relevance by expanding its collection of background information on students, their attitudes, and school conditions. FRSS, despite its rather low level of expenditure, appears to have maintained an acceptable level of timeliness, relevance, and technical adequacy, although not all FRSS requests have been granted, because of budgetary constraints and other issues. In contrast, CCD has not received sustained support. In 1987, state education agency data-gathering went unfunded. The consequences were clearly demonstrated in our review of the data that have been gathered. Some data elements appear to be consistent with NCES Specifications, consistent over time (as indicated by small projection errors), and comparable across many states, but the system has been vigorously criticized with respect to its overall quality. The criticism hardly seems surprising. CCD appears to have been underfunded for its purposes and expectations. In the mid-1970's, the department requested $10 million to $23 million for CCD, a fraction of what one study estimated would be necessary. Funding never exceeded $900,000 per year-less than $20,000 per state agency. Judging from the discrepancy between proposed and actual funding, the idea of having a common core of data that could be used by all relevant policymakers was not adequately implemented. These cases show that while it is necessary to provide a stable source of funds, it may not be necessary to increase fiscal resources in order to Influences on Production and Quality, Agency enhance the relevance, timeliness, and technical adequacy of information. Rather, funding levels should match expectations for the nature of the data to be collected and the intended uses of the information. Congressional Table 4.1: National Institute of Education Research Obligations in Fiscal Years 1980 and 1984 In addition to establishing general statutory missions for informationproducing units, the Congress has used specific legislation, conference reports, and hearings as ways of requesting information about the condition of education. NAEP, for example, is a congressionally mandated, ongoing activity. TIERS and the National Vocational Education Data System (NVEDS) were mandated and intended as ongoing information systems. The Congress has also asked for special studies and surveys. Examples include the 1974 NIE title I compensatory education study and the 1985 survey of teacher supply and demand. Congressionally mandated activities can have a variety of influences on Our review of the role of mandates as a request strategy suggests that they can protect an activity by ensuring a sustained level of support, even during periods of budget-cutting, but other activities may be affected by insufficient funding or staffing or both. Our analyses of shifts in priorities showed that information-gathering activities that do Influences on Production and Quality, Agency not carry a mandate were more vulnerable when faced with fiscal constraints. For example, in the late 1970's, the Bureau of Education for the Handicapped devised a multistudy, longitudinal evaluation plan to assess aspects of Public Law 94-142, the Education for All Handicapped Children Act. One component of this plan, a set of longitudinal case studies of the effect of the law, was initiated but was terminated early. Budget cuts, changes in priorities, the study design, and the lack of an explicit mandate were among the reasons offered for the termination. Another type of vulnerability concerns the ability to collect data when studies are not mandated. In our study period, at least one FRSS request could not be honored: OMB required evidence that the information to be collected was requested under the mandated activity, but the proposed projects could not be justified on this ground. Further, OMB increased its efforts to reduce information collection by deleting data elements from OPBE evaluation questionnaires on the basis that these data were not legislatively required. Available instances suggest that unmandated information-gathering activities are vulnerable to changes in policies and priorities and to funding constraints. Second, specially mandated studies have a large but transient effect on the operations of information-producing agencies. They are less predictable than ongoing activities such as NAEP. Depending on a study's size or timing, it can consume a substantial amount of a unit's resources, incurring opportunity costs with regard to other activities. A unit reduced in staff size 16-20 percent may not be able to expand in response to certain requests. Further, in some instances, the questions to be answered, timeliness, and methods to be used were explicitly stated in the request in ways that were difficult to carry out. This is not to say that some mandates may not have a positive influence by altering normal agency routine. For example, in one case the mandate directed that reports were to be sent without prior department review to the requesting group, thus reducing the department's influence on the study design and final report. This has been seen as shortening time and increasing relevance in the sense that departmental policy concerns and congressional interests did not have to be reconciled. With regard to quality, NAEP has produced reports that are viewed as timely, technically adequate, and relevant descriptions of the condition of education. However, mandates alone are not sufficient to ensure that |