CENTER FOR EDUCATION STATISTICS
OFFICE OF EDUCATIONAL RESEARCH AND IMPROVEMENT
U. S. DEPARTMENT OF EDUCATION


 

SUBJECT: STANDARD FOR INDEPENDENT EVALUATION OF SURVEYS

EFFECTIVE DATE: 03/16/87 CES STANDARD 87-01-02

PURPOSE: To ensure that the data collected are of the highest quality consistent with cost and other constraints, and that corroborating evidence can be found regarding the information collected in CES surveys.
o At the time a survey is being planned and developed, a research plan should also be developed that will allow the independent evaluation of the survey. This plan should identify possible sources of nonsampling error in the survey and indicate how these will be dealt with and measured. In addition, the plan should list how data will be obtained for the evaluation of the survey: through checks for internal consistency, external comparison against other data sources, and experiments built into the survey to study alternative data collection methodologies.

o Where possible, the pretest for a survey should include some experiments designed to study sources and control of nonsampling error. Normally, no survey should be initiated without a formal pretest and evaluation of the pretest results.

o Items should be identified that can be used in the internal checking process once the data has been collected in a survey. Tables should be generated to study this data before any other set of data from the survey, and results of this analysis should be used to correct the data if outliers are found or data are found to be inconsistent.

o Sources outside the data collected in a survey should be identified and put to use to analyze the survey results. Again, tables should be generated compared to these outside sources to help ensure the
validity of the data before analysis of the data begins.

o The study plan for the survey or data collection operation should have an ongoing research plan built into the study design. This means that consideration should be given to having a small independent replicate of the sample preidentified for use as a test group while the survey continues. This is especially important for longitudinal studies, where design changes, new items, or other effects introduced to the survey without prior testing can severely affect the time series.

RELATED STANDARDS:

o CES Standard for Planning Statistical Surveys 86-01-01
o CES Standard for Testing Data Collection Instruments 87-03-01
o CES Standard for Technical Documentation in Data Releases 87-04-02

RELATED CHECKLISTS AND FORMS: