At the time a survey is being planned and developed, a research plan
should also be developed that will allow the independent evaluation of
the survey. This plan should identify possible sources of nonsampling
error in the survey and indicate how these will be dealt with and
measured. In addition, the plan should list how data will be obtained
for the evaluation of the survey: through checks for internal
consistency, external comparison against other data sources, and
experiments built into the survey to study alternative data collection
Where possible, the pretest for a survey should include some experiments
designed to study sources and control of nonsampling error. Normally, no
survey should be initiated without a formal pretest and evaluation of
the pretest results.
Items should be identified that can be used in the internal checking
process once the data has been collected in a survey. Tables should be
generated to study this data before any other set of data from the
survey, and results of this analysis should be used to correct the data
if outliers are found or data are found to be inconsistent.
Sources outside the data collected in a survey should be identified and
put to use to analyze the survey results. Again, tables should be
compared to these outside sources to help ensure the
validity of the data before analysis of the data begins.
The study plan for the survey or data collection operation should have
an ongoing research plan built into the study design. This means that
consideration should be given to having a small independent replicate of
the sample preidentified for use as a test group while the survey
continues. This is especially important for longitudinal studies, where
design changes, new items, or other effects introduced to the survey
without prior testing can severely affect the time series.