Design effect


In survey research, the design effect is a number that shows how well a sample of people may represent a larger group of people for a specific measure of interest. This is important when the sample comes from a sampling method that is different than just picking people using a simple random sample.
The design effect is a positive real number, represented by the symbol. If, then the sample was selected in a way that is just as good as if people were picked randomly. When, then inference from the data collected is not as accurate as it could have been if people were picked randomly.
When researchers use complicated methods to pick their sample, they use the design effect to check and adjust their results. It may also be used when planning a study in order to determine the sample size.

Introduction

In survey methodology, the design effect is a measure of the expected impact of a sampling design on the variance of an estimator for some parameter of a population. It is calculated as the ratio of the variance of an estimator based on a sample from an complex sampling design, to the variance of an alternative estimator based on a simple random sample of the same number of elements. The can be used to evaluate the variance of an estimator in cases where the sample is not drawn using simple random sampling. It may also be useful in sample size calculations and for quantifying the representativeness of samples collected with various sampling designs.
The design effect is a positive real number that indicates an inflation, or deflation in the variance of an estimator for some parameter, that is due to the study not using SRS. Intuitively we can get when we have some a-priori knowledge we can exploit during the sampling process. And, in contrast, we often get when we need to compensate for some limitation in our ability to collect data. Some sampling designs that could introduce generally greater than 1 include: cluster sampling, stratified sampling, cluster randomized controlled trial, disproportional sample, statistical [|adjustments of the data] for non-coverage or non-response, and many others. Stratified sampling can yield that is smaller than 1 when using Proportionate allocation to strata sizes or Optimum allocation.
Many calculations have been proposed in the literature for how a known sampling design influences the variance of estimators of interest, either increasing or decreasing it. Generally, the design effect varies among different statistics of interests, such as the total or ratio mean. It also matters if the sampling design is correlated with the outcome of interest. For example, a possible sampling design might be such that each element in the sample may have a different probability to be selected. In such cases, the level of correlation between the probability of selection for an element and its measured outcome can have a direct influence on the subsequent design effect. Lastly, the design effect can be influenced by the distribution of the outcome itself. All of these factors should be considered when estimating and using design effect in practice.

History

The term "design effect" was coined by Leslie Kish in his 1965 book "Survey Sampling." In it, Kish proposed the general definition for the design effect, as well as formulas for the design effect of cluster sampling ; and the famous design effect formula for unequal probability sampling. These are often known as "Kish's design effect", and were later combined into a single formula.
In a 1995 paper, Kish mentions that a similar concept, termed "Lexis ratio", was described at the end of the 19th century. The closely related Intraclass correlation was described by Fisher in 1950, while computations of ratios of variances were already published by Kish and others from the late 1940s to the 1950s. One of the precursors to Kish's definition was work done by Cornfield in 1951.
In his 1995 paper, Kish proposed that considering the design effect is necessary when averaging the same measured quantity from multiple surveys conducted over a period of time. He also suggested that the design effect should be considered when extrapolating from the error of simple statistics to more complex ones. However, when analyzing data, values are less useful nowadays due to the availability of specialized software for analyzing survey data. Prior to the development of software that computes standard errors for many types of designs and estimates, analysts would adjust standard errors produced by software that assumed all records in a dataset were i.i.d by multiplying them by a .

Definitions

Notations

SymbolDescription
Variance of an estimator under a given sampling design
Variance of an estimator under simple random sampling without replacement
Variance of an estimator under simple random sampling with replacement
,Design effect, a measure of the impact of a sampling design on the variance of an estimator compared to simple random sampling without replacement,
,Design effect factor, the square root of the ratio of variances under a given sampling design and SRS with replacement,
Sample size
Population size
Effective sample size, the sample size under SRS needed to achieve the same variance as the given sampling design,
Weight for the -th unit
Sample size for stratum
Population size for stratum
Weight for stratum
Total number of strata
,Average cluster size
Total number of clusters
Sample size for cluster
Intraclass correlation coefficient for cluster sampling
,,Measures of variation in weights using the coefficient of variation squared
Estimated correlation between the outcome variable and the selection probabilities
Estimated intercept in the linear regression of the outcome variable on the selection probabilities
Estimated standard deviation of the outcome variable
,Coefficient of variation for the weights and selection probabilities, respectively
Sampling fraction,
Population variance of the outcome variable
,Selection probability for the -th unit
Inclusion probability for the -th unit

''Deff''

The design effect, commonly [|denoted] by , is the ratio of two theoretical variances for estimators of some parameter :
So that:
In other words, measures the extent to which the variance has increased because the sample was drawn and adjusted to a specific sampling design compared to if the sample was from a simple random sample. Notice how the definition of is based on parameters of the population that are often unknown, and that are hard to estimate directly. Specifically, the definition involves the variances of estimators under two different sampling designs, even though only a single sampling design is used in practice.
For example, when estimating the population mean, the is:
Where is the sample size, is the fraction of the sample from the population, is the finite population correction, is the unbiassed sample variance, and is some estimator of the variance of the mean under the sampling design. The issue with the above formula is that it is extremely rare to be able to directly estimate the variance of the estimated mean under two different sampling designs, since most studies rely on only a single sampling design.
There are many ways of calculation, depending on the parameter of interest, the estimator used, and the sampling design. The process of estimating for specific designs will be described in the following section.

''Deft''

A related quantity to, proposed by Kish in 1995, is the Design Effect Factor, abbreviated as . It is defined as the square root of the variance ratios while also having the denominator use a simple random sample with replacement, instead of without replacement :
In this later definition Kish argued in favor of using over for several reasons. It was argued that SRS "without replacement", it was claimed that using will be simpler than writing. It is also said that for many cases when the population is very large, is the square root of , hence it is easier to use than exactly calculating the finite population correction.
Even so, in various cases a researcher might approximate the by calculating the variance in the numerator while assuming SRS with replacement instead of SRS without replacement, even if it is not precise. For example, consider a multistage design with primary sampling units selected systematically with probability proportional to some measure of size from a list sorted in a particular way. Also, let it be combined with an estimator that uses raking to match the totals for several demographic variables. In such a design, the joint selection probabilities for the PSUs, which are needed for a without replacement variance estimator, are 0 for some pairs of PSUs - implying that an exact design-based variance estimator does not exist. Another example is when a public use file issued by some government agency is used for analysis. In such a case the information on joint selection probabilities of first-stage units is almost never released. As a result, an analyst cannot estimate a with replacement variance for the numerator even if desired. The standard workaround is to compute a variance estimator as if the PSUs were selected with replacement. This is the default choice in software packages such as Stata, the R survey package, and the SAS survey procedures.