Design effect
In survey research, the design effect is a number that shows how well a sample of people may represent a larger group of people for a specific measure of interest. This is important when the sample comes from a sampling method that is different than just picking people using a simple random sample.
The design effect is a positive real number, represented by the symbol. If, then the sample was selected in a way that is just as good as if people were picked randomly. When, then inference from the data collected is not as accurate as it could have been if people were picked randomly.
When researchers use complicated methods to pick their sample, they use the design effect to check and adjust their results. It may also be used when planning a study in order to determine the sample size.
Introduction
In survey methodology, the design effect is a measure of the expected impact of a sampling design on the variance of an estimator for some parameter of a population. It is calculated as the ratio of the variance of an estimator based on a sample from an complex sampling design, to the variance of an alternative estimator based on a simple random sample of the same number of elements. The can be used to evaluate the variance of an estimator in cases where the sample is not drawn using simple random sampling. It may also be useful in sample size calculations and for quantifying the representativeness of samples collected with various sampling designs.The design effect is a positive real number that indicates an inflation, or deflation in the variance of an estimator for some parameter, that is due to the study not using SRS. Intuitively we can get when we have some a-priori knowledge we can exploit during the sampling process. And, in contrast, we often get when we need to compensate for some limitation in our ability to collect data. Some sampling designs that could introduce generally greater than 1 include: cluster sampling, stratified sampling, cluster randomized controlled trial, disproportional sample, statistical [|adjustments of the data] for non-coverage or non-response, and many others. Stratified sampling can yield that is smaller than 1 when using Proportionate allocation to strata sizes or Optimum allocation.
Many calculations (and estimators) have been proposed in the literature for how a known sampling design influences the variance of estimators of interest, either increasing or decreasing it. Generally, the design effect varies among different statistics of interests, such as the total or ratio mean. It also matters if the sampling design is correlated with the outcome of interest. For example, a possible sampling design might be such that each element in the sample may have a different probability to be selected. In such cases, the level of correlation between the probability of selection for an element and its measured outcome can have a direct influence on the subsequent design effect. Lastly, the design effect can be influenced by the distribution of the outcome itself. All of these factors should be considered when estimating and using design effect in practice.
History
The term "design effect" was coined by Leslie Kish in his 1965 book "Survey Sampling." In it, Kish proposed the general definition for the design effect, as well as formulas for the design effect of cluster sampling ; and the famous design effect formula for unequal probability sampling. These are often known as "Kish's design effect", and were later combined into a single formula.In a 1995 paper, Kish mentions that a similar concept, termed "Lexis ratio", was described at the end of the 19th century. The closely related Intraclass correlation was described by Fisher in 1950, while computations of ratios of variances were already published by Kish and others from the late 1940s to the 1950s. One of the precursors to Kish's definition was work done by Cornfield in 1951.
In his 1995 paper, Kish proposed that considering the design effect is necessary when averaging the same measured quantity from multiple surveys conducted over a period of time. He also suggested that the design effect should be considered when extrapolating from the error of simple statistics to more complex ones. However, when analyzing data, values are less useful nowadays due to the availability of specialized software for analyzing survey data. Prior to the development of software that computes standard errors for many types of designs and estimates, analysts would adjust standard errors produced by software that assumed all records in a dataset were i.i.d by multiplying them by a .
Definitions
Notations
| Symbol | Description |
| Variance of an estimator under a given sampling design | |
| Variance of an estimator under simple random sampling without replacement | |
| Variance of an estimator under simple random sampling with replacement | |
| , | Design effect, a measure of the impact of a sampling design on the variance of an estimator compared to simple random sampling without replacement, |
| , | Design effect factor, the square root of the ratio of variances under a given sampling design and SRS with replacement, |
| Sample size | |
| Population size | |
| Effective sample size, the sample size under SRS needed to achieve the same variance as the given sampling design, | |
| Weight for the -th unit | |
| Sample size for stratum | |
| Population size for stratum | |
| Weight for stratum | |
| Total number of strata | |
| , | Average cluster size |
| Total number of clusters | |
| Sample size for cluster | |
| Intraclass correlation coefficient for cluster sampling | |
| ,, | Measures of variation in weights using the coefficient of variation squared |
| Estimated correlation between the outcome variable and the selection probabilities | |
| Estimated intercept in the linear regression of the outcome variable on the selection probabilities | |
| Estimated standard deviation of the outcome variable | |
| , | Coefficient of variation for the weights and selection probabilities, respectively |
| Sampling fraction, | |
| Population variance of the outcome variable | |
| , | Selection probability for the -th unit |
| Inclusion probability for the -th unit |
''Deff''
The design effect, commonly [|denoted] by, is the ratio of two theoretical variances for estimators of some parameter :So that:
In other words, measures the extent to which the variance has increased because the sample was drawn and adjusted to a specific sampling design compared to if the sample was from a simple random sample. Notice how the definition of is based on parameters of the population that are often unknown, and that are hard to estimate directly. Specifically, the definition involves the variances of estimators under two different sampling designs, even though only a single sampling design is used in practice.
For example, when estimating the population mean, the is:
Where is the sample size, is the fraction of the sample from the population, is the finite population correction, is the unbiassed sample variance, and is some estimator of the variance of the mean under the sampling design. The issue with the above formula is that it is extremely rare to be able to directly estimate the variance of the estimated mean under two different sampling designs, since most studies rely on only a single sampling design.
There are many ways of calculation, depending on the parameter of interest, the estimator used, and the sampling design. The process of estimating for specific designs will be described in the following section.
''Deft''
A related quantity to, proposed by Kish in 1995, is the Design Effect Factor, abbreviated as . It is defined as the square root of the variance ratios while also having the denominator use a simple random sample with replacement, instead of without replacement :In this later definition Kish argued in favor of using over for several reasons. It was argued that SRS "without replacement", it was claimed that using will be simpler than writing. It is also said that for many cases when the population is very large, is the square root of, hence it is easier to use than exactly calculating the finite population correction.
Even so, in various cases a researcher might approximate the by calculating the variance in the numerator while assuming SRS with replacement instead of SRS without replacement, even if it is not precise. For example, consider a multistage design with primary sampling units selected systematically with probability proportional to some measure of size from a list sorted in a particular way. Also, let it be combined with an estimator that uses raking to match the totals for several demographic variables. In such a design, the joint selection probabilities for the PSUs, which are needed for a without replacement variance estimator, are 0 for some pairs of PSUs - implying that an exact design-based variance estimator does not exist. Another example is when a public use file issued by some government agency is used for analysis. In such a case the information on joint selection probabilities of first-stage units is almost never released. As a result, an analyst cannot estimate a with replacement variance for the numerator even if desired. The standard workaround is to compute a variance estimator as if the PSUs were selected with replacement. This is the default choice in software packages such as Stata, the R survey package, and the SAS survey procedures.
Effective sample size
The effective sample size, defined by Kish in 1965, is calculated by dividing the original sample size by the design effect. Namely:This quantity reflects what would be the sample size that is needed to achieve the current variance of the estimator with the existing design, if the sample design were based on a simple random sample.
A related quantity is the effective sample size ratio, which can be calculated by simply taking the inverse of .
For example, let the design effect, for estimating the population mean based on some sampling design, be 2. If the sample size is 1,000, then the effective sample size will be 500. It means that the variance of the weighted mean based on 1,000 samples will be the same as that of a simple mean based on 500 samples obtained using a simple random sample.
The design effect for well-known sampling designs
The design effect depends on sampling design and statistical adjustments
Different sampling designs and statistical adjustments may have substantially different impact on the bias and variance of estimators.An example of a design which can lead to estimation efficiency, compared to simple random sampling, is Stratified sampling. This efficiency is gained by leveraging information about the composition of the population. For example, if it is known that gender is correlated with the outcome of interest, and also that the male-female ratio for some population is 50%-50%, then sampling exactly half of the sample from each gender will reduce the variance of the outcome's estimator. Similarly, if a particular sub-population is of special interest, deliberately over-sampling from that sub-population will decrease the variance for estimations made about it.
Improvement in variance efficiency might sometimes be sacrificed for convenience or cost. For example, in the cluster sampling case the units may have equal or unequal selection probabilities, irrespective of their intra-class correlation. We might decide to collect responses from only 2 people of each household, which could lead to more complex post-sampling adjustment to deal with unequal selection probabilities. Also, such decisions could lead to less efficient estimators than just taking a fixed proportion of responses from a cluster.
When the sampling design isn’t set in advance and needs to be figured out from the data we have, this can lead to an increase of both the variance and bias of the weighted estimator. This might happen when making adjustments for issues like non-coverage, non-response, or an unexpected strata split of the population that wasn’t available during the initial sampling stage. In these cases, we might use statistical procedures such as post-stratification, raking, or inverse propensity score weighting, among other methods. Using these methods requires assumptions about the initial design model. For example, when we use post-stratification based on age and gender, it is assumed that these variables can explain a significant portion of the bias in the sample. The quality of these estimators is closely tied to the quality of the additional information and the missing at random assumptions used when making them. Either way, even when estimators do a good job capturing most of the sampling design, using the weights can make a small or a large difference, depending on the specific data-set.
Due to the large variety in sampling designs, different formulas have been developed to capture the potential design effect, as well as to estimate the variance of estimators when accounting for the sampling designs. Sometimes, these different design effects can be compounded together. Whether or not to use these formulas, or just assume SRS, depends on the expected amount of bias reduction vs. the increase in estimator variance.
| Formula Name | Equation | Description |
| Kish's design effect for unequal weights | Measures the loss in precision due to unequal weights, where is the weight for the -th unit. | |
| Kish's design effect for cluster sampling | Measures the loss in precision due to cluster sampling, where is the average cluster size and is the intraclass correlation. | |
| Kish's combined design effect | Measures the combined effect of unequal weights and cluster sampling, where and are the sample size and weight for the -th stratum, respectively. | |
| Spencer's design effect for estimated total | Measures the design effect for estimating a total when there is a correlation between the outcome and the selection probabilities, where is the estimated correlation, is the relvariance of the weights, is the estimated intercept, and is the estimated standard deviation of the outcome. | |
| Park and Lee's design effect for estimated ratio mean | Measures the design effect for estimating a ratio mean when there is a correlation between the outcome and the selection probabilities, where and are the coefficients of variation for the weights and selection probabilities, respectively. | |
| Henry's design effect for calibration weighting | Extends Kish's design effect to include calibration weighting in single-stage samples | Proposes a model-assisted design effect measure for single-stage sampling with calibration weighting, considering the correlation between the outcome and the calibration variables. |
| Lohr's design effect for regression slope | Provides design effect formulas for OLS and GLS regression slope estimators in cluster sampling | Presents design effect formulas for ordinary least squares and generalized least squares regression slope estimators in the context of cluster sampling, using a random coefficient model. |
Unequal selection probabilities
"Design based" vs "model based" for describing properties of estimators
Adjusting for unequal probability selection through "individual case weights", yields various types of estimators for quantities of interest. Estimators such as Horvitz–Thompson estimator yield unbiased estimators, for total and the mean of the population. Deville and Särndal coined the term "calibration estimator" for estimators using weights such that they satisfy some condition, such as having the sum of weights equal the population size. And more generally, that the weighted sum of weights is equal some quantity of an auxiliary variable: .The two primary ways to argue about the properties of calibration estimators are:
- randomization based - in this case, the weights and values of the outcome of interest that are measured in the sample are all treated as known. In this framework, there is variability in the values of the outcome. However, the only randomness comes from which of the elements in the population were picked into the sample. For a simple random sample, each will be an IID Bernoulli distribution with some parameter. For general EPSEM will still be Bernoulli with some parameter, but they may no longer be independent random variables. I.e., knowing that a sample is EPSEM means that it maintains marginally equal probability of selection, but it does not inform us about the joint probability of selection. For something like post stratification, the number of elements at each stratum can be modeled as a multinomial distribution with different inclusion probabilities for each element belonging to some stratum. In these cases, the sample size itself can be a random variable.
- model based - in this case, the sample is fixed, the weights are fixed, but the outcome of interest is treated as a random variable. For example, in the case of post-stratification, the outcome can be modeled as some linear regression function where the independent variables are indicator variables mapping each observation to its relevant stratum, and the variability comes with the error term.
Common types of weights
| Weight Type | Description | Interpretation |
| Frequency weights | Each weight is an integer indicating the absolute frequency of an item in the sample | Specific value has an absolute meaning; weights represent the amount of information in the dataset |
| Inverse-variance weights | Each element is assigned a weight that is the inverse of its known variance | When all elements have the same expectancy, using such weights for weighted averages has the least variance |
| Normalized weights | Weights form a convex combination ; can be normalized to sum to sample size | Weights that sum to n have a relative interpretation: elements with weights > 1 are more "rare" than average and have larger influential on the average, while weights < 1 are more "common" and less influential |
| Inverse probability weights | Each element is given a weight proportional to the inverse of its selection probability | Weights represent how many items each element "represents" in the target population; sum of weights equals the size of the target population |
There are many types of weights, with different ways to use and interpret them. With some weights their absolute value has some important meaning, while with other weights the important part is the relative values of the weights to each other. This section introduces some of the more common types of weights so that they can be referenced in follow-up sections.Frequency weights are a basic type of weighting presented in introductory statistics courses. With these, each weight is an integer number that indicates the absolute frequency of an item in the sample. These are also sometimes termed repeat weights. The specific value has an absolute meaning that is lost if the weights are transformed, such as when scaling. For example: if we have the numbers 10 and 20 with the frequency weights values of 2 and 3, then when "spreading" our data it is: 10,10, 20, 20, 20. Frequency weights includes the amount of information contained in a dataset, and thus allows things like creating unbiased weighted variance estimation using Bessel's correction. Notice that such weights are often random variables, since the specific number of items we will see from each value in the dataset is random.inverse-variance weighting, also known as analytic weights, is when each element is assigned a weight that is the inverse of its variance. When all elements have the same expectancy, using such weights for calculating weighted averages has the least variance among all weighted averages. In the common formulation, these weights are known and not random.Normalized weights is a set of weights that form a convex combination, i.e., each weight is a number between 0 and 1, and the sum of all weights is equal to 1. Any set of weights can be turned into normalized weights by dividing each weight with the sum of all weights, making these weights normalized to sum to 1.Inverse probability weighting, or simply probability weights, is when each element is given a weight that is to the inverse probability of selecting that element. E.g., by using. With inverse probability weights, we learn how many items each element "represents" in the target population. Hence, the sum of such weights returns the size of the target population of interest. Inverse probability weights can be normalized to sum to 1 or normalized to sum to the sample size, and many of the calculations from the following sections will yield the same results.
There are also indirect ways of applying "weighted" adjustments. For example, the existing cases may be duplicated to impute missing observations, with variance estimated using methods such as multiple imputation. An alternative approach is to remove some cases. For example, when wanting to reduce the influence of over-sampled groups that are less essential for some analysis. Both cases are similar in nature to inverse probability weighting but the application in practice gives more/less rows of data, instead of applying an extra column of weights. Nevertheless, the consequences of such implementations are similar to just using weights. So while in the case of removing observations the data can easily be handled by common software implementations, the case of adding rows requires special adjustments for the uncertainty estimations. Not doing so may lead to erroneous conclusions.
The term "Haphazard weights", coined by Kish, is used to refer to weights that correspond to unequal selection probabilities, but ones that are not related to the expectancy or variance of the selected elements.
Haphazard weights with estimated ratio-mean - Kish's design effect
Formula
When taking an unrestricted sample of elements, we can then randomly split these elements into disjoint strata, each of them containing some size of elements so that. All elements in each stratum has some non-negative weight assigned to them. The weight can be produced by the inverse of some unequal selection probability for elements in each stratum . In this setting, Kish's design effect, for the increase in variance of the sample weighted mean due to this design, versus SRS of some outcome variable y is:By treating each item as coming from its own stratum, Kish simplified the above formula to the following version:
This version of the formula is valid when one stratum had several observations taken from it, or when there are just many strata were each one had one observation taken from it, but several of them had the same probability of selection. While the interpretation is slightly different, the calculation of the two scenarios comes out to be the same.
When using Kish's design effect for unequal weights, you may use the following simplified formula for "Kish's Effective Sample Size"
Assumptions and proofs
The above formula, by Kish, gives the increase in the variance of the weighted mean based on "haphazard" weights. This can also be written as the following formula where y are observations selected using unequal selection probabilities, and y' are the observations we would have had if we got them from a simple random sample:It can be shown that the ratio of variances formula can be reduced to Kish's formula by using a model based perspective. In it, Kish's formula will hold when all n observations are uncorrelated, with the same variance in the response variable of interest. It will also be required to assume the weights themselves are not a random variable but rather some known constants.
The following is a simplified proof for when there are no clusters and each stratum includes only one observation:
Transitions:
- from definition of the weighted mean.
- using normalized (convex) weights definition :.
- sum of uncorrelated random variables.
- If the weights are constants. Another way to say it is that the weights are known upfront for each observation i. Namely that we are actually calculating
- when all observations have the same variance.
- Some algebra: Moving to the left, adding a multiplication term of, and opening back.
- Back to the definitions.
If different s values have different variances, then while the weighted variance could capture the correct population-level variance, Kish's formula for the design effect may no longer be true.
A similar issue happens if there is some correlation structure in the samples.
Relation to the coefficient of variation
Notice that Kish's definition of the design effect is closely tied to the coefficient of variation of the weights. This has several notations in the literature:Where is the population variance of, and is the mean. When the weights are normalized to sample size, then and the formula reduces to. While it is true we assume the weights are fixed, we can think of their variance as the variance of an empirical distribution defined by sampling one weight from our set of weights.
Relation to disproportionate stratified sampling
Kish's original definition compared the variance under some sampling design to the variance achieved through a simple random sample. Some literature provide the following alternative definition for Kish's design effect: "the ratio of the variance of the weighted survey mean under disproportionate stratified sampling to the variance under proportionate stratified sampling when all stratum unit variances are equal". Reflecting on this, Park and Lee stated that "The rationale behind derivation is that the loss in precision of due to haphazard unequal weighting can be approximated by the ratio of the variance under disproportionate stratified sampling to that under the proportionate stratified sampling".Note that this alternative definition only approximated since if the denominator is based on "proportionate stratified sampling" then such a selection will yield a reduced variance as compared with simple random sample. This is since stratified sampling removes some of the variability in the specific number of elements per stratum, as occurs under SRS.
Relatedly, Cochran provides a formula for the proportional increase in variance due to deviation from optimum allocation.
Alternative naming conventions
Early papers used the term. As more definitions of the design effect appeared, Kish's design effect for unequal selection probabilities was denoted or simply for short. Kish's design effect is also known as the "Unequal Weighting Effect", termed by Liu et al. in 2002.When the outcome correlates with the selection probabilities
Spencer's ''Deff'' for estimated total
The estimator for the total is the "p-expanded with replacement" estimator. It is based on a simple random sample of n items from a population of size N. Each item has a probability of to be drawn in a single draw. The probability that a specific will appear in the sample is. The "p-expanded with replacement" value is with the following expectancy:. Hence, the pwr-estimator, is an unbiased estimator for the sum total of y.In 2000, Bruce D. Spencer proposed a formula for estimating the design effect for the variance of estimating the total of some quantity, when there is correlation between the selection probabilities of the elements and the outcome variable of interest.
In this setup, a sample of size n is drawn from a population of size N. Each item is drawn with probability . The selection probabilities are used to define the Normalized (convex) weights:. Notice that for some random set of n items, the sum of weights will be equal to 1 only by expectation with some variability of the sum around it. The relationship between and is defined by the following simple linear regression:
Where is the outcome of element i, which linearly depends on with the intercept and slope. The residual from the fitted line is. We can also define the population variances of the outcome and the residuals as and. The correlation between and is.
Spencer's design effect for estimating the total of y is:
Where:
- estimates
- estimates the slope
- estimates the population variance, and
- L is the relvariance of the weights, as defined in Kish's formula:.
When the population size is very large, the formula can be written as:
This approximation assumes that the linear relationship between P and y holds. And also that the correlation of the weights with the errors, and the errors squared, are both zero. I.e., and.
We notice that if, then . In such a case, the formula reduces to
Only if the variance of y is much larger than its mean, then the right-most term is close to 0, which reduces Spencer's design effect to be equal to Kish's design effect :. Otherwise, the two formulas will yield different results, which demonstrates the difference between the design effect of the total vs. the design effect of the mean.
Park and Lee's Deff for estimated ratio-mean
In 2001, Park and Lee extended Spencer's formula to the case of the ratio-mean. It is:Where:
- is the squared coefficient of variation of the probabilities of selection.
In general, the for the total tends to be less efficient than the for the ratio mean when is small. And in general, impacts the efficiency of both design effects.
Cluster sampling
For data collected using cluster sampling we assume the following structure:- observations in each cluster and K clusters, and with a total of observations.
- The observations have a block diagonal correlation matrix in which every pair of observations from the same cluster is correlated with an intra-class correlation of, while every pair from difference clusters are uncorrelated. I.e., for every pair of observations, and, if they belong to the same cluster, we get. And two items from two different clusters are not correlated, i.e.i.e.:.
- An element from any cluster is assumed to have the same variance:.
It is sometimes also denoted as.
In various papers, when cluster sizes are not equal, the above formula is also used with as the average cluster size. In such cases, Kish's formula serves as a conservative of the exact design effect.
Alternative formulas exists for unequal cluster sizes. Followup work had discussed the sensitivity of using the average cluster size with various assumptions.
The design effect for complex designs
Unequal selection probabilities times Cluster sampling
In a 1987 paper, Kish proposed a combined design effect that incorporates both the effects due to weighting that accounts for unequal selection probabilities and cluster sampling:The above uses notations similar to what is used in this article. A model based justification for this formula was provided by Gabler et al.
Stratified sampling times unequal selection probabilities times Cluster sampling
In 2000, Liu and Aragon proposed a decomposition of unequal selection probabilities design effect for different strata in stratified sampling. In 2002, Liu et al. extended that work to account for stratified samples, where within each stratum is a set of unequal selection probability weights. The cluster sampling is either global or per stratum. Similar work was done also by Park et al. in 2003.Chen-Rust Deff: Design effects to two- and three-stage designs with stratification
The Chen-Rust extends the model-based justification of Kish’s 1987 formula for design effects proposed by Gabler, el. al., applying it to two-stage designs with stratification at the first stage and to three-stage designs without stratification. The modified formulae define the overall design effect using survey weights and population intracluster correlations. These formulae allow for insightful interpretations of design effects from various sources and can estimate intracluster correlations in completed surveys or predict design effects in future surveys.Henry's Deff: a design effect measure for calibration weighting in single-stage samples
Henry's proposes an extended model-assisted weighting design-effect measure for single-stage sampling and calibration weight adjustments for a case where, where is a vector of covariates, the model errors are independent, and the estimator of the population total is the general regression estimator of Särndal, Swensson, and Wretman. The new measure considers the combined effects of non-epsem sampling design, unequal weights from calibration adjustments, and the correlation between an analysis variable and the auxiliaries used in calibration.Lohr's Deff: a design effect for a regression slope in a cluster sample
Lohr's is for ordinary least squares and generalized least squares estimators in the context of cluster sampling, using a random coefficient regression model. Lohr presents conditions under which the GLS estimator of the regression slope has a design effect less than 1, indicating higher efficiency. However, the design effect of the GLS estimator is highly sensitive to model specification. If an underlying random coefficient model is incorrectly specified as a random intercept model, the design effect can be seriously understated. In contrast, the OLS estimator of the regression slope and the design effect calculated from a design-based perspective are robust to misspecification of the variance structure, making them more reliable in situations where the model specification may not be accurate.Uses
may be used when planning a future data collection, as well as a diagnostic tool:When planning a future data collection - may be used to evaluate the sampling efficiency. E.g. if there is potentially "too much" increase in variance due to some sampling design decision, or if some alternative design is more efficient. This also influences the sample size. When planning the sample size, work may be done to correct the design effect so as to separate the interviewer effect from the effects of the sampling design on the sampling variance.As a diagnostic tool - may help in evaluating potential problems with a post-hoc weighting analysis. For example, if the value is especially high, then it might indicate an issue with the sampling or weighting scheme. This can also assist when performing some manipulation on the weights, the design effect could be used to evaluate the influence of the manipulation on the effective sample size. And also in identifying glaring issues with the data or its analysis. Although some literature suggests that is likely to require some attention, there is no universal rule of thumb for which design effect value is "too high". Practical considerations of values are often context dependent.Considering the design effect is unnecessary when the source population is closely IID, or when the sample design of the data was drawn as a simple random sample. It is also less useful when the sample size is relatively small.
While Kish originally hoped to have the design effect be as agnostic as possible to the underlying distribution of the data, sampling probabilities, their correlations, and the statistics of interest, followup research has shown that these do influence the design effect. Hence, these properties should be carefully considered when deciding which calculation to use, and how to use it.
The design effect is rarely applied when constructing confidence intervals. Ideally, one would be able to determine, for an estimator of a particular parameter, both the variance under Simple Random Sample with replacement and the design effect. In such scenarios, the basic variance and the design effect could have been multiplied to compute the variance of the estimator for the specific design. This computed value can then be employed to form confidence intervals. However, in real-world applications, it is uncommon to estimate both values simultaneously. As a result, other methods are favored. For instance, Taylor linearization is utilized to construct confidence intervals based on the variance of the weighted mean. More broadly, the bootstrap method, also known as replication weights, is applied for a range of weighted statistics.