External validity

External validity is the validity of applying the conclusions of a scientific study outside the context of that study. In other words, it is the extent to which the results of a study can generalize or transport to other situations, people, stimuli, and times. Generalizability refers to the applicability of a predefined sample to a broader population while transportability refers to the applicability of one sample to another target population. In contrast, internal validity is the validity of conclusions drawn within the context of a particular study.
Mathematical analysis of external validity concerns a determination of whether generalization across heterogeneous populations is feasible, and devising statistical and computational methods that produce valid generalizations.
In establishing external validity, scholars tend to identify the "scope" of the study, which refers to the applicability or limitations of the theory or argument of the study. This entails defining the sample of the study and the broader population that the sample represents.

Threats

"A threat to external validity is an explanation of how you might be wrong in making a generalization from the findings of a particular study." In most cases, generalizability is limited when the effect of one factor depends on other factors. Therefore, all threats to external validity can be described as statistical interactions. Some examples include:

Aptitude by treatment interaction: The sample may have certain features that interact with the independent variable, limiting generalizability. For example, comparative psychotherapy studies often employ specific samples. If psychotherapy is found effective for these sample patients, will it also be effective for non-volunteers or the mildly depressed or patients with concurrent other disorders? If not, the external validity of the study would be limited.
Situation by treatment interactions: All situational specifics of a study potentially limit generalizability.
Pre-test by treatment interactions: If cause-effect relationships can only be found when pre-tests are carried out, then this also limits the generality of the findings. This sometimes goes under the label "sensitization", because the pretest makes people more sensitive to the manipulation of the treatment.

Note that a study's external validity is limited by its internal validity. If a causal inference made within a study is invalid, then generalizations of that inference to other contexts will also be invalid.
Cook and Campbell made the crucial distinction between generalizing to some population and generalizing across subpopulations defined by different levels of some background factor. Lynch has argued that it is almost never possible to generalize to meaningful populations except as a snapshot of history, but it is possible to test the degree to which the effect of some cause on some dependent variable generalizes across subpopulations that vary in some background factor. That requires a test of whether the treatment effect being investigated is moderated by interactions with one or more background factors.

Disarming threats

Whereas enumerating threats to validity may help researchers avoid unwarranted generalizations, many of those threats can be disarmed, or neutralized in a systematic way, so as to enable a valid generalization. Specifically, experimental findings from one population can be "re-processed", or "re-calibrated" so as to circumvent population differences and produce valid generalizations in a second population, where experiments cannot be performed. Pearl and Bareinboim classified generalization problems into two categories: those that lend themselves to valid re-calibration, and those where external validity is theoretically impossible. Using graph-based causal inference calculus, they derived a necessary and sufficient condition for a problem instance to enable a valid generalization, and devised algorithms that automatically produce the needed re-calibration, whenever such exists. This reduces the external validity problem to an exercise in graph theory, and has led some philosophers to conclude that the problem is now solved.
An important variant of the external validity problem deals with selection bias, also known as sampling bias—that is, bias created when studies are conducted on non-representative samples of the intended population. For example, if a clinical trial is conducted on college students, an investigator may wish to know whether the results generalize to the entire population, where attributes such as age, education, and income differ substantially from those of a typical student. The graph-based method of Bareinboim and Pearl identifies conditions under which sample selection bias can be circumvented and, when these conditions are met, the method constructs an unbiased estimator of the average causal effect in the entire population. The main difference between generalization from improperly sampled studies and generalization across disparate populations lies in the fact that disparities among populations are usually caused by preexisting factors, such as age or ethnicity, whereas selection bias is often caused by post-treatment conditions, for example, patients dropping out of the study, or patients selected by severity of injury. When selection is governed by post-treatment factors, unconventional re-calibration methods are required to ensure bias-free estimation, and these methods are readily obtained from the problem's
graph.

Examples

If age is judged to be a major factor causing treatment effect to vary from individual to individual, then age differences between the sampled students and the general population would lead to a biased estimate of the average treatment effect in that population. Such bias can be corrected though by a simple re-weighing procedure: We take the age-specific effect in the student subpopulation and compute its average using the age distribution in the general population. This would give us an unbiased estimate of the average treatment effect in the population. If, on the other hand, the relevant factor that distinguishes the study sample from the general population is in itself affected by the treatment, then a different re-weighing scheme need be invoked. Calling this factor Z, we again average the z-specific effect of X on Y in the experimental sample, but now we weigh it by the "causal effect" of X on Z. In other words, the new weight is the proportion of units attaining level Z=z had treatment X=x been administered to the entire population. This interventional probability, often written using Do-calculus, can sometimes be estimated from
observational studies in the general population.
A typical example of this nature occurs when Z is a mediator between the treatment and outcome, For instance, the treatment may be a cholesterol-reducing drug, Z may be cholesterol level, and Y life expectancy. Here, Z is both affected by the treatment and a major factor in determining the outcome, Y. Suppose that subjects selected for the experimental study
tend to have higher cholesterol levels than is typical in the general population. To estimate the average effect of the drug on survival in the entire population, we first compute the z-specific treatment effect in the experimental study, and then average it using as a weighting function. The estimate obtained will be bias-free even when Z and Y are confounded—that is, when there is an unmeasured common factor that affects both Z and Y.
The precise conditions ensuring the validity of this and other weighting schemes are formulated in Bareinboim and Pearl, 2016 and Bareinboim et al., 2014.

External, internal, and ecological validity

In many studies and research designs, there may be a trade-off between internal validity and external validity: Attempts to increase internal validity may also limit the generalizability of the findings, and vice versa.
This situation has led many researchers call for "ecologically valid" experiments. By that they mean that experimental procedures should resemble "real-world" conditions. They criticize the lack of ecological validity in many laboratory-based studies with a focus on artificially controlled and constricted environments. Some researchers think external validity and ecological validity are closely related in the sense that causal inferences based on ecologically valid research designs often allow for higher degrees of generalizability than those obtained in an artificially produced lab environment. However, this again relates to the distinction between generalizing to some population and generalizing across subpopulations that differ on some background factor. Some findings produced in ecologically valid research settings may hardly be generalizable, and some findings produced in highly controlled settings may claim near-universal external validity. Thus, external and ecological validity are independent—a study may possess external validity but not ecological validity, and vice versa.

Qualitative research

Within the qualitative research paradigm, external validity is replaced by the concept of transferability. Transferability is the ability of research results to transfer to situations with similar parameters, populations and characteristics.

In experiments

It is common for researchers to claim that experiments are by their nature low in external validity. Some claim that many drawbacks can occur when following the experimental method. By the virtue of gaining enough control over the situation so as to randomly assign people to conditions and rule out the effects of extraneous variables, the situation can become somewhat artificial and distant from real life.
There are two kinds of generalizability at issue:

The extent to which we can generalize from the situation constructed by an experimenter to real-life situations, and
The extent to which we can generalize from the people who participated in the experiment to people in general

However, both of these considerations pertain to Cook and Campbell's concept of generalizing to some target population rather than the arguably more central task of assessing the generalizability of findings from an experiment across subpopulations that differ from the specific situation studied and people who differ from the respondents studied in some meaningful way.
Critics of experiments suggest that external validity could be improved by the use of field settings and by the use of true probability samples of respondents. However, if one's goal is to understand generalizability across subpopulations that differ in situational or personal background factors, these remedies do not have the efficacy in increasing external validity that is commonly ascribed to them. If background factor X treatment interactions exist of which the researcher is unaware, these research practices can mask a substantial lack of external validity. Dipboye and Flanagan, writing about industrial and organizational psychology, note that the evidence is that findings from one field setting and from one lab setting are equally unlikely to generalize to a second field setting. Thus, field studies are not by their nature high in external validity and laboratory studies are not by their nature low in external validity. It depends in both cases whether the particular treatment effect studied would change with changes in background factors that are held constant in that study. If one's study is "unrealistic" on the level of some background factor that does not interact with the treatments, it has no effect on external validity. It is only if an experiment holds some background factor constant at an unrealistic level and if varying that background factor would have revealed a strong Treatment x Background factor interaction, that external validity is threatened.