Selection limits
A selection limit is a term from animal breeding and quantitative genetics that refers to a cessation of progress even when continued directional selection is being applied to a trait, such as body size. In other words, a breeder or scientist is using selective breeding and choosing individuals as breeders within a population based on some phenotypic trait or traits. If this is done, then the average value of the population typically evolves across generations in the direction being favored by selection, but then at some point the population stops evolving. The trait under selection is then said to have reached a limit or plateau at that value.
Details
The existence of limits in artificial selection experiments was discussed in the scientific literature in the 1940s or earlier. The most obvious possible cause of reaching a limit when a population is under continued directional selection is that all of the additive-genetic variation related to that trait gets "used up" or fixed. For example, if a trait, such as body mass, is under selection to increase, then, over time, the alleles at all loci that tend to make individuals larger than average will increase in frequency, while those that tend to make an individual smaller than average will decrease in frequency. Eventually, in principle, the favored alleles at all relevant loci will become the only ones remaining at those loci. In reality, mutation, random genetic drift, and gene flow from immigrants may stop some loci from becoming fixed for the "good" alleles.However, other factors may interfere with the realization of genetic gains before loss of genetic variation causes a selection limit. As noted by Lerner and Dempster, these factors are generally one of two types: 1) negative relations with Darwinian fitness; 2) non-additive gene action and/or genotype-environment interaction.
A negative relation with Darwinian fitness is a situation in which an allele that is "good" for the trait under directional selection is "bad" with respect to lifetime reproductive success. For example, an allele that tends to confer larger body size might also lead to infertility, thus reducing the ability of individuals with that allele to produce offspring, limiting further response to selection, and sometimes even leading to extinction of the selected line.
Non-additive gene action refers to such situations as heterozygote advantage, where heterozygous individuals have higher values for a trait than do either of the two homozygotes. In such a case, selection will tend to maintain more than one allele in the population, and a selection limit may be reached while additive-genetic variation remains for the trait under directional selection.
Genotype-environment interaction occurs when the phenotype produced by a particular set of alleles confers relatively higher or lower values of a trait depending on the environmental circumstances in which an individual is born or raised, or under which the trait is measured. For instance, somewhat different genes tend to give the highest value of a trait depending on the season. If this occurs, then directional selection will act to favor some genes in winter and others in summer, for example. Again, the result may be that a selection plateau is attained while the population retains some additive-genetic variance for the trait under directional selection.
Some traits have a natural physical limit beyond which a trait cannot possibly go. For example, replicated selection for the building of small thermoregulatory nests in mice reached a limit at or near zero. Similarly, lines of maize selected for low oil or protein content in the kernels reached limits near to zero percent.
Aside from absolute physical limits, and whatever their cause, limits or plateaus have often been observed in artificial selection experiments with animals, including: bristle number in fruit flies ; avoidance behavior in laboratory rats; and large body size, large litter size, large nest size, and high voluntary wheel-running behavior in laboratory house mice.