Independence (probability theory)
Independence is a fundamental notion in probability theory, as in statistics and the theory of stochastic processes. Two events are independent, statistically independent, or stochastically independent if, informally speaking, the occurrence of one does not affect the probability of occurrence of the other or, equivalently, does not affect the odds. Similarly, two random variables are independent if the realization of one does not affect the probability distribution of the other. Conversely, dependence is when the occurrence of one event affect the likelihood of another.
When dealing with collections of more than two events, two notions of independence need to be distinguished. The events are called pairwise independent if any two events in the collection are independent of each other, while mutual independence of events means, informally speaking, that each event is independent of any combination of other events in the collection. A similar notion exists for collections of random variables. Mutual independence implies pairwise independence, but not the other way around. In the standard literature of probability theory, statistics, and stochastic processes, independence without further qualification usually refers to mutual independence.
Definition
For events
Two events
Two events and are independent if and only if their joint probability equals the product of their probabilities:indicates that two independent events and have common elements in their sample space so that they are not mutually exclusive. Why this defines independence is made clear by rewriting with conditional probabilities as the probability at which the event occurs provided that the event has or is assumed to have occurred:
and similarly
Thus, the occurrence of does not affect the probability of, and vice versa. In other words, and are independent of each other. Although the derived expressions may seem more intuitive, they are not the preferred definition, as the conditional probabilities may be undefined if or are 0. Furthermore, the preferred definition makes clear by symmetry that when is independent of, is also independent of.
Odds
Stated in terms of odds, two events are independent if and only if the odds ratio of and is unity. Analogously with probability, this is equivalent to the conditional odds being equal to the unconditional odds:or to the odds of one event, given the other event, being the same as the odds of the event, given the other event not occurring:
The odds ratio can be defined as
or symmetrically for odds of given, and thus is 1 if and only if the events are independent.
More than two events
A finite set of events is pairwise independent if every pair of events is independent—that is, if and only if for all distinct pairs of indices,A finite set of events is mutually independent if every event is independent of any intersection of the other events—that is, if and only if for every and for every k indices,
This is called the multiplication rule for independent events. It is [|not a single condition] involving only the product of all the probabilities of all single events; it must hold true for all subsets of events.
For more than two events, a mutually independent set of events is pairwise independent; but the converse is [|not necessarily true].
Log probability and information content
Stated in terms of log probability, two events are independent if and only if the log probability of the joint event is the sum of the log probability of the individual events:In information theory, negative log probability is interpreted as information content, and thus two events are independent if and only if the information content of the combined event equals the sum of information content of the individual events:
See for details.
For real valued random variables
Two random variables
Two random variables and are independent if and only if the elements of the -system generated by them are independent; that is to say, for every and, the events and are independent events. That is, and with cumulative distribution functions and, are independent iff the combined random variable has a joint cumulative distribution functionor equivalently, if the probability densities and and the joint probability density exist,
More than two random variables
A finite set of random variables is pairwise independent if and only if every pair of random variables is independent. Even if the set of random variables is pairwise independent, it is not necessarily mutually independent as defined next.A finite set of random variables is mutually independent if and only if for any sequence of numbers, the events are mutually independent events. This is equivalent to the following condition on the joint cumulative distribution function A finite set of random variables is mutually independent if and only if
It is not necessary here to require that the probability distribution factorizes for all possible subsets as in the case for events. This is not required because e.g. implies.
The measure-theoretically inclined reader may prefer to substitute events for events in the above definition, where is any Borel set. That definition is exactly equivalent to the one above when the values of the random variables are real numbers. It has the advantage of working also for complex-valued random variables or for random variables taking values in any measurable space.
For real valued random vectors
Two random vectors and are called independent ifwhere and denote the cumulative distribution functions of and and denotes their joint cumulative distribution function. Independence of and is often denoted by.
Written component-wise, and are called independent if
For stochastic processes
For one stochastic process
The definition of independence may be extended from random vectors to a stochastic process. Therefore, it is required for an independent stochastic process that the random variables obtained by sampling the process at any times are independent random variables for any.Formally, a stochastic process is called independent, if and only if for all and for all
where Independence of a stochastic process is a property within a stochastic process, not between two stochastic processes.
For two stochastic processes
Independence of two stochastic processes is a property between two stochastic processes and that are defined on the same probability space. Formally, two stochastic processes and are said to be independent if for all and for all, the random vectors and are independent, i.e. ifIndependent σ-algebras
The definitions above are both generalized by the following definition of independence for σ-algebras. Let be a probability space and let and be two sub-σ-algebras of. and are said to be independent if, whenever and,Likewise, a finite family of σ-algebras, where is an index set, is said to be independent if and only if
and an infinite family of σ-algebras is said to be independent if all its finite subfamilies are independent.
The new definition relates to the previous ones very directly:
- Two events are independent if and only if the σ-algebras that they generate are independent. The σ-algebra generated by an event is, by definition,
- Two random variables and defined over are independent if and only if the σ-algebras that they generate are independent. The σ-algebra generated by a random variable taking values in some measurable space consists, by definition, of all subsets of of the form, where is any measurable subset of.
Properties
Self-independence
Note that an event is independent of itself if and only ifThus an event is independent of itself if and only if it almost surely occurs or its complement almost surely occurs; this fact is useful when proving zero–one laws.
Similarly, a random variable is independent of itself if and only if it is almost surely constant.
Expectation and covariance
If and are statistically independent random variables, then the expectation operator has the propertyand the covariance is zero, as follows from
The converse does not hold: if two random variables have a covariance of 0 they still may be not independent.
Similarly for two stochastic processes and : If they are independent, then they are uncorrelated.
Characteristic function
Two random variables and are independent if and only if the characteristic function of the random vector satisfiesIn particular the characteristic function of their sum is the product of their marginal characteristic functions:
though the reverse implication is not true. Random variables that satisfy the latter condition are called subindependent.