Continuous Bernoulli distribution
In probability theory, statistics, and machine learning, the continuous Bernoulli distribution is a family of continuous probability distributions parameterized by a single shape parameter, defined on the unit interval, by:
The continuous Bernoulli distribution arises in deep learning and computer vision, specifically in the context of variational autoencoders, for modeling the pixel intensities of natural images. As such, it defines a proper probabilistic counterpart for the commonly used binary cross entropy loss, which is often applied to continuous, -valued data. This practice amounts to ignoring the normalizing constant of the continuous Bernoulli distribution, since the binary cross entropy loss only defines a true log-likelihood for discrete, -valued data.
The continuous Bernoulli also defines an exponential family of distributions. Writing for the natural parameter, the density can be rewritten in canonical form:
.
Statistical inference
Given an independent sample of points with from continuous Bernoulli, the log-likelihood of the natural parameter isand the maximum likelihood estimator of the natural parameter is the solution of, that is, satisfies
where the left hand side is the expected value of continuous Bernoulli with parameter. Although does not admit a closed-form expression, it can be easily calculated with numerical inversion.
Further properties
The entropy of a continuous Bernoulli distribution isRelated distributions
Bernoulli distribution
The continuous Bernoulli can be thought of as a continuous relaxation of the Bernoulli distribution, which is defined on the discrete set by the probability mass function:where is a scalar parameter between 0 and 1. Applying this same functional form on the continuous interval results in the continuous Bernoulli probability density function, up to a normalizing constant.