Floating-point arithmetic
In computing, floating-point arithmetic is arithmetic using formulaic representation of real numbers as an approximation to support a trade-off between range and precision. For this reason, floating-point computation is often found in systems which include very small and very large real numbers, which require fast processing times. A number is, in general, represented approximately to a fixed number of significant digits and scaled using an exponent in some fixed base; the base for the scaling is normally two, ten, or sixteen. A number that can be represented exactly is of the following form:
where significand is an integer, base is an integer greater than or equal to two, and exponent is also an integer.
For example:
The term floating point refers to the fact that a number's radix point can "float"; that is, it can be placed anywhere relative to the significant digits of the number. This position is indicated as the exponent component, and thus the floating-point representation can be thought of as a kind of scientific notation.
A floating-point system can be used to represent, with a fixed number of digits, numbers of different orders of magnitude: e.g. the distance between galaxies or the diameter of an atomic nucleus can be expressed with the same unit of length. The result of this dynamic range is that the numbers that can be represented are not uniformly spaced; the difference between two consecutive representable numbers varies with the chosen scale.
Over the years, a variety of floating-point representations have been used in computers. In 1985, the IEEE 754 Standard for Floating-Point Arithmetic was established, and since the 1990s, the most commonly encountered representations are those defined by the IEEE.
The speed of floating-point operations, commonly measured in terms of FLOPS, is an important characteristic of a computer system, especially for applications that involve intensive mathematical calculations.
A floating-point unit is a part of a computer system specially designed to carry out operations on floating-point numbers.
Overview
Floating-point numbers
A number representation specifies some way of encoding a number, usually as a string of digits.There are several mechanisms by which strings of digits can represent numbers. In common mathematical notation, the digit string can be of any length, and the location of the radix point is indicated by placing an explicit "point" character there. If the radix point is not specified, then the string implicitly represents an integer and the unstated radix point would be off the right-hand end of the string, next to the least significant digit. In fixed-point systems, a position in the string is specified for the radix point. So a fixed-point scheme might be to use a string of 8 decimal digits with the decimal point in the middle, whereby "00012345" would represent 0001.2345.
In scientific notation, the given number is scaled by a power of 10, so that it lies within a certain range—typically between 1 and 10, with the radix point appearing immediately after the first digit. The scaling factor, as a power of ten, is then indicated separately at the end of the number. For example, the orbital period of Jupiter's moon Io is seconds, a value that would be represented in standard-form scientific notation as seconds.
Floating-point representation is similar in concept to scientific notation. Logically, a floating-point number consists of:
- A signed digit string of a given length in a given base. This digit string is referred to as the significand, mantissa, or coefficient. The length of the significand determines the precision to which numbers can be represented. The radix point position is assumed always to be somewhere within the significand—often just after or just before the most significant digit, or to the right of the rightmost digit. This article generally follows the convention that the radix point is set just after the most significant digit.
- A signed integer exponent, which modifies the magnitude of the number.
Using base-10 as an example, the number, which has ten decimal digits of precision, is represented as the significand together with 5 as the exponent. To determine the actual value, a decimal point is placed after the first digit of the significand and the result is multiplied by to give, or. In storing such a number, the base need not be stored, since it will be the same for the entire range of supported numbers, and can thus be inferred.
Symbolically, this final value is:
where is the significand, is the precision, is the base, and is the exponent.
Historically, several number bases have been used for representing floating-point numbers, with base two being the most common, followed by base ten, and other less common varieties, such as base sixteen, base eight, base four, base three and even base 256 and base.
A floating-point number is a rational number, because it can be represented as one integer divided by another; for example is ×1000 or /100. The base determines the fractions that can be represented; for instance, 1/5 cannot be represented exactly as a floating-point number using a binary base, but 1/5 can be represented exactly using a decimal base. However, 1/3 cannot be represented exactly by either binary or decimal, but in base 3, it is trivial . The occasions on which infinite expansions occur depend on the base and its prime factors.
The way in which the significand and exponent are stored in a computer is implementation-dependent. The common IEEE formats are described in detail later and elsewhere, but as an example, in the binary single-precision floating-point representation,, and so the significand is a string of 24 bits. For instance, the number π's first 33 bits are:
In this binary expansion, let us denote the positions from 0 to 32. The 24-bit significand will stop at position 23, shown as the underlined bit above. The next bit, at position 24, is called the round bit or rounding bit. It is used to round the 33-bit approximation to the nearest 24-bit number. This bit, which is in this example, is added to the integer formed by the leftmost 24 bits, yielding:
When this is stored in memory using the IEEE 754 encoding, this becomes the significand. The significand is assumed to have a binary point to the right of the leftmost bit. So, the binary representation of π is calculated from left-to-right as follows:
where is the precision, is the position of the bit of the significand from the left and is the exponent.
It can be required that the most significant digit of the significand of a non-zero number be non-zero. This process is called normalization. For binary formats, this non-zero digit is necessarily. Therefore, it does not need to be represented in memory; allowing the format to have one more bit of precision. This rule is variously called the leading bit convention, the implicit bit convention, the hidden bit convention, or the assumed bit convention.
Alternatives to floating-point numbers
The floating-point representation is by far the most common way of representing in computers an approximation to real numbers. However, there are alternatives:- Fixed-point representation uses integer hardware operations controlled by a software implementation of a specific convention about the location of the binary or decimal point, for example, 6 bits or digits from the right. The hardware to manipulate these representations is less costly than floating point, and it can be used to perform normal integer operations, too. Binary fixed point is usually used in special-purpose applications on embedded processors that can only do integer arithmetic, but decimal fixed point is common in commercial applications.
- Logarithmic number systems represent a real number by the logarithm of its absolute value and a sign bit. The value distribution is similar to floating point, but the value-to-representation curve is smooth. Conversely to floating-point arithmetic, in a logarithmic number system multiplication, division and exponentiation are simple to implement, but addition and subtraction are complex. The level-index arithmetic of Charles Clenshaw, Frank Olver and Peter Turner is a scheme based on a generalized logarithm representation.
- Tapered floating-point representation, which does not appear to be used in practice.
- Where greater precision is desired, floating-point arithmetic can be implemented with variable-length significands that are sized depending on actual need and depending on how the calculation proceeds. This is called arbitrary-precision floating-point arithmetic.
- Floating-point expansions are another way to get a greater precision, benefiting from the floating-point hardware: a number is represented as an unevaluated sum of several floating-point numbers. An example is double-double arithmetic, sometimes used for the C type
long double
. - Some simple rational numbers cannot be represented exactly in binary floating point, no matter what the precision is. Using a different radix allows one to represent some of them, but the possibilities remain limited. Software packages that perform rational arithmetic represent numbers as fractions with integral numerator and denominator, and can therefore represent any rational number exactly. Such packages generally need to use "bignum" arithmetic for the individual integers.
- Interval arithmetic allows one to represent numbers as intervals and obtain guaranteed bounds on results. It is generally based on other arithmetics, in particular floating point.
- Computer algebra systems such as Mathematica, Maxima, and Maple can often handle irrational numbers like or in a completely "formal" way, without dealing with a specific encoding of the significand. Such a program can evaluate expressions like "" exactly, because it is programmed to process the underlying mathematics directly, instead of using approximate values for each intermediate calculation.
History
In 1938, Konrad Zuse of Berlin completed the Z1, the first binary, programmable mechanical computer; it uses a 24-bit binary floating-point number representation with a 7-bit signed exponent, a 17-bit significand, and a sign bit. The more reliable relay-based Z3, completed in 1941, has representations for both positive and negative infinities; in particular, it implements defined operations with infinity, such as, and it stops on undefined operations, such as.
, architect of the Z3 computer, which uses a 22-bit binary floating-point representation.
Zuse also proposed, but did not complete, carefully rounded floating-point arithmetic that includes and NaN representations, anticipating features of the IEEE Standard by four decades. In contrast, von Neumann recommended against floating-point numbers for the 1951 IAS machine, arguing that fixed-point arithmetic is preferable.
The first commercial computer with floating-point hardware was Zuse's Z4 computer, designed in 1942–1945. In 1946, Bell Laboratories introduced the Mark V, which implemented decimal floating-point numbers.
The Pilot ACE has binary floating-point arithmetic, and it became operational in 1950 at National Physical Laboratory, UK. Thirty-three were later sold commercially as the English Electric DEUCE. The arithmetic is actually implemented in software, but with a one megahertz clock rate, the speed of floating-point and fixed-point operations in this machine were initially faster than those of many competing computers.
The mass-produced IBM 704 followed in 1954; it introduced the use of a biased exponent. For many decades after that, floating-point hardware was typically an optional feature, and computers that had it were said to be "scientific computers", or to have "scientific computation" capability. It was not until the launch of the Intel i486 in 1989 that general-purpose personal computers had floating-point capability in hardware as a standard feature.
The UNIVAC 1100/2200 series, introduced in 1962, supported two floating-point representations:
- Single precision: 36 bits, organized as a 1-bit sign, an 8-bit exponent, and a 27-bit significand.
- Double precision: 72 bits, organized as a 1-bit sign, an 11-bit exponent, and a 60-bit significand.
Initially, computers used many different representations for floating-point numbers. The lack of standardization at the mainframe level was an ongoing problem by the early 1970s for those writing and maintaining higher-level source code; these manufacturer floating-point standards differed in the word sizes, the representations, and the rounding behavior and general accuracy of operations. Floating-point compatibility across multiple computing systems was in desperate need of standardization by the early 1980s, leading to the creation of the IEEE 754 standard once the 32-bit word had become commonplace. This standard was significantly based on a proposal from Intel, which was designing the i8087 numerical coprocessor; Motorola, which was designing the 68000 around the same time, gave significant input as well.
In 1989, mathematician and computer scientist William Kahan was honored with the Turing Award for being the primary architect behind this proposal; he was aided by his student and a visiting professor.
Among the x86 innovations are these:
- A precisely specified floating-point representation at the bit-string level, so that all compliant computers interpret bit patterns the same way. This makes it possible to accurately and efficiently transfer floating-point numbers from one computer to another.
- A precisely specified behavior for the arithmetic operations: A result is required to be produced as if infinitely precise arithmetic were used to yield a value that is then rounded according to specific rules. This means that a compliant computer program would always produce the same result when given a particular input, thus mitigating the almost mystical reputation that floating-point computation had developed for its hitherto seemingly non-deterministic behavior.
- The ability of exceptional conditions to propagate through a computation in a benign manner and then be handled by the software in a controlled fashion.
Range of floating-point numbers
On a typical computer system, a double-precision binary floating-point number has a coefficient of 53 bits, an exponent of 11 bits, and 1 sign bit. Since 2^{10} = 1024, the complete range of the positive normal floating-point numbers in this format is from 2^{−1022} ≈ 2 × 10^{−308} to approximately 2^{1024} ≈ 2 × 10^{308}.
The number of normalized floating-point numbers in a system where
- B is the base of the system,
- P is the precision of the system to P numbers,
- L is the smallest exponent representable in the system,
- and U is the largest exponent used in the system)
There is a smallest positive normalized floating-point number,
which has a 1 as the leading digit and 0 for the remaining digits of the significand, and the smallest possible value for the exponent.
There is a largest floating-point number,
which has B − 1 as the value for each digit of the significand and the largest possible value for the exponent.
In addition, there are representable values strictly between −UFL and UFL. Namely, positive and negative zeros, as well as denormalized numbers.
IEEE 754: floating point in modern computers
The IEEE standardized the computer representation for binary floating-point numbers in IEEE 754 in 1985. This first standard is followed by almost all modern machines. It was revised in 2008. IBM mainframes support IBM's own hexadecimal floating point format and IEEE 754-2008 decimal floating point in addition to the IEEE 754 binary format. The Cray T90 series had an IEEE version, but the SV1 still uses Cray floating-point format.The standard provides for many closely related formats, differing in only a few details. Five of these formats are called basic formats, and others are termed extended precision formats and extendable precision format. Three formats are especially widely used in computer hardware and languages:
- Single precision, usually used to represent the "float" type in the C language family. This is a binary format that occupies 32 bits and its significand has a precision of 24 bits.
- Double precision, usually used to represent the "double" type in the C language family. This is a binary format that occupies 64 bits and its significand has a precision of 53 bits.
- Double extended, also ambiguously called "extended precision" format. This is a binary format that occupies at least 79 bits and its significand has a precision of at least 64 bits. The C99 and C11 standards of the C language family, in their annex F, recommend such an extended format to be provided as "long double". A format satisfying the minimal requirements is provided by the x86 architecture. Often on such processors, this format can be used with "long double", though extended precision is not available with MSVC. For alignment purposes, many tools store this 80-bit value in a 96-bit or 128-bit space. On other processors, "long double" may stand for a larger format, such as quadruple precision, or just double precision, if any form of extended precision is not available.
Less common IEEE formats include:
- . This is a binary format that occupies 128 bits and its significand has a precision of 113 bits.
- Decimal64 and decimal128 floating-point formats. These formats, along with the decimal32 format, are intended for performing decimal rounding correctly.
- Half precision, also called binary16, a 16-bit floating-point value. It is being used in the NVIDIA Cg graphics language, and in the openEXR standard.
The standard specifies some special values, and their representation: positive infinity, negative infinity, a negative zero distinct from ordinary zero, and "not a number" values.
Comparison of floating-point numbers, as defined by the IEEE standard, is a bit different from usual integer comparison. Negative and positive zero compare equal, and every NaN compares unequal to every value, including itself. All values except NaN are strictly smaller than +∞ and strictly greater than −∞. Finite floating-point numbers are ordered in the same way as their values.
Internal representation
Floating-point numbers are typically packed into a computer datum as the sign bit, the exponent field, and the significand or mantissa, from left to right. For the IEEE 754 binary formats which have extant hardware implementations, they are apportioned as follows:Type | Sign | Exponent | Significand field | Total bits | Exponent bias | Bits precision | Number of decimal digits | |
Half | 1 | 5 | 10 | 16 | 15 | 11 | ~3.3 | |
Single | 1 | 8 | 23 | 32 | 127 | 24 | ~7.2 | |
Double | 1 | 11 | 52 | 64 | 1023 | 53 | ~15.9 | |
x86 extended precision | 1 | 15 | 64 | 80 | 16383 | 64 | ~19.2 | |
Quad | 1 | 15 | 112 | 128 | 16383 | 113 | ~34.0 |
While the exponent can be positive or negative, in binary formats it is stored as an unsigned number that has a fixed "bias" added to it. Values of all 0s in this field are reserved for the zeros and subnormal numbers; values of all 1s are reserved for the infinities and NaNs. The exponent range for normalized numbers is for single precision, for double, or for quad. Normalized numbers exclude subnormal values, zeros, infinities, and NaNs.
In the IEEE binary interchange formats the leading 1 bit of a normalized significand is not actually stored in the computer datum. It is called the "hidden" or "implicit" bit. Because of this, single precision format actually has a significand with 24 bits of precision, double precision format has 53, and quad has 113.
For example, it was shown above that π, rounded to 24 bits of precision, has:
- sign = 0 ; e = 1 ; s = 110010010000111111011011
- 0 10000000 10010010000111111011011 = 40490FDB as a hexadecimal number.
and the 64 bit layout is similar.
Special values
Signed zero
In the IEEE 754 standard, zero is signed, meaning that there exist both a "positive zero" and a "negative zero". In most run-time environments, positive zero is usually printed as "0" and the negative zero as "-0". The two values behave as equal in numerical comparisons, but some operations return different results for +0 and −0. For instance, 1/ returns negative infinity, while 1/+0 returns positive infinity. Other common functions with a discontinuity at x=0 which might treat +0 and −0 differently include log, signum, and the principal square root of for any negative number y. As with any approximation scheme, operations involving "negative zero" can occasionally cause confusion. For example, in IEEE 754, x = y does not always imply 1/x = 1/y, as 0 = −0 but 1/0 ≠ 1/−0.Subnormal numbers
Subnormal values fill the underflow gap with values where the absolute distance between them is the same as for adjacent values just outside the underflow gap.This is an improvement over the older practice to just have zero in the underflow gap, and where underflowing results were replaced by zero.
Modern floating-point hardware usually handles subnormal values, and does not require software emulation for subnormals.
Infinities
The infinities of the extended real number line can be represented in IEEE floating-point datatypes, just like ordinary floating-point values like 1, 1.5, etc.They are not error values in any way, though they are often used as replacement values when there is an overflow. Upon a divide-by-zero exception, a positive or negative infinity is returned as an exact result. An infinity can also be introduced as a numeral.
IEEE 754 requires infinities to be handled in a reasonable way, such as
- + =
- × =
- × 0 = NaN – there is no meaningful thing to do
NaNs
The representation of NaNs specified by the standard has some unspecified bits that could be used to encode the type or source of error; but there is no standard for that encoding. In theory, signaling NaNs could be used by a runtime system to flag uninitialized variables, or extend the floating-point numbers with other special values without slowing down the computations with ordinary values, although such extensions are not common.
IEEE 754 design rationale
It is a common misconception that the more esoteric features of the IEEE 754 standard discussed here, such as extended formats, NaN, infinities, subnormals etc., are only of interest to numerical analysts, or for advanced numerical applications; in fact the opposite is true: these features are designed to give safe robust defaults for numerically unsophisticated programmers, in addition to supporting sophisticated numerical libraries by experts. The key designer of IEEE 754, William Kahan notes that it is incorrect to "... features of IEEE Standard 754 for Binary Floating-Point Arithmetic that... not appreciated to be features usable by none but numerical experts. The facts are quite the opposite. In 1977 those features were designed into the Intel 8087 to serve the widest possible market... Error-analysis tells us how to design floating-point arithmetic, like IEEE Standard 754, moderately tolerant of well-meaning ignorance among programmers".- The special values such as infinity and NaN ensure that the floating-point arithmetic is algebraically completed, such that every floating-point operation produces a well-defined result and will not—by default—throw a machine interrupt or trap. Moreover, the choices of special values returned in exceptional cases were designed to give the correct answer in many cases, e.g. continued fractions such as R := 7 − 3/ will give the correct answer in all inputs under IEEE 754 arithmetic as the potential divide by zero in e.g. R = 4.6 is correctly handled as +infinity and so can be safely ignored. As noted by Kahan, the unhandled trap consecutive to a floating-point to 16-bit integer conversion overflow that caused the loss of an Ariane 5 rocket would not have happened under the default IEEE 754 floating-point policy.
- Subnormal numbers ensure that for finite floating-point numbers x and y, x − y = 0 if and only if x = y, as expected, but which did not hold under earlier floating-point representations.
- On the design rationale of the x87 80-bit format, Kahan notes: "This Extended format is designed to be used, with negligible loss of speed, for all but the simplest arithmetic with float and double operands. For example, it should be used for scratch variables in loops that implement recurrences like polynomial evaluation, scalar products, partial and continued fractions. It often averts premature Over/Underflow or severe local cancellation that can spoil simple algorithms". Computing intermediate results in an extended format with high precision and extended exponent has precedents in the historical practice of scientific calculation and in the design of scientific calculators e.g. Hewlett-Packard's financial calculators performed arithmetic and financial functions to three more significant decimals than they stored or displayed. The implementation of extended precision enabled standard elementary function libraries to be readily developed that normally gave double precision results within one unit in the last place at high speed.
- Correct rounding of values to the nearest representable value avoids systematic biases in calculations and slows the growth of errors. Rounding ties to even removes the statistical bias that can occur in adding similar figures.
- Directed rounding was intended as an aid with checking error bounds, for instance in interval arithmetic. It is also used in the implementation of some functions.
- The mathematical basis of the operations enabled high precision multiword arithmetic subroutines to be built relatively easily.
- The single and double precision formats were designed to be easy to sort without using floating-point hardware. Their bits as a two's-complement integer already sort the positives correctly, and the negatives reversed. If that integer is negative, xor with its maximum positive, and the floats are sorted as integers.
Other notable floating-point formats
- The Bfloat16 format requires the same amount of memory as the IEEE 754 half-precision format, but allocates 8 bits to the exponent instead of 5, thus providing the same range as a single-precision IEEE 754 number. The tradeoff is a reduced precision, as the significand field is reduced from 10 to 7 bits. This format is mainly used in the training of machine learning models, where range is more valuable than precision. Many machine learning accelerators provide hardware support for this format.
- The TensorFloat-32 format provides the best of the Bfloat16 and half-precision formats, having 8 bits of exponent as the former and 10 bits of significand field as the latter. This format was introduced by Nvidia, which provides hardware support for it in the Tensor Cores of its GPUs based on the Nvidia Ampere architecture. The drawback of this format is its total size of 19 bits, which is not a power of 2. However, according to Nvidia, this format should only be used internally by hardware to speed up computations, while inputs and outputs should be stored in the 32-bit single-precision IEEE 754 format.
Type | Sign | Exponent | Significand field | Total bits |
Half-precision | 1 | 5 | 10 | 16 |
Bfloat16 | 1 | 8 | 7 | 16 |
TensorFloat-32 | 1 | 8 | 10 | 19 |
Single-precision | 1 | 8 | 23 | 32 |
Representable numbers, conversion and rounding
By their nature, all numbers expressed in floating-point format are rational numbers with a terminating expansion in the relevant base. Irrational numbers, such as π or √2, or non-terminating rational numbers, must be approximated. The number of digits of precision also limits the set of rational numbers that can be represented exactly. For example, the decimal number 123456789 cannot be exactly represented if only eight decimal digits of precision are available.When a number is represented in some format which is not a native floating-point representation supported in a computer implementation, then it will require a conversion before it can be used in that implementation. If the number can be represented exactly in the floating-point format then the conversion is exact. If there is not an exact representation then the conversion requires a choice of which floating-point number to use to represent the original value. The representation chosen will have a different value from the original, and the value thus adjusted is called the rounded value.
Whether or not a rational number has a terminating expansion depends on the base. For example, in base-10 the number 1/2 has a terminating expansion while the number 1/3 does not. In base-2 only rationals with denominators that are powers of 2 are terminating. Any rational with a denominator that has a prime factor other than 2 will have an infinite binary expansion. This means that numbers which appear to be short and exact when written in decimal format may need to be approximated when converted to binary floating-point. For example, the decimal number 0.1 is not representable in binary floating-point of any finite precision; the exact binary representation would have a "1100" sequence continuing endlessly:
where, as previously, s is the significand and e is the exponent.
When rounded to 24 bits this becomes
which is actually 0.100000001490116119384765625 in decimal.
As a further example, the real number π, represented in binary as an infinite sequence of bits is
but is
when approximated by rounding to a precision of 24 bits.
In binary single-precision floating-point, this is represented as s = 1.10010010000111111011011 with e = 1.
This has a decimal value of
whereas a more accurate approximation of the true value of π is
The result of rounding differs from the true value by about 0.03 parts per million, and matches the decimal representation of π in the first 7 digits. The difference is the discretization error and is limited by the machine epsilon.
The arithmetical difference between two consecutive representable floating-point numbers which have the same exponent is called a unit in the last place. For example, if there is no representable number lying between the representable numbers 1.45a70c22_{hex} and 1.45a70c24_{hex}, the ULP is 2×16^{−8}, or 2^{−31}. For numbers with a base-2 exponent part of 0, i.e. numbers with an absolute value higher than or equal to 1 but lower than 2, an ULP is exactly 2^{−23} or about 10^{−7} in single precision, and exactly 2^{−53} or about 10^{−16} in double precision. The mandated behavior of IEEE-compliant hardware is that the result be within one-half of a ULP.
Rounding modes
Rounding is used when the exact result of a floating-point operation would need more digits than there are digits in the significand. IEEE 754 requires correct rounding: that is, the rounded result is as if infinitely precise arithmetic was used to compute the value and then rounded. There are several different rounding schemes. Historically, truncation was the typical approach. Since the introduction of IEEE 754, the default method is more commonly used. This method rounds the ideal result of an arithmetic operation to the nearest representable value, and gives that representation as the result. In the case of a tie, the value that would make the significand end in an even digit is chosen. The IEEE 754 standard requires the same rounding to be applied to all fundamental algebraic operations, including square root and conversions, when there is a numeric result. It means that the results of IEEE 754 operations are completely determined in all bits of the result, except for the representation of NaNs.Alternative rounding options are also available. IEEE 754 specifies the following rounding modes:
- round to nearest, where ties round to the nearest even digit in the required position
- round to nearest, where ties round away from zero
- round up
- round down
- round toward zero
The alternative rounding modes are also useful in diagnosing numerical instability: if the results of a subroutine vary substantially between rounding to + and − infinity then it is likely numerically unstable and affected by round-off error.
Floating-point arithmetic operations
For ease of presentation and understanding, decimal radix with 7 digit precision will be used in the examples, as in the IEEE 754 decimal32 format. The fundamental principles are the same in any radix or precision, except that normalization is optional. Here, s denotes the significand and e denotes the exponent.Addition and subtraction
A simple method to add floating-point numbers is to first represent them with the same exponent. In the example below, the second number is shifted right by three digits, and one then proceeds with the usual addition method:123456.7 = 1.234567 × 10^5
101.7654 = 1.017654 × 10^2 = 0.001017654 × 10^5
Hence:
123456.7 + 101.7654 = +
= +
= × 10^5
= 1.235584654 × 10^5
In detail:
e=5; s=1.234567
+ e=2; s=1.017654
e=5; s=1.234567
+ e=5; s=0.001017654
--------------------
e=5; s=1.235584654
This is the true result, the exact sum of the operands. It will be rounded to seven digits and then normalized if necessary. The final result is
e=5; s=1.235585
The lowest three digits of the second operand are essentially lost. This is round-off error. In extreme cases, the sum of two non-zero numbers may be equal to one of them:
e=5; s=1.234567
+ e=−3; s=9.876543
e=5; s=1.234567
+ e=5; s=0.00000009876543
----------------------
e=5; s=1.23456709876543
e=5; s=1.234567
In the above conceptual examples it would appear that a large number of extra digits would need to be provided by the adder to ensure correct rounding; however, for binary addition or subtraction using careful implementation techniques only two extra guard bits and one extra sticky bit need to be carried beyond the precision of the operands.
Another problem of loss of significance occurs when two nearly equal numbers are subtracted. In the following example e = 5; s = 1.234571 and e = 5; s = 1.234567 are representations of the rationals 123457.1467 and 123456.659.
e=5; s=1.234571
− e=5; s=1.234567
----------------
e=5; s=0.000004
e=−1; s=4.000000
The best representation of this difference is e = −1; s = 4.877000, which differs more than 20% from e = −1; s = 4.000000. In extreme cases, all significant digits of precision can be lost. This cancellation illustrates the danger in assuming that all of the digits of a computed result are meaningful. Dealing with the consequences of these errors is a topic in numerical analysis; see also [|Accuracy problems].
Multiplication and division
To multiply, the significands are multiplied while the exponents are added, and the result is rounded and normalized.e=3; s=4.734612
× e=5; s=5.417242
-----------------------
e=8; s=25.648538980104
e=8; s=25.64854
e=9; s=2.564854
Similarly, division is accomplished by subtracting the divisor's exponent from the dividend's exponent, and dividing the dividend's significand by the divisor's significand.
There are no cancellation or absorption problems with multiplication or division, though small errors may accumulate as operations are performed in succession. In practice, the way these operations are carried out in digital logic can be quite complex.
For a fast, simple method, see the Horner method.
Dealing with exceptional cases
Floating-point computation in a computer can run into three kinds of problems:- An operation can be mathematically undefined, such as ∞/∞, or division by zero.
- An operation can be legal in principle, but not supported by the specific format, for example, calculating the square root of −1 or the inverse sine of 2.
- An operation can be legal in principle, but the result can be impossible to represent in the specified format, because the exponent is too large or too small to encode in the exponent field. Such an event is called an overflow, underflow or denormalization.
of trap that the programmer might be able to catch. How this worked was system-dependent,
meaning that floating-point programs were not portable..
Here, the required default method of handling exceptions according to IEEE 754 is discussed. Arithmetic exceptions are required to be recorded in "sticky" status flag bits. That they are "sticky" means that they are not reset by the next operation, but stay set until explicitly reset. The use of "sticky" flags thus allows for testing of exceptional conditions to be delayed until after a full floating-point expression or subroutine: without them exceptional conditions that could not be otherwise ignored would require explicit testing immediately after every floating-point operation. By default, an operation always returns a result according to specification without interrupting computation. For instance, 1/0 returns +∞, while also setting the divide-by-zero flag bit.
The original IEEE 754 standard, however, failed to recommend operations to handle such sets of arithmetic exception flag bits. So while these were implemented in hardware, initially programming language implementations typically did not provide a means to access them. Over time some programming language standards have been updated to specify methods to access and change status flag bits. The 2008 version of the IEEE 754 standard now specifies a few operations for accessing and handling the arithmetic flag bits. The programming model is based on a single thread of execution and use of them by multiple threads has to be handled by a means outside of the standard.
IEEE 754 specifies five arithmetic exceptions that are to be recorded in the status flags :
- inexact, set if the rounded value is different from the mathematically exact result of the operation.
- underflow, set if the rounded value is tiny and inexact, returning a subnormal value including the zeros.
- overflow, set if the absolute value of the rounded value is too large to be represented. An infinity or maximal finite value is returned, depending on which rounding is used.
- divide-by-zero, set if the result is infinite given finite operands, returning an infinity, either +∞ or −∞.
- invalid, set if a real-valued result cannot be returned e.g. sqrt or 0/0, returning a quiet NaN.
Overflow and invalid exceptions can typically not be ignored, but do not necessarily represent errors: for example, a root-finding routine, as part of its normal operation, may evaluate a passed-in function at values outside of its domain, returning NaN and an invalid exception flag to be ignored until finding a useful start point.
Accuracy problems
The fact that floating-point numbers cannot precisely represent all real numbers, and that floating-point operations cannot precisely represent true arithmetic operations, leads to many surprising situations. This is related to the finite precision with which computers generally represent numbers.For example, the non-representability of 0.1 and 0.01 means that the result of attempting to square 0.1 is neither 0.01 nor the representable number closest to it. In 24-bit representation, 0.1 was given previously as e = −4; s = 110011001100110011001101, which is
Squaring this number gives
Squaring it with single-precision floating-point hardware gives
But the representable number closest to 0.01 is
Also, the non-representability of π means that an attempted computation of tan will not yield a result of infinity, nor will it even overflow. It is simply not possible for standard floating-point hardware to attempt to compute tan, because π/2 cannot be represented exactly. This computation in C:
/* Enough digits to be sure we get the correct approximation. */
double pi = 3.1415926535897932384626433832795;
double z = tan;
will give a result of 16331239353195370.0. In single precision, the result will be −22877332.0.
By the same token, an attempted computation of sin will not yield zero. The result will be 0.1225 in double precision, or −0.8742 in single precision.
While floating-point addition and multiplication are both commutative, they are not necessarily associative. That is, + c is not necessarily equal to a +. Using 7-digit significand decimal arithmetic:
a = 1234.567, b = 45.67834, c = 0.0004
+ c:
1234.567
+ 45.67834
____________
1280.24534 rounds to 1280.245
1280.245
+ 0.0004
____________
1280.2454 rounds to 1280.245 ← + c
a + :
45.67834
+ 0.0004
____________
45.67874
1234.567
+ 45.67874
____________
1280.24574 rounds to 1280.246 ← a +
They are also not necessarily distributive. That is, × c may not be the same as a × c + b × c:
1234.567 × 3.333333 = 4115.223
1.234567 × 3.333333 = 4.115223
4115.223 + 4.115223 = 4119.338
but
1234.567 + 1.234567 = 1235.802
1235.802 × 3.333333 = 4119.340
In addition to loss of significance, inability to represent numbers such as π and 0.1 exactly, and other slight inaccuracies, the following phenomena may occur:
- Cancellation: subtraction of nearly equal operands may cause extreme loss of accuracy. When we subtract two almost equal numbers we set the most significant digits to zero, leaving ourselves with just the insignificant, and most erroneous, digits. For example, when determining a derivative of a function the following formula is used:
- Conversions to integer are not intuitive: converting to integer yields 7, but converting may yield 6. This is because conversions generally truncate rather than round. Floor and ceiling functions may produce answers which are off by one from the intuitively expected value.
- Limited exponent range: results might overflow yielding infinity, or underflow yielding a subnormal number or zero. In these cases precision will be lost.
- Testing for safe division is problematic: Checking that the divisor is not zero does not guarantee that a division will not overflow.
- Testing for equality is problematic. Two computational sequences that are mathematically equal may well produce different floating-point values.
Incidents
- On 25 February 1991, a loss of significance in a MIM-104 Patriot missile battery prevented it from intercepting an incoming Scud missile in Dhahran, Saudi Arabia, contributing to the death of 28 soldiers from the U.S. Army's 14th Quartermaster Detachment.
Machine precision and backward error analysis
With rounding to zero,
whereas rounding to nearest,
This is important since it bounds the relative error in representing any non-zero real number x within the normalized range of a floating-point system:
Backward error analysis, the theory of which was developed and popularized by James H. Wilkinson, can be used to establish that an algorithm implementing a numerical function is numerically stable. The basic approach is to show that although the calculated result, due to roundoff errors, will not be exactly correct, it is the exact solution to a nearby problem with slightly perturbed input data. If the perturbation required is small, on the order of the uncertainty in the input data, then the results are in some sense as accurate as the data "deserves". The algorithm is then defined as backward stable. Stability is a measure of the sensitivity to rounding errors of a given numerical procedure; by contrast, the condition number of a function for a given problem indicates the inherent sensitivity of the function to small perturbations in its input and is independent of the implementation used to solve the problem.
As a trivial example, consider a simple expression giving the inner product of vectors and, then
and so
where
where
by definition, which is the sum of two slightly perturbed input data, and so is backward stable. For more realistic examples in numerical linear algebra, see Higham 2002 and other references below.
Minimizing the effect of accuracy problems
Although, as noted previously, individual arithmetic operations of IEEE 754 are guaranteed accurate to within half a ULP, more complicated formulae can suffer from larger errors due to round-off. The loss of accuracy can be substantial if a problem or its data are ill-conditioned, meaning that the correct result is hypersensitive to tiny perturbations in its data. However, even functions that are well-conditioned can suffer from large loss of accuracy if an algorithm numerically unstable for that data is used: apparently equivalent formulations of expressions in a programming language can differ markedly in their numerical stability. One approach to remove the risk of such loss of accuracy is the design and analysis of numerically stable algorithms, which is an aim of the branch of mathematics known as numerical analysis. Another approach that can protect against the risk of numerical instabilities is the computation of intermediate values in an algorithm at a higher precision than the final result requires, which can remove, or reduce by orders of magnitude, such risk: IEEE 754 quadruple precision and extended precision are designed for this purpose when computing at double precision.For example, the following algorithm is a direct implementation to compute the function A = / which is well-conditioned at 1.0, however it can be shown to be numerically unstable and lose up to half the significant digits carried by the arithmetic when computed near 1.0.
double A
If, however, intermediate computations are all performed in extended precision, then up to full precision in the final double result can be maintained. Alternatively, a numerical analysis of the algorithm reveals that if the following non-obvious change to line is made:
if Z = log/;
then the algorithm becomes numerically stable and can compute to full double precision.
To maintain the properties of such carefully constructed numerically stable programs, careful handling by the compiler is required. Certain "optimizations" that compilers might make can work against the goals of well-behaved software. There is some controversy about the failings of compilers and language designs in this area: C99 is an example of a language where such optimizations are carefully specified to maintain numerical precision. See the external references at the bottom of this article.
A detailed treatment of the techniques for writing high-quality floating-point software is beyond the scope of this article, and the reader is referred to, and the other references at the bottom of this article. Kahan suggests several rules of thumb that can substantially decrease by orders of magnitude the risk of numerical anomalies, in addition to, or in lieu of, a more careful numerical analysis. These include: as noted above, computing all expressions and intermediate results in the highest precision supported in hardware ; and rounding input data and results to only the precision required and supported by the input data. Brief descriptions of several additional issues and techniques follow.
As decimal fractions can often not be exactly represented in binary floating-point, such arithmetic is at its best when it is simply being used to measure real-world quantities over a wide range of scales, and at its worst when it is expected to model the interactions of quantities expressed as decimal strings that are expected to be exact. An example of the latter case is financial calculations. For this reason, financial software tends not to use a binary floating-point number representation. The "decimal" data type of the C# and Python programming languages, and the decimal formats of the IEEE 754-2008 standard, are designed to avoid the problems of binary floating-point representations when applied to human-entered exact decimal values, and make the arithmetic always behave as expected when numbers are printed in decimal.
Expectations from mathematics may not be realized in the field of floating-point computation. For example, it is known that, and that, however these facts cannot be relied on when the quantities involved are the result of floating-point computation.
The use of the equality test requires care when dealing with floating-point numbers. Even simple expressions like
0.6/0.2-30
will, on most computers, fail to be true. Consequently, such tests are sometimes replaced with "fuzzy" comparisons. The wisdom of doing this varies greatly, and can require numerical analysis to bound epsilon. Values derived from the primary data representation and their comparisons should be performed in a wider, extended, precision to minimize the risk of such inconsistencies due to round-off errors. It is often better to organize the code in such a way that such tests are unnecessary. For example, in computational geometry, exact tests of whether a point lies off or on a line or plane defined by other points can be performed using adaptive precision or exact arithmetic methods.Small errors in floating-point arithmetic can grow when mathematical algorithms perform operations an enormous number of times. A few examples are matrix inversion, eigenvector computation, and differential equation solving. These algorithms must be very carefully designed, using numerical approaches such as Iterative refinement, if they are to work well.
Summation of a vector of floating-point values is a basic algorithm in scientific computing, and so an awareness of when loss of significance can occur is essential. For example, if one is adding a very large number of numbers, the individual addends are very small compared with the sum. This can lead to loss of significance. A typical addition would then be something like
3253.671
+ 3.141276
-----------
3256.812
The low 3 digits of the addends are effectively lost. Suppose, for example, that one needs to add many numbers, all approximately equal to 3. After 1000 of them have been added, the running sum is about 3000; the lost digits are not regained. The Kahan summation algorithm may be used to reduce the errors.
Round-off error can affect the convergence and accuracy of iterative numerical procedures. As an example, Archimedes approximated π by calculating the perimeters of polygons inscribing and circumscribing a circle, starting with hexagons, and successively doubling the number of sides. As noted above, computations may be rearranged in a way that is mathematically equivalent but less prone to error.
Two forms of the recurrence formula for the circumscribed polygon are:
Here is a computation using IEEE "double" arithmetic:
i 6 × 2^{i} × t_{i}, first form 6 × 2^{i} × t_{i}, second form
---------------------------------------------------------
0 '.4641016151377543863 '.4641016151377543863
1 '.2153903091734710173 '.2153903091734723496
2 '596599420974940120 '596599420975006733
3 '60862151314012979 '60862151314352708
4 '27145996453136334 '27145996453689225
5 '8730499801259536 '8730499798241950
6 '6627470548084133 '6627470568494473
7 '6101765997805905 '6101766046906629
8 '70343230776862 '70343215275928
9 '37488171150615 '37487713536668
10 '9278733740748 '9273850979885
11 '7256228504127 '7220386148377
12 '717412858693 '707019992125
13 '189011456060 '78678454728
14 '717412858693 '46593073709
15 '19358822321783 '8571730119
16 '717412858693 '6566394222
17 '810075796233302 '6065061913
18 '717412858693 '939728836
19 '4061547378810956 '908393901
20 '05434924008406305 '900560168
21 '00068646912273617 '8608396
22 '349453756585929919 '8122118
23 '00068646912273617 '95552
24 '.2245152435345525443 '68907
25 '62246
26 '62246
27 '62246
28 '62246
The true value is
While the two forms of the recurrence formula are clearly mathematically equivalent, the first subtracts 1 from a number extremely close to 1, leading to an increasingly problematic loss of significant digits. As the recurrence is applied repeatedly, the accuracy improves at first, but then it deteriorates. It never gets better than about 8 digits, even though 53-bit arithmetic should be capable of about 16 digits of precision. When the second form of the recurrence is used, the value converges to 15 digits of precision.