SHA-3


SHA-3 is the latest member of the Secure Hash Algorithm family of standards, released by NIST on August 5, 2015. Although part of the same series of standards, SHA-3 is internally different from the MD5-like structure of SHA-1 and SHA-2.
SHA-3 is a subset of the broader cryptographic primitive family Keccak, designed by Guido Bertoni, Joan Daemen, Michaël Peeters, and Gilles Van Assche, building upon RadioGatún. Keccak's authors have proposed additional uses for the function, not standardized by NIST, including a stream cipher, an authenticated encryption system, a "tree" hashing scheme for faster hashing on certain architectures, and AEAD ciphers Keyak and Ketje.
Keccak is based on a novel approach called sponge construction. Sponge construction is based on a wide random function or random permutation, and allows inputting any amount of data, and outputting any amount of data, while acting as a pseudorandom function with regard to all previous inputs. This leads to great flexibility.
NIST does not plan to withdraw SHA-2 or remove it from the revised Secure Hash Standard. The purpose of SHA-3 is that it can be directly substituted for SHA-2 in current applications if necessary, and to significantly improve the robustness of NIST's overall hash algorithm toolkit.
For small message sizes, the creators of the Keccak algorithms and the SHA-3 functions suggest using the faster function KangarooTwelve with adjusted parameters and a new tree hashing mode without extra overhead.

History

The Keccak algorithm is the work of Guido Bertoni, Joan Daemen, Michaël Peeters, and Gilles Van Assche. It is based on earlier hash function designs PANAMA and RadioGatún. PANAMA was designed by Daemen and Craig Clapp in 1998. RadioGatún, a successor of PANAMA, was designed by Daemen, Peeters, and Van Assche, and was presented at the NIST Hash Workshop in 2006. The reference implementation was released to the public domain.
In 2006, NIST started to organize the NIST hash function competition to create a new hash standard, SHA-3. SHA-3 is not meant to replace SHA-2, as no significant attack on SHA-2 has been publicly demonstrated. Because of the successful attacks on MD5, SHA-0 and SHA-1,
NIST perceived a need for an alternative, dissimilar cryptographic hash, which became SHA-3.
After a setup period, admissions were to be submitted by the end of 2008. Keccak was accepted as one of the 51 candidates. In July 2009, 14 algorithms were selected for the second round. Keccak advanced to the last round in December 2010.
During the competition, entrants were permitted to "tweak" their algorithms to address issues that were discovered. Changes that have been made to Keccak are:
  • The number of rounds was increased from to to be more conservative about security.
  • The message padding was changed from a more complex scheme to the simple 10*1 pattern described below.
  • The rate r was increased to the security limit, rather than rounding down to the nearest power of 2.
On October 2, 2012, Keccak was selected as the winner of the competition.
In 2014, the NIST published a draft FIPS 202 "SHA-3 Standard: Permutation-Based Hash and Extendable-Output Functions". FIPS 202 was approved on August 5, 2015.
On August 5, 2015, NIST announced that SHA-3 had become a hashing standard.

Weakening controversy

In early 2013 NIST announced they would select different values for the "capacity", the overall strength vs. speed parameter, for the SHA-3 standard, compared to the submission. The changes caused some turmoil.
The hash function competition called for hash functions at least as secure as the SHA-2 instances. It means that a d-bit output should have d/2-bit resistance to collision attacks and d-bit resistance to preimage attacks, the maximum achievable for d bits of output. Keccak's security proof allows an adjustable level of security based on a "capacity" c, providing c/2-bit resistance to both collision and preimage attacks. To meet the original competition rules, Keccak's authors proposed. The announced change was to accept the same d/2-bit security for all forms of attack and standardize. This would have sped up Keccak by allowing an additional d bits of input to be hashed each iteration. However, the hash functions would not have been drop-in replacements with the same preimage resistance as SHA-2 any more; it would have been cut in half, making it vulnerable to advances in quantum computing, which effectively would cut it in half once more.
In September 2013, Daniel J. Bernstein suggested on the NIST hash-forum mailing list to strengthen the security to the 576-bit capacity that was originally proposed as the default Keccak, in addition to and not included in the SHA-3 specifications. This would have provided at least a SHA3-224 and SHA3-256 with the same preimage resistance as their SHA-2 predecessors, but SHA3-384 and SHA3-512 would have had significantly less preimage resistance than their SHA-2 predecessors. In late September, the Keccak team responded by stating that they had proposed 128-bit security by setting as an option already in their SHA-3 proposal. Although the reduced capacity was justifiable in their opinion, in the light of the negative response, they proposed raising the capacity to bits for all instances. This would be as much as any previous standard up to the 256-bit security level, while providing reasonable efficiency, but not the 384-/512-bit preimage resistance offered by SHA2-384 and SHA2-512. The authors stated that "claiming or relying on security strength levels above 256 bits is meaningless".
In early October 2013, Bruce Schneier criticized NIST's decision on the basis of its possible detrimental effects on the acceptance of the algorithm, saying:
He later retracted his earlier statement, saying:
Paul Crowley, a cryptographer and senior developer at an independent software development company, expressed his support of the decision, saying that Keccak is supposed to be tunable and there is no reason for different security levels within one primitive. He also added:
There was some confusion that internal changes may have been made to Keccak, which were cleared up by the original team, stating that NIST's proposal for SHA-3 is a subset of the Keccak family, for which one can generate test vectors using their reference code submitted to the contest, and that this proposal was the result of a series of discussions between them and the NIST hash team.
In response to the controversy, in November 2013 John Kelsey of NIST proposed to go back to the original proposal for all SHA-2 drop-in replacement instances. The reversion was confirmed in subsequent drafts and in the final release.

Design

SHA-3 uses the sponge construction, in which data is "absorbed" into the sponge, then the result is "squeezed" out. In the absorbing phase, message blocks are XORed into a subset of the state, which is then transformed as a whole using a permutation function . In the "squeeze" phase, output blocks are read from the same subset of the state, alternated with the state transformation function. The size of the part of the state that is written and read is called the "rate", and the size of the part that is untouched by input/output is called the "capacity". The capacity determines the security of the scheme. The maximum security level is half the capacity.
Given an input bit string, a padding function, a permutation function that operates on bit blocks of width, a rate and an output length, we have capacity and the sponge construction. This yields a bit string of length as follows:
  • pad the input N using the pad function, yielding a padded bit string P with a length divisible by
  • break P into n consecutive r-bit pieces P0,..., Pn−1
  • initialize the state S to a string of b zero bits
  • absorb the input into the state: for each block Pi:
  • * extend Pi at the end by a string of c zero bits, yielding one of length b
  • * XOR that with S
  • * apply the block permutation f to the result, yielding a new state S
  • initialize Z to be the empty string
  • while the length of Z is less than d:
  • * append the first r bits of S to Z
  • * if Z is still less than d bits long, apply f to S, yielding a new state S
  • truncate Z to d bits
The fact that the internal state S contains c additional bits of information in addition to what is output to Z prevents the length extension attacks that SHA-2, SHA-1, MD5 and other hashes based on the Merkle–Damgård construction are susceptible to.
In SHA-3, the state S consists of a array of w-bit words, b = 5 × 5 × w = 5 × 5 × 64 = 1600 bits total. Keccak is also defined for smaller power-of-2 word sizes w down to 1 bit. Small state sizes can be used to test cryptanalytic attacks, and intermediate state sizes can be used in practical, lightweight applications.
For SHA3-224, SHA3-256, SHA3-384, and SHA3-512 instances, r is greater than d, so there is no need for additional block permutations in the squeezing phase; the leading d bits of the state are the desired hash. However, SHAKE128 and SHAKE256 allow an arbitrary output length, which is useful in applications such as optimal asymmetric encryption padding.

Padding

To ensure the message can be evenly divided into r-bit blocks, padding is required. SHA-3 uses the pattern 10...01 in its padding function: a 1 bit, followed by zero or more 0 bits and a final 1 bit.
The maximum of zero bits occurs when the last message block is bits long. Then another block is added after the initial 1 bit, containing zero bits before the final 1 bit.
The two 1 bits will be added even if the length of the message is already divisible by r. In this case, another block is added to the message, containing a 1 bit, followed by a block of zero bits and another 1 bit. This is necessary so that a message with length divisible by r ending in something that looks like padding does not produce the same hash as the message with those bits removed.
The initial 1 bit is required so messages differing only in a few additional 0 bits at the end do not produce the same hash.
The position of the final 1 bit indicates which rate r was used, which is required for the security proof to work for different hash variants. Without it, different hash variants of the same short message would be the same up to truncation.