High Efficiency Video Coding
High Efficiency Video Coding, also known as H.265 and MPEG-H Part 2, is a proprietary video compression standard designed as part of the MPEG-H project as a successor to the widely used Advanced Video Coding. The standard was published in 2013. In comparison to AVC, HEVC offers from 25% to 50% better data compression at the same level of video quality, or substantially improved video quality at the same bit rate. It supports resolutions up to 8192×4320, including 8K UHD, and unlike the primarily eight-bit AVC, HEVC's higher-fidelity Main 10 profile has been incorporated into nearly all supporting hardware. The High Efficiency Image Format is a container format whose default codec is HEVC.
While AVC uses the integer discrete cosine transform with 4×4 and 8×8 block sizes, HEVC uses both integer DCT and discrete sine transform with varied block sizes between 4×4 and 32×32.
Concept
In most ways, HEVC is an extension of the concepts in H.264/MPEG-4 AVC. Both work by comparing different parts of a frame of video to find areas that are redundant, both within a single frame and between consecutive frames. These redundant areas are then replaced with a short description instead of the original pixels. The primary changes for HEVC include the expansion of the pattern comparison and difference-coding areas from 16×16 pixel to sizes up to 64×64, improved variable-block-size segmentation, improved "intra" prediction within the same picture, improved motion vector prediction and motion region merging, improved motion compensation filtering, and an additional filtering step called sample-adaptive offset filtering. Effective use of these improvements requires much more signal processing capability for compressing the video but has less impact on the amount of computation needed for decompression.HEVC was standardized by the Joint Collaborative Team on Video Coding, a collaboration between the ISO/IEC MPEG and ITU-T Study Group 16 VCEG. The ISO/IEC group refers to it as MPEG-H Part 2 and the ITU-T as H.265. The first version of the HEVC standard was ratified in January 2013 and published in June 2013. The second version, with multiview extensions, range extensions, and scalability extensions, was completed and approved in 2014 and published in early 2015. Extensions for 3D video were completed in early 2015, and extensions for screen content coding were completed in early 2016 and published in early 2017, covering video containing rendered graphics, text, or animation as well as camera-captured video scenes. In October 2017, the standard was recognized by a Primetime Emmy Engineering Award
as having had a material effect on the technology of television.
HEVC contains technologies covered by patents owned by the organizations that participated in the JCT-VC. Implementing a device or software application that uses HEVC may require a license from HEVC patent holders. The ISO/IEC and ITU require companies that belong to their organizations to offer their patents on reasonable and non-discriminatory licensing terms. Patent licenses can be obtained directly from each patent holder, or through patent licensing bodies, such as MPEG LA, Access Advance, and Velos Media.
The combined licensing fees currently offered by all of the patent licensing bodies are higher than for AVC. The licensing fees are one of the main reasons HEVC adoption has been low on the web and is why some of the largest tech companies have joined the Alliance for Open Media, which finalized royalty-free alternative video coding format AV1 on March 28, 2018.
History
The HEVC format was jointly developed by more than a dozen organisations across the world. The majority of active patent contributions towards the development of the HEVC format came from five organizations: Samsung Electronics, General Electric, M&K Holdings, NTT, and JVC Kenwood. Other patent holders include Fujitsu, Apple, Canon, Columbia University, KAIST, Kwangwoon University, MIT, Sungkyunkwan University, Funai, Hikvision, KBS, KT and NEC.Previous work
In 2004, the ITU-T Video Coding Experts Group began a major study of technology advances that could enable the creation of a new video compression standard. In October 2004, various techniques for potential enhancement of the H.264/MPEG-4 AVC standard were surveyed. In January 2005, at the next meeting of VCEG, VCEG began designating certain topics as "Key Technical Areas" for further investigation. A software codebase called the KTA codebase was established for evaluating such proposals. The KTA software was based on the Joint Model reference software that was developed by the MPEG & VCEG Joint Video Team for H.264/MPEG-4 AVC. Additional proposed technologies were integrated into the KTA software and tested in experiment evaluations over the next four years.Two approaches for standardizing enhanced compression technology were considered: either creating a new standard or creating extensions of H.264/MPEG-4 AVC. The project had tentative names H.265 and H.NGVC, and was a major part of the work of VCEG until it evolved into the HEVC joint project with MPEG in 2010.
The preliminary requirements for NGVC were the capability to have a bit rate reduction of 50% at the same subjective image quality compared with the H.264/MPEG-4 AVC High profile, and computational complexity ranging from 1/2 to 3 times that of the High profile. NGVC would be able to provide 25% bit rate reduction along with 50% reduction in complexity at the same perceived video quality as the High profile, or to provide greater bit rate reduction with somewhat higher complexity.
The ISO/IEC Moving Picture Experts Group started a similar project in 2007, tentatively named High-performance Video Coding. An agreement of getting a bit rate reduction of 50% had been decided as the goal of the project by July 2007. Early evaluations were performed with modifications of the KTA reference software encoder developed by VCEG. By July 2009, experimental results showed average bit reduction of around 20% compared with AVC High Profile; these results prompted MPEG to initiate its standardization effort in collaboration with VCEG.
Joint Collaborative Team on Video Coding
MPEG and VCEG established a Joint Collaborative Team on Video Coding to develop the HEVC standard.Standardization
A formal joint Call for Proposals on video compression technology was issued in January 2010 by VCEG and MPEG, and proposals were evaluated at the first meeting of the MPEG & VCEG Joint Collaborative Team on Video Coding, which took place in April 2010. A total of 27 full proposals were submitted. Evaluations showed that some proposals could reach the same visual quality as AVC at only half the bit rate in many of the test cases, at the cost of 2–10× increase in computational complexity, and some proposals achieved good subjective quality and bit rate results with lower computational complexity than the reference AVC High profile encodings. At that meeting, the name High Efficiency Video Coding was adopted for the joint project. Starting at that meeting, the JCT-VC integrated features of some of the best proposals into a single software codebase and a "Test Model under Consideration", and performed further experiments to evaluate various proposed features. The first working draft specification of HEVC was produced at the third JCT-VC meeting in October 2010. Many changes in the coding tools and configuration of HEVC were made in later JCT-VC meetings.On January 25, 2013, the ITU announced that HEVC had received first stage approval in the ITU-T Alternative Approval Process. On the same day, MPEG announced that HEVC had been promoted to Final Draft International Standard status in the MPEG standardization process.
On April 13, 2013, HEVC/H.265 was approved as an ITU-T standard. The standard was formally published by the ITU-T on June 7, 2013, and by the ISO/IEC on November 25, 2013.
On July 11, 2014, MPEG announced that the 2nd edition of HEVC will contain three recently completed extensions which are the multiview extensions, the range extensions, and the scalability extensions.
On October 29, 2014, HEVC/H.265 version 2 was approved as an ITU-T standard. It was then formally published on January 12, 2015.
On April 29, 2015, HEVC/H.265 version 3 was approved as an ITU-T standard.
On June 3, 2016, HEVC/H.265 version 4 was consented in the ITU-T and was not approved during a vote in October 2016.
On December 22, 2016, HEVC/H.265 version 4 was approved as an ITU-T standard.
Patent licensing
On September 29, 2014, MPEG LA announced their HEVC license which covers the essential patents from 23 companies. The first 100,000 "devices" are royalty-free, and after that the fee is $0.20 per device up to an annual cap of $25 million. This is significantly more expensive than the fees on AVC, which were $0.10 per device, with the same 100,000 waiver, and an annual cap of $6.5 million. MPEG LA does not charge any fee on the content itself, something they had attempted when initially licensing AVC, but subsequently dropped when content producers refused to pay it. The license has been expanded to include the profiles in version 2 of the HEVC standard.When the MPEG LA terms were announced, commenters noted that a number of prominent patent holders were not part of the group. Among these were AT&T, Microsoft, Nokia, and Motorola. Speculation at the time was that these companies would form their own licensing pool to compete with or add to the MPEG LA pool. Such a group was formally announced on March 26, 2015, as HEVC Advance. The terms, covering 500 essential patents, were announced on July 22, 2015, with rates that depend on the country of sale, type of device, HEVC profile, HEVC extensions, and HEVC optional features. Unlike the MPEG LA terms, HEVC Advance reintroduced license fees on content encoded with HEVC, through a revenue sharing fee.
The initial HEVC Advance license had a maximum royalty rate of US$2.60 per device for Region 1 countries and a content royalty rate of 0.5% of the revenue generated from HEVC video services. Region 1 countries in the HEVC Advance license include the United States, Canada, European Union, Japan, South Korea, Australia, New Zealand, and others. Region 2 countries are countries not listed in the Region 1 country list. The HEVC Advance license had a maximum royalty rate of US$1.30 per device for Region 2 countries. Unlike MPEG LA, there was no annual cap. On top of this, HEVC Advance also charged a royalty rate of 0.5% of the revenue generated from video services encoding content in HEVC.
When they were announced, there was considerable backlash from industry observers about the "unreasonable and greedy" fees on devices, which were about seven times that of the MPEG LA's fees. Added together, a device would require licenses costing $2.80, twenty-eight times as expensive as AVC, as well as license fees on the content. This led to calls for "content owners band together and agree not to license from HEVC Advance". Others argued the rates might cause companies to switch to competing standards such as Daala and VP9.
On December 18, 2015, HEVC Advance announced changes in the royalty rates. The changes include a reduction in the maximum royalty rate for Region 1 countries to US$2.03 per device, the creation of annual royalty caps, and a waiving of royalties on content that is free to end users. The annual royalty caps for a company is US$40 million for devices, US$5 million for content, and US$2 million for optional features.
On February 3, 2016, Technicolor SA announced that they had withdrawn from the HEVC Advance patent pool and would be directly licensing their HEVC patents. HEVC Advance previously listed 12 patents from Technicolor. Technicolor announced that they had rejoined on October 22, 2019.
On November 22, 2016, HEVC Advance announced a major initiative, revising their policy to allow software implementations of HEVC to be distributed directly to consumer mobile devices and personal computers royalty free, without requiring a patent license.
On March 31, 2017, Velos Media announced their HEVC license which covers the essential patents from Ericsson, Panasonic, Qualcomm Incorporated, Sharp, and Sony.
the MPEG LA HEVC patent list is 164 pages long.