Closed-GOP prediction structure

Let us first look at a conventional closed-GOP prediction structure. Since nothing was decoded yet, the first frame in a bitstream is an Instantaneous Decoding Refresh (IDR) frame. If an IDR frame is received, the decoder is instantaneously reset (refreshed) and all frame buffers or other internal buffers are cleared. Since the frame has no dependencies on other frames, it can always be decoded. An IDR frame is also a Random Access (RA) point or keyframe. At RA points decoding can be started. (RAs are marked in orange).

The following frames are then encoded using predictive (P) coding. This means that they use data from the already decoded frames. This includes pixel data for motion compensation but also motion vectors or prediction modes. But let’s illustrate this with an example:

In this example we are encoding a total of 9 frames. The frames are marked from 0-9 in the order that they are displayed to the viewer. (The vertical offset of the uneven frame numbers is just for illustration purposes.) However, they are not encoded in the order that they are displayed. In this example, frame 2 uses only frame 0, which has been decoded already, for prediction. Next, frame 1 is decoded which is displayed between frames 0 and 2 and uses both frames for prediction. This so-called Bi-prediction is much more efficient than prediction only from frames in the temporal past and is a key feature that makes modern video codecs so efficient. 

Of course, it is impractical to have only one keyframe at the very beginning of a video. We also want to be able to start decoding at frequent points within a bitstream. This allows us to seek in a video as well as to switch between different renditions as it is done in adaptive streaming. So, we can just insert multiple IDR frames in a video:

In this example, frame 4 is also an IDR frame. We can start decoding at frame 0 as well as at frame 4. Frames 0-3 form a Group of Pictures (GOP) which is completely self-contained and can be decoded completely independently of any other GOPs. The same is true for the following GOP of frames 4-9. As there are no dependencies between these GOPs, this is also referred to as a closed-GOP configuration. 

The closed-GOP configuration is widely used in adaptive bitrate streaming applications where the ubiquitous approach is to split the video into segments of a certain length. Each segment is then encoded using a predetermined set of different resolutions and bitrates called renditions. Since every segment starts with an IDR frame, it is possible to start decoding at each segment which therefore enables seeking. Furthermore, the video player can also freely switch to any of the other renditions at every segment boundary.

Another benefit emerges at the encoder side where each long video is split into small pieces (segments). If these segments can be independently decoded, then they can also be independently encoded. And if we mention “many segments” and “independently encodable” then the next thought is “scalability” and “cloud compute”. And this is exactly the principle that the Bitmovin cloud encoder is based on. We take all these individual encoding tasks and then scale horizontally in the cloud. 

Open-GOP prediction structure

So the opposite of a closed-GOP is an open-GOP configuration. The key difference is that in an open-GOP prediction structure predictions between the GOPs are allowed. Let’s again look at an example:

So, the frames 0, 1 and 2 are decoded in a hierarchical fashion as before. But then something different happens. The next frame in decoding order is frame 4 which is a random access point (RA). However, it is not an IDR but a Clean Random Access (CRA) point. While a CRA can also be decoded independently of any other frame, it does not reset the decoder as an IDR does and the reference picture buffer is not cleared. Next, we have frame 3 in coding order. As before, this frame uses frames 2 and 4 as reference. The rest of the frames are coded as before.

As in the closed-GOP example we can start the decoding process at frame 4 because it is not using any other frames as a reference. But this time, the decoder cannot be reset if this frame is received because the following frame (frame 3) uses previously decoded frames as a reference which as a result must remain in the picture buffer. The process of starting decoding from the second GOP is therefore a bit more complex:

Frame 4 is a Clean Random Access (CRA) point so decoding can be started with this frame. For the next frame in coding order (frame 3) we now have an issue. Since one of its references (frame 2) has not been decoded, frame 3 cannot be decoded. If we start decoding at the Random Access (RA) point of frame 4, the decoding of the leading picture 3 must be skipped. Consequently, the frame type is Random Access Skipped Leading (RASL). The remaining frames can be decoded as before. 

So, what are the advantages of an open-GOP configuration? So far, we just observed that decoding is more complicated. Moreover, it is now also impossible to switch to a different rendition at the CRA because we have not decoded the reference frames that are needed for the leading frames of the open-GOP, and we will have to skip decoding them. But there are two substantial advantages:

Coding performance

As I already mentioned, bidirectional prediction into the temporal past and future is one of the key features that make modern video codecs so efficient. Generally, the more past and future reference frames a frame can use for prediction, the higher the compression efficiency. Frames that do not use any other frames as reference (RA frames like IDR and CRA frames) typically have the worst compression efficiency.

While we cannot avoid having regular random access points in the bitstream for seeking, we can increase the coding efficiency of the leading pictures significantly in the open-GOP configuration. This leads to a significant reduction in overall bitrate at the same quality. In the experiments from the HHI, an overall BD-rate reduction of up to 9% could be observed. Of course, these results depend on many factors like the general coding structure, the resolution and bitrate as well as the content itself.

Coding performance across segment boundaries

In a closed-GOP configuration, the decoder must be reset with every IDR frame. An unwanted side effect of this is that the quality as well as the visual representation of a scene changes very abruptly at this point. Especially at lower bitrates, this can be perceived as a sudden jump or pumping in the video. Things that are generally hard to encode like water, clouds and trees are particularly susceptible to this effect. 

In this example, the difference between closed-GOP on the left and open-GOP on the right is nicely visible. The pumping is especially notable in the exhaust clouds of the rocket launch in the first scene and in the trees in the background in the second scene. In the open-GOP configuration this effect is hardly visible. 

Open-GOP resolution switching

I mentioned before that switching to a different rendition is only possible at IDR frames in a closed-GOP configuration. We also saw that if we start decoding a CRA frame with RASL frames, we must skip decoding of the leading frames. Obviously, we don’t want to skip decoding frames whenever the player switches to a different resolution. This would be a horrible experience for the viewer.

Fortunately, VVC has a trick up its sleeve for exactly this scenario. In the example above we noted that decoding of the RASL frame (frame 3) is not possible because it uses frame 2 as a reference which has not been decoded when switching renditions. But what has been decoded is a different version of frame 2 from a different rendition. While this frame may have been decoded at a different quality or even at a different spatial resolution it is a representation of the exact same frame. So, with a bit of high level syntax, the VVC decoder can use this frame from another rendition as a reference frame for decoding frame 3. Even if the frame uses a different resolution, the decoder has a standardized set of up/down scaling filters. Let’s look at this:

In this example we are decoding frames 0 to 2 from a rendition at a lower resolution. Then the player decides to switch to a rendition with a higher resolution and bitrate. Decoding of the CRA (frame 4) is no problem since RA frames can be decoded independently of other frames. For frame 3, the decoder will now upscale frame 2 from the lower rendition and use this frame as a reference instead of the unavailable frame from the higher rendition. Decoding of the remaining frames is unchanged.

As mentioned before, the open-GOP prediction structure significantly reduces quality pumping effects. But there is another advantage. When switching to a higher or lower rendition in a closed-GOP configuration, there is a visible jump of the quality of the video. Of course, the bigger the jump is, the more pronounced the visible quality jump becomes. However, in open-GOP resolution switching, the intermediate leading frames that are using references from both renditions act as a sort of “quality interpolation” between the renditions which results in a much smoother transition between the renditions. 

Here, we can see an example of the PSNR for the resolution switching behavior. We have 3 renditions of 1920×800 (gray), 1280×534 (red) and 640×268 (blue). In conventional adaptive streaming implementations with closed-GOPs, a rendition switch would result in an abrupt jump in quality at the switching points. The yellow graph shows how the quality has a much smoother transition between the renditions when using open-GOP resolution switching in VVC.

IBC 2023

At the IBC, we are presenting a technical paper about practical implementations and considerations when implementing open-GOP resolution switching with VVC in real world environments. This is a joint effort of Fraunhofer HHI, Spin Digital and Bitmovin. Please join us at the IBC in the “Advances in video coding and processing” session on Sep 16th starting at 14:15 in room E102. Here we will present what technical challenges arise when deploying this feature for low latency live transcoding as well as in the highly scalable Bitmovin cloud encoder.

Related Links

IBC 2023 Website

VVC codec background

VVC benefits and supported devices

The post VVC: Open-GOP Resolution Switching appeared first on Bitmovin.

Versatile Video Coding (VVC) is a codec that was made to help reduce bitrates without compromising video quality. As a result, VVC offers a number of benefits to video streaming, including:

• Efficient streaming of demanding content, including UHD (8k+) and 360 video
• Lower distribution costs due to the lower bitrate
• Maintaining high visual quality even on slower networks

As the benefits sound extremely promising, we’ve taken to encode streams in VVC, which you can read more about in our previous article. Until now, however, there have been limited playback implementations from device makers, making it difficult to see the true gains. Due to this, we believe it’s equally important to evaluate the state of playback on different devices and advancement in playback technology because, ultimately, if you can encode VVC but can’t play it back, then viewers & streaming services don’t get to experience the full value of the codec.

What devices support VVC now and will in the future?

As VVC is a new codec standard, no device currently has hardware support for it, and only a limited number of devices can software decode it in real time. In our testing, we found that high and mid-end devices usually have enough processing power to play 1080p VVC videos such as Google Pixel 5 or Samsung Galaxy S8. Researchers from Ericsson presented similar findings at the NAB show, also sharing that they needed to use AMD EPYC or Intel Xeon servers for 8K playback testing, which shows how important hardware decoding will be for UHD VVC content.

Even though the current device ecosystem that can decode VVC is limited, with the net gains VVC is projected to offer, it will definitely be a priority addition for device makers in the coming versions of their devices.

In the future, VVC is expected to be supported in devices such as:

• Smartphones and tablets (Samsung, Apple, Huawei, + more)
• Smart TVs (LG, Samsung, Sony, + more)
• Streaming devices (Roku, Amazon Fire TV, Apple TV, + more)
• Recent computers and laptops

*sidenote for future Smart TV’s above – LG 8K models officially support VVC already.

How did we test VVC in an actual streaming environment?

To evaluate VVC playback, we teamed up with the Fraunhofer Heinrich-Hertz-Institut (HHI) to integrate their software-based decoder VVdeC. You can check out VVdeC on GitHub.

We’ve integrated the VVC decoder into an internal version of our Android demo app to evaluate VVC playback on a range of streams and devices. This demo implementation has allowed us to test the decode performance on a variety of devices, for example, low-end, mid-tier, and high-end models. As optimizations are gradually made to the decoding, we will continue expanding our evaluation progress and share that with video developers. Ultimately we believe this will help streaming video apps decide whether VVC is a suitable codec for their audience and the devices where their content is streamed.

Homepage of the Bitmovin Android Application testing VVC

The Bitmovin Player on the Bitmovin Android demo application testing VVC

What have we learned?

Firstly, we must say the integration of VVdeC was a smooth process. It took approximately 3 days to achieve the first video playback, and from there, we rolled the implementation into an internal version of our Android demo app. We evaluated the viewer experience offered by VVC with our findings below.

To test the VVC low bitrate performance. We encoded the same assets in both VVC and HEVC at the same bitrate of 470, 350, and 207 kb/s using Bitmovin Encoder. Upon playback on Android devices, we compared the performance and image quality of the streams. VVC is a clear winner in fast scenes without motion blur. For example, in the explosion frame below (from Charge), VVC shows less aliasing and much more detail, especially in the bullet ricochet that spans only 3 frames and is brightly visible during playback. The difference:

Explosion frame comparison between HEVC (left) and VVC (right) at 470 kb/s

While evaluating the implementation, it became clear that software-based decoding requires greater CPU performance and increases memory consumption. This finding is partly due to the emerging nature of VVC, and the current implementations aren’t optimized for processing efficiency. As a result, we used a Samsung Galaxy S8 tablet to achieve smooth playback and showcased it in our demo booth at NAB 2023.

What’s next?

Now that we’ve tested Playback with Android-powered tablets and phones, we intend to further our evaluation of VVC playback on other devices that are in widespread use. For example, Android-based TVs. As we found performance to be challenging at this stage on lower-powered devices, we hope to identify optimizations to improve performance. Clearly, this will be driven by the advantages of having bitrate savings at comparable qualities that are seen when using other codecs.

We’re continuing our VVC research and collaboration with Fraunhofer and have a joint research paper that will be presented later this year at IBC2023. We will share more details as soon as the conference makes them publicly available.

Further, we’re also looking into emerging audio codecs which sit alongside video codec innovations like VVC. This includes technologies such as MPEG-H. You can read more about MPEG-H from Fraunhofer’s Institute for Integrated Circuits (IIS) group.

Want to know more?

If you’re interested in learning more, you can always ask a question on the Bitmovin Community, and we’d be happy to speak further about the demo we showcased at NAB 2023.

Check out this presentation from Mile High Video 2022 on our collaboration with Fraunhofer to bring their VVenC encoder to the cloud.

Bitmovin’s Chief Architect Igor Oreper will be speaking at the Streaming Media East conference on May 17 as part of the session VVC: Ready For Action? A Comprehensive Guide To Deployment & Implementation, which is sure to be valuable for anyone considering VVC.

The post VVC – Its Benefits, Supported Devices, and How Bitmovin is Implementing it appeared first on Bitmovin.

So it’s happening. After their previous work on h.264/AVC and h.265/HEVC, the ITU-T Video Coding Experts Group (VCEG) and ISO Moving Picture Experts Group (MPEG) have again joined forces to create another video codec named Versatile Video Coding (VVC).
The first goal for the VVC Video Codec was to significantly reduce bitrate expenditure while maintaining the same visual quality compared to HEVC. When considering PSNR as a quality metric, the reference VVC encoder outperforms the reference HEVC encoder by about 40% in BD-rate. However, some subjective tests were also performed which demonstrated overall bit-savings of closer to 50%. The second goal in the standardization was versatility. On this front, VVC facilitates coding and transport for a wide range of applications and content types such as conventional video streaming, optimizations for screen content, 360-degree video, as well as live and ultra-low delay applications.

Evolution or Revolution?

Since the development of VVC was started from the basis of HEVC, the first question to ask about VVC is: Is it an evolution based on the technologies that were used in the former coding standards, or is it a really new and revolutionary way of compressing video?
Answer: It’s more or less an evolution of the basic building blocks that were already used in HEVC and various other codecs before.

• It is still a hybrid, block-based video coding standard
• Most technologies are based on HEVC and are further refined and improved
• But there are also a lot of new coding tools which have not been seen in the context of video coding

But how was this achieved? Like other video coding standards (e.g. AVC, HEVC, or AV1), VVC is based on the hybrid block-based video coding approach with conventional intra and inter prediction. As with former evolutions of video coding standards, these gains can not be attributed to one single technique that was added in VVC but to smaller improvements in all the building blocks of the coding scheme. So here is a (not even close to complete) list of advancements in VVC:

• The maximum block size of Coding Tree Units (CTU) that can be processed was increased. The maximum block size is now 128×128 pixels. Also, the maximum block sizes for intra prediction and transformations were increased. This is particularly beneficial as the resolution of content that is encoded also is increasing. 
• After the initial split, each CTU is further split into Coding Units. This splitting algorithm is now much more flexible and allows for more different block sizes both square and non-square.
• The number of directions that can be used in directional intra prediction was further increased. Intra prediction can now also be performed for non-square blocks. 
• Many aspects of motion compensated prediction (inter prediction) were improved as well like better motion vector prediction, decoder side motion vector refinement, and overlapped block motion compensation (OBMC).
• More different types of transformation are available by using a separable transform combining Discrete Cosine and Sine Transform.
• The in-loop filters were improved as well with the addition of a new filter – Adaptive Loop Filter – for which the encoder can signal optimal parameters on a CTU basis.

As I already mentioned this list is far from complete and there are many many more new and adapted technologies that make VVC so efficient. If you want to learn more then there is plenty of material out there. But a good place to start is this very detailed overview paper.

What’s new in VVC?

So while the standard was finalized in late 2020, there is no widespread use of VVC in the market yet. But this was also not to be expected after such a short time. History has shown that adoption time for new video codecs is long and usually follows the following scheme: First, devices must support playback of a new standard. While software decoding on devices with high compute capabilities and no restrictions on power usage can be implemented quickly, most consumption of media is performed on devices that rely on hardware decoders. And while it takes some time to develop and deploy new decoding hardware, it takes even longer until these new devices reach a critical mass of deployment “in the wild”. At the same time, encoding solutions must be developed, tuned, and deployed. And finally, there is the question of royalties that must be paid. And only when all of these issues are resolved will it make sense to deploy actual video streaming using the new video codec. 
For VVC, we are seeing the first practical implementations of VVC. There are some software-based encoder and decoder solutions and the first vendors have released hardware decoders in their System on a Chip (SoC) devices. Furthermore, some vendors (like Bitmovin) have deployed VVC Video Codec as an option in their transcoding as a service product. On the patent side, there are a few players moving the codec into production, with MPEG LA introducing the first license in early 2021. While some patent pools are forming it is still very much unknown how much the usage of VVC will cost. And in the face of the falling cost of bandwidth, this is the biggest problem of VVC. To put it simply: If the price is not worth the bitrate savings, it will not be a thing. 
So as I mentioned, Bitmovin already deployed VVC video encoding in its cloud-based transcoding solution. For this, we teamed up with the Fraunhofer Heinrich-Hertz-Institut (HHI) to integrate their software-based encoder VVenC which is an open source VVC encoder and is freely available on Gihub. While this is working great, there was no easy way to create a VVC Video and watch it. This is where the vvDecPlayer project comes in.

Introducing the VVC Video Player: BitvvDecPlayer

What I wanted to create was a simple demonstration player that was able to stream and decode a VVC video stream in real-time. All parts of the decoder are based on other open-source projects. Many of the rendering routines were copied from the YUView player (another project of mine) which in turn is using the Qt framework.
When opening a playlist in the vvDecPlayer, four threads are launched that build up the decoding pipeline:

• Download: This thread performs the download of the VVC video segments using HTTP. It has an internal buffer of 5 segments that it will try to keep full. The actual download was implemented using the Qt network module.

• Bitstream parsing: After the download is done, we perform parsing of the high-level syntax of the bitstream. This gives us information about the segment like resolution and more importantly the number of frames in the segment.

• Decode: The decode thread decodes the compressed bitstream into raw YUV frames. We are using the Fraunhofer VVdeC software decoder here. The decoder is quite fast and is able to do real-time decoding of UHD content, provided that enough CPU power is available. The decoded frames are all stored in temporary buffers.

• Conversion: While the decoded frames are in the YUV domain, we require RGB pixel data for display. This is done with a native C++ function that was copied from YUView.

Finally, there is a timer running in the main thread that is trying to update the screen ‘FPS’ times per second by drawing the next converted RGB pixel buffer to the screen.

The screenshot shows the player in action. On the top left, the available renditions are shown. The currently selected rendition is marked with an arrow and the currently visible rendition is highlighted in green. Use the up/down arrows to switch renditions. On the top right are the fps counter and the status of the threads. The thread status display can be enabled in the menu or by pressing Ctrl+D. On the bottom is the progress graph (Ctrl+P). This is showing many values from the decoder pipeline. The dark cyan blocks indicate the compressed data of each segment where the height corresponds to the bitrate of each segment. Within each block, there is one bar that indicates the bitrate per frame. On the bottom, the status of each frame is shown: Downloaded (grey), Decoded (blue), converted to RGB, and ready for display (green). Playback can also be paused using the space bar and playback can be switched to full-screen view by double-clicking the video.
If you want to give it a try check out the project on Github. The only prerequisites you need are a compiler, CMake, and the Qt libraries. How to build really depends on the platform that you are compiling for. But it goes something like this:

• Get Qt. Either from the Qt page or if you are on Linux then probably your distro’s package manager can install it for you. On the mac homebrew is a good option.
• Check out the source code of the player, and create a new directory to build in (e.g. ‘build’). Go into the directory and call qmake: ‘qmake ../’. After that, you start compilation. On Linux/mac this is probably ‘make’, while on windows it is likely ‘nmake’.
• Next, you also need the VVdeC decoder library. Compiling that is also easy. Get the sources from the Github repo and create a build directory. In there, call ‘cmake -DBUILD_SHARED_LIBS=1 ..’ and then ‘cmake -–build . –config Release’. This should build the shared decoder library in the ‘bin’ or the ‘lib’ folder.
• Lastly, you can start the player. Go to ‘Settings->Select VVdeC library’ and browse to the shared VVdeC library you just built.

Now everything should be ready to stream some videos. The player comes with some sample stream provided by us and our encoder. Just select a sample from ‘File->Bitmovin Streams’. If you encounter a bug please feel free to open a bug report on Github. To learn more about the BitvvDecPlayer, view our Webinar with Fraunhoffer HHI.

Conclusion

There is no doubt that VVC has some exciting potential and is already showing interesting results. Two years is a long time, and there is a lot of opportunities to further increase the coding performance and lower the encoding complexity, as well as perfect some of the new tools that are already adopted into the new standard. It will be exciting to check in again in 6 to 12 months to see how the implementations are developing.
If you like to learn more about how the VVC Video Codec works check out our introductory blog: What is VVC and how does it work
Open projects like these are a staple of the benefits of working as an Engineer for Bitmovin. Come join our team to work on your exciting standard-setting projects.

Video technology guides and articles

• Back to Basics: Guide to the HTML5 Video Tag 
• What is a VoD Platform?A comprehensive guide to Video on Demand (VOD)
• Video Technology [2022]: Top 5 video technology trends
• HEVC vs VP9: Modern codecs comparison
• What is the AV1 Codec?
• Video Compression: Encoding Definition and Adaptive Bitrate
• What is adaptive bitrate streaming
• MP4 vs MKV: Battle of the Video Formats
• AVOD vs SVOD; the “fall” of SVOD and Rise of AVOD & TVOD (Video Tech Trends)
• MPEG-DASH (Dynamic Adaptive Streaming over HTTP)
• Container Formats: The 4 most common container formats and why they matter to you.
• Quality of Experience (QoE) in Video Technology [2022 Guide]

The post VVC Video Codec – The Next Generation Codec appeared first on Bitmovin.