Many streaming providers are looking for ways to offer a more premium and high quality experience to their users. One often overlooked component in streaming quality is audio – and more specifically which audio bitrates, channel layouts, and even audio languages are available and how these options can be delivered to the viewers on a range of devices. While there many ways of improving the video streaming quality & experience such as Per-Title Encoding, Multi-Bitrate Video, High Dynamic Range (HDR), and high resolutions, there are also some some great ways of enhancing a user’s experience with premium hls audio. Some of the most important considerations for audio streaming are:

• Adaptive Streaming: serving multiple audio bitrates for various streaming conditions
• Reduced Bandwidth & Device Compatibility: multi-codec audio for better compression at reduced bitrates
• Improved User Experience: 5.1(or greater) surround sound or even lossless audio
• Accessibility and Localization: such as multi-language or descriptive audio

You can learn even more about how audio encoding affects the streaming experience in this blog.

In Bitmovin’s 2023-24 Video Developer Report, we saw that immersive audio ranked in the top 15 areas for innovation; while audio transcription was the #1 ranked use-case for AI and ML. Furthermore, though AAC remains the the most widely used audio codec – mostly due to it’s wide device support, we see that both Dolby Digital/+ and Dolby Atmos are the #2 and #3 ranked audio codecs that streaming companies are either currently supporting or planning on supporting in the near future.

With HLS and its multivariant approach, this is all possible; but understanding just how to construct and organize your HLS multivariant playlist can be tricky at first. In this tutorial we will take a look at some best practices in HLS for serving alternate audio renditions as well as an example at the end of this article showcasing how to simply do this using the Bitmovin Encoder.

Basic audio stream packaging

The most basic way to package audio for HLS is to mux the audio track with each video track. This works for very simple configurations where you are only dealing with outputting a single AAC Stereo audio track at a single given bitrate. While the benefit of this approach is simplicity, it has many limitations such as not being able to support multi-channel surround sound, advanced codecs, and multi-language support. Additionally demuxing audio and video comes with benefit of using other muxing containers like fragmented MP4 or CMAF which don’t require client-side transmuxing. Additionally, keeping audio and video muxed together comes with inefficient storage and delivery as each video variant will have the audio duplicated. Similarly, demuxed audio and video allows for the use MP4 and CMAF containers which are more performant for client devices since they won’t have to demux or transmux the segments real-time.

A multivariant playlist output for this would look something like:

#EXTM3U
#EXT-X-VERSION:3
#EXT-X-INDEPENDENT-SEGMENTS
#EXT-X-STREAM-INF:BANDWIDTH=4255267,AVERAGE-BANDWIDTH=4255267,CODECS="avc1.4d4032,mp4a.40.2",RESOLUTION=2560x1440
manifest_1.m3u8

#EXT-X-STREAM-INF:BANDWIDTH=3062896,AVERAGE-BANDWIDTH=3062896,CODECS="avc1.4d4028,mp4a.40.2",RESOLUTION=1920x1080
manifest_2.m3u8

#EXT-X-STREAM-INF:BANDWIDTH=1591232,AVERAGE-BANDWIDTH=1591232,CODECS="avc1.4d4028,mp4a.40.2",RESOLUTION=1600x900
manifest_3.m3u8

#EXT-X-STREAM-INF:BANDWIDTH=1365632,AVERAGE-BANDWIDTH=1365632,CODECS="avc1.4d401f,mp4a.40.2",RESOLUTION=1280x720
manifest_4.m3u8

#EXT-X-STREAM-INF:BANDWIDTH=862995,AVERAGE-BANDWIDTH=862995,CODECS="avc1.4d401f,mp4a.40.2",RESOLUTION=960x540
manifest_5.m3u8

Audio/Video demuxing

A better approach is to demux the Audio and Video tracks – luckily HLS makes this simple by the use of HLS EXT-X-MEDIA playlists which is the standard way of declaring alternate content renditions for audio, subtitle, closed-captions, or video(mostly used alternative viewing angles such as in live sports). With the use of EXT-X-MEDIA to decouple audio from video, we can add in many great audio features such as supporting alternate/dubbed language tracks, surround sound tracks, multiple audio qualities, and multi-codec audio.

By supplying audio tracks with EXT-X-MEDIA tags, we can explicitly add each audio track that we want to output as well as group them together – Then we can correlate each Video Variant(EXT-X-STREAM-INF) to one of the grouped Audio Media Playlists.

Using the previous example of a single AAC Stereo Audio track, a demuxed audio/video output would look like:

#EXTM3U
#EXT-X-VERSION:3
#EXT-X-INDEPENDENT-SEGMENTS

#EXT-X-MEDIA:TYPE=AUDIO,GROUP-ID="AAC_Stereo",LANGUAGE="en",NAME="English - Stereo",AUTOSELECT=YES,DEFAULT=YES,URI="audio_aac.m3u8"

#EXT-X-STREAM-INF:...,CODECS="avc1.4d4032,mp4a.40.2",RESOLUTION=2560x1440,AUDIO="AAC_Stereo"
manifest_1.m3u8

#EXT-X-STREAM-INF:...,CODECS="avc1.4d4028,mp4a.40.2",RESOLUTION=1920x1080,AUDIO="AAC_Stereo"
manifest_2.m3u8

#EXT-X-STREAM-INF:...,CODECS="avc1.4d4028,mp4a.40.2",RESOLUTION=1600x900,AUDIO="AAC_Stereo"
manifest_3.m3u8

#EXT-X-STREAM-INF:...,CODECS="avc1.4d401f,mp4a.40.2",RESOLUTION=1280x720,AUDIO="AAC_Stereo"
manifest_4.m3u8

#EXT-X-STREAM-INF:...,CODECS="avc1.4d401f,mp4a.40.2",RESOLUTION=960x540,AUDIO="AAC_Stereo"
manifest_5.m3u8

Here, you can first see we declare a single Audio Media(EXT-X-MEDIA) playlist for our audio track and give it a group-id attribute value of “AAC_Stereo“. Then each Video Variant EXT-X-STREAM-INF tag uses the “AUDIO” attribute to associate its video track to the Audio Media group “AAC_Stereo“.

Multiple audio bitrates

But now let’s imagine we want to better optimize our Adaptive Streaming to deliver our AAC Stereo audio in multiple bitrates such as a high(196kbps) and low(64kbps) so that the higher resolution Video Variants can take advantage of higher quality+bitrate audio given the increase in bandwidth when streaming those variants. We can accomplish this by encoding our audio with both a low and high bitrate outputs and group them separately – then decide which Video Variant gets which Audio bitrate/quality. – For example, our 720p or below variants get the lower quality audio by default, and our full HD or above variants get the higher quality audio by default. Just think of that as defaults though, because most modern Players that stream HLS, will allow for independently picking which audio quality to play based on Adaptive-Bitrate streaming conditions.

An example of utilizing a low and a high AAC Stereo tracks would look like:

#EXTM3U
#EXT-X-VERSION:3
#EXT-X-INDEPENDENT-SEGMENTS

#EXT-X-MEDIA:TYPE=AUDIO,GROUP-ID="aac-stereo-64",LANGUAGE="en",NAME="English - Stereo",AUTOSELECT=YES,DEFAULT=YES,URI="audio_aac_64k.m3u8"
#EXT-X-MEDIA:TYPE=AUDIO,GROUP-ID="aac-stereo-196",LANGUAGE="en",NAME="English - Stereo",AUTOSELECT=YES,DEFAULT=NO,URI="audio_aac_196k.m3u8"

#EXT-X-STREAM-INF:...,CODECS="avc1.4d4032,mp4a.40.2",RESOLUTION=2560x1440,AUDIO="aac-stereo-196"
manifest_1.m3u8

#EXT-X-STREAM-INF:...,CODECS="avc1.4d4028,mp4a.40.2",RESOLUTION=1920x1080,AUDIO="aac-stereo-196"
manifest_2.m3u8

#EXT-X-STREAM-INF:...,CODECS="avc1.4d4028,mp4a.40.2",RESOLUTION=1600x900,AUDIO="aac-stereo-196"
manifest_3.m3u8

#EXT-X-STREAM-INF:...,CODECS="avc1.4d401f,mp4a.40.2",RESOLUTION=1280x720,AUDIO="aac-stereo-64"
manifest_4.m3u8

#EXT-X-STREAM-INF:...,CODECS="avc1.4d401f,mp4a.40.2",RESOLUTION=960x540,AUDIO="aac-stereo-64"
manifest_5.m3u8

In this example, we now have two audio tracks, one for each bitrate, and therefore have two Audio Media (EXT-X-MEDIA) playlists defined, each having unique GROUP-ID attribute, but the same NAME attribute. This is a good way declaring that the audio tracks are the same language, channel config, and codec, but at different qualities. Now, we can declare that each Video Variant(EXT-X-STREAM-INF) that is 720p or less sets the AUDIO group for that variant to the low bitrate Audio Track(GROUP-ID="aac-stereo-64") and those variants above 720p get the higher bitrate AUDIO group(GROUP-ID="aac-stereo-196") by default (but again, most Players can manage the audio tracks independently for optimal adaptive streaming).

This is at least an improvement on the previous single-bitrate audio packaging – But still, there are plenty of enhancements we can make!

More efficient AAC

The previous examples are all relying on Low Complexity AAC(AAC-LC) because this basic audio codec is supported by every playback device. It is necessary to always have at least one AAC-LC track to be able support older devices. However, most devices these days can support more efficient versions of AAC such as High Efficiency AAC(AAC-HE) which comes in two main versions: v2 which is used for bitrates up to 48kbps and v1 which is used for bitrates up to 96kbps.

So let’s adapt our previous example to not rely on 2 (or more) different AAC-LC audio tracks, and instead output one AAC-HE v1, one AAC-HE v2, and one AAC-LC rendition. The tricky part here is that we will want to group each of the above into a different GROUP-ID so that the Player client can decide which to use based on which codecs it supports – but we also will want each Video Variant to be able to use any of those audio tracks. To accomplish this, all we need to do is duplicate each Video Variant for each of the 3 unique Audio Media GROUP-IDs.

A note on grouping audio renditions

The apple authoring spec recommends creating one audio group for each pair of codec and channel count.

We now have have 3 different versions of the AAC codec so we will have 3 different audio groups.

#EXTM3U
#EXT-X-VERSION:3
#EXT-X-INDEPENDENT-SEGMENTS

#EXT-X-MEDIA:TYPE=AUDIO,GROUP-ID="aac_lc-stereo-128k",LANGUAGE="en",NAME="English - Stereo",AUTOSELECT=YES,DEFAULT=YES,URI="audio_aaclc_128k.m3u8"
#EXT-X-MEDIA:TYPE=AUDIO,GROUP-ID="aac_he1-stereo-64k",LANGUAGE="en",NAME="English - Stereo",AUTOSELECT=YES,DEFAULT=NO,URI="audio_aache1_64k.m3u8"
#EXT-X-MEDIA:TYPE=AUDIO,GROUP-ID="aac_he2-stereo-32k",LANGUAGE="en",NAME="English - Stereo",AUTOSELECT=YES,DEFAULT=NO,URI="audio_aache2_32k.m3u8"

#EXT-X-STREAM-INF:...,CODECS="avc1.4d4032,mp4a.40.2",RESOLUTION=2560x1440,AUDIO="aac_lc-stereo-128k"
manifest_1.m3u8

#EXT-X-STREAM-INF:...,CODECS="avc1.4d4032,mp4a.40.5",RESOLUTION=2560x1440,AUDIO="aac_he1-stereo-64k"
manifest_1.m3u8

#EXT-X-STREAM-INF:...,CODECS="avc1.4d4032,mp4a.40.29",RESOLUTION=2560x1440,AUDIO="aac_he2-stereo-32k"
manifest_1.m3u8

## Repeat above approach for each additional Video Variant

In this example, you can see that we replicated the 1440p variant 3 times – 1 for reach Audio Media GROUP-ID which would then be repeated for each additional Video Variant. This will allow the client Player to decide for a given Video Variant, which audio track group to use based upon codec support and streaming conditions. Also take note how each Video Variant’s CODECS attribute is updated to represent the necessary audio codec identifier.

Surround sound audio

Now, let’s say we also want to be able to support 5.1 surround sound for those clients which can benefit from it. For this we can decide on which surround sound codec we want to support. Let’s use Dolby Digital AC-3 for this example. Since we are now relying on a more advanced audio codec for optimal surround experience, it is also be important to consider devices that may have 5.1 or greater speaker setups, but that can NOT support Dolby Digital. For this we will also include a secondary 5.1 track using basic AAC-LC codec. Now, we will create 2 new Audio Media playlists with unique GROUP-ID and NAME attributes.

A note on downmixing from 5.1 audio sources

In this example, we will assume the source has a Dolby Digital surround audio track. From that single audio source, we will create create our AC-3 surround track, implicitly convert to our AAC surround track, and automatically downmix the source 5.1 to our various AAC 2.0 Stereo outputs using the Bitmovin Encoder which is shown in sample code at the bottom of this article. Alternatively you can do all sorts of mixing, channel-swapping, as well as work with distinct audio input files like separate files for each channel for example. You can learn more about that here.

Don’t forget about grouping audio renditions

As previously mentioned, the apple authoring spec recommends creating one audio group for each pair of codec and channel count.

We now have have 5 different unique combinations of codecs and channel counts so we will have 5 different audio groups.

#EXTM3U
#EXT-X-VERSION:3
#EXT-X-INDEPENDENT-SEGMENTS

#EXT-X-MEDIA:TYPE=AUDIO,GROUP-ID="aac_lc-stereo-128k",LANGUAGE="en",NAME="English - Stereo",AUTOSELECT=YES,DEFAULT=YES,URI="audio_aac_128k.m3u8"
#EXT-X-MEDIA:TYPE=AUDIO,GROUP-ID="aac_he1-stereo-64k",LANGUAGE="en",NAME="English - Stereo",AUTOSELECT=YES,DEFAULT=NO,URI="audio_aache1_64k.m3u8"
#EXT-X-MEDIA:TYPE=AUDIO,GROUP-ID="aac_he2-stereo-32k",LANGUAGE="en",NAME="English - Stereo",AUTOSELECT=YES,DEFAULT=NO,URI="audio_aache2_32k.m3u8"
#EXT-X-MEDIA:TYPE=AUDIO,GROUP-ID="aac_lc-5_1-320k",LANGUAGE="en",NAME="English - 5.1",AUTOSELECT=YES,DEFAULT=NO,URI="audio_aac_lc_5_1_320k.m3u8"
#EXT-X-MEDIA:TYPE=AUDIO,GROUP-ID="dolby",LANGUAGE="en",NAME="English - Dolby",CHANNELS="6",URI="audio_dolbydigital.m3u8"

#EXT-X-STREAM-INF:...,CODECS="avc1.4d4032,mp4a.40.2",RESOLUTION=2560x1440,AUDIO="aac_lc-stereo-128k"
manifest_1.m3u8

#EXT-X-STREAM-INF:...,CODECS="avc1.4d4032,mp4a.40.5",RESOLUTION=2560x1440,AUDIO="aac_he1-stereo-64k"
manifest_1.m3u8

#EXT-X-STREAM-INF:...,CODECS="avc1.4d4032,mp4a.40.29",RESOLUTION=2560x1440,AUDIO="aac_he2-stereo-32k"
manifest_1.m3u8

#EXT-X-STREAM-INF:...,CODECS="avc1.4d4032,mp4a.40.29",RESOLUTION=2560x1440,AUDIO="aac_lc-5_1-320k"
manifest_1.m3u8

#EXT-X-STREAM-INF:...,CODECS="avc1.4d4032,ac-3",RESOLUTION=2560x1440,AUDIO="dolby"
manifest_1.m3u8


## Repeat above approach for each additional Video Variant

Here you can see that now we have the 1440p variant replicated a total of 5 times, once for each Audio Media GROUP-ID which allows the client Player to select the most appropriate audio and video track combination.

Again, note how each duplicated Video Variant has an updated CODECS attribute to represent the appropriate audio codec associated to it. One major reason we duplicate each Video Variant for each Audio Media GROUP-ID is that most devices cannot handle switching between audio codec’s during playback; so as Adaptive-Bitrate logic on the Player switches between different Video Variant’s it will pick the variant that has the same audio codec that it has been using. Additionally, in HLS, we cannot simply list the Video Variant once and add all of the various audio codecs to the CODECS attribute. This is because per HLS, the client device MUST be able to support all of the CODECS mentioned on a given Video Variant(EXT-X-STREAM-INF) to avoid possible playback failures. So instead, we separate out the Video Variants per each codec + channel number set.

Multi-language audio

This is all great, but what if I want to support additional dubbed audio language tracks or even Descriptive Audio tracks? Luckily, that is rather simple to do. We can just create additional AudioMedia playlists for each language and utilize the existing GROUP-IDs depending on which codecs and formats we want to support. We can use the existing GROUP-IDs which are logically grouped by Codec and Channel pairing per the Apple authoring spec, then we can add our additional language tracks to those existing groups.

#EXTM3U
#EXT-X-INDEPENDENT-SEGMENTS
#EXT-X-VERSION:6
#EXT-X-MEDIA:TYPE=AUDIO,GROUP-ID="AAC-HE-V1-Stereo",NAME="English-Stereo",LANGUAGE="en",DEFAULT=NO,URI="audio_aache1_stereo.m3u8"
#EXT-X-MEDIA:TYPE=AUDIO,GROUP-ID="AAC-HE-V1-Stereo",NAME="Spanish-Stereo",LANGUAGE="es",DEFAULT=NO,URI="audio_aache1_stereo_es.m3u8"

#EXT-X-MEDIA:TYPE=AUDIO,GROUP-ID="AAC-HE-V2-Stereo",NAME="English-Stereo",LANGUAGE="en",DEFAULT=NO,URI="audio_aache2_stereo.m3u8"
#EXT-X-MEDIA:TYPE=AUDIO,GROUP-ID="AAC-HE-V2-Stereo",NAME="Spanish-Stereo",LANGUAGE="es",DEFAULT=NO,URI="audio_aache2_stereo_es.m3u8"

#EXT-X-MEDIA:TYPE=AUDIO,GROUP-ID="AAC-LC-5.1",NAME="English-5.1",LANGUAGE="en",DEFAULT=NO,URI="audio_aaclc-5_1.m3u8"
#EXT-X-MEDIA:TYPE=AUDIO,GROUP-ID="AAC-LC-5.1",NAME="Spanish-5.1",LANGUAGE="es",DEFAULT=NO,URI="audio_aaclc-5_1_es.m3u8"

#EXT-X-MEDIA:TYPE=AUDIO,GROUP-ID="AAC-LC-Stereo",NAME="English-Stereo",LANGUAGE="en",DEFAULT=NO,URI="audio_aaclc_stereo.m3u8"
#EXT-X-MEDIA:TYPE=AUDIO,GROUP-ID="AAC-LC-Stereo",NAME="Spanish-Stereo",LANGUAGE="es",DEFAULT=NO,URI="audio_aaclc_stereo_es.m3u8"

#EXT-X-MEDIA:TYPE=AUDIO,GROUP-ID="AC-3-5.1",NAME="English-Dolby",LANGUAGE="en",CHANNELS="6",DEFAULT=NO,URI="dolby-ac3-5_1.m3u8"
#EXT-X-MEDIA:TYPE=AUDIO,GROUP-ID="AC-3-5.1",NAME="Spanish-Dolby",LANGUAGE="es",CHANNELS="6",DEFAULT=NO,URI="dolby-ac3-5_1_es.m3u8"

#EXT-X-STREAM-INF:...,CODECS="avc1.4D401F,ac-3",RESOLUTION=1280x720,AUDIO="AC-3-5.1".0
video_720_3000000.m3u8

#EXT-X-STREAM-INF:...,CODECS="avc1.4D401F,mp4a.40.29",RESOLUTION=1280x720,AUDIO="AAC-HE-V2-Stereo".0
video_720_3000000.m3u8

#EXT-X-STREAM-INF:...,CODECS="avc1.4D401F,mp4a.40.2",RESOLUTION=1280x720,AUDIO="AAC-LC-Stereo".0
video_720_3000000.m3u8

#EXT-X-STREAM-INF:...,CODECS="avc1.4D401F,mp4a.40.2",RESOLUTION=1280x720,AUDIO="AAC-LC-5.1".0
video_720_3000000.m3u8

#EXT-X-STREAM-INF:...,CODECS="avc1.4D401F,mp4a.40.5",RESOLUTION=1280x720,AUDIO="AAC-HE-V1-Stereo".0
video_720_3000000.m3u8

How does this differ from DASH?

In DASH, demuxed Audio and Video tracks are grouped into separate AdaptationSets for a given period. This means each given Video AdaptationSet is not directly linked to one specific Audio track, but rather the client Player independently picks a Video Representation from the Video AdaptationSet and a Audio Representation from the Audio AdaptationSet. So with DASH, we don’t have to worry about re-stating Video tracks for each group of Audio tracks as they are managed independently of each other.

Additional notes

The video codecs you choose to support may also determine which audio codecs and container formats you use. For example if you encode video to VP9 you may want to consider using vorbis or opus audio codecs.

In this example, we used AC-3 for Dolby Digital 5.1, but you may consider using Enhanced AC-3 or more commonly referred to as E-AC-3 for additional channel support(such as 7.1 or more) or spatial audio support like Dolby Atmos. Other premium surround sound codec options are DTS:HD and DTS:X.

Premium HLS audio example with the Bitmovin Encoder & Manifest Generator

Below linked GitHub sample is a pseudo-code example using the Bitmovin Javascript/Typescript SDK that demonstrates outputting multi-bitrate, multi-codec, multi-channel, and multi-language audio tracks. This can greatly enhance user’s experience as it allows for streaming the best quality and most appropriate audio for each device’s codec support and speaker channel configuration.

With the Bitmovin Encoder, we can use one master (Dolby Digital surround in this example) audio file/stream for each language and easily downmix it to 2.0 stereo or implicitly convert it to AAC 5.1. Then, once we simply create each desired audio track, we will use the Bitmovin Manifest Generator to create our HLS multivariant playlists.

Encoding Example For HLS With Multiple Audio Layers

The post Providing a Premium Audio Experience in HLS with the Bitmovin Encoder appeared first on Bitmovin.

Table of Contents

This post covers some recent updates, focusing on the new Firefox AV1 support in Encrypted Media Extensions. Bitmovin has been supporting and advocating for use of the AV1 codec for several years, even though there have been gaps in playback support preventing adoption for some workflows. Slowly but surely, those gaps are being filled and the reasons not to use AV1 are going away. Keep reading to learn more.

Firefox 125 adds support for encrypted AV1

A couple of years ago, Bitmovin began testing several different combinations of AV1 encoding, muxing and DRM support across browsers and playback devices. We were somewhat surprised to learn that even though Firefox was the first major browser to support AV1 playback, they had not implemented support for encrypted AV1 as they had for other codecs. We found there was actually an open bug/request filed 5 years ago. 

Shortly after we began watching closely, there was an update…

Ouch. Once the ticket got reassigned, Bitmovin got involved and gave our feedback that for premium/studio content, this support would be needed soon. We also provided a Widevine-protected sample for them to use in testing. Fast-forward to this spring, we saw some action on the ticket and support for AV1 with Encrypted Media Extensions was officially added to Firefox 125!

This means premium content workflows can now use AV1 on all of the major desktop browsers. Apple added support to Safari last fall, including with FairPlay Streaming, but for now it’s limited to devices with AV1 hardware decoders (iPhone 15 Pro, iPad Pro, new Macs with M3 processors).

Previous Bitmovin and Firefox AV1 collaboration

Way back in 2017, before the AV1 spec was finalized, Bitmovin and Firefox collaborated on the first HTML5 AV1 playback. Because the bitstream was still under development and subject to change, Bitmovin and Mozilla agreed on a common codec string to ensure compatibility between the version in the Bitmovin encoder and the decoder in Mozilla Firefox. It was made available in Mozilla’s experimental development version, Firefox Nightly, for users to manually enable. 

Even earlier in 2017, Bitmovin demonstrated the first broadcast quality AV1 live stream at NAB, winning a Best of Show award from Streaming Media Magazine. 

Other recent AV1 playback updates

Android adds dav1d decoder

In March 2024, VideoLAN’s “dav1d” became available to all Android devices running Android 12 or higher. Apps need to opt-in to using AV1 for now, but according to Google, most devices can at least keep up with software decoding of 720p 30fps video. YouTube initially opted to begin using dav1d on devices without a hardware decoder, but may have reverted that decision, likely due to battery concerns on phones. For plug-in Android devices, dav1d is still a great option and a welcome addition to the ecosystem.

iPad Pro gets AV1 playback support with M4 processor

In early May 2024, Apple continued their march toward full AV1 support with the announcement of their new M4 chip, which will power the new iPad Pro. The Media Engine of M4 is the most advanced to come to iPad, supporting several popular video codecs, like H.264, HEVC, and ProRes, in addition to AV1.

Ready to get started with AV1?

Bitmovin has added AV1 codec support to our Per-Title and 3-pass encoding optimizations and made AV1 encoding available in our dashboard UI, so now you can perform your first AV1 encode without any code, API calls, or configuration necessary! Bitmovin’s AV1 encoding has supported DASH streaming together with Widevine content protection for a long time, but we’ve now also added support for fMP4 in HLS playlists together with FairPlay content protection to take advantage of Apple AV1 support for premium content. It’s also available in our free trial, so there’s never been a better time to check it out and begin taking advantage of the bandwidth savings and quality improvements that AV1 can provide.

Related Links

Website: Bitmovin’s AV1 hub   

Blog: State of AV1 Playback Support

Blog: Everything you need to know about Apple’s AV1 Support

Blog: 4K video at SD bitrates with AV1

The post New Firefox AV1 support for Encrypted Media Extensions appeared first on Bitmovin.

We concluded the event with our first annual Bitmovin Innovators Network partner awards to recognize and celebrate the amazing work of our partners who embrace the fact that the industry is “Better Together”, by creating solutions with partners that are designed to simplify customers’ video workload needs and advance the viewing experience for audiences.

I am incredibly proud to share the winners of the Bitmovin Innovators Network partner awards below, and the contributions they’ve made: 

Accenture – Global Systems Integrator of the Year:

Accenture and Bitmovin exemplify the “better together” approach through their close strategic partnership, including an ongoing collaboration with the world’s largest motorsports content owners that led to joint engagements with several of the largest sports and media brands in the world.

Broadpeak – Global ISV Partner of the Year:

Broadpeak embodies the “Better Together” spirit through its unwavering strategic collaboration with Bitmovin. This powerful partnership has yielded several key benefits. Together, they have developed solutions that integrated with Bitmovin’s encoder, player, and analytics, resulting in improved workflows for customers; created a consistent two-way communication between sales teams which has resulted in successful deals with European media brands, and joint marketing and PR initiatives at local events to strengthen their joint brand presence.

MediaKind –  Global Service Provider Partner of the Year:

MediaKind and Bitmovin have developed and maintained a robust strategic partnership that has launched sports applications for world-renowned sports leagues. These applications, including launching an app with a sports league on Apple Vision Pro that garnered rave reviews at the Apple launch event, have significantly boosted market visibility for both brands.

Microsoft Azure Marketplace – Cloud Marketplace of the Year:

Bitmovin has had unprecedented success with the Microsoft Azure Marketplace, including more than 200 new customer wins since June 2023. Azure Marketplace has quickly become Bitmovin’s largest and most successful sales channel.

Nomad Media – Americas Regional Channel Partner of the Year:

Nomad Media has deployed over 30 customers on the Bitmovin Play platform in 2023 alone as part of its Nomad Media platform. Nomad Media has also innovated on the player capabilities with dynamic multi-view capabilities. These advancements were showcased to major US clients, propelling both companies forward. This collaboration not only built a strong pipeline but also significantly boosted brand recognition in the US market.

G&L Geißendörfer & Leschinsky – EMEA Regional Channel Partner of the Year:

G&L is a proactive and committed industry partner, who has worked with Bitmovin on both successful sales and marketing initiatives. The collaboration between the two companies resulted in joint revenue, a new logo, and G&L also exhibited on the Bitmovin stand at IBC 2023 where it highlighted how the two companies’ solutions work together. Bitmovin and G&L also hosted a joint CMCD webinar together, which attracted attendees from key German broadcasters and various telecoms and content providers, and it recently published an e-commerce case study with Home Shopping Europe.

Viet Communications – APAC Regional Channel Partner of the Year

Vietcoms was the first licensee for the Bitmovin Player in the Asia Pacific region. Vietcoms was selected for its hard work and efforts in securing our impressive player business in Vietnam and developing agile operational models to meet the specific customer and TelCo business needs and technical requirements.

Once again, I’d like to give huge congratulations to all the winners. A huge thank you to everyone who attended the Bitmovin Innovators Network Partner Executive Networking Event, and to every single one of our partners who continue to embrace the spirit of “Better Together.” IBC is just around the corner, and we will have some exciting initiatives and announcements coming soon to share with you ahead of the show.

The post The Bitmovin Innovators Network “Better Together” Award Winners! appeared first on Bitmovin.

We’re very proud of how our encoder maintains the visual quality of the source files, while significantly reducing the amount of data used, but now we’re exploring how we can actually improve on the quality of the source file for older and standard definition content. Super Resolution implementations have come a long way in the past few years and have the potential to give older content new life and make it look amazing on Ultra-High Definition screens. Keep reading to learn about Bitmovin’s progress and results. 

What is video Super Resolution and how does it work? 

Super Resolution refers to the process of enhancing the quality or increasing the resolution of an image or video beyond its original resolution. The original methods of upscaling images and video involved upsampling by using mathematical functions like bilinear and bicubic interpolation to predict new data points in between sampled data points. Some techniques used multiple lower-resolution images or video frames to create a composite higher resolution image or frame. Now AI and machine learning (ML) based methods involve training deep neural networks (DNNs) with large libraries of low and high-resolution image pairs. The networks learn to map the differences between the pairs, and after enough training they are able to accurately generate a high-resolution image from a lower-resolution one. 

Bitmovin’s AI video Super Resolution exploration and testing

Super Resolution upscaling is something that Bitmovin has been investigating and testing with customers for several years now. We published a 3-part deep dive back in 2020 that goes into detail about the principles behind Super Resolution, how it can be incorporated into video workflows and the practical applications and results. We won’t fully rehash those posts here, so check them out if you’re interested in the details. But one of the conclusions we came to back then, was that Super Resolution was an especially well-suited application for machine learning techniques. This is even more true now, as GPUs have gotten exponentially more powerful over the past 4 years, while becoming more affordable and accessible as cloud resources. 

ATHENA Super Resolution research

Bitmovin’s ATHENA research lab partner has also been looking into various AI video Super Resolution approaches. In a proposed method called DeepStream, they demonstrated how a DNN enhancement-layer could be included with a stream to perform Super Resolution upscaling on playback devices with capable GPUs. The results showed this method could save ~35% bitrate while delivering equivalent quality. See this link for more detail. 

Other Super Resolution techniques the ATHENA team has looked at involve upscaling on mobile devices that typically can’t take advantage of DNNs due to lack of processing power and power consumption/battery concerns. Lightweight Super Resolution networks specifically tailored for mobile devices like LiDeR and SR-ABR Net have shown positive early outcomes and performance. 

AI-powered video enhancement with Bitmovin partner Pixop

Bitmovin partner Pixop specializes in AI and ML video enhancement and upscaling. They’re also cloud native and fellow members of NVIDIA’s Inception Startup Program. They offer several AI-powered services and filters including restoration, Super Resolution upscaling, denoising, deinterlacing, film grain and frame rate conversion that automate tedious processes that used to be painstaking and time consuming. We’ve found them to be very complementary to Bitmovin’s VOD Encoding and have begun trials with Bitmovin customers. 

One application we’re exploring is digital remastering of historic content. We’ve been able to take lower resolution, grainy and generally lower quality content (by today’s standards) through Pixop’s upscaling and restoration, with promising results. The encoded output was not only a higher resolution, but also the application of cropping, graining and color correction resulted in a visually more appealing result, allowing our customer to re-monetize their aged content. The image below shows a side-by-side comparison of remastered content with finer details.

Interested in giving your older content new life with the power of AI video Super Resolution? Get in touch here.

Related Links

Blog: Super Resolution Tech Deep Dive Part 1

Blog: Super Resolution Tech Deep Dive Part 2

Blog: Super Resolution Tech Deep Dive Part 3

Blog: AI Video Research

ATHENA research lab – Super Resolution projects and publications

pixop.com

The post AI-powered Video Super Resolution and Remastering appeared first on Bitmovin.

When it comes to content scale and audience reach, Globo is on par with Hollywood and the big US broadcasters with over 3,000 hours of entertainment content being produced each year. The viewership numbers are equally impressive with forty-nine million Brazilians watching the daily, one-hour newscast and Globo’s Digital Hub attracting eight out of ten Brazilians with internet access. The Digital Hub hosts a variety of content categories, from news, sports, and entertainment to live events such as the Olympics, Carnival, and the FIFA World Cup. Globo also runs a subscription video on demand (SVOD) service called Globoplay that streams live sports, licensed content, as well as movies and television series produced by Estúdios Globo, the largest content production studio in Latin America.

Globo standard of quality

Globo has worked hard to build and become known for the “Globo Standard of Quality”. This included creating the optimal viewing experience together with award-winning content, delivered in stunning visual quality. To develop that reputation, Globo became one of the first mainstream broadcasters outside of the US to offer content in 4K, adopting it as a new standard across its platforms and devices. It has already produced hundreds of hours of 4K content (including HDR) with over a thousand hours of encoding output with its telenovelas and original series. The early adoption of 4K is even more impressive for Globo as Brazil is ranking 79th on the list of countries by Internet connection speed. In order to deliver high-quality video, operators cannot just work with higher bitrates but rather have to find the optimal encoder that achieves both quality, speed, and cost-efficiency at the same time. In the past, 4K encoding was accomplished with on-premises hardware encoders. As the next update cycle of the appliances was fast approaching, Igor Macaubas, Head of Online Video Platform, and Lucas Stephanou, Video Platform Product Owner at Globo, decided to conduct a thorough evaluation of vendors, and ultimately chose Bitmovin.

“We are not willing to compromise the visual integrity of our content and we hold ourselves to strict perception-quality standards. Bitmovin’s renowned 3-Pass Encoding exceeded our expectations and ensures that high perceptual quality can still be delivered while streaming at optimal bandwidth levels.”

– Lucas Stephanou (Video Platform Product Owner, Globo)

Globoplay, powered by Bitmovin VOD Encoding on Google Cloud

Globo handles a massive VOD library of over a million titles, and with 12 variants in their HEVC bitrate stack — encoding demands are high. Bitmovin’s VOD encoding service running on Google Cloud gave Globo the capability to encode a 90-minute video asset in 14 minutes across the entire HEVC ladder. This is a realtime factor of 6.4 times, which resulted in a quantifiable impact on time-to-market. Globo saw the business need for fast turnaround time in encodes and chose Bitmovin as the clear front runner in this regard. 

Bitmovin VOD Encoding on Google Cloud is an easy-to-use, fully-managed video transcoding software-as-a-service (SaaS). Bitmovin VOD Encoding allows customers to efficiently stream any type of on-demand content to any viewing device. Customers use Bitmovin VOD Encoding for a wide range of on-demand streaming use cases, including Subscription Video on Demand (SVOD), Transactional VOD (TVOD), and Ad-supported VOD (AVOD) services, online training, and other use cases. Bitmovin’s Emmy Award® winning multi-codec outputs and per-scene and per-title content-aware transcoding produce higher visual quality video outputs at lower bit rates than other file-based transcoding SaaS to optimize content delivery and reduce streaming cost. Bitmovin VOD Encoding is available for purchase on Google Cloud Marketplace.

Bitmovin’s 3-Pass Encoding algorithm uses machine learning and AI to examine the video on a scene-by-scene basis. It analyzes the content’s complexity multiple times to optimize intra-frame and inter-frame compression. This helps determine the ideal resolution and bitrate combinations that maximize the quality and efficiency. All together, this ensures the visual elements of the video are not degraded in the encoding process and prevents unnecessary overhead data that might impact the viewing experience. 

Processing HD and 4K video with Globo’s volume requires computing resources that would exceed the CapEx budgets of most companies. This is where the Google Cloud’s flexibility and on-demand compute power really shine. Together with Bitmovin’s split-and-stitch technology, single encoding jobs run significantly faster with parallel processing and spikes in demand are handled with ease and throughput that is just not possible with on-premises encoding. Customers also have the option to deploy Bitmovin VOD Encoding as a managed service running in the Bitmovin account or as a single tenancy running in the customer’s Google Cloud account. This allows encoding costs to be applied toward any annual spending commitments.

“Globo is known to set quality standards. We want our viewers to experience our great content in stunning video quality. Our 4K workflows have been relying on hardware encoders, but we wanted to test the power of the cloud and conducted a thorough vendor evaluation based on video quality. Bitmovin’s encoding quality and speed convinced us across the board. And, since using Bitmovin’s encoding service running on Google Cloud, we are spending a fraction of the cost by bringing our capital cost down without spending more on operational cost.”

– Igor Macaubas (Head of Online Video Platform, Globo)

Olympics in 8K

One prime example of this collaboration innovating and pushing the boundaries of video quality is from the Tokyo Olympics in 2021, where 8K VOD content from the Olympics was delivered to viewers at home via Globoplay. This marked the first time that the Olympics were viewable in 8K resolution outside of Japan. 8K video has 16x the resolution of HD and 4x that of 4K, so it requires an enormous amount of processing power and advanced compression to lower the data rates for delivery to end users. 4K and 8K content is also referred to as Ultra High Definition (UHD) and is usually mastered in a High Dynamic Range (HDR) format that allows for brighter highlights, more contrast and a wider color palette. Hybrid-Log Gamma (HLG) is an HDR format that was developed for broadcast applications and backward compatibility with Standard Dynamic Range (SDR) television sets.    

After receiving the HLG mastered content from Intel in Japan, Globo utilized Bitmovin VOD Encoding on Google Cloud’s compute instances for efficient parallel processing with Bitmovin’s VOD Encoding API. 8K/60p transcoding was performed using the High Efficiency Video Coding (HEVC) codec, creating an optimized adaptive bitrate ladder. At this stage, Bitmovin’s 3-pass encoding was key for transforming the content into a compatible size for transport over broadband internet connections, without sacrificing the stunning 8K visual quality. The 8K content was then delivered via Globo’s own Content Delivery Network (CDN) infrastructure to subscribers of Globoplay with 8K Samsung TVs.

“Our 3-Pass Encoding proved to be the right encoding mode. It ensured high perceptual quality could still be delivered while streaming at optimal bandwidth levels. With our split-and-stitch technology running on Google Cloud’s scalable infrastructure, we were able to deliver both speed and quality for this time-sensitive content.”

– Stefan Lederer (CEO, Bitmovin)

Learn more about Bitmovin’s VOD Encoding SaaS here.

Related Links

Google Cloud Media & Entertainment Blog

Bitmovin on Google Cloud Marketplace

Globo – Bitmovin Customer Showcase and Case Study

The post Globo, Google Cloud and Bitmovin: Taking Quality to New Heights appeared first on Bitmovin.

In the early days of digital video, encoding a full-length movie could take several hours or even days to complete, depending on the settings and techniques that were used. Over time, as processor speeds increased and specialized hardware was introduced, encoding turnaround times decreased, but it was usually an incremental, linear response to the advancements in technology. Once cloud computing resources became readily available and opened new possibilities, cloud-native encoding services like Bitmovin disrupted the status quo with massive gains for encoding speed and turnaround times. This potential was unlocked by developing an innovative new technique known as split-and-stitch encoding. 

What is split-and-stitch encoding? 

As the name suggests, split-and-stitch encoding is a method of encoding that involves splitting a file into smaller chunks, encoding those chunks separately, and then stitching them back together. These smaller chunks being encoded in parallel with separate cloud computing resources led to huge leaps in shortening turnaround times. Prior to that, digital videos were processed linearly, which was an unnecessary limitation carried over from film and tape processing workflows, where the physical medium was actually a limiting factor.  

How fast is split-and-stitch encoding?

Back in 2015 when Bitmovin first implemented our encoder on the Google Compute Engine (now Google Cloud Platform) we were able to achieve encoding speeds of 66x real-time running in their cloud, as mentioned here. With some further optimization, we became the first to reach 100x real-time encoding speeds. 

The actual turn-around times for your encoding jobs will depend on a lot of factors including source format, codec(s), resolution, duration and advanced features like Dolby Vision, but even with very complex 4K HDR workflows, your encodes will run faster than real-time using split and stitch. Below is a real-world example of an H.264/AAC encoding that ran faster than 92x real-time.

Running split-and-stitch encoding in the cloud means your individual encoding jobs run faster than real-time, but it also means that you can scale to run many jobs in parallel which allows large backlogs to be cleared in hours instead of weeks. You also have the capacity to handle spikes of content with no impact on queue time.

What are the advantages of Bitmovin’s split-and-stitch encoding?

Bitmovin has over a decade of experience developing and refining our split-and-stitch implementation. We built our system to take advantage of spot and preemptible instances to keep costs down, while surpassing the quality of single instance encodes with innovations like 3-pass encoding and Smart Chunking.  Our intelligent workload orchestration allows you to manage priority and resource scheduling with capacity for thousands of jobs per hour.

Bitmovin also supports using multiple codecs and packaging formats together with split-and-stitch, including H.264 (AVC), H265 (HEVC), VP9 and AV1 with both HLS and DASH, where other platforms may be limited to H.264 and HLS. We’ve also implemented fast decode enhancements for large J2K and ProRes mezzanine source files that reduce the overall turnaround time even further. 

What is Smart Chunking?

In 2023, Bitmovin made some key changes and updates to our VOD Encoder with a new feature called Smart Chunking. This further increased the potential visual quality and turnaround times that were possible with split and stitch by decoupling the split-and-stitch chunk duration from the user-defined segment duration. This allows for variable chunk size depending on the type of codec and the complexity of encoding, enabling many immediate improvements and future optimizations. Using Smart Chunking means we can segment chunks at the optimal points with better bitrate distribution, providing more consistent quality without any noticeable dips. 

In the graph below, you can see a comparison of an encoding job run with and without Smart Chunking. While the overall quality is similar, in the blue version (without Smart Chunking) there are several lower quality outlier frames. By using Smart Chunking (orange version) the lowest 1% of frames in terms of quality were improved by an average of 6 VMAF points, which is a noticeable difference. The lowest 0.1% improved by 22 VMAF points and the single worst frame gained a massive 60 VMAF points.

Is split-and-stitch always the best approach? 

The steps of analyzing, splitting and reassembling chunks of video do add some overhead processing time to the encoding process. For longer episodic content or movies, the added time is negligible compared to the time saved by using split-and-stitch. But, for shorter videos like ads and news clips that are time-sensitive, the pre-processing can make using split-and-stitch less advantageous. 

For these cases, Bitmovin has 2 solutions. First, we’ve added support for hardware encoding with Nvidia T4 GPUs. They can deliver the same quality of video encoding, up to four times faster than CPUs, with H.264 (AVC) and H.265 (HEVC) codec support. We also have a new “accelerated mode” that uses pre-warmed cloud compute resources, so you no longer have to wait for new instances to be started. This has made a huge impact on overall encoding job turnaround time, lowering queuing times from minutes to <10 seconds.

Ready to get started with split-and-stitch encoding?

Bitmovin’s split-and-stitch encoding with Smart Chunking is enabled by default and doesn’t require any special configuration. You can get started quickly with our dashboard encoding wizard without any coding required. Get going today with our free trial and see the results for yourself by clicking here!

The post Split-and-Stitch Encoding with incredible speed, quality and scale appeared first on Bitmovin.

The AV1 Video Codec

First things first, what is AV1 and where does it come from? In September 2015 the Alliance for Open Media (AOMedia) was founded by leading companies from various industries with an association with media technology. Among them are browser vendors like Google, Mozilla, and Microsoft, hardware vendors like AMD, ARM, Intel, and NVIDIA, and content providers like Amazon and Netflix. The goal of the AOMedia is to develop an open, royalty-free, next-generation video coding format that is:

• Interoperable and open
• Optimized for the Internet
• Scalable to any modern device at any bandwidth
• Designed with a low computational footprint and optimized for hardware
• Capable of consistent, highest-quality, real-time video delivery, and
• Flexible for both commercial and non-commercial content, including user-generated content.

The new video coding format AOMedia Video 1 (AV1) is meant to replace Google’s VP9 and compete with HEVC/H265 from MPEG. The Alliance is targeting an improvement of about 50% over VP9/HEVC with only reasonable increases in encoding and playback complexity.
When comparing AV1 with HEVC, probably the biggest competitive advantage of AV1 will be that it is royalty-free, especially if we look at the still very uncertain royalty situation with HEVC. Currently, there are two patent pools with MPEG LA and MPEG Advance, plus some unknown HEVC IP owners who have not joined a pool yet. In the end, nobody will know how much you will need to pay in royalties for HEVC. This situation is obviously not satisfactory for the industry and especially, encoding, distribution, content, and hardware companies. (Download the AV1 Datasheet)

Bitmovin and AV1 Video Encoding as of 2024

We have made improvements to the core AV1 encoder in itself and have extensively benchmarked it against multiple practical use cases. The turnaround time and speed of encoding have improved by several orders of magnitude. And in regards to the quality, for the encoder version release v2.110.0, we found that AV1 can offer the same visual quality at 50% less bitrate for H.264/AVC and 30% less bitrate for H.265/HEVC respectively. That is a pretty significant gain!
In addition to the improvements to the core encoder itself, we have integrated AV1 with all the popular features that our customers have come to love. Here is a quick rundown : 

• Since encoder version 2.104.0, 3-pass encoding with AV1 is generally available. We have found that three-pass AV1 video encoding provides significantly better bitrate distribution compared to the regular 2-pass encoding.
• Since encoder version 2.109.0, Per-Title encoding with AV1 is available now. Per-Title is one of our biggest competitive advantages. We are proud to offer this also for AV1. 
• Since encoder version 2.110.0, AV1 video encoding offers three smart presets. This allows customers to choose an optimal tradeoff between the quality and speed of the AV1 encodings. 
• Since encoder version 2.187.0, AV1 video encoding can be used in HLS playlists, together with FairPlay content protection. This enables support for AV1 playback on compatible Apple devices like the iPhone 15 Pro and new laptops with Apple’s M3 processor.

Also at Bitmovin, we like to keep our promises . We promised seven years ago that we will not stop innovating around AV1 and that we will enable our customers in the best possible way with our AV1 solutions. We are excited to announce that we have kept our end of the bargain. We have developed two patent-pending technologies around AV1. We cannot delve into the details now but just to tease you out, it significantly improves the turnaround times for Per-Title and 3-pass encodings. Keep watching this space for more details about this soon!
And here is the cherry on top of all this. It’s easy to get all this awesome Per-Title ABR encoding together with the AV1 codec and DASH packaging in a SINGLE API call! Yes, it’s not a typo. We said SINGLE. Can you believe that !? What are you waiting for you? It’s easier than ever to get started with AV1. Try it and reach out to us if you have any questions! We are happy and excited to get you onboard with AV1.

How AV1 Video Encoding Development Works

The AV1 codec has its roots in the codebase of Google’s VP9/VP10 codec with an additional 77 experimental coding tools that have been added and are under consideration. Out of that 77 experimental coding tools, only 8 are currently enabled by default (adapt_scan, ref_mv, filter_7bit, reference_buffer, delte_q, tile_groups, rect_tx, cdef), but the performance of the codec is already appealing. The final goal is to get as many promising coding tools into the final version of the codec and afterward freeze the bitstream specification.
The following procedure explains the high-level process on how experiments can be added to the AV1 codec:

1. Coding tools are added as experiments into the AV1 codebase. They are controlled at build-time by flags (e.g., –enable-experimental –enable-<experiment-name>).
2. The hardware team (group of hardware members inside of AOMedia) reviews the experiments to ensure it can be implemented in hardware.
3. Each experiment needs to pass an IP review to ensure no IPs are violated.
4. Once reviews are passed the experiment can be enabled by default.

As of today, it is not sure which experiments will make it into the final codec. However, we want to highlight a few that look promising today:

Directional Deringing

It is an effective algorithm for removing ringing artifacts from a coded frame. It plugs in right at the end of the decoding process, so it is easy to integrate. Blocks are searched for an overall direction that is taken into account when applying a conditional replacement filter (CRF) to reduce the risk of blurring and only take obvious ringing patterns into account. It is currently enabled by default.

PVQ (Perceptual Vector Quantization)

This experiment was originally developed for the Daala codec and has the potential to bring a lot of gains, however, it is also quite difficult to integrate into AV1 because PVQ interacts with many other parts of a codec. Compared to the usual scalar quantization, PVQ offers a lot more flexibility to control quantization. It makes techniques like Chroma from Luma or Activity Masking easier. Activity Masking is trying to provide better resolution in low contrast areas. This can be achieved by varying the codebook which is possible with PVQ.

Chroma from Luma (CfL)

CfL is based on a rather simple idea: Take advantage of the fact that edges in the chroma plane are usually well correlated with those in the luma plane. As CfL works entirely in the frequency domain, it can be easily implemented using PVQ. Using PVQ, the chroma coefficients can be predicted from injected luma coefficients. It is a very promising tool as it is quite simple to compute and provides nice benefits with much cleaner colors.

Bitmovin AV1 VoD and Live Encoding

The Bitmovin encoding service now supports AV1 video encoding for VoD and Live. It is possible to encode AV1 with our cloud encoding service. Currently, AV1 video encoding with common encoding tools is a very time-consuming process, as can be seen in the below screenshot taken from a Lenovo T540p notebook with an i7-4800MQ, 8GB RAM running Ubuntu 14.04. It would take 8 hours and 42 minutes to encode a 1080p@24fps 40-second long sequence (Tears of Steel Teaser) with a target bitrate of 1.5Mbps.

The encoding runs with about 1.93 fpm (frames per minute) which would translate to 0.032 fps (frames per second). If you want to achieve real-time with 24 fps you would need at least 746 times the computing power on a single machine, which is not very practical in a real-world scenario. Clearly, we need another approach to encode with reasonable speeds, especially when it comes to live streaming.
Thanks to our chunk-based encoding approach that allows us to scale a single encoding among multiple instances we can encode AV1 with reasonable turnaround times and it’s also possible to use AV1 for live streams. Our chunked encoding allows us to speed up the encoding almost linearly with the number of instances that are added to the encoding cluster and this approach works with our cloud encoding the same way it works with our on-premise setups that are based on Kubernetes and Docker. Consequently, we can reach the same encoding speeds for AV1 that our customers have come to expect for H264, VP9, and HEVC encoding, which makes the codec effectively usable for media companies and content providers throughout the industry.

We also encoded the ToS teaser with our AV1 encoder in the cloud with the default configuration where we achieved 7 fps, which is about 219 times faster than what was achieved in the test with the Lenovo notebook. This is already pretty impressive however, we were not satisfied with the speed as it was still below real-time. So we tried with an enterprise set-up by just adding more instances to the encoding process. The resulting encoding speed was at 36 fps, which is about 1125 times faster than with the single Lenovo notebook.

In addition, we don’t have to compromise on quality for speed because our encoder does not need to sacrifice quality to reach a certain speed on a single instance as other encoding vendors typically do. With our approach we are not bound to the hardware restrictions of a single instance, we can add more instances to an encoding cluster to generate the quality that our customers have configured in a reasonable time or in real-time for live streams. With our chunk-based implementation of the AV1 video codec, we can encode videos with AV1 even faster than in real-time without compromising quality.

How to implement an AV1 Livestream

In most cases, to run live stream encodings you would need around 4 to 15 Mbps with traditional codecs like H264 to deliver the same quality. So AV1 could reduce your CDN and storage cost by up to 10x.
The setup of our AV1 live workflow that we will showcase consists of the following components:

• OBS RTMP mezzanine stream, 12Mbps 1080p@30fps
• Bitmovin Distributed AV1 Cloud Encoder running in Google Cloud receives an RTMP ingest and transcodes to 1.5Mbps 1080p@30fps segmented WebM. Segments will be directly transferred to a Google Cloud Storage bucket.
• The Bitmovin Distributed AV1 Cloud Encoder also generates HLS and MPEG-DASH manifests that will be transferred to the Google Cloud Storage bucket. Enabled experiments of the AV1 codec are: adapt_scan, ref_mv, filter_7bit, reference_buffer, delte_q, tile_groups, rect_tx, cdef
• Native playback on a desktop with a Bitmovin Player based on aomdec and ffplay

Our AV1 encoder generates WebM segmented output that could be used with HLS or MPEG-DASH for VoD and Live. However, as AV1 is currently not supported by any browser, we had to write our own player that is able to playback our AV1 live stream. We updated the aomdec application to be able to download and decode the AV1 chunks which can be seen in the left console window. Fortunately, decoding is not as resource intensive as the encoding, which allows you to decode the AV1 stream on normal hardware without special requirements, e.g., the same Lenovo notebook (i7-4800MQ, 8GB RAM running Ubuntu 14.04) that was not capable of encoding this video just near to realtime could easily playback AV1 in software. After the decoding step, we pipe the decoded YUV frames to ffplay to display the stream in a window as you see in the screenshot above. We plan to contribute this functionality back to aomdec after a technical cleanup of the current implementation.

A Practical Quality Comparison

Although the bitstream from AV1 is not finalized yet and much work needs to be done to further improve the quality of the codec, we wanted to get a snapshot of the current state and compare its quality with AVC/H264, HEVC/H265, and VP9. For that purpose, we made two different quality comparisons, the first one with two objective metrics, PSNR and SSIM. PSNR does not always correlate well with perceived quality but is the de-facto standard for video quality comparisons. SSIM is a perception-based quality metric that should give better results in regard to perceived quality.
For the second comparison, we chose to make a side-by-side quality comparison between AV1 and the other codecs. This quality comparison targets a practical use case where the resulting content can be used for Adaptive Bitrate Streaming (ABR). Therefore we have used a fixed Group of Pictures (GOP) size for our experiments and also used Variable Bitrate (VBR) encodings with a target bitrate. This approach is established in the industry but results can vary from scientific evaluations that purely target abstract use cases and theoretical encoder performance through the HM (HEVC reference software) and JM (AVC reference software) reference software that has no practical relevance in the industry.
Let’s first start with the objective quality comparison with PSNR. We encoded the open-source movie Sintel from the Blender Foundation with VBR to the following target bitrates: 100Kbps, 250Kbps, 500Kbps, 1Mbps, 2Mbps, 4Mbps and calculated PSNR and SSIM for the bitrate that has actually been achieved by the individual codec (typical codecs in VBR mode do not hit the target bitrate exactly).
The following encoding settings for the different codecs were used in the Bitmovin Encoding Service:

• AVC/H264:
  GOP Size: 96 frames (4 seconds), Me_range: 16, Cabac: true, B-Adapt: 2, Me: UMH, Rc-Lookahead: 50, Subme: 8, Trellis: 1, Partitions: All, BFrames: 3, ReferenceFrames: 5, Profile: High, Direct-Pred: Auto

• HEVC/H265:
  GOP Size: 96 frames (4 seconds), Sao: 1, B-Adapt: 2, CTU: 64, Profile: Main, BFrames: 4, Rc-Lookahead: 25, WeightP: 1, MeRange: 57, Ref: 4, Subme: 3, Tu-Inter-Depth: 1, Me: 3, No-WeightB: 1, Tu-Intra-Depth: 1

• VP9:
  GOP Size: 96 frames (4 seconds), Cpu-used: 1, Tile-columns: 4, Arnr-Type: Centered, Threads: 4, Arnr-maxframes: 0, Quality: Good, Frame-Parallel: 0, AQ-Mode: none, Arnr-Strength: 3, Tile-Rows: 0

• AV1:
  Build f3477635d3d44a2448b5298255ee054fa71d7ad9, Enabled experiments by default: adapt_scan, ref_mv, filter_7bit, reference_buffer, delte_q, tile_groups, rect_tx, cdef
  Passes: 1, Quality: Good, Threads: 1, Cpu-used: 1, KeyFrame-Mode: Auto, Lag-In-Frames: 25, End-Usage: VBR

The above diagram clearly shows that AV1 already outperforms all the other codecs for each bitrate setting. For bitrates from 1Mbps and higher the quality difference is already pretty big (> 0.5db which is usually clearly visible). VP9 and HEVC/H265 are very similar from a PSNR perspective, however, VP9 was the codec that overshot the target bitrate by far the most.

We also compared the four codecs with SSIM. The results can be seen in the above diagram and are quite similar to PSNR with some slight differences. AV1 is still the best performing codec over all bitrates, and AVC/H264 lags behind. However, interestingly AVC/H264 catches up with increased bitrate. An explanation for that could be that in the higher bitrates we can reach nearly the quality of the source material with all codecs, which results in only minor differences between the codecs.
Additionally, we created several side-by-side quality comparisons where we experimentally changed the target bitrate for each codec to reach an average of 500 Kbps. Below you can see the quality comparisons between the encodings comparing the quality of Bitmovin AV1 video encoding with AVC/H264, HEVC/H265, and VP9. We used the well-known Tears of Steel teaser that is 40 seconds long with a 1080p resolution for the comparison, selecting a complex scene that is hard to encode.

When comparing AV1 video encoding with AVC/H264 the quality difference is very obvious as expected. We can clearly see multiple encoding artifacts and blocking in the right part of the image that has been encoded with AVC/H264. In contrast, the left part with AV1 Video Encoding looks much cleaner without obvious encoding artifacts.

Looking at the quality difference between AV1 and VP9 it is not as obvious as with AVC/H264, but still quite visible. Especially the borders of the tiles of the sphere show encoding artifacts and the overall picture in VP9 seems to have quite some noise. We can also identify some blocking artifacts that are not visible in AV1.

HEVC/H265 visually looks a bit better than VP9, however, it still has visible encoding artifacts, especially in the lower part of the image and around the arm of the guy with the red coat. When we look closely at the arm we can see that the color is not encoded as nicely as with AV1 and shows some noise.

Conclusion

Bitmovin’s culture and vision have always been to be a technology leader and our passion for video means we consistently tackle the most complex video problems. Why? Because it’s fun and challenging and our team loves a challenge!
Besides that, there are already use cases for an AV1 video encoding where you could use it as your mezzanine format to preserve a high-quality version of your video at a low bit rate that can be used to create your adaptive bitrate renditions or other formats. Using AV1 for that use case would decrease your storage footprint and speed up transfer times inside of your data center or for upload to the cloud.
Furthermore, with the companies behind AOMedia, like AMD, ARM, Intel, NVIDIA, Google, Microsoft, Mozilla, Netflix, and Amazon, it should not take too long to get broad support for AV1. AMD, Intel, and NVIDIA cover the desktop market quite nicely, and ARM and Intel the mobile market. Additionally, the major browser vendors, Google, Microsoft, and Mozilla will make sure that the codec finds its way into the browsers soon after the bitstream freeze. Google, Netflix, and Amazon will make sure that AV1 content will be available quickly and that will further drive adoption and hardware support.
AV1 is the next generation video codec and it’s on track to deliver a 30% improvement over VP9 & HEVC – Learn More

More AV1 Resources:

• 4K Video at SD Bitrates with AV1
• Bitmovin’s AV1 Encoding Gift Guide
• Cool New Video Tools: Five Encoding Advancements Coming in AV1

The post Bitmovin Improves Support AV1 Video Encoding for VoD appeared first on Bitmovin.

Table of Contents

Introduction

The post will explain how Per-Title Encoding works and the advantages of using Per-Title Encoding compared to using the same bitrate ladder for all your content. Per-Title often requires fewer ABR ladder renditions and lower bitrates that translate into storage, egress and CDN cost savings. It also improves QoE with less buffering and quality drops for viewers, along with better visual quality. On top of that, it can make 4K streaming viable, turning it from a loss leader and financial burden into a revenue generator. Keep reading to learn more. 

Per-Title Encoding is key for cutting streaming costs

For the past couple of years, “controlling costs” has been among the top challenges for video streaming, according to the results of Bitmovin’s annual video developer survey. While the pandemic years created a boom for streaming services and content creation, things have now shifted toward cost-cutting in a few different ways. Several platforms have cut back their budgets for original content and are removing shows and films from their libraries to save on licensing and other operational costs. 

Another trend highlighted by industry analyst Dan Rayburn, has been the lowering of bitrates, including removal of 4K streaming in some cases. Services that do still offer 4K often restrict it to their highest-priced subscription tier. Back in 2014, Dan called out the cost and QoS challenges services would face when delivering 4K video, and many are still struggling with that reality, especially those using a fixed bitrate ladder for their encoding. 

Per-Title Encoding can have a huge impact on 4K content, something that can be seen in the recommended internet connection speeds for 4K streaming:

Netflix: 15 Mbps (they use their own version of per-title encoding)

Disney+: 25 Mbps

Paramount+: 25 Mbps

Max: 50+ Mbps 

For long form content that gets even in the tens of thousands views, the difference between 15 Mbps and 25 or 50 Mbps will add up quickly in the form of excess network egress and CDN costs. With non-optimized encoding at those high bitrates, a viral hit that ends up getting hundreds of thousands or millions of views can end up being a financial burden. Using Per-Title Encoding ensures each video uses only the bits needed for its content and complexity and when combined with more advanced codecs like HEVC and AV1, it can make a game-changing difference. When Bitmovin added support for using Per-Title Encoding with the AV1 codec, I was shocked to see just how low the bitrate could go (often under 2 Mbps). 

How does Per-Title Encoding work?

In 2012, Bitmovin’s co-founders published a research paper titled “Dynamic Adaptive Streaming over HTTP Dataset” that, among other things, provided data for per-genre encoding that would further evolve into Bitmovin’s Per-Title Encoding. Per-Title Encoding is an optimization of adaptive bitrate encoding that analyzes the complexity of a video file and determines the encoding settings needed to maintain the highest level of visual quality together with the most efficient adaptive bitrate ladder. 

In 2015, Netflix published a tech blog that detailed their research and development of their own per-title encoding. Through brute force encoding of content at different resolutions and quality levels, they found that the ideal adaptive bitrate ladder for each video would form a smooth convex hull when plotting quality vs bitrate. When the bitrate and resolution pairs in their ABR ladder fell along the convex hull, it maximized quality for the viewer and meant that data was being distributed efficiently. Bitmovin’s Per-Title complexity analysis spares you the excessive testing and experimentation and automatically determines the ideal ABR ladder and convex hull for each file. 

Per-Title Encoding ABR ladder vs fixed bitrate ladder

The graph below shows how Per-Title Encoding provides better QoE with lower bitrates than the competition’s static bitrate ladder for a 4K source. Per-Title Encoding matches the source at 3840x2160p, with a bitrate of 6050 kbps and a VMAF score of 95.5. The static ladder is capped at 1080p and requires 7830 kbps for a lower VMAF score of 90.9. That’s 22.7% bitrate savings with better quality by using Per-Title.

The next example uses the HEVC codec for the customer’s UHD ladder vs Bitmovin Per-Title Encoding. The highest rendition on the Per-Title ladder only needs 1.9 Mbps to hit a VMAF score of 94.9, while the static ladder uses 15 Mbps, an increase of 13.1 Mbps in bandwidth for an undetectable VMAF difference. This equates to 87% savings on the CDN bill for viewers of the top rendition, without sacrificing quality. 

With a duration of 44:39, the top rendition for Per-Title would mean 0.622 GB in data transfer while the top rendition for the fixed ladder would require 5.023 GB. For popular content tens of thousands of views (or more) those savings add up quickly. At a time when some services are removing 4K renditions, these optimizations make it feasible to provide UHD and improve margins on premium subscription tiers. 

Next we have some medium-complexity 1080p content where using Bitmovin Per-Title with a more advanced codec like HEVC can make a huge difference. Throughout the ladder, using Bitmovin Per-Title with H.264 provides some quality gains and bitrate savings compared to the customer’s static ladder with ffmpeg, but the results from Per-Title with HEVC highlight the impact of using a newer generation codec. HEVC delivers 1080p in the 90+ VMAF range with only 2 Mbps while ffmpeg with H.264 needs over 6.5 Mbps for the same quality. That’s around 70% bandwidth savings for viewers of the top rendition. At the lower end of the spectrum, a viewer with 1 Mbps available bandwidth would be limited to 432p with the static H.264 ladder, but would still receive 1080p with Per-Title HEVC.

Storage savings with Per-Title Encoding

Bitmovin’s Per-Title Encoding can deliver massive storage savings when compared to fixed bitrate ladders, by removing unnecessary renditions from the ABR ladder and ensuring the most efficient bitrate is used for each piece of content. The chart below shows the potential savings on your storage bill from using Per-Title Encoding over a fixed ladder with AVC encoding. 

Improve quality without increasing bitrates 

Per-Title Encoding can also improve quality without needing to use additional data. The chart below references our customer’s fixed ABR ladder using the AVC codec and shows the quality improvements (% VMAF score) that Bitmovin’s Per-Title provided with different codecs at the same bitrate. 

Bitmovin Smart Chunking prevents lower quality outlier frames

The graphs below plot the VMAF quality scores of every frame in our customer’s sample content. Bitmovin’s smart chunking virtually eliminates all of the lower quality outlier frames that are present in our competitor’s encoding and would be noticeable by viewers. Smart Chunking is now active for all Bitmovin VOD Encoding without any additional configuration or cost to the user.

Conclusion

In the past, balancing cost and quality has always been a tradeoff, but using Per-Title Encoding may be the single most effective way for streaming services to reduce their total cost of ownership without sacrificing their viewers’ quality of experience. With consumers having an abundance of options, the QoE improvements Per-Title provides can mean the difference between renewal and churn and its cost savings can tip the scales toward profitability. With streaming firmly in a cost conscious era, using Per-Title Encoding makes more sense than ever before.   

Ready to see what difference Per-Title Encoding can make with your content? Anyone can test it out for free with no coding required using our VOD encoding wizard. We also have a comparison tool in our dashboard where you can input your own content or use common test videos. Try it out today!

Related Links

Choosing the Best Per-Title Encoding Technology

What is Per-Title Encoding and how does it work 

How to Create a Per-Title Encoding

Advanced Per-Title configuration 

The post Game-Changing Savings with Per-Title Encoding appeared first on Bitmovin.

Whether you’re calling it Virtual Reality (VR) or 360 video or Metaverse content, there are a lot of details that should be taken into consideration in order to guarantee a good immersive experience. Things like video resolution, bitrates and codec settings all need to be set in a way that creates a high quality of experience for the viewers, while being conscious of storage and delivery costs that can come with these huge files. Although all this stuff has been widely discussed for 2D displays, like mobile phones and TVs, VR streaming differs enormously from those traditional screens, using different display technology that drastically shortens the viewing distance from eye to screen. In addition to that, VR headset specs may differ from one device to another, so the same video may produce a different visual experience depending on the model or device. In this post we are going to share the things you need to consider, along with tips and best practices for how to encode great looking VR content, specifically for playback on Meta Quest (formerly known as Oculus) headsets.

Visual quality requirements of 3D-VR vs 2D videos

Unlike traditional 2D screens, where viewers are located at a considerable distance from the screen, VR viewers are looking at a smaller screen much closer to the eyes. This drastically changes the way a video should be encoded in order to guarantee good visual quality for an immersive 3D experience. For this same reason, the traditional 2D video quality metrics such as VMAF and PSNR are not usually useful to measure the visual perception for 3-D VR content, for instance:

VMAF for 3D-VR

VMAF considers 2D viewers located at a viewing distance in the order of magnitude of the screen size, for example:

• 4K VMAF model – vmaf_4k_v0.6.1,  takes into consideration that the viewer is located at 1.5H from the screen, where H is the TV screen high. 

• HD VMAF model – vmaf_v0.6.1, considers  a viewer located at 3H from the screen.

The previous models resulted in a pixel density of about 60 pixels per degree (ppd) and 75 ppd – for 4K and HD respectively. However, when talking about VR videos, the pixel density is highly magnified, for instance, for Meta Quest 2 headsets the specs mention a pixel density of 20 ppd. Therefore, the predefined VMAF models are not suitable.  Actually, if you do use VMAF to get the visual quality (VQ) for a VR video intended for headset playback, you’ll probably find it does not look good enough even though it has a high VMAF score – this is because of the “zoom in” that Quest does in comparison to the traditional screens. 

PSNR for 3D-VR

Even when it is not a rule, it is expected to have good VQ on 2D videos when PSNR values are between 39 dB and 42 dB – for average to high complexity videos. See [1] [2] However, this PSNR range is usually not enough to create a good immersive experience with Quest headsets. For instance, according to some empirical tests we did, we found that at least a PSNR above 48 dB is required for good VQ with Quest devices.

The Best Encoding Settings for Meta Quest devices

A general overview of the Video Requirements can be found at the Meta Quest website. Additionally, the following encoding settings may be useful when building your encoding workflow: 

Resolution

The minimal resolution suggested by Meta is 3840 x 3840 px for stereoscopic content and 3840 x 1920 px for monoscopic content, which is much higher than earlier generations or mobile devices.  

H265 Video Codec Settings 

Video Codec – Meta Quest devices support H264(AVC) and H265(HEVC) codecs, however given that they require resolutions above 3840 px, we strongly recommend H265 due to the high encoding efficiency it has when comparing it to H264. 

GOP Length – In our tests we successfully achieved a good VQ within the recommending bitrate range, using a 2-second GOP length for 30 fps content. However, since the VR experience is not as latency sensitive for video on demand, we suggest using greater GOP lengths in order to improve the encoding efficiency even more if needed.

Target bitrate and CRF – Meta suggests a target bitrate between 25-60 Mbps and as mentioned, we strongly suggest using the H265 codec to maintain high visual quality within that range. If the bitrate goes too far above the suggested maximum, customers may experience slow playback or stalling due to device performance issues. 

Having said all that, it is worth mentioning that setting a proper bitrate to meet the VQ expectations is really challenging, mainly because the bitrates necessary may change from one piece of content to another depending on their visual complexity. Because of that, we suggest using a CRF based encoding instead of a fixed bitrate. Specifically, we found that when talking about H265, a CRF between 17-18 would produce videos that are suitable for viewing on Quest headsets without excessively high bitrates. 

Building 360-VR encoding workflows with Bitmovin VOD Encoding

Bitmovin’s VOD Encoding provides a set of highly flexible APIs for creating workflows that fully meet Meta Quest encoding requirements. For instance:

• If adaptive bitrate streaming is required at the output, Bitmovin Per-Title encoding can be used to automatically create the ABR ladder with the top rendition driven by the desired CRF target.
• If progressive file output is required, a traditional CRF encoding can be used by capping the bitrates properly.
• Additionally, Bitmovin filters can be used to create monoscopic content based on a stereoscopic input, for instance, cropping the original/stereoscopic video to convert it from a top-and-bottom or side-by-side array into a single one. Monoscopic outputs can be viewed on 2D displays, extending the reach of your 360 content beyond headsets.

Per-Title Encoding configuration for VR

The following per-title configuration may be used as a reference for encoding a VR content. Depending on the content complexity, the output may include from 4 to 7 renditions with the top rendition targeting a CRF value of 17.

perTitle: {
     h265Configuration: {
       minBitrate: 5000000,
       maxBitrate: 60000000,
       targetQualityCrf: 17,
       minBitrateStepSize: 1.5,
       maxBitrateStepSize: 2,
       codecMinBitrateFactor: 0.6,
       codecMaxBitrateFactor: 1.4,
       codecBufsizeFactor: 2,
       autoRepresentations: {
         adoptConfigurationThreshold: 0,
         },
       },
     }

Theres also full code samples here if you would like to dig deeper.

The same configuration can be used to encode any VR format such as top-and-bottom, side-by-side or monoscopic 360 content. The per-title algorithm will automatically propose a proper bitrate and resolution for each VR format based on the input details. Additionally, it is strongly recommended to use VOD_HIGH_QUALITY as an encoding preset and THREE_PASS as encoding mode. This will assure the Bitmovin Encoder delivers the best possible visual quality. 

In our tests using typical medium-high complexity content, we found that using a CRF of 17 produces good VQ for Meta Quest playback, with PSNR values above 48 dB and bitrates that are usually below the suggested maximum of 60 Mbps. 

Alternatively, traditional CRF encoding can be used instead of Per-title, for instance if only one rendition is desired at the output – with no ABR.

Creating monoscopic outputs from stereoscopic inputs

Usually, VR 360 cameras record the content in stereoscopic format either in top-and-bottom or side-by-side arrangements. However, depending on the customer use case, it could be required to convert the content from stereoscopic to monoscopic formats. This can be easily solved with the Bitmovin VOD Encoding API by applying cropping filters to remove the required pixels or frame percentage from the stereoscopic content, turning it into monoscopic format, i. e., by removing the left/right or the bottom/top side from the input asset.

For instance, the following javascript snippet would remove the top side of a 3840 x 3840 stereoscopic content:

.....
.....
// Crop filter definition
const cropTopSideFilter = new CropFilter({
  name: "stereo-to-mono-filter-example",
  left: 0,
  right: 0,
  bottom: 0,
  top: 1920,
 })
 
// Crop filter creation 
cropTopSideFilter = await bitmovinApi.encoding.filters.crop.create( cropTopSideFilter)

// Stream Filter definition
const cropTopSideStreamFilter = new StreamFilter({
  id : cropTopSideFilter.id,
  position: 0,
})

// StreamFilter creation
bitmovinApi.encoding.encodings.streams.filters.create(<encoding.id>, <videoStream.id>, [cropTopSideStreamFilter] )

AV1 Codec Support on Meta Quest 3

In the recommended settings above, we strongly suggested using HEVC over H.264 because the newer generation codec offers greater compression efficiency that turns into bandwidth savings and a better quality of experience for users. Now with the Quest 3, you can take advantage of AV1, an even newer codec that outperforms HEVC. On average, our testing has shown that you can maintain equivalent quality while using around 30% lower bitrate with AV1. This will depend on the type of content you’re working with, so if you’re experimenting with AV1 for the Quest 3, choosing a bitrate that’s ~25% lower than your HEVC encoding would be a good place to start.  DEOVR shared a 2900p sample .mp4 file encoded with AV1, but you can also create your own with a Bitmovin trial account.

Ready to start encoding your own 360 content for Meta Quest headsets? Sign up for a free trial and get going today! 

Related links:

Bitmovin Docs – Encoding Tutorials | Per-Title Configuration Options explained

Bitmovin Player 360 video demo

The post Encoding VR and 360 Immersive Video for Meta Quest Headsets appeared first on Bitmovin.

Table of Contents

Introduction

The story of Bitmovin began with video research and innovation back in 2012, when our co-founders Stefan Lederer and Christopher Mueller were students at Alpen-Adria-Universität (AAU) Klagenfurt. Together with their professor Dr. Christian Timmerer, the three co-founded Bitmovin in 2013, with their research providing the foundation for Bitmovin’s groundbreaking MPEG-DASH player and Per-Title Encoding. Five years later in 2018, a joint project between Bitmovin and AAU called ATHENA was formed, with a new laboratory and research program that would be led by Dr. Timmerer. The aim of ATHENA was to research and develop new approaches, tools and evaluations for all areas of HTTP adaptive streaming, including encoding, delivery, playback and end-to-end quality of experience (QoE). Bitmovin could then take advantage of the knowledge gained to further innovate and enhance its products and services. In the late spring and summer of 2023, the first cohort of ATHENA PhD students completed their projects and successfully defended their dissertations. This post will highlight their work and its potential applications. 

Video Research Projects

Optimizing QoE and Latency of Live Video Streaming Using Edge Computing and In-Network Intelligence

Dr. Alireza Erfanian

The work of Dr. Erfanian focused on leveraging edge computing and in-network intelligence to enhance the QoE and reduce end-to-end latency in live ABR streaming. The research also addresses improving transcoding performance and optimizing costs associated with running live streaming services and network backhaul utilization. 

1. Optimizing resource utilization – Two new methods ORAVA and OSCAR, utilize edge computing, network function virtualization, and software-defined networking (SDN). At the network’s edge, virtual reverse proxies collect clients’ requests and send them to an SDN controller, which creates a multicast tree to deliver the highest requested bitrate efficiently. This approach minimizes streaming cost and resource utilization while considering delay constraints. ORAVA, a cost-aware approach, and OSCAR, an SDN-based live video streaming method, collectively save up to 65% bandwidth compared to state-of-the-art approaches, reducing OpenFlow commands by up to 78% and 82%, respectively.
2. Light-Weight Transcoding – These three new approaches utilize edge computing and network function virtualization to significantly improve transcoding efficiency. LwTE is a novel light-weight transcoding approach at the edge that saves time and computational resources by storing optimal results as metadata during the encoding process. It employs store and transcode policies based on popularity, caching popular segments at the edge. CD-LwTE extends LwTE by proposing Cost- and Delay-aware Light-weight Transcoding at the Edge, considering resource constraints, introducing a fetch policy, and minimizing total cost and serving delay for each segment/bitrate. LwTE-Live investigates the cost efficiency of LwTE in live streaming, leveraging the approach to save bandwidth in the backhaul network. Evaluation results demonstrate LwTE processes transcoding at least 80% faster, while CD-LwTE reduces transcoding time by up to 97%, decreases streaming costs by up to 75%, and reduces delay by up to 48% compared to state-of-the-art approaches.

Slides and more detail

Video Coding Enhancements for HTTP Adaptive Streaming using Machine Learning

Dr. Ekrem Çetinkaya

The research of Dr. Çetinkaya involved several applications of machine learning techniques for improving the video coding process across 4 categories:

1. Fast Multi-Rate Encoding with Machine Learning – These two techniques address the challenge of encoding multiple representations of a video for ABR streaming. FaME-ML utilizes convolutional neural networks to guide encoding decisions, reducing parallel encoding time by 41%. FaRes-ML extends this approach to multi-resolution scenarios, achieving a 46% reduction in overall encoding time while preserving visual quality.
2. Enhancing Visual Quality on Mobile Devices – These three methods focused on improving visual quality on mobile devices with limited hardware. SR-ABR integrates super-resolution into adaptive bitrate selection, saving up to 43% bandwidth. LiDeR addresses computational complexity, achieving a 428% increase in execution speed while maintaining visual quality. MoViDNN facilitates the evaluation of machine learning solutions for enhanced visual quality on mobile devices.
3. Light-Field Image Coding with Super-Resolution – This new approach addresses the data size challenge of light field images in emerging media formats. LFC-SASR utilizes super-resolution to reduce data size by 54%, ensuring a more immersive experience while preserving visual quality.
4. Blind Visual Quality Assessment Using Vision Transformers – A new technique, BQ-ViT, tackles the blind visual quality assessment problem for videos. Leveraging the vision transformer architecture, BQ-ViT achieves a high correlation (0.895 PCC) in predicting video visual quality using only the encoded frames.

Slides and more detail

Policy-driven Dynamic HTTP Adaptive Streaming Player Environment

Dr. Minh Nguyen

The work of Dr. Ngyuen addressed critical issues impacting QoE in adaptive bitrate (ABR) streaming, with four main contributions:

1. Days of Future Past Plus (DoFP+) – This approach uses HTTP/3 features to enhance QoE by upgrading low-quality segments during streaming sessions, resulting in a 33% QoE improvement and a 16% reduction in downloaded data.
2. WISH ABR – This is a weighted sum model that allows users to customize their ABR switching algorithm by specifying preferences for parameters like data usage, stall events, and video quality. WISH considers throughput, buffer, and quality costs, enhancing QoE by up to 17.6% and reducing data usage by 36.4%.
3. WISH-SR – This is an ABR scheme that extends WISH by incorporating a lightweight Convolutional Neural Network (CNN) to improve video quality on high-end mobile devices. It can reduce downloaded data by up to 43% and enhance visual quality with client-side Super Resolution upscaling. 
4. New CMCD Approach – This new method for determining Common Media Client Data (CMCD) parameters, enables the server to generate suitable bitrate ladders based on clients’ device types and network conditions. This approach reduces downloaded data while improving QoE by up to 2.6 times

Slides and more detail  

Multi-access Edge Computing for Adaptive Video Streaming

Dr. Jesús Aguilar Armijo

The network plays a crucial role for video streaming QoE and one of the key technologies available on the network side is Multi-access Edge Computing (MEC). It has several key characteristics: computing power, storage, proximity to the clients and access to network and player metrics, that make it possible to deploy mechanisms at the MEC node to assist video streaming.

This thesis of Dr. Aguilar Armijo investigates how MEC capabilities can be leveraged to support video streaming delivery, specifically to improve the QoE, reduce latency or increase savings on storage and bandwidth. 

1. ANGELA Simulator – A new simulator is designed to test mechanisms supporting video streaming at the edge node. ANGELA addresses issues in state-of-the-art simulators by providing access to radio and player metrics, various multimedia content configurations, Adaptive Bitrate (ABR) algorithms at different network locations, and a range of evaluation metrics. Real 4G/5G network traces are used for radio layer simulation, offering realistic results. ANGELA demonstrates a significant simulation time reduction of 99.76% compared to the ns-3 simulator in a simple MEC mechanism scenario.
2. Dynamic Segment Repackaging at the Edge – The proposal suggests using the Common Media Application Format (CMAF) in the network’s backhaul, performing dynamic repackaging of content at the MEC node to match clients’ requested delivery formats. This approach aims to achieve bandwidth savings in the network’s backhaul and reduce storage costs at the server and edge side. Measurements indicate potential reductions in delivery latency under certain expected conditions.
3. Edge-Assisted Adaptation Schemes – Leveraging radio network and player metrics at the MEC node, two edge-assisted adaptation schemes are proposed. EADAS improves ABR decisions on-the-fly to enhance clients’ Quality of Experience (QoE) and fairness. ECAS-ML shifts the entire ABR algorithm logic to the edge, managing the tradeoff among bitrate, segment switches, and stalls through machine learning techniques. Evaluations show significant improvements in QoE and fairness for both schemes compared to various ABR algorithms.
4. Segment Prefetching and Caching at the Edge – Segment prefetching, a technique transmitting future video segments closer to the client before being requested, is explored at the MEC node. Different prefetching policies, utilizing resources and techniques such as Markov prediction, machine learning, transrating, and super-resolution, are proposed and evaluated. Results indicate that machine learning-based prefetching increases average bitrate while reducing stalls and extra bandwidth consumption, offering a promising approach to enhance overall performance.

Slides and more detail

Potential applications for Bitmovin products

The WISH ABR algorithm presented by Dr. Nguyen is already available in the Bitmovin Web Player SDK as of version 8.136.0, which was released in early October 2023. It can be enabled via AdaptationConfig.logic. Use of CMCD metadata is still gaining momentum throughout the industry, but Bitmovin and Akamai have already demonstrated a joint solution and the research above will help improve our implementation.

Bitmovin has experimented with server-side Super Resolution upscaling with some customers, mainly focusing on upscaling SD content to HD for viewing on TVs and larger monitors, but the techniques investigated by Dr. Çetinkaya take advantage of newer models that can extend Super Resolution to the client side on mobile devices. These have the potential to reduce data usage which is especially important to users with limited data plans and bandwidth. They can also improve QoE and visual quality while saving service providers on delivery costs. 

Controlling costs has been at or near the top of the list of challenges video developers and streaming service providers have faced over the past couple of years according to Bitmovin’s annual Video Developer Report. This trend will likely continue into 2024 and the resource management and transcoding efficiency improvements developed by Dr. Erfanian will help optimize and reduce operational costs for Bitmovin and its services. 

Edge computing is becoming more mainstream, with companies like Bitmovin partners Videon and Edgio delivering new applications that take advantage of available compute resources closer to the end user. The contributions developed by Dr. Aguilar Armijo address different facets of content delivery and provide a comprehensive approach to optimizing video streaming in edge computing environments. This has the potential to provide more actionable analytics data and enable more intelligent and robust adaptation during challenging network conditions. 

Conclusion

Bitmovin was born from research and innovation and 10 years later is still breaking new ground. We were honored to receive a Technology & Engineering Emmy Award for our efforts and remain committed to improving every part of the streaming experience. Whether it’s taking advantage of the latest machine learning capabilities or developing novel approaches for controlling costs, we’re excited for what the future holds. We’re also grateful for all of the researchers, engineers, technology partners and customers who have contributed along the way and look forward to the next 10 years of progress and innovation.

The post PhD video research: From the ATHENA lab to Bitmovin products appeared first on Bitmovin.