v4.0.0-beta.2

Release v4.0.0-beta.2 (2023-06-27)

Quick Links:
📖 API documentation - ⏯ Demo - 🎓 Migration guide from v3

• 🔍 Overview
• 📑 Changelog
• NO_PLAYABLE_REPRESENTATION behavior change
• New representationListUpdate event
• Automatic reloading when stuck on DRM issue

🔍 Overview

The new v4 beta release, based on the v3.31.0, is here.
It contains all improvements from previous v4 alpha and beta releases, as well as some further improvements mostly related to DRM, all described in this release note.

About v4 beta releases

As a reminder, beta v4 versions are RxPlayer pre-releases (as some minor API changes are still done, see changelog) for the future official v4 release, a successor to the current v3 releases of the RxPlayer.

We're currently testing it on several applications. As we're doing it, our team as well as people porting it can propose some small API improvements, which may then be added to the next beta releases. After enough people have made the switch and are satisfied with the new API, the first official v4 release will be published (we're also in the process to have some applications running it in production to ensure its stability with a large enough population).

We will still continue maintaining and providing improvements to the v3 for at least as long as the v4 is in beta (and we will probably continue to provide bug fixes for the v3 for some time after the official v4.0.0 is released).
This process is long on purpose to be sure that we're providing a useful v4 API for applications and also to avoid alienating application developers, as the migration from the v3 might take time.

📑 Changelog

Changes

• If all Representations from the current track become undecipherable, automatically switch to another track (also send a trackUpdate event) instead of stopping on error [#1234]
• Only send MediaError errors with the NO_PLAYABLE_REPRESENTATION error code when no Representation from all tracks of a given type can be played [#1234]

Features

• Add representationListUpdate event for when the list of available Representation for a current track changes [#1240]
• Add "no-playable-representation" as a reason for trackUpdate events when the track switch is due to encrypted Representations [#1234]

Other improvements

• DRM: Reload when playback is unexpectedly frozen with encrypted but only decipherable data in the buffer to work-around rare encryption-related issues [#1236]

NO_PLAYABLE_REPRESENTATION behavior change

In the v3 and previous v4 beta releases, if you could not play any Representation (i.e. quality) from your chosen audio or video track due to encryption matters, you would obtain a MediaError error with the NO_PLAYABLE_REPRESENTATION error code.

Stopping on error when no quality of the chosen track can be played seemed logical at the time, but we're now encountering use cases where it would be better if the RxPlayer automatically took the decision to change the current track instead, to one that perhaps has decipherable Representation(s).
The main example we encountered was cases where we had separate video tracks, each linked to another dynamic range (e.g. an HDR and a SDR track) and with different security policies (tracks with a high dynamic range would have more drastic security policies for example).
Here, I would guess that an application would prefer that by default we switch to the SDR video track if no Representation in the HDR one is decipherable, instead of just stopping playback with a NO_PLAYABLE_REPRESENTATION error.

Before:
-------

+----------------+
|    Selected    |  Not decipherable
|   Video Track  |  --------------------> NO_PLAYABLE_REPRESENTATION
| (example: HDR) |                        Error
+----------------+

Now:
----

+----------------+                        +---------------------+
|    Selected    |  Not decipherable      |      Automatic      |
|   Video Track  |  --------------------> |     fallback to     |
| (example: HDR) |                        | another video Track |
+----------------+                        |   (example: SDR)    |
                                          +---------------------+

Note that the NO_PLAYABLE_REPRESENTATION error might still be thrown, only now it is when no Representation of all tracks for the given type are decipherable.

New trackUpdate reason

Because an application might still want to be notified or even stop playback by itself when the initially-chosen track has no playable Representation, we also brought added the "no-playable-representation" reason to the trackUpdate event, which indicates that the current track for any Period of the current content was updated due to this situation.

player.addEventListener("trackUpdate", (payload) => {
  if (payload.reason === "no-playable-representation") {
    console.warn(
      `A ${payload.trackType} track was just changed ` +
      "because it had no playable Representation"
    );
  }
});

New representationListUpdate event

This new beta version of the v4 also brings a new event: representationListUpdate.

The problem without this event

Let's consider for example an application storing information on the currently available qualities (a.k.a. Representation) for the chosen video track of the currently-playing Period (said another way: being played right now).
With the v4 that application can simply get the initial list of Representation when that Period begins to be played through the periodChange event and update this list any time the video track changes, by listening to the trackUpdate event:

let currentVideoRepresentations = null;

function updateVideoRepresentations() {
  const videoTrack = player.getVideoTrack();
  if (videoTrack === undefined || videoTrack === null) {
    currentVideoRepresentations = null;
  } else {
    currentVideoRepresentations = videoTrack.representations;
  }
}

// Set it when the Period is initially played
player.addEventListener("periodChange", () => {
  updateVideoRepresentations();
});

// Set it when the video track is changed
player.addEventListener("trackUpdate", (t) => {
  // We only want to consider the currently-playing Period
  const currentPeriodId = player.getCurrentPeriod()?.id;
  if (t.trackType === "video" && t.period.id === currentPeriodId) {
    updateVideoRepresentations();
  }
});

// Remove it when no content is loaded
player.addEventListener("playerStateChange", () => {
  if (!player.isContentLoaded()) {
    currentVideoRepresentations = null;
  }
});

This seems sufficient at first.

But now let's consider that we're playing encrypted contents, and that one of the Representation of those current video tracks became un-decipherable at some point (e.g. after its license has been fetched and communicated to your browser's Content Decryption Module).
Here, an application won't be able to select that Representation anymore, so it generally will want to remove it from its internal state. However, there was no event to indicate that the list of available Representations had changed when neither the video track itself nor the Period has changed

The new event

We thus decided to add the representationListUpdate event. Exactly like the trackUpdate event, it is only triggered when the Representation list changes, i.e. not initially, when the track is chosen.

So taking into consideration that point, the previous code can be written:

let currentVideoRepresentations = null;

function updateVideoRepresentations() {
  const videoTrack = player.getVideoTrack();
  if (videoTrack === undefined || videoTrack === null) {
    currentVideoRepresentations = null;
  } else {
    currentVideoRepresentations = videoTrack.representations;
  }
}

// Set it when the Period is initially played
player.addEventListener("periodChange", () => {
  updateVideoRepresentations();
});

// Set it when the video track is changed
player.addEventListener("trackUpdate", (t) => {
  // We only want to consider the currently-playing Period
  const currentPeriodId = player.getCurrentPeriod()?.id;
  if (t.trackType === "video" && t.period.id === currentPeriodId) {
    updateVideoRepresentations();
  }
});

// Remove it when no content is loaded
player.addEventListener("playerStateChange", () => {
  if (!player.isContentLoaded()) {
    currentVideoRepresentations = null;
  }
});

// What's new:
// Set it if the list of Representation ever changes during playback
player.addEventListener("representationListUpdate", (r) => {
  // We only want to consider the currently-playing Period
  const currentPeriodId = player.getCurrentPeriod()?.id;
  if (t.trackType === "video" && t.period.id === currentPeriodId) {
    updateVideoRepresentations();
  }
});

This event is documented here.

Note about its usability

While developping that feature, we thought that its usage could be simplified if the representationListUpdate event was also sent when the track was initially chosen. We would here have no need in the previous code examples to also listen to trackUpdate events, as the representationListUpdate event would also be sent during track change.

However, it appeared to us that also sending such events on the initial track choice could quickly become complex in the RxPlayer's code, due to all the side-effects an event listener can perform (for example, you could be changing the video track inside your representationListUpdate listener, which would then have to be directly considered by the RxPlayer).
Handling all kinds of side-effect inside the RxPlayer was possible, but it would have brought very complex code and potentially performance inneficiencies.

We thus decided to keep this behavior of sending that event ONLY if the Representation list changes, meaning that application developpers will most of the time also need to react to at least one other event for knowing about the initial Representations, like we did in our examples with periodChange and trackUpdate listeners.

Automatic reloading when stuck on DRM issue

The need for a last-resort solution

The majority of RxPlayer issues are now DRM-related and device-specific, generally its source being platform lower-level (CDM, browser integration...) bugs.
Even if we frequently exchange with partners to obtain fixes, this is not always possible (sometimes because there are too many people relying on the older logic and thus risks in changing it, sometimes because there are a lot of higher priorities on their side).

We are now encountering relatively frequently, for some contents with DRM, what we call a "playback freeze": playback does not advance despite having data in the buffer, all being known to be decipherable.

Screenshot: Example of what we call a "freeze". We have data in the buffer, yet the position is stuck in place". In v4 versions, we have there the "FREEZING" player state.

Recently we've seen this similar issue with specific contents on Microsoft Edge, UWP applications (Windows, XBOX...) and LG TV.
In all of those cases, what we call "reloading" the content after the license has been pushed always fixes the issue. The action of reloading means principally to re-create the audio and video buffers, to then push again segments on it.
Reloading leads to a bad experience, as we might go to a black screen in the meantime. Thankfully, the issue mostly appeared at load, where reloading is much less noticeable.

Although we prefer providing more targeted fixes or telling to platform developers to fix their implementation, this issue was so frequent that we began to wonder if we should provide some heuristic in the RxPlayer, to detect if that situation arises and reload as a last resort mechanism in that case.

Risks and how we mitigate them

What we are most afraid here is the risk of false positives: falsely considering that we are in a "decipherability freeze" situation, where it's in fact just a performance issue on the hardware side or an issue with the content being played.

To limit greatly the risk of false positives, we added a lot of rules that will lead to a reload under that heuristic.
For the curious, here they are (all sentences after dashes here are mandatory):

• we have a readyState set to 1 (meaning that the browser announces that it has enough metadata to be able to play, but does not seem to have any media data to decode)
• we have at least 6 seconds in the buffer ahead of the current position (so compared with the previous dash, we DO have enough data to play - generally this happens when pushed media data is not being decrypted).
• The playhead (current position) is not advancing and has been in this frozen situation for at least 4 seconds
• We buffered at least one audio and/or video Representation with DRM.
• One of the following is true:
  1. There is at least one segment that is known to be undecipherable in the buffer (which should never happen at that point, this was added as a security)
  2. There are ONLY segments that are known to be decipherable (licenses have been pushed AND their key-id are all "usable") in the buffer, for both audio and video.

If all those conditions are "true" we will reload. For now with no limit (meaning we could have several reloads for one content if the situation repeats).
Note that at the request level, this only might influence segment requests (which will have to be reloaded after 4 seconds) and not DRM-related requests nor Manifest requests.

Results

We actually tested that logic for some time on most devices to a large population.
After some initial tweaking, we seem to have "fixed" the more difficult issues with no much false positive left.

This has encouraged us to include it in this v4 beta release so it can be present in the first official v4.0.0 release.
Note that we cannot simply bring this improvement also to the v3 under SEMVER rules, as "reloading" without a specific new API is a v3 breaking change (the RELOADING state didn't exist in the `v3.0.0).