Behavioral matrix for DEFAULT and AUTOSELECT

[defagos]

As a development team working on subtitles in Pillarbox it would be helpful to better understand the change of behavior created by these flags delivered in the playlist.

Acceptance criteria

• A matrix documenting the result of setting / not setting the DEFAULT and AUTOSELECT flags has been made available, including the 3 platforms we support.

Tasks

• Run on all platforms and document what happens in each case:
  • Apple
  • Android
  • Web
• Document AUTOSELECT and DEFAULT recommendations for SRG SSR packaging if meaningful.

[defagos]

2 examples are readily available: https://github.com/SRGSSR/srgletterbox-web/issues/769#issuecomment-1482557458

[defagos]

I already documented packaging advice for Apple platforms in #502 and #504. We should likely share this documentation on the Pillarbox documentation repository, though.

[defagos]

From HLS RFC 8216:

DEFAULT

The value is an enumerated-string; valid strings are YES and NO. If the value is YES, then the client SHOULD play this Rendition of the content in the absence of information from the user indicating a different choice. This attribute is OPTIONAL. Its absence indicates an implicit value of NO.

AUTOSELECT

The value is an enumerated-string; valid strings are YES and NO. This attribute is OPTIONAL. Its absence indicates an implicit value of NO. If the value is YES, then the client MAY choose to play this Rendition in the absence of explicit user preference because it matches the current playback environment, such as chosen system language.

Moreover from HTTP Live Streaming (HLS) authoring specification for Apple devices:

4.6. If a subtitles track is intended to provide accessibility for people who are deaf or hard of hearing, the AUTOSELECT attribute MUST have a value of "YES".
5.11. Forced subtitles SHOULD always have AUTOSELECT=YES.

Non-directly related but also important is:

4.5. If a subtitles track is intended to provide accessibility for people who are deaf or hard of hearing, it MUST be marked with the attribute CHARACTERISTICS="public.accessibility.transcribes-spoken-dialog,public.accessibility.describes-music-and-sound". (Subtitles with this attribute value are treated the same as closed captions.)

[defagos]

Apple

Here is how AVPlayer deals with DEFAULT and AUTOSELECT.

AUTOSELECT behavior

By default AVPlayer picks an audio track matching the device language, provided the track has AUTOSELECT set to YES.

For subtitles the behavior is similar, though the resolution logic is more complex. In all cases subtitles must have AUTOSELECT set to YES so that the player considers them:

• .forcedOnly: Auto-selectable forced subtitles are automatically enabled if available, based on the current audio language. If the audio language is changed forced subtitles are updated accordingly.
• .automatic: Auto-selectable subtitles are automatically enabled based on various considerations (device language, accessibility system settings).

Automatic selection requires appliesMediaSelectionCriteriaAutomatically to be set to true on the AVPlayer instance (which the default). This is the behavior we currently implement and we won't provide any customization API until a real need (i.e. which cannot be solved by fixing the packaging) is identified.

Adjusting AUTOSELECT behavior

The default selection behavior can be changed using AVPlayerMediaSelectionCriteria, e.g. to force the player to play an audio track different from the device language.

This works both for legible and audible renditions.

DEFAULT behavior

DEFAULT is not taken into account by AVPlayer. Only renditions with AUTOSELECT set to YES are considered. Still DEFAULT and AUTOSELECT must be properly configured otherwise the player will fail to parse the playlist.

[defagos]

Some issues with how our content is currently packaged.

RTS - 19h30

😭 DEFAULT=YES for FR subtitles. Not required and affecting Pillarbox Android behavior.
😭 public.accessibility.describes-spoken-dialog is missing if subtitles are meant to be SDH.

#EXTM3U

#EXT-X-MEDIA:TYPE=AUDIO,GROUP-ID="audio0",NAME="Français",LANGUAGE="fr",AUTOSELECT=YES,DEFAULT=YES,CHANNELS="2",URI="index-f1-a1.m3u8"


#EXT-X-MEDIA:TYPE=SUBTITLES,GROUP-ID="subs0",NAME="Français (SDH)",LANGUAGE="fr",AUTOSELECT=YES,DEFAULT=YES,CHARACTERISTICS="public.accessibility.describes-music-and-sound",URI="index-f6.m3u8"

#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=1328647,RESOLUTION=640x360,FRAME-RATE=25.000,CODECS="avc1.4d401f,mp4a.40.2",AUDIO="audio0",SUBTITLES="subs0"
index-f1-v1.m3u8
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=628170,RESOLUTION=484x272,FRAME-RATE=25.000,CODECS="avc1.4d401f,mp4a.40.2",AUDIO="audio0",SUBTITLES="subs0"
index-f2-v1.m3u8
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=2129146,RESOLUTION=960x540,FRAME-RATE=25.000,CODECS="avc1.4d401f,mp4a.40.2",AUDIO="audio0",SUBTITLES="subs0"
index-f3-v1.m3u8
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=3628915,RESOLUTION=1280x720,FRAME-RATE=25.000,CODECS="avc1.4d401f,mp4a.40.2",AUDIO="audio0",SUBTITLES="subs0"
index-f4-v1.m3u8
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=6129843,RESOLUTION=1920x1080,FRAME-RATE=25.000,CODECS="avc1.640029,mp4a.40.2",AUDIO="audio0",SUBTITLES="subs0"
index-f5-v1.m3u8

RTS - Top Models

😭 Poor rendition names.
😭 No AUTOSELECT on audible renditions, preventing optimal automatic selection on Apple devices.

#EXTM3U
#EXT-X-VERSION:5
#EXT-X-MEDIA:TYPE=SUBTITLES,GROUP-ID="subs",LANGUAGE="fr",NAME="text_fra_sdh",DEFAULT=NO,AUTOSELECT=YES,FORCED=NO,URI="https://rtsvod-euwe.akamaized.net:443/6894ae3a-d1eb-4c7f-9398-d8cc14960edf/0aa681e29326261dfa81f6579ed12874/topmo_20230823_full_fra_sdh.m3u8"
#EXT-X-MEDIA:TYPE=AUDIO,GROUP-ID="audio",NAME="audio_eng_128000",LANGUAGE="en",URI="QualityLevels(128000)/Manifest(audio_eng,format=m3u8-aapl)"
#EXT-X-MEDIA:TYPE=AUDIO,GROUP-ID="audio",NAME="audio_fra_128000",LANGUAGE="fr",DEFAULT=YES,URI="QualityLevels(128000)/Manifest(audio_fra,format=m3u8-aapl)"
#EXT-X-STREAM-INF:SUBTITLES="subs",CLOSED-CAPTIONS=NONE,BANDWIDTH=658539,RESOLUTION=484x272,CODECS="avc1.4d401f,mp4a.40.2",AUDIO="audio"
QualityLevels(500395)/Manifest(video,format=m3u8-aapl)
#EXT-X-I-FRAME-STREAM-INF:BANDWIDTH=658539,RESOLUTION=484x272,CODECS="avc1.4d401f",URI="QualityLevels(500395)/Manifest(video,format=m3u8-aapl,type=keyframes)"
#EXT-X-STREAM-INF:SUBTITLES="subs",CLOSED-CAPTIONS=NONE,BANDWIDTH=1374572,RESOLUTION=640x360,CODECS="avc1.4d401f,mp4a.40.2",AUDIO="audio"
QualityLevels(1201014)/Manifest(video,format=m3u8-aapl)
#EXT-X-I-FRAME-STREAM-INF:BANDWIDTH=1374572,RESOLUTION=640x360,CODECS="avc1.4d401f",URI="QualityLevels(1201014)/Manifest(video,format=m3u8-aapl,type=keyframes)"
#EXT-X-STREAM-INF:SUBTITLES="subs",CLOSED-CAPTIONS=NONE,BANDWIDTH=2192870,RESOLUTION=960x540,CODECS="avc1.4d401f,mp4a.40.2",AUDIO="audio"
QualityLevels(2001697)/Manifest(video,format=m3u8-aapl)
#EXT-X-I-FRAME-STREAM-INF:BANDWIDTH=2192870,RESOLUTION=960x540,CODECS="avc1.4d401f",URI="QualityLevels(2001697)/Manifest(video,format=m3u8-aapl,type=keyframes)"
#EXT-X-STREAM-INF:SUBTITLES="subs",CLOSED-CAPTIONS=NONE,BANDWIDTH=3726776,RESOLUTION=1280x720,CODECS="avc1.4d401f,mp4a.40.2",AUDIO="audio"
QualityLevels(3502584)/Manifest(video,format=m3u8-aapl)
#EXT-X-I-FRAME-STREAM-INF:BANDWIDTH=3726776,RESOLUTION=1280x720,CODECS="avc1.4d401f",URI="QualityLevels(3502584)/Manifest(video,format=m3u8-aapl,type=keyframes)"
#EXT-X-STREAM-INF:SUBTITLES="subs",CLOSED-CAPTIONS=NONE,BANDWIDTH=6283672,RESOLUTION=1920x1080,CODECS="avc1.640029,mp4a.40.2",AUDIO="audio"
QualityLevels(6004439)/Manifest(video,format=m3u8-aapl)
#EXT-X-I-FRAME-STREAM-INF:BANDWIDTH=6283672,RESOLUTION=1920x1080,CODECS="avc1.640029",URI="QualityLevels(6004439)/Manifest(video,format=m3u8-aapl,type=keyframes)"

Play CH - Sample stream

😭 AUTOSELECT set to NO for EN audio, preventing optimal automatic selection on Apple devices.
😭 Same for subtitles (but forced subtitles can correctly be selected automatically).
😭 public.accessibility.describes-spoken-dialog is missing if subtitles are meant to be SDH.

#EXTM3U

#EXT-X-MEDIA:TYPE=AUDIO,GROUP-ID="audio0",NAME="Deutsch",LANGUAGE="de",AUTOSELECT=YES,DEFAULT=YES,CHANNELS="2",URI="index-f1-a1.m3u8"
#EXT-X-MEDIA:TYPE=AUDIO,GROUP-ID="audio0",NAME="English",LANGUAGE="en",AUTOSELECT=NO,DEFAULT=NO,CHANNELS="2",URI="index-f8-a1.m3u8"


#EXT-X-MEDIA:TYPE=SUBTITLES,GROUP-ID="subs0",NAME="Deutsch",LANGUAGE="de",AUTOSELECT=YES,DEFAULT=YES,URI="index-f9.m3u8"
#EXT-X-MEDIA:TYPE=SUBTITLES,GROUP-ID="subs0",NAME="Français",LANGUAGE="fr",AUTOSELECT=NO,DEFAULT=NO,URI="index-f10.m3u8"
#EXT-X-MEDIA:TYPE=SUBTITLES,GROUP-ID="subs0",NAME="Italiano",LANGUAGE="it",AUTOSELECT=NO,DEFAULT=NO,URI="index-f11.m3u8"
#EXT-X-MEDIA:TYPE=SUBTITLES,GROUP-ID="subs0",NAME="Italiano FORCED",LANGUAGE="it",AUTOSELECT=YES,DEFAULT=NO,FORCED=YES,URI="index-f12.m3u8"
#EXT-X-MEDIA:TYPE=SUBTITLES,GROUP-ID="subs0",NAME="Français FORCED",LANGUAGE="fr",AUTOSELECT=YES,DEFAULT=NO,FORCED=YES,URI="index-f13.m3u8"
#EXT-X-MEDIA:TYPE=SUBTITLES,GROUP-ID="subs0",NAME="Deutsch FORCED",LANGUAGE="de",AUTOSELECT=YES,DEFAULT=NO,FORCED=YES,URI="index-f14.m3u8"
#EXT-X-MEDIA:TYPE=SUBTITLES,GROUP-ID="subs0",NAME="Deutsch (SDH)",LANGUAGE="de",AUTOSELECT=YES,DEFAULT=NO,CHARACTERISTICS="public.accessibility.describes-music-and-sound",URI="index-f15.m3u8"
#EXT-X-MEDIA:TYPE=SUBTITLES,GROUP-ID="subs0",NAME="Italiano (SDH)",LANGUAGE="it",AUTOSELECT=YES,DEFAULT=NO,CHARACTERISTICS="public.accessibility.describes-music-and-sound",URI="index-f16.m3u8"
#EXT-X-MEDIA:TYPE=SUBTITLES,GROUP-ID="subs0",NAME="Français (SDH)",LANGUAGE="fr",AUTOSELECT=YES,DEFAULT=NO,CHARACTERISTICS="public.accessibility.describes-music-and-sound",URI="index-f17.m3u8"

#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=333299,RESOLUTION=640x360,FRAME-RATE=25.000,CODECS="avc1.42c01e,mp4a.40.2",AUDIO="audio0",SUBTITLES="subs0",CLOSED-CAPTIONS=NONE
index-f1-v1.m3u8
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=468971,RESOLUTION=854x480,FRAME-RATE=25.000,CODECS="avc1.42c01e,mp4a.40.2",AUDIO="audio0",SUBTITLES="subs0",CLOSED-CAPTIONS=NONE
index-f2-v1.m3u8
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=562532,RESOLUTION=960x540,FRAME-RATE=25.000,CODECS="avc1.4d401f,mp4a.40.2",AUDIO="audio0",SUBTITLES="subs0",CLOSED-CAPTIONS=NONE
index-f3-v1.m3u8
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=875991,RESOLUTION=1280x720,FRAME-RATE=25.000,CODECS="avc1.4d4020,mp4a.40.2",AUDIO="audio0",SUBTITLES="subs0",CLOSED-CAPTIONS=NONE
index-f4-v1.m3u8
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=959084,RESOLUTION=1280x720,FRAME-RATE=25.000,CODECS="avc1.640028,mp4a.40.2",AUDIO="audio0",SUBTITLES="subs0",CLOSED-CAPTIONS=NONE
index-f5-v1.m3u8
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=2510818,RESOLUTION=1920x1080,FRAME-RATE=25.000,CODECS="avc1.640032,mp4a.40.2",AUDIO="audio0",SUBTITLES="subs0",CLOSED-CAPTIONS=NONE
index-f6-v1.m3u8

TV livestreams (e.g. RTS 1)

😭 public.accessibility.describes-music-and-sound is missing if subtitles are meant to be SDH.
😭 Livestreams without DVR support (e.g. SRF 1): Media playlists have more than 3 chunks.

#EXTM3U
#EXT-X-VERSION:5
#EXT-X-MEDIA:TYPE=AUDIO,GROUP-ID="audio",NAME="audio1",LANGUAGE="fra",DEFAULT=YES,AUTOSELECT=YES,URI="QualityLevels(128000)/Manifest(audio1,format=m3u8-aapl,filter=nodvr)"
#EXT-X-MEDIA:TYPE=SUBTITLES,GROUP-ID="subs",NAME="Francais",LANGUAGE="fra",DEFAULT=YES,AUTOSELECT=YES,CHARACTERISTICS="public.accessibility.transcribes-spoken-dialog",URI="QualityLevels(9600)/Manifest(Francais,format=m3u8-aapl,filter=nodvr)"
#EXT-X-STREAM-INF:BANDWIDTH=657318,RESOLUTION=480x272,CODECS="avc1.4d4020,mp4a.40.2",AUDIO="audio",SUBTITLES="subs"
QualityLevels(499200)/Manifest(video,format=m3u8-aapl,filter=nodvr)
#EXT-X-I-FRAME-STREAM-INF:BANDWIDTH=657318,RESOLUTION=480x272,CODECS="avc1.4d4020",URI="QualityLevels(499200)/Manifest(video,format=m3u8-aapl,filter=nodvr,type=keyframes)"
#EXT-X-STREAM-INF:BANDWIDTH=402227,RESOLUTION=320x180,CODECS="avc1.42e020,mp4a.40.2",AUDIO="audio",SUBTITLES="subs"
QualityLevels(249600)/Manifest(video,format=m3u8-aapl,filter=nodvr)
#EXT-X-I-FRAME-STREAM-INF:BANDWIDTH=402227,RESOLUTION=320x180,CODECS="avc1.42e020",URI="QualityLevels(249600)/Manifest(video,format=m3u8-aapl,filter=nodvr,type=keyframes)"
#EXT-X-STREAM-INF:BANDWIDTH=1373536,RESOLUTION=640x360,CODECS="avc1.4d4020,mp4a.40.2",AUDIO="audio",SUBTITLES="subs"
QualityLevels(1200000)/Manifest(video,format=m3u8-aapl,filter=nodvr)
#EXT-X-I-FRAME-STREAM-INF:BANDWIDTH=1373536,RESOLUTION=640x360,CODECS="avc1.4d4020",URI="QualityLevels(1200000)/Manifest(video,format=m3u8-aapl,filter=nodvr,type=keyframes)"
#EXT-X-STREAM-INF:BANDWIDTH=2190727,RESOLUTION=960x544,CODECS="avc1.4d4029,mp4a.40.2",AUDIO="audio",SUBTITLES="subs"
QualityLevels(1999600)/Manifest(video,format=m3u8-aapl,filter=nodvr)
#EXT-X-I-FRAME-STREAM-INF:BANDWIDTH=2190727,RESOLUTION=960x544,CODECS="avc1.4d4029",URI="QualityLevels(1999600)/Manifest(video,format=m3u8-aapl,filter=nodvr,type=keyframes)"

[StaehliJ]

Android

Here is how Exoplayer default track selector deals with DEFAULT and AUTOSELECT.

AUTOSELECT behavior

Format with selection flag set to AUTOSELECT are elegible for selection automatically by the Exoplayer if no track overrides is specified.

For forced subtitles for exemple if they have AUTOSELECT`, they will be automatically selected and corresponding to the preferred audio language or the default one.Track overrides are ignored to determine audio language.

DEFAULT behavior

Format with selection flag set to DEFAULT will be automatically selected if nothing is specified in the track selection parameters.