Integrate with libass-android to support SSA/ASS subtitles in ExoPlayer

[damontecres]

Description

Add better ASS/SSA support to ExoPlayer by using libass-android.

This is enabled with the "Use libass for ASS subtitles" advanced settings for ExoPlayer. It is enabled by default.

Related issues

Related to #22
Replaces reverted #569
Fixes #1107

Try it out

See instructions here

[ethaldeman]

@damontecres I have tested the pre-release of this build and for me it is 100% working. I tested it with various tv shows and my test mkv which includes complex karaoke animations and utilizes embedded fonts.

While I hope others can test and report back to support my findings, I believe it is ready to be merged.

[nx6]

This build does utilize fonts, including fonts attached to mkvs with application/octet-stream instead of more "correct" methods. A clear improvement over ExoPlayer's own support before. 👍

Performance is comparable to other players utilizing ASS with ExoPlayer. For most tasks it is fine, but the slight delay in rendering it exhibits will cause issues for signs. Particularly more complicated ones with masking where frame-exact timing is needed. For simpler work it's very serviceable. Again, this is an issue on all players that implement this ASS support in ExoPlayer. Mpv will try to maintain rendering to timing accurately despite limited performance of streaming devices, usually at the expense of smooth video playback.

Examples: https://files.catbox.moe/ja9sdz.7z

• Ex1 - Signs on stalls and banners in background at a festival (masking around characters and timed to end on scene change).
• Ex2 - Translation of smartphone screen motion tracked, and timed to the zooms in the latter half.
• Ex3 - Simple text matched to an object in motion (text on a book cover).
• Ex4 - More extreme on-screen text motion tracking/masking.

[damontecres]

Thanks @ethaldeman & @nx6, I appreciate the testing feedback for ExoPlayer!

Have either of you tried using MPV on this build as well? The MPV player will use this libass build as well which technically shouldn't be significantly different, but you never know.

And no one has run into anything like #1049? This bug report is why I reverted this change originally.

[nx6]

And no one has run into anything like #1049? This bug report is why I reverted this change originally.

Have not seen anything like that.

Device
Sony Bravia 9

Kinda leaning towards that being a device issue, honestly. Would be good if they could compare to other apps (and supplied a sample). I've only tested this build on one device myself though (a Fire Stick 4K Max, Gen 2) so possible something will happen somewhere else, but I doubt it. Can check more later today.

[ethaldeman]

It seems my "100% working" may have been a little premature.......

And no one has run into anything like #1049? This bug report is why I reverted this change originally.

I too have not seen anything like that.

https://www.dropbox.com/scl/fi/uss9qa7hxzskoguodc0sn/ASS_Sample_Test.mkv?rlkey=038nzj2czoxnj5vymmth54vrn&st=o9lx2o7n&dl=0

Above is a link to one of my test mkv that is copyright free and free use. It tests various karaoke effects, but is not a good test of frame perfect complex signs.

I downloaded nx6's 4 sample mkv and tested them. All four had the ASS Sign's render correctly, but they were pretty off timing wise, roughly .5 seconds. I also tested the official jellyfin android tv implementation of libass-android and it had the same issues. I also tested in Wholphin's MPV and official jellyfin android tv ffmpeg transcoding. Both of these did not have the timing issues. Thus it seems to be an libass-android Exoplayer issue. All my testing was performed on the Onn 4k Pro (Amlogic S905X5M and 2 GB RAM)

I plan on opening an issue upstream in libass-android, noting this may not be fixable in Exoplayer citing androidx/media#2289. However in the meantime, I still think the functionality is good enough in most use cases to be merged, maybe just not enabled by default, i.e the default is Exoplayer will still transcode ASS.

[nx6]

I plan on opening an issue upstream in libass-android, noting this may not be fixable in Exoplayer citing androidx/media#2289. However in the meantime, I still think the functionality is good enough in most use cases to be merged, maybe just not enabled by default, i.e the default is Exoplayer will still transcode ASS.

This has been a known issue for awhile. The Plex dev primarily responsible for the ASS support on the client there attributed it to limitations of ExoPlayer itself, essentially saying that it wasn't capable of frame-exact timing on subtitles. They wanted to work on improving the efficiency in rendering ASS subs but that never came to pass (this was a few years ago).

[nx6]

I've only tested this build on one device myself though (a Fire Stick 4K Max, Gen 2) so possible something will happen somewhere else, but I doubt it. Can check more later today.

I've had the opportunity to try this on a first-gen Fire Stick 4K Max and a Shield Pro now. I was going to report that on the Shield two of my four test clips are not using the included fonts, but after some more tests and looking at my Dashboard I see now that those two clips are not direct playing and the Jellyfin server is incorrectly rendering the subs in the transcode. Ex3 and Ex4 are Hi10p video (10-bit h264), so I should have known better since the Shield cannot direct-play that format.

I have not messed with the mpv setting on the first-gen Max, because there is a known chroma-corruption issue on 10-bit HEVC, which is the most-common type so really reduces the usefulness. This is not a Wholphin issue, it's device-specific and impacts Plezy and mpv-android, too (but no such issue on the second-gen Max). The Shield with mpv setting played back subs as expected (better performance than any sticks but not up to a normal PC's performance).

The only thing I've really noticed is alpha-value rendering seems to be slightly off. In my Ex1 there's masking over the names of the stalls in the background ("Octo-Balls" and "Candy Apples") the ExoPlayer render is allowing the original Japanese banner to show faintly throgh. In mpv the masking appears fully opaque. But this is very minor and doesn't impact readability.