8000 Support SSA/ASS styling · Issue #8435 · google/ExoPlayer · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

Support SSA/ASS styling #8435

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
jakobkukla opened this issue Jan 7, 2021 · 68 comments
Open

Support SSA/ASS styling #8435

jakobkukla opened this issue Jan 7, 2021 · 68 comments

Comments

@jakobkukla
Copy link

Add support for SSA/ASS styling. Currently there is only "minimal" support for ASS subtitles since styling was removed from the initial PR. (See #2281)

It would be great to have full support of the spec. Many subtitles use these styling features to work directly with the video (e.g. to substitute text in a video). SSA/ASS styling is also very common in certain genres of media like Cartoons, Anime or Karaoke.

@jakobkukla jakobkukla changed the title Support SSA/ASS subtitles styling Support SSA/ASS styling Jan 7, 2021
@andrewlewis
Copy link
Collaborator

@icbaker Tentatively marking as an enhancement, but please dupe or close as needed. Thanks!

@icbaker
Copy link
Collaborator
icbaker commented Jan 7, 2021

Thanks for the request.

Unfortunately we don't currently have any capacity to improve ExoPlayer's SSA/ASS styling support. However we would welcome high quality PRs in this area.

I'm also happy to discuss various approaches with PR authors if a new feature doesn't fit easily into the existing code structure (either before the PR, or in the review, whichever is easier).

@13steinj
Copy link
13steinj commented Jan 8, 2021

Could we determine a basic checklist of what isn't supported, to determine what features of the spec can be focused on in order of size and/or ease of implementation?

(I'm not familiar with what exactly even is supported currently).

@icbaker
Copy link
Collaborator
icbaker commented Jan 8, 2021

Could we determine a basic checklist of what isn't supported, to determine what features of the spec can be focused on in order of size and/or ease of implementation?

That sounds like a good idea - the best way to do that is probably to compare the SsaDecoder to a copy of the spec. I've found the SSA/ASS spec to be fairly loose, and I'm not sure there's a canonical definition, but I've generally referred to this when working with these files in the past: http://moodub.free.fr/video/ass-specs.doc (linked from Matroska's page on SSA).

In case it's helpful, we would probably prioritise things roughly in this order:

  • Features that look like they're implemented but don't work properly (basically bugs).
  • Features that are similar to existing ones, and so can be implemented with small incremental changes.
  • Features that don't fit into the existing code architecture - so implementing them requires significant refactoring.

I don't know if SSA has anything in the final category - but if it looks like it does then please come talk to us before spending ages refactoring SsaDecoder, as we might have some ideas about the best way to approach the problem.

@cosmo321
Copy link

I'm not a developer, so I can't exactly contribute a lot, but proper support of ass/ssa in Exoplayer is something I'd very much would like to have, as Plex for Android currently makes me sad...
I don't know if this makes sense in the context of Exoplayer, but it seems to me that the closest thing we have to a "standard" for ass/ssa is with libass, as a lot of notable players have implemented it and it is actively developed. I think at least being compliant with how libass renders subtitles would be a good idea.

@icbaker
Copy link
Collaborator
icbaker commented Jan 13, 2021

I don't know if this makes sense in the context of Exoplayer, but it seems to me that the closest thing we have to a "standard" for ass/ssa is with libass, as a lot of notable players have implemented it and it is actively developed. I think at least being compliant with how libass renders subtitles would be a good idea.

Thanks for the pointer to libass, I took a quick look. I think you're possibly suggesting two alternative things:

  • ExoPlayer could depend on libass and use it's logic for all the SSA rendering.
  • People working on improving ExoPlayer's existing SSA rendering logic could use libass as a reference implementation to see how different pieces of content should be rendered, as an augmentation to the "spec".

In case anyone is considering trying the first of these, I'd like to pre-emptively warn against it. libass looks like it's written in C, and we're very reluctant to pull in native/JNI dependencies unless a pure Java implementation of the same behaviour is deemed ~impossible. Given SSA/ASS is a relatively simple plaintext-based format, it seems completely feasible to parse it fully using pure Java - so it's extremely unlikely we'd accept a PR that introduced a native dependency to achieve this. There's more discussion about (not) using native deps for parsing subtitles in #7746 and #7749.

The second suggestion (use it as a reference implementation) makes perfect sense - and afaict it's used by VLC, so it should be enough to just play any content with VLC and compare to ExoPlayer's behaviour.

@cosmo321
Copy link
cosmo321 commented Jan 13, 2021

I leave that up to actual developers to decide. I just thought I'd point out that libass exists and is probably the best reference we have, and that rendering subs differently from libass could potentially make people like me wonder why the subs behave differently in ExoPlayer. The second alternative is probably the better option for ExoPlayer. :)

@szaboa
Copy link
Contributor
szaboa commented Jan 14, 2021

hey @icbaker , I'll check what the current SsaDecoder supports and what is missing in regards of styling, based on the specs you linked above, then I could probably start and implement those step by step (with my limited time).

@jakobkukla did you have anything specific in mind? I mean is there a specific styling option which you would like to see first?

@icbaker
Copy link
Collaborator
icbaker commented Jan 15, 2021

Thanks for taking a look! Breaking down the work into small PRs definitely sounds good.

@jakobkukla
Copy link
Author

@szaboa To be honest I'm not really aware what is currently supported and what not. But the ability to display fonts and colors properly would be nice!

@szaboa
Copy link
Contributor
szaboa commented Jan 18, 2021

@icbaker
I've checked out and it seems that only the v4+ "alignment" is supported. Here is the list of all the style attributes of v4 and v4+ from the specs. Do you see specific styles that would be very cumbersome to implement at first sight? E.g. thinking of "fontname", and "encoding"..

Style name v4 v4+ Note Format line support Override support PR
Name - - -
Fontname - -
Fontsize - #8615
PrimaryColour v4 BBGGRR, v4+ AABBGGRR #8490
SecondaryColour v4 BBGGRR, v4+ AABBGGRR -
OutlineColour v4 BBGGRR, v4+ AABBGGRR
Called OutlineColour in v4+, and TertiaryColour in v4
-
BackColour v4 BBGGRR, v4+ AABBGGRR -
Bold - #8654
Italic - #8654
Underline - #8851
Strikeout - #8851
ScaleX - -
ScaleY - -
Spacing - -
Angle - -
BorderStyle - -
Outline - -
Shadow - -
BorderStyle - -
Alignment values may be 1=Left, 2=Centered, 3=Right. Add 4 to the value for a "Toptitle". Add 8 to the value for a "Midtitle" -
Alignment layout of the numpad (1-3 sub, 4-6 mid, 7-9 top) -
MarginL, MarginR, MarginV - - #10169
AlphaLevel - -
Encoding - -

My plan is to start implement some easy ones first :)

@icbaker
Copy link
Collaborator
icbaker commented Jan 19, 2021

The spec I linked above (#8435 (comment)) seems to think that Underline, Strikeout, ScaleX, ScaleY, Spacing and Angle are supported in v4+. In fact are possibly only supported there and not in v4. Maybe those rows in your table are accidentally flipped? Or maybe I'm misreading the spec...

The SsaDecoder currently ignores v4 styling completely. It's not clear if that's sensible - I don't know how prevalent each version is in the wild.

If you're going to add v4 support we probably need to do think about whether we can reuse the SsaStyle class or split it into two. It's up to you, but for now imo it would be easiest to focus on v4+ only.

These all look like they should be relatively easy (I've referenced the relevant Android styling span where it seems obvious):

  • Fontname (TypefaceSpan)
  • PrimaryColour (ForegroundColorSpan)
  • BackColour (BackgroundColorSpan)
  • Bold (StyleSpan)
  • Italic (StyleSpan)
  • Underline (UnderlineSpan)
  • Strikeout (StrikethroughSpan)
  • AlphaLevel (read together with PrimaryColour, though the spec I'm reading says "SSA does not use this yet" so maybe not worth bothering with)
  • MarginL, R and V (in fact there's already a TODO for these :)). These will feed into Cue's position & line fields. This might be a bit messy - suggest you don't start here.

The spec I'm looking at doesn't really define these clearly enough to be sure:

  • Fontsize (I suspect you can use RelativeSizeSpan or AbsoluteSizeSpan but I don't know which is relevant)
    • EDIT: Small correction - if we work out how the spec defines this, we can use Cue.textSize and textSizeType for encoding size information from the "Format" section of the file that applies to the whole cue. We only need to use spans for overrides that affect substrings of the cue text.

These look harder. They have basically no support in the existing Cue interchange format:

  • SecondaryColour
  • OutlineColour
  • ScaleX and ScaleY
  • Spacing
  • Angle
  • BorderStyle
  • Outline
  • Shadow

These are hard for other reasons:

  • Encoding - I'm not sure how you're supposed to read this since it's in a text file - so you've already had to guess the encoding (right?). Currently we assume it's always UTF-8.

@szaboa
Copy link
Contributor
szaboa commented Jan 19, 2021

Yep, you are right, the Underline, Strikeout, ScaleX, ScaleY, Spacing and Angle are supported in v4+ only, and not in v4. I will correct the table.

Thanks for the tips, will definitely start with the easy ones.

@szaboa
Copy link
Contributor
szaboa commented Jan 21, 2021

Initial PR for "Primary Colour" is here #8490.

@moneytoo
Copy link
Contributor

Is there any option/API to disable, customize or detect the styling? If an app sets custom CaptionStyleCompat, it now accepts all styling while the text color is now variable based on line from subtitle file. Also, how about when user uses platform CaptioningManager to customize subtitle look - again, text color is now variable. With static background/border colors and variable text color, the subtitle line may end up low contrast on unreadable (right...?). Does my thinking here make any sense? (I've never used SSA/ASS outside some testing.)

@icbaker
Copy link
Collaborator
icbaker commented Feb 11, 2021

@moneytoo Assuming you're using the ExoPlayer UI components, you can force the SubtitleView to ignore 'embedded' styles (i.e. those encoded in subtitle files) by calling subtitleView.setApplyEmbeddedStyles(false). You can get the SubtitleView from playerView.getSubtitleView().

@szaboa
Copy link
Contributor
szaboa commented Feb 18, 2021

I just want to note that I haven't forgotten about this, just need to find time to continue it :)
Updated the table above with the PrimaryColour support.

@szaboa
Copy link
Contributor
szaboa commented Feb 21, 2021

Initial PR for "FontSize" is here: #8615

@GlassedSilver
Copy link

Re-implement SSA/ASS render would be a pure way, but takes much more time to get full feature support, I think import libass with JNI would be the easiest way to make things done.

That would be not only the easiest way, but also the only way to correctly render all those obscure and advanced feature-rich ASS files that typesetting enthusiasts can come up with. There are a lot .ass files with over 25MB of animated signs, and at this point not even those long-standing libass alternatives like vsfilter or its derivates can render them properly. Trying to recreate libass would be a neverending task and a huge magnet for complaints about any kind of advanced signs not rendering correctly. Not to mention, libass has years of performance optimizations.

Just adopting libass instead of making a new subtitle renderer could solve a lot of problems.

I wholeheartedly agree.

If this issue hadn't been such a long-standing one and been actually looked into by Google itself rather than them watching from the sidelines and hoping for the community to fully do this job by the looks of it, I'd be a lot more optimistic about exoplayer re-implementing and keeping up with changes in-time or mostly in-time. But apparently relying on constantly staying in-sync with ass extensions is a futile effort, so just using libass seems like the most reliable, future-proof-ish effort there is from my limited understanding.

It's a shame, but the dynamic nature of ass is both a curse, but also a blessing because it offers features not found elsewhere. The cost may be annoying. But when libass is doing the work needed why don't we just rely on it like what I think any sensible player does with proper ass-support?

Why re-invent the wheel? It's not like there's much to gain from it right? Even if using libass might not be as performant as a fully-optimized re-implementation, what sorts of differences that can be felt are we talking about here? Subtitle rendering isn't something that needs to scale endlessly, especially not when it's per player, at humanly digestible play-speeds, etc...

To anyone who is investing any sort of work into this effort: you have my most honest gratitude. The day Android has player-independent ACE ass support will be a good day! Not only for us anime and FOSS app fans.

@FongMi
Copy link
FongMi commented May 27, 2023

Is there way to fix overlapping subtitles ?

@GlassedSilver
Copy link

Is there way to fix overlapping subtitles ?

It's a known issue, any fix would be supplied by exoplayer extending ass support. I'm not aware of user settings that can fix this.

@Sparh4wk
Copy link

so its been two years already and we still dont have proper ssa support.. if using libass can solve this, can some dev do something about it? I think a lot of ppl would be rly happy. I know you are doing this for free, but still. thanks

@happyTonakai
Copy link

Is there any progress on it? Many other multimedia apps are based on ExoPlayer and thus have limited support on ass/ssa subtitles.

@FongMi
Copy link
FongMi commented Jan 15, 2024

Is there any progress on it? Many other multimedia apps are based on ExoPlayer and thus have limited support on ass/ssa subtitles.

#10169

@happyTonakai
Copy link

Is there any progress on it? Many other multimedia apps are based on ExoPlayer and thus have limited support on ass/ssa subtitles.

#10169

I'm not sure if this can be called as a progress since nearly two years have passed and this pr is still not merged. In addition, there are still many styles that are not supported. Low priority label makes me desperate...

@e1ker
Copy link
e1ker commented May 13, 2024

I'd love to get this implemented as it would make this player the only one with DV and proper .ass support in Android besides Kodi which is way too heavy for anything other than an Nvidia Shield.

@zhuhang-jasper
Copy link

Looking forward to this!

@peerless2012
Copy link
peerless2012 commented Jan 21, 2025

L

@icbaker I looked into what SSA/ASS supports and what is implemented and what not. So what I think about it. Currently there are two "subtitle" renderers. CanvasSubtitleOutput (the default) and WebViewSubtitleOutput (which I'm not sure how it's launched, I guess it's done by the app which uses ExoPlayer). Anyway - to play with it I forced the WebViewSubtitleOutput, just because doing some styling is easier for me personally with CSS. Many features ASS supports are not so easily done with CSS as well. I even played with doing stuff like BorderStyle, Outline, Shadow with SVGs, as CSS doesn't support BorderStyle-Outline the way ASS requires it. And I'm not even talking about Effects -- which are basically animations. I then went further down wanting to check how libass works and came across their project, which uses webassembly. Btw according to libass github page, Crunchyroll is using this project. I even entertained myself with using said webassembly with WebViewSubtitleOutput, but it's a no go. As it stands - ExoPlayer looks at subtitles as a series of cues being rendered on a WebView and/or Canvas. Libass creates just another "video" stream based on description of text file. This makes sense. Check this example page. It showcases some capabilities of SSA/ASS which I'm not sure are at all possible with cues. So what are next steps?

  1. One can hope that someone ports libass to Java -- which, probably will never happen :)
  2. Quoting you from above. I now think that parsing is indeed easy, but rendering is next to impossible (with cues).

libass looks like it's written in C, and we're very reluctant to pull in native/JNI dependencies unless a pure Java implementation of the same behaviour is deemed ~impossible. Given SSA/ASS is a relatively simple plaintext-based format, it seems completely feasible to parse it fully using pure Java

  1. Keep it as it is -- as in, not ever adding the full list of features.

wdyt?

Good news, the libass for exoplayer is on the way, please check our progress in libass-android

@peerless2012
Copy link
peerless2012 commented Jan 21, 2025

Is there any progress on it? Many other multimedia apps are based on ExoPlayer and thus have limited support on ass/ssa subtitles.

You can find the libass for exoplayer at libass-android

@icbaker
Copy link
Collaborator
icbaker commented Jan 21, 2025

As it stands - ExoPlayer looks at subtitles as a series of cues being rendered on a WebView and/or Canvas. Libass creates just another "video" stream based on description of text file.

This is why I suggested it might fit better to use libass to drive a custom effect (androidx/media#2053 (comment)) - my intuition is that plumbing the subtitle data into a custom effect will be easier than trying to pass video-like data through ExoPlayer's Cue machinery.

@peerless2012
Copy link

As it stands - ExoPlayer looks at subtitles as a series of cues being rendered on a WebView and/or Canvas. Libass creates just another "video" stream based on description of text file.

This is why I suggested it might fit better to use libass to drive a custom effect (androidx/media#2053 (comment)) - my intuition is that plumbing the subtitle data into a custom effect will be easier than trying to pass video-like data through ExoPlayer's Cue machinery.

Libass produce bitmap as output, witch is compat for Cue with bitmap. I already finish this with ARGB_888 bitmap.
But when I want to change bitmap to ALPHA_8, the Cue encode is error, and SubtitlePainter also need Blend bitmap.

Why do I stick with this approach?

Because this is a most simple way to render ass with libass, I don't need to worry about subtitles syncing with the video,and compat with current sutitle render process,with a few change in cue and SubtitlePainter.

If we use Effect (If it can work), there is more work I need todo, and to hack exo extractor and prase process.

@moi15moi
Copy link
moi15moi commented Mar 11, 2025

@icbaker I'm currently trying to add LibassSubtitleRenderer, which would use libass, and eventually open a PR.
As you already mentioned, the Cue machinery doesn't "fit" how libass works.

If I am not mistaken, ASS/SSA subtitles are only available for the Matroska container, so all the decisions I made were based on this observation.

Libass Method Media3 Function
ass_render_frame ExoPlayer.setVideoEffects() + BitmapOverlay. I would return the bitmap generated by libass in BitmapOverlay.getBitmap(). Also, in the bitmap alpha blending, I would call ExoPlayer.getVideoFormat().colorInfo to do the "VSFilter mangled colors" explained here
ass_set_frame_size Use the value obtained from Player.getSurfaceSize() + Player.Listener.onSurfaceSizeChanged().
ass_set_storage_size Use the value obtained from Player.getVideoSize() + Player.Listener.onVideoSizeChanged().
ass_process_codec_private SsaParser constructor, but SsaParser implements SubtitleParser, which requires me to use Cues, which I cannot do. So I'm not sure what I should do.
ass_process_chunk TrackOutput.sampleMetadata() seems to receive the Matroska block, but media3 seems to modify the block data and add the duration. It should be fine, but it is something I need to verify.
ass_add_font (Libass absolutely needs the font muxed into the MKV) It is fairly simple to extract the font from the MKV (I did it here), but I don't see how I can pass the fonts to libass.

I saw your comment here:

This would involve a custom ExtractorsFactory and a customized ForwardingExtractorOutput that lets you intercept the subtitle data.

By default, Media3 uses DefaultExtractorsFactory, so I assume I would need to modify this class.

How could I add a special case to use libass in this class?

Also, I don't quite understand what I should edit, so this line returns my custom TrackOutput. Could you give me some detail?

@icbaker
Copy link
Collaborator
icbaker commented Mar 12, 2025

and eventually open a PR.

Just to set expectations, based on the sketched design in your comment, I'm afraid we are unlikely to merge such a PR.

The design relies on media3's lib-effect, and libass native code, neither of which are currently depended on by lib-exoplayer, so either we'd need to add these deps (which then affects every user of the player library), or this would need to be in a new lib-ssa-parser module or similar, which comes with significant maintenance burden that I'm afraid probably isn't justified.

The design being so 'different' to all other subtitle parsers (by necessity) also increases the maintenance burden.

So I'm afraid a design like this will have to exist as an extension maintained outside the main media3 repository.

@peerless2012
Copy link
peerless2012 commented Mar 12, 2025

Support extract fonts in mkv and pass them to SubtitleParser maybe available?

In our project, we use AssMatroskaExtractor
witch extend MatroskaExtractor to extract the fonts attachments.

@e1ker
Copy link
e1ker commented Mar 13, 2025

So I'm afraid a design like this will have to exist as an extension maintained outside the main media3 repository.

I'm not sure if this means it wouldn't be officially integrated into the player, but if that's the case even an unofficial fork would be great. I feel this is the only thing missing to have a player that "just works" not only on Android but on almost every other OS, since most have very big compromises, are too heavy or too hard to set up.

@peerless2012
Copy link

So I'm afraid a design like this will have to exist as an extension maintained outside the main media3 repository.

I'm not sure if this means it wouldn't be officially integrated into the player, but if that's the case even an unofficial fork would be great. I feel this is the only thing missing to have a player that "just works" not only on Android but on almost every other OS, since most have very big compromises, are too heavy or too hard to set up.

There is already a third plugin to bring libass in media3, you just need to add only one method to your players construct.

Here's an example:

player = ExoPlayer.Builder(this)
            .buildWithAssSupport(
                this,
                AssRenderType.OPEN_GL
            )

See buildWithAssSupport

@GlassedSilver
Copy link

and eventually open a PR.

Just to set expectations, based on the sketched design in your comment, I'm afraid we are unlikely to merge such a PR.

The design relies on media3's lib-effect, and libass native code, neither of which are currently depended on by lib-exoplayer, so either we'd need to add these deps (which then affects every user of the player library), or this would need to be in a new lib-ssa-parser module or similar, which comes with significant maintenance burden that I'm afraid probably isn't justified.

The design being so 'different' to all other subtitle parsers (by necessity) also increases the maintenance burden.

So I'm afraid a design like this will have to exist as an extension maintained outside the main media3 repository.

Honest question from someone very grateful exoplayer exists to begin with <3

Why is it unreasonable to include those dependencies?

If the goal is to support more subtitles which is a goal of accessibility I don't see how that's something that shouldn't be universal.

And before anyone says it... Text-only subs that don't rely on styling are not always an equal but not as pretty path. Quite frequently subs need to be placed in various different places and styling is often done to differentiate between characters, underline the mood/atmosphere /etc...

@nigelmegitt
Copy link

Text-only subs that don't rely on styling are not always an equal but not as pretty path. Quite frequently subs need to be placed in various different places and styling is often done to differentiate between characters, underline the mood/atmosphere /etc...

ExoPlayer already supports other formats than SSA/ASS that allow for positioning and styling. The argument for SSA/ASS is, as I understand it, to do with specific styling options that are only available in that format and that are particularly popular with some genres of content.

@GlassedSilver
Copy link
GlassedSilver commented Mar 14, 2025

Text-only subs that don't rely on styling are not always an equal but not as pretty path. Quite frequently subs need to be placed in various different places and styling is often done to differentiate between characters, underline the mood/atmosphere /etc...

ExoPlayer already supports other formats than SSA/ASS that allow for positioning and styling. The argument for SSA/ASS is, as I understand it, to do with specific styling options that are only available in that format and that are particularly popular with some genres of content.

Correct, however when all that you have is SSA/ASS for a given piece of content (at least properly detailed subs, a lot of times rather mediocre compromise subs exist in text-only) then wishing for the other options getting used will get you nowhere.

How many genres are affected doesn't change that this does make some content less accessible.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

0