GNOME Bugzilla – Bug 612312
typefind: Post AAC profile in caps
Last modified: 2010-04-30 19:57:00 UTC
This tag will be set by codecs such as AAC and AVC to expose what profile is being used in a stream (Main/Low Complexity/...).
Created attachment 155662 [details] [review] Defined a 'codec-profile' tag
Should this instead be part of the caps? I think so, since some decoders may be unable to handle all profiles. If it's in a tag, and has no defined format, it's really only useful for displaying to a user - which might be useful for some applications, but isn't very flexible for things that want to restrict capabilities appropriately.
I too would be more in favor of having it in the caps, maybe in addition to having it as a tag. Also, some codecs define profiles as strings, and some others as integers.
Makes sense - I'm rewriting this to put the profile in the caps at typefind time. For codecs that have an integer profile, it can still be expressed in string form. The advantage of this is that it'll make the type consistent, but it will make the caps more verbose for codecs with numeric profiles.
Created attachment 155744 [details] [review] typefind: Post AAC profile in caps Patch for the AAC typefind function to post the AAC profile in caps.
Comment on attachment 155744 [details] [review] typefind: Post AAC profile in caps Maybe mark the string array const, other than that looks good :)
Posting a patch updated for comments soon. In the mean time, here's some documentation on how these profiles are to be used. For AAC-LC/Main/SSR/LTP, the usage is straightforward - the streams will have the corresponding profile string in caps, and decoders will (eventually) list the profiles they support in their caps, and linking will proceed as usual. The HE-AAC v1/v2 case is slightly more complicated. HE-AAC v1 is AAC-LC with an additional SBR payload, and HE-AAC v2 is AAC-LC with an SBR and PS payload. The AudioObjectType in the header can: 1. explicitly signal that this is HE-AAC (MP4 container only, ADTS doesn't have enough bits) 2. implicitly signal by an AAC-LC object with SBR/PS data just showing up in the bitstream 3. implicitly signal in backward-compatible mode via an extensionAudioObject field in the header (again MP4 only) indicating the HE-AAC profile type The expected behaviour for an AAC-only decoder is that it should not play a stream which explicitly signals HE-AAC, but for an implicitly signaled HE-AAC stream, it should play only the AAC bits. To make this work with the caps-based approach, I propose that streams with explicitly signaled HE-AAC have profile set as "he-aac-v1" or "he-aac-v2", while implicitly signaled HE-AAC streams have these as "aac-lc+sbr" and "aac-lc+sbr+ps" respectively. Non-HE AAC decoders would register one element for handling aac-lc, and another, with low rank supporting "aac-lc+sbr" and "aac-lc+sbr+ps". HE-AAC decoders would register support for the AAC profiles, "aac-lc+sbr", "he-aac-v1" (and possibly "aac-lc+sbr+ps" and "he-aac-v2" if supported) with higher rank. In this way, when an implicitly signaled HE-AAC stream needs to be decoded, if an HE-AAC decoder is available it will be used, but if not, the fallback AAC decoder will be used. To make that easier to grok, the table lists the type of decoder and the supported profiles: Decoder > non-HE AAC HE-AAC v1 HE-AAC v2 Caps v aac-lc, etc. Y* Y* Y* aac-lc+sbr Y (degraded) Y Y aac-lc+sbr+ps Y (degraded) Y (degraded) Y he-aac-v1 N Y Y he-aac-v2 N N Y '*' => decoders would support what profiles they can decode, of course, but I'm assuming all decoders we're speaking of can decode aac-lc 'degraded' => the decoder _can_ play the profile type, but since it doesn't support all stream features, it will not be of the maximum possible quality
Created attachment 156564 [details] [review] Updated patch Patch with feedback incorporated, and a pointer to this bug for notes on how to use the profile string.
(In reply to comment #7) > The expected behaviour for an AAC-only decoder is that it should not play a > stream which explicitly signals HE-AAC, but for an implicitly signaled HE-AAC > stream, it should play only the AAC bits. Why is this the expected behavior? I would expect the behavior for an AAC-only decoder to always play the base layer.
(In reply to comment #9) > (In reply to comment #7) > > The expected behaviour for an AAC-only decoder is that it should not play a > > stream which explicitly signals HE-AAC, but for an implicitly signaled HE-AAC > > stream, it should play only the AAC bits. > > Why is this the expected behavior? I would expect the behavior for an AAC-only > decoder to always play the base layer. This is the behaviour specified in the ISO 14496-3. I presume the intention is to allow content creators to say "this would sound really crappy if played on an AAC-only decoder, so don't".
Ping?
In that case, I think it would be better to list the actual profile, and a separate boolean "lc-playable" ("lc-useful", "lc-ok", "have-base-layer", someone come up with a good name) to indicate whether the base layer is intended to be useful.
How about two fields, something like this (I left out the "aac" bits): base-layer = { "lc", "main", "ssr", "ltp", "he-v1", "he-v2" } profile = { "lc", "main", "ssr", "ltp", "he-v1", "he-v2" } so then you'd get: base-layer = lc, profile = lc base-layer = ltp, profile = ltp base-layer = ssr, profile = ssr base-layer = main, profile = main base-layer = he-v1, profile = he-v1 (for explicitly signalled HE-AAC) base-layer = he-v2, profile = he-v2 (for explicitly signalled HE-AAC) base-layer = lc, profile = { he-v1, he-v2 } (This is assuming an HE-AAC v1 decoder should not decode an explicitly signalled HE-AAC v2 stream either.) Would that do the trick?
This would work, and I guess it's a bit cleaner than profiles like 'aac+sbr+ps'. Only thing here is that for implicitly signaled he-v2, you'd have: base-layer = { lc, he-v1 }, profile = { he-v2 } This is because in the implicitly signaled case, he-v2 should be decoded by an he-v1 decoder if available, and if not, then an lc decoder. I'd go with "fallback-profile" over "base-layer" since it conveys the meaning more accurately.
In fact, only lc, he-v1 and he-v2 decoders would define "fallback-profile" in their caps, and only implicitly signaled he-v1/v2 streams would need to define these in stream caps. For all the others, either a match on "profile" will be found, or linking will be impossible
I forgot to add earlier: I'm not so sure about the boolean solution suggested by Dave, because it seems less flexible/extensible as other solutions, and it doesn't look obvious to me that we might not want to add other (obscure?) profiles at some point. I still think base-something (layer? profile?) is a good name for this field, since I expect most decoders to just specify that by default. It'd be weird to see a decoder announce caps like: audio/mpeg, mpegversion=4, fallback-profile={foo, bar} without a profile field (which could be added, but is probably not really needed). I'm assuming that the usual case is that we want to play everything we can by default, even if it's degraded. Complex setups with multiple different decoders should be possible, but these are special cases where the vendor is in control of the setup.
The attached patch can go in as is, as it can be extended to accommodated either solution ('base-layer' or boolean flag). I'll implement the consensus when adding HE-AAC support.
Ok, so I think the general agreement was that a bool might not be the best idea, and the base-foo approach should be acceptable. I believe the preference was for base-profile rather than base-layer though.
commit 34dcb8458e2dfee37f83e0a3bf37195a4d503bb2 Author: Arun Raghavan <arun.raghavan@collabora.co.uk> Date: Wed Mar 10 13:32:53 2010 +0000 typefinding: add AAC profile to ADTS caps This looks at the AAC profile for ADTS streams and adds the profile as a string in the corresponding caps. Profile is the actual profile, base-profile denotes the minimum codec requirements to decode this stream. In this case they're always the same, but they may differ e.g. in case of certain HE-AAC streams that can be partially decoded by LC decoders (with loss of quality of course) if no suitable HE-AAC decoder is available. Fixes #612312.