Proposal for media encoding profiles

Problem
=======
We need an easy yet powerful way to select media encoding. However, media
encoding is very diverse, so the system needs to be modular. This document
describes a few sample problems and a proposal for a solution.

Currently, GNOME contains audio profiles, which are a way of selecting an
audio encoding for audio authoring applications such as GNOME Sound
Recorder (audio capture), Sound Juicer (CD ripping) and so on. This system
is not powerful enough for video, though. Cupid (video capture) and Pitivi
(video editing) both use their own (almost identical) way of media encoding
selection. however, both the GNOME audio profiles as well as the selections
offered by Cupid/Pitivi have serious limitations.

GNOME audio profiles:
* are hard to create from a user point of view
* are basically 1-to-1, making video hard to impossible

Cupid/Pitivi's selections:
* are very static in their configuration (1x audio, 1x video, 1x muxer)
* don't really offer a way for profile sharing
* don't allow predefined profiles
* are therefore generally quite hard for end users

Cases
=====
As said before, a general profiles system should allow for more than just
audio, but preferrably also more than just 1x video and 1x audio. Also, just
offering a video selection box isn't enough, since sometimes, you'll need an
additional parsing element for specific muxers (e.g. divxenc ! ogmvideoenc !
oggmux). Additionally, you would ideally allow for more than one media type
encoding. E.g. for DVD authoring, you'd want to be able to choose between
DTS, AC-3 and LCPM audio. Lastly, we shouldn't limit ourselves to elements
only. The GNOME audio profiles already allow for filtered caps, we should
allow this too, so that we can separate between PAL DVD MPEG-2 and NTSC DVD
MPEG-2 (720x576 at 25.0 fps vs. 720x480 at 29.97 fps). In pseudo-gst-launch
syntax, all of those should be possible:
* lame
* audio/x-raw-int,channels=2,rate=48000 ! wavparse
* ( vorbisenc ) and ( theoraenc ) each connected to oggmux
* ( vorbisenc ) and ( xvidenc ! ogmvideoenc ) each connected to oggmux
* ( audio/x-raw-int,channels=6,channel-positions=<setup-for-5.1-audio> !
  ffenc_ac3 ) and ( audio/x-raw-int,channels=2 ! ffenc_ac3 ) and (
  audio/x-raw-int,channels=2 ! lpcmenc ) and (
  audio/x-raw-int,channels=6,channel-positions=<setup-for-5.1-audio> !
  dtsenc ) and ( video/x-raw-{yuv,rgb},width=720,height=576,framerate=25 !
  mpeg2enc ! video/mpeg,mpegversion=2 ) and (
  video/x-raw-{yuv,rgb},width=720,height=480,framerate=29.97 ! mpeg2enc !
  video/mpeg,mpegversion=2 ) each connected to mplex

The difficulty with 5, which is used for DVD encoding, is the fact that we can
have multiple inputs of the same type (e.g. 2 audio streams), and each may
have a different encoding, but may also be the same. The issue with 4 is that
there are two elements for video encoding. None of the currently existing
solutions handle that case at all. GNOME audio profiles currently allow for
method 1 and 2, so we should continue allowing that.

As for the end user, GNOME audio profiles does pretty much the right thing. We
should offer a selectionbox to the user to select a specific profile. The
application should, in some way, be able/allowed to get only the encoding
profiles that match his specific use. For Sound Juicer, only 1 and 2 would be
useful, whereas for Cupid, all of 3, 4 and 5 would be useful.
For applications such as Pitivi, it would be useful to be able to select a
specific "input profile" for each of the elementary streams (e.g. for audio,
you could choose between DTS 5.1, AC-3 2.0, AC-3 5.1 or LPCM 2.0 for each
audio elemtary stream).

For reference, use cases with profiles are:
* getting a list of available profiles
* selecting and using a profile in your authoring application
* modifying an existing profile or creating a new profile

Backend implementation
======================
A profile would consist of a series of subprofiles, each of which is a 1-to-1
chain of elements and filters (like what GNOME audio profiles do now), at
least for the backend. Those subgroups can be ordered and linked in a N-to-N
fashion with each other. Example subprofiles for the above-given cases would
be everything within one set of brackets, or the whole thing if no brackets
are given. No "general conversion elements" are used (e.g. ffmpegcolorspace,
videoscale, videorate, audioscale, audioconvert), applications are expected
to be aware of this themselves. Specific filtering elements (e.g. deinterlace)
can be part of a subprofile, though.
Subprofiles can be chained to form the complete profile, e.g. { dts-5.1,
ac-3-5.1, ac-3-2.0, lpcm-2.0, pal-dvd, ntsc-dvd } ! mplex.
Profiles could be stored in XML format, similar to the registry (maybe even
as part of the registry). A Profile consists of a pretty-name and a
combination of subprofiles. Each subprofile has a pretty-name, a gst-launch
syntax and optionally an input/output link to another subprofile.

Frontend API
============
The application frontend will be presented a list of strings, each of which
represents one single profile. A profile can be rendered to a bin, which has
one output pad and a number (where N = number of entry subprofiles) of sink
always/request pad templates. Applications can then use this to link it in
their regular pipeline that they're working with. In addition, it should be
possible to get a pretty-name for each entry subprofile. I don't know how
yet.
This system is essentially very similar to what the GNOME audio profiles do
right now.

Adding new profiles
===================
The hard part with the current GNOME audio profiles are profile editing and
creation. The subprofile <-> profile distinction doesn't make this any
easier. However, a possible future profile editor could start as a series of
gst-launch syntax pipelines (each of which represents a subprofile), and a
second part where groups can be linked with one another.

Later on, the subprofile editor should become more sophisticated. It doesn't
exactly have to be gst-editor, because it's still only a 1-to-1 pipeline,
which simplifies everything a lot. We could try to create a 'filter' and
'element' graphical object and chain those horizontally to represent the
chain of filters and elements. Gst-editor already has an element selector
(treeview), which could be used to edit or add a new element to the chain.
Gst-editor also provides a widget for element property editing. A filter
editor (caps) would be harder, but should still not be impossible. Chaining
those together in some specific order would then finish the editing/creating
of a subprofile. Give them a pretty name and represent them by that name in
the 'subprofile linking UI'.

Subprofile linking UI
=====================
As mentionned above, subprofile are 1-to-1 pipelines, which contain at least one
element or one caps-filter. One proposition to graphically create/modify
subprofiles is to use a "sandwich" view, each layer being either an element, or
else a filter. The top being the input, and the bottom being the output of the
pipeline.

Ex : "Wave Stereo 48kHz" (audio/x-raw-int,channels=2,rate=48000 ! wavparse)

/==================(input)===================\
| [ audio/x-raw-int,channels=2,rate=480000 ] |
| [                wavparse                ] |
\==================(output)==================/

Ex : "Progressive MPEG2 Video" (deinterlace ! ffenc_mpeg2video)

/======(input)=========\
| [    deinterlace   ] |
| [ ffenc_mpeg2video ] |
\======(output)========/

Ex : "Mpeg2 NTSC" (video/x-raw-yuv,width=720,height=480,framerate=29.97 ! 
mpeg2enc,format=9 ! video/mpeg,mpegversion=2)

/=========================(input)================================\
| [ video/x-raw-{yuv,rgb},width=720,height=480,framerate=29.97 ] |
| [                  mpeg2enc,format=9                         ] |
| [               video/mpeg,mpegversion=2                     ] |
\=========================(output)===============================/

The trick is to be able to set some filters/elements properties (like in the
example above with the format option of mpeg2enc).
The boxes above are listviews, with a single possible selection. On selection,
the single selected component in the sandwitch becomes editable in a separate
widget opening up to the right of this listview.

 _____________________________________________________
/ \/  mpeg2enc (High-quality MPEG-1/2 video encoder)  \
|                                                     |
| [x] format       [ DVD MPEG-2                    ]  |
| [ ] framerate    [xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx]  |
| ....                                                |
\_____________________________________________________/


For setting filters, it's a tiny bit trickier, in the sense that by default the
links between elements are already filtered (the intersection between the srcpad
 (if any) and the sinkpad (if any) caps). This of course only works if the
subprofile contains at least one element.
So once there's an element added to the subprofile, we need a way to "refine"
the existing caps if desired. One proposition is to have a thick line above,
between and under the elements that represent the existing caps. When you click
on it, it expands (like the element layers) and show the available caps
properties where you can specify some of the parameters.

Ex : for the input caps of mpeg2enc,
 _________________________________
/ \/ video/x-raw-yuv              \
| [x] width     [           720 ] |
| [x] height    [           480 ] |
| [x] framerate [ 29.970...  \/ ] |
\_________________________________/

To link subprofiles together and form a profile, all "linkable" (i.e. matching
caps) profiles on in- and output should show up in a togglebutton listview
inside this interface. So for DVD, it would look like this for the mplex
component:

 _________________________________   ______________________________
/ \/ Input for mplex              \ / \/ Output for mplex          \
| [x] DVD PAL MPEG-2              | | [ ] nothing selected         |
| [x] DVD NTSC MPEG-2             | |                              |
| [ ] some dummy disabled subprof | |                              |
| ...                             | |                              |
\_________________________________/ \______________________________/

This last bit may look somewhat geeky (or, rather, non-intuitive).
Suggestions for improvement are welcome.

By
==
Ronald S. Bultje <rbultje@gnome.org>
Edward Hervey <bilboed@bilboed.com>
16/3/'05
