After an evaluation, GNOME has moved from Bugzilla to GitLab. Learn more about GitLab.
No new issues can be reported in GNOME Bugzilla anymore.
To report an issue in a GNOME project, go to GNOME GitLab.
Do not go to GNOME Gitlab for: Bluefish, Doxygen, GnuCash, GStreamer, java-gnome, LDTP, NetworkManager, Tomboy.
Bug 711087 - rtpbin: Support Auxiliary streams
rtpbin: Support Auxiliary streams
Status: RESOLVED FIXED
Product: GStreamer
Classification: Platform
Component: gst-plugins-good
git master
Other All
: Normal enhancement
: 1.3.1
Assigned To: GStreamer Maintainers
GStreamer Maintainers
http://cgit.collabora.com/git/user/gk...
Depends on: 711084
Blocks:
 
 
Reported: 2013-10-29 17:05 UTC by Julien Isorce
Modified: 2013-12-31 14:14 UTC
See Also:
GNOME target: ---
GNOME version: ---


Attachments
rtpbin: add support for "receiver" auxiliary stream in rtpbin (10.41 KB, patch)
2013-10-29 17:05 UTC, Julien Isorce
none Details | Review
rtpbin: add support for "sender" auxiliary stream in rtpbin (9.23 KB, patch)
2013-10-29 17:05 UTC, Julien Isorce
none Details | Review
tests/check: add rtprtx::test_simple_rtpbin_aux unit test (11.38 KB, patch)
2013-10-29 17:06 UTC, Julien Isorce
none Details | Review
rtpbin: add support for "receiver" auxiliary stream in rtpbin (11.87 KB, patch)
2013-11-01 17:34 UTC, Julien Isorce
none Details | Review
rtpbin: add support for "sender" auxiliary stream in rtpbin (11.73 KB, patch)
2013-11-01 17:35 UTC, Julien Isorce
none Details | Review
tests/check: add rtpaux::test_simple_rtpbin_aux unit test (14.87 KB, patch)
2013-11-01 17:36 UTC, Julien Isorce
none Details | Review
rtpbin: generate "set-aux-send" and "set-aux-receive" action signals in gtk doc (1.07 KB, patch)
2013-11-01 17:37 UTC, Julien Isorce
none Details | Review
gstrtpsession: send custom upstream event "GstRTPCollision" on send_rtp_sink pad (1.57 KB, patch)
2013-11-01 17:37 UTC, Julien Isorce
none Details | Review
gstrtpbasepayload: regenerate ssrc when receiving GstRTPCollision event (4.61 KB, patch)
2013-11-01 17:38 UTC, Julien Isorce
none Details | Review
tests/check: add rtpsession::test_master_ssrc_collision unit test (10.66 KB, patch)
2013-11-01 17:39 UTC, Julien Isorce
none Details | Review
tests/check: add rtpsession::test_rtx_ssrc_collision unit test (8.25 KB, patch)
2013-11-01 17:39 UTC, Julien Isorce
none Details | Review
rtpbin: add support for "receiver" auxiliary stream in rtpbin (11.87 KB, patch)
2013-11-06 17:24 UTC, George Kiagiadakis
none Details | Review
rtpbin: add support for "sender" auxiliary stream in rtpbin (11.73 KB, patch)
2013-11-06 17:24 UTC, George Kiagiadakis
none Details | Review
tests/check: add rtpaux::test_simple_rtpbin_aux (15.21 KB, patch)
2013-11-06 17:25 UTC, George Kiagiadakis
none Details | Review
rtpbin: generate "set-aux-send" and "set-aux-receive" action signals in gtk doc (1.07 KB, patch)
2013-11-06 17:26 UTC, George Kiagiadakis
rejected Details | Review
doc: add design-rtpauxiliary.txt to describe how rtpbin deals with auxiliary elements (5.18 KB, patch)
2013-11-06 17:26 UTC, George Kiagiadakis
none Details | Review
tests/check: add rtpaux::test_simple_rtpbin_aux (14.95 KB, patch)
2013-11-08 11:02 UTC, Julien Isorce
none Details | Review
rtpbin: add support for "sender" auxiliary stream in rtpbin (7.45 KB, patch)
2013-11-18 16:36 UTC, George Kiagiadakis
rejected Details | Review
rtpbin: add support for "receiver" auxiliary stream in rtpbin (11.87 KB, patch)
2013-11-20 12:53 UTC, George Kiagiadakis
rejected Details | Review
tests/check: add rtpaux::test_simple_rtpbin_aux (15.25 KB, patch)
2013-11-20 12:54 UTC, George Kiagiadakis
none Details | Review
examples: rtp: Add end-to-end rtpbin example with RTX elements (23.57 KB, patch)
2013-11-25 16:07 UTC, George Kiagiadakis
none Details | Review

Description Julien Isorce 2013-10-29 17:05:04 UTC
So that rtpbin can configure auxiliary streams for sender and receiver.

Examples:
rtprtxsend and rtprtxreceive are auxiliary elements. But they are SSRC-multiplexed.

Auxiliary streams may also be sessions-multiplexed
Comment 1 Julien Isorce 2013-10-29 17:05:32 UTC
Created attachment 258463 [details] [review]
rtpbin: add support for "receiver" auxiliary stream in rtpbin
Comment 2 Julien Isorce 2013-10-29 17:05:59 UTC
Created attachment 258464 [details] [review]
rtpbin: add support for "sender" auxiliary stream in rtpbin
Comment 3 Julien Isorce 2013-10-29 17:06:30 UTC
Created attachment 258465 [details] [review]
tests/check: add rtprtx::test_simple_rtpbin_aux unit test
Comment 4 Julien Isorce 2013-10-29 17:41:30 UTC
So my kind of API to setup auxiliary elements inside rtpbin works like that:

* Instanciate any aux receivers or senders in your application.
* build a structure with name "rtpauxreceive" for aux receivers or "rtpauxsend" for aux senders.
* add "element" field of type GST_TYPE_ELEMENT to that structure to pass the gst element.
* add "sessions" field of type GST_TYPE_LIST to that structure to pass the sessions ids that have to use this aux element. (ids are the field uses in "recv_rtp_sink_%u" for receivers, and "send_rtp_sink_%u" with "send_rtp_src_%u") for senders
* use new rtpbin's "add-aux-element" property to pass this structure.
* rtpbin then handle those aux elements in a generic way so that they can be session-multiplxed or SSRC-multiplexed

See checks/test/elements/rtprtx.c::test_simple_rtpbin_aux for an example.

Notes: currently not possible to use from gst-launch. We can discuss about having the ability for rtpbin to instanciate itself the special aux elements rtprtxsend and rtprtxreceive but they need to be configured ("payload-type" and "payload-types" properties) to make retransmission work. So having several rtprtxsend and rtprtxreceive in a rtpbin would require a complete api/properties. And for each aux elements.
Comment 5 Olivier Crête 2013-10-29 18:17:26 UTC
As I mentioned in person, I'm not fan of GST_TYPE_LIST in APIs. I wonder if the API couldn'T be something like this, and allow it to be called multiple times and rtpbin would check if the element is already there and re-use it, etc ?

g_signal_emit(rtpbin, "set-aux-element", stream_id, element);
Comment 6 Julien Isorce 2013-10-30 09:59:42 UTC
I guess you mean a G_SIGNAL_ACTION ?

Make sense, also because there are not a lot of stream_id per aux element:

For SSRC-multiplexed auxiliary elements like rtprtxsend and rtprtxreceive, it's always one. 

For sessions-multiplexed auxiliary elements, in concrete cases it will be not much more than two I think.
Comment 7 Julien Isorce 2013-11-01 17:34:02 UTC
Created attachment 258765 [details] [review]
rtpbin: add support for "receiver" auxiliary stream in rtpbin
Comment 8 Julien Isorce 2013-11-01 17:35:49 UTC
Created attachment 258766 [details] [review]
rtpbin: add support for "sender" auxiliary stream in rtpbin
Comment 9 Julien Isorce 2013-11-01 17:36:31 UTC
Created attachment 258767 [details] [review]
tests/check: add rtpaux::test_simple_rtpbin_aux unit test
Comment 10 Julien Isorce 2013-11-01 17:37:08 UTC
Created attachment 258768 [details] [review]
rtpbin: generate "set-aux-send" and "set-aux-receive" action signals in gtk doc
Comment 11 Julien Isorce 2013-11-01 17:37:47 UTC
Created attachment 258769 [details] [review]
gstrtpsession: send custom upstream event "GstRTPCollision" on send_rtp_sink pad
Comment 12 Julien Isorce 2013-11-01 17:38:41 UTC
Created attachment 258770 [details] [review]
gstrtpbasepayload: regenerate ssrc when receiving GstRTPCollision event
Comment 13 Julien Isorce 2013-11-01 17:39:21 UTC
Created attachment 258771 [details] [review]
tests/check: add  rtpsession::test_master_ssrc_collision unit test
Comment 14 Julien Isorce 2013-11-01 17:39:49 UTC
Created attachment 258772 [details] [review]
tests/check: add rtpsession::test_rtx_ssrc_collision unit test
Comment 15 George Kiagiadakis 2013-11-06 17:24:24 UTC
Created attachment 259118 [details] [review]
rtpbin: add support for "receiver" auxiliary stream in rtpbin
Comment 16 George Kiagiadakis 2013-11-06 17:24:57 UTC
Created attachment 259119 [details] [review]
rtpbin: add support for "sender" auxiliary stream in rtpbin
Comment 17 George Kiagiadakis 2013-11-06 17:25:54 UTC
Created attachment 259120 [details] [review]
tests/check: add rtpaux::test_simple_rtpbin_aux
Comment 18 George Kiagiadakis 2013-11-06 17:26:25 UTC
Created attachment 259122 [details] [review]
rtpbin: generate "set-aux-send" and "set-aux-receive" action signals in gtk doc
Comment 19 George Kiagiadakis 2013-11-06 17:26:53 UTC
Created attachment 259123 [details] [review]
doc: add design-rtpauxiliary.txt to describe how rtpbin deals with auxiliary elements
Comment 20 Julien Isorce 2013-11-08 11:02:21 UTC
Created attachment 259246 [details] [review]
tests/check: add rtpaux::test_simple_rtpbin_aux
Comment 21 Sebastian Dröge (slomo) 2013-11-11 15:35:12 UTC
Comment on attachment 259122 [details] [review]
rtpbin: generate "set-aux-send" and "set-aux-receive" action signals in gtk doc

This is automatically generated by make update in the docs, and is not necessary to change manually. Will be done together with releases usually.
Comment 22 George Kiagiadakis 2013-11-18 16:36:05 UTC
Created attachment 260141 [details] [review]
rtpbin: add support for "sender" auxiliary stream in rtpbin

Removed useless changes that caused ABI break.
Comment 23 George Kiagiadakis 2013-11-20 12:53:32 UTC
Created attachment 260307 [details] [review]
rtpbin: add support for "receiver" auxiliary stream in rtpbin
Comment 24 George Kiagiadakis 2013-11-20 12:54:26 UTC
Created attachment 260308 [details] [review]
tests/check: add rtpaux::test_simple_rtpbin_aux
Comment 25 George Kiagiadakis 2013-11-25 16:07:16 UTC
Created attachment 261459 [details] [review]
examples: rtp: Add end-to-end rtpbin example with RTX elements
Comment 26 Olivier Crête 2013-12-11 22:20:31 UTC
Review of attachment 260141 [details] [review]:

I think the API here is wrong (also the same for the receiver), it needs to be possible to add more than one aux session per real session (for example to do both RTX and FEC), in the multicast case, there may be more than one level/type of FEC (so different aux session with different FEC settings and the receiver picks the right multicast group based on its needs).

::: gst/rtpmanager/gstrtpbin.c
@@ +3090,3 @@
+    GstPad *srcauxpad = NULL;
+
+    /* if user has set a parent and if it's a bin then remove it */

Why do you do that ?

@@ +3098,3 @@
+
+    /* add auxiliary element to rtpbin */
+    gst_bin_add (GST_BIN_CAST (rtpbin), aux_element);

Need to check the return value

@@ +3111,3 @@
+        "linking auxiliary sender RTP src pad to session");
+    gst_pad_link_full (srcauxpad, session->send_rtp_sink,
+        GST_PAD_LINK_CHECK_NOTHING);

Please check the return value here too
Comment 27 Olivier Crête 2013-12-11 23:53:06 UTC
I had a chat with Wim on how this API should work.

1. - aux sender elements would have a static "sink" pad and n "src_%u" request pads
   - aux receiver elements would have a static "src" pad and n "sink_%u" request pads
2. The application would be responsible for requesting pads on the aux elements elements before giving them to rtpbin
3. the %u in the request pad must correspond to the rtpbin session numbers of the rtpsession that will be linked to them.

Multiple aux elements are handled by having a bin containing multiple aux elements.


For the sender:

1. The application creates a sender aux element and configures it by requesting the "src_%u" pads and setting properties
2. the application requests a send_rtp_sink_%u pad
3. rtpbin emits a "request-aux-sender" signal, the paremeter of the signal would be the session number, the return value is the aux elements from step 1, one of the pads of this aux pad must have a number corresponding to the session number requested in step 2
4. rtpbin checks that no session currently exist with numbers matching the pads from the aux element (so if it has pads src_1, src_2, src_3, it checks that sessions 1, 2 and 3 don't exist). rtpbin also checks that the aux element has no parent, etc. For any failure, it return NULL as the request pad.
5. rtpbin connects the aux element between the requested send_rtcp_src_%u and the rtpsession, emits new sometimes pad for this new session.
6. rtpbin creates a new rtpsession for each extra pad on the aux element and emits sometimes pads with matching numbers
7. rtpbin returns the pad requested in step 2

If the application tries to request the send_rtcp_sink_%u for any aux session, NULL is returned.

For the receiver:

1. The application creates a sender aux element and configures it by requesting the "sink_%u" pads and setting properties
2. the application requests a recv_rtp_sink_%u pad
3. rtpbin emits a "request-aux-receiver" signal, the paremeter of the signal would be the session number, the return value is the aux elements from step 1, one of the pads of this aux pad must have a number corresponding to the session number requested in step 2
4. rtpbin checks that no session currently exist with numbers matching the pads from the aux element (so if it has pads sink_1, sink_2, sink_3, it checks that sessions 1, 2 and 3 don't exist). rtpbin also checks that the aux element has no parent, etc. For any failure, it return NULL as the request pad.
5. rtpbin connects the aux element between rtpsession and ssrcdemux
6. rtpbin returns the pad requested in step 2
7. the application requested another recv_rtp_sink_%u pad for the aux session configured on the aux element in step 1
8. rtpbin emits a "request-aux-receiver" signal, the paremeter of the signal would be the session number, the return value is the aux elements from step 1
9. if the element is not the same, request_pad returns NULL
10. if the element is the same, check that it has a sink_%u pad matching the one requested in step 7, otherwise also return NULL
11. create a new rtpsession, connect it to the aux element
12. return the request pad
13. for any other pad, goto 7

This API has the advantage that it's thread safe (two thread can request different sessions and just try again if it returns NULL, there can be no clash), etc.
Comment 28 Wim Taymans 2013-12-31 14:13:07 UTC
Implemented like that now:

commit c83ed4f61e882b0170d4c5a4b38d1883aebc6551
Author: Wim Taymans <wtaymans@redhat.com>
Date:   Tue Dec 31 14:57:27 2013 +0100

    tests: add AUX receiver unit test

commit b91e0096b7a4adfa162172542d3b3ef3d11a08d0
Author: Wim Taymans <wtaymans@redhat.com>
Date:   Tue Dec 31 13:20:01 2013 +0100

    tests: improve rtpbin test

commit bb2d37b11d68b788d1ddaeecdc90ae4985cab764
Author: Wim Taymans <wtaymans@redhat.com>
Date:   Tue Dec 31 13:16:46 2013 +0100

    rtpbin: add some docs about AUX elements

commit 3e83e6a33d0430bdfdf4cc754e44bace80cb56cc
Author: Wim Taymans <wtaymans@redhat.com>
Date:   Tue Dec 31 13:01:22 2013 +0100

    tests: add AUX sender unit test

commit d08e05b4ef8dae23fd8b5f551c6514db5fe222cb
Author: Wim Taymans <wtaymans@redhat.com>
Date:   Tue Dec 31 12:31:25 2013 +0100

    rtpbin: add support for AUX sender and receiver
    
    AUX elements are elements that can be inserted into the rtpbin
    pipeline right before or after 1 or more session elements.
    
    The AUX elements are essential for implementing functionality such
    as error correction (FEC) and retransmission (RTX).
    
    Fixes https://bugzilla.gnome.org/show_bug.cgi?id=711087