GNOME Bugzilla – Bug 711087
rtpbin: Support Auxiliary streams
Last modified: 2013-12-31 14:14:44 UTC
So that rtpbin can configure auxiliary streams for sender and receiver. Examples: rtprtxsend and rtprtxreceive are auxiliary elements. But they are SSRC-multiplexed. Auxiliary streams may also be sessions-multiplexed
Created attachment 258463 [details] [review] rtpbin: add support for "receiver" auxiliary stream in rtpbin
Created attachment 258464 [details] [review] rtpbin: add support for "sender" auxiliary stream in rtpbin
Created attachment 258465 [details] [review] tests/check: add rtprtx::test_simple_rtpbin_aux unit test
So my kind of API to setup auxiliary elements inside rtpbin works like that: * Instanciate any aux receivers or senders in your application. * build a structure with name "rtpauxreceive" for aux receivers or "rtpauxsend" for aux senders. * add "element" field of type GST_TYPE_ELEMENT to that structure to pass the gst element. * add "sessions" field of type GST_TYPE_LIST to that structure to pass the sessions ids that have to use this aux element. (ids are the field uses in "recv_rtp_sink_%u" for receivers, and "send_rtp_sink_%u" with "send_rtp_src_%u") for senders * use new rtpbin's "add-aux-element" property to pass this structure. * rtpbin then handle those aux elements in a generic way so that they can be session-multiplxed or SSRC-multiplexed See checks/test/elements/rtprtx.c::test_simple_rtpbin_aux for an example. Notes: currently not possible to use from gst-launch. We can discuss about having the ability for rtpbin to instanciate itself the special aux elements rtprtxsend and rtprtxreceive but they need to be configured ("payload-type" and "payload-types" properties) to make retransmission work. So having several rtprtxsend and rtprtxreceive in a rtpbin would require a complete api/properties. And for each aux elements.
As I mentioned in person, I'm not fan of GST_TYPE_LIST in APIs. I wonder if the API couldn'T be something like this, and allow it to be called multiple times and rtpbin would check if the element is already there and re-use it, etc ? g_signal_emit(rtpbin, "set-aux-element", stream_id, element);
I guess you mean a G_SIGNAL_ACTION ? Make sense, also because there are not a lot of stream_id per aux element: For SSRC-multiplexed auxiliary elements like rtprtxsend and rtprtxreceive, it's always one. For sessions-multiplexed auxiliary elements, in concrete cases it will be not much more than two I think.
Created attachment 258765 [details] [review] rtpbin: add support for "receiver" auxiliary stream in rtpbin
Created attachment 258766 [details] [review] rtpbin: add support for "sender" auxiliary stream in rtpbin
Created attachment 258767 [details] [review] tests/check: add rtpaux::test_simple_rtpbin_aux unit test
Created attachment 258768 [details] [review] rtpbin: generate "set-aux-send" and "set-aux-receive" action signals in gtk doc
Created attachment 258769 [details] [review] gstrtpsession: send custom upstream event "GstRTPCollision" on send_rtp_sink pad
Created attachment 258770 [details] [review] gstrtpbasepayload: regenerate ssrc when receiving GstRTPCollision event
Created attachment 258771 [details] [review] tests/check: add rtpsession::test_master_ssrc_collision unit test
Created attachment 258772 [details] [review] tests/check: add rtpsession::test_rtx_ssrc_collision unit test
Created attachment 259118 [details] [review] rtpbin: add support for "receiver" auxiliary stream in rtpbin
Created attachment 259119 [details] [review] rtpbin: add support for "sender" auxiliary stream in rtpbin
Created attachment 259120 [details] [review] tests/check: add rtpaux::test_simple_rtpbin_aux
Created attachment 259122 [details] [review] rtpbin: generate "set-aux-send" and "set-aux-receive" action signals in gtk doc
Created attachment 259123 [details] [review] doc: add design-rtpauxiliary.txt to describe how rtpbin deals with auxiliary elements
Created attachment 259246 [details] [review] tests/check: add rtpaux::test_simple_rtpbin_aux
Comment on attachment 259122 [details] [review] rtpbin: generate "set-aux-send" and "set-aux-receive" action signals in gtk doc This is automatically generated by make update in the docs, and is not necessary to change manually. Will be done together with releases usually.
Created attachment 260141 [details] [review] rtpbin: add support for "sender" auxiliary stream in rtpbin Removed useless changes that caused ABI break.
Created attachment 260307 [details] [review] rtpbin: add support for "receiver" auxiliary stream in rtpbin
Created attachment 260308 [details] [review] tests/check: add rtpaux::test_simple_rtpbin_aux
Created attachment 261459 [details] [review] examples: rtp: Add end-to-end rtpbin example with RTX elements
Review of attachment 260141 [details] [review]: I think the API here is wrong (also the same for the receiver), it needs to be possible to add more than one aux session per real session (for example to do both RTX and FEC), in the multicast case, there may be more than one level/type of FEC (so different aux session with different FEC settings and the receiver picks the right multicast group based on its needs). ::: gst/rtpmanager/gstrtpbin.c @@ +3090,3 @@ + GstPad *srcauxpad = NULL; + + /* if user has set a parent and if it's a bin then remove it */ Why do you do that ? @@ +3098,3 @@ + + /* add auxiliary element to rtpbin */ + gst_bin_add (GST_BIN_CAST (rtpbin), aux_element); Need to check the return value @@ +3111,3 @@ + "linking auxiliary sender RTP src pad to session"); + gst_pad_link_full (srcauxpad, session->send_rtp_sink, + GST_PAD_LINK_CHECK_NOTHING); Please check the return value here too
I had a chat with Wim on how this API should work. 1. - aux sender elements would have a static "sink" pad and n "src_%u" request pads - aux receiver elements would have a static "src" pad and n "sink_%u" request pads 2. The application would be responsible for requesting pads on the aux elements elements before giving them to rtpbin 3. the %u in the request pad must correspond to the rtpbin session numbers of the rtpsession that will be linked to them. Multiple aux elements are handled by having a bin containing multiple aux elements. For the sender: 1. The application creates a sender aux element and configures it by requesting the "src_%u" pads and setting properties 2. the application requests a send_rtp_sink_%u pad 3. rtpbin emits a "request-aux-sender" signal, the paremeter of the signal would be the session number, the return value is the aux elements from step 1, one of the pads of this aux pad must have a number corresponding to the session number requested in step 2 4. rtpbin checks that no session currently exist with numbers matching the pads from the aux element (so if it has pads src_1, src_2, src_3, it checks that sessions 1, 2 and 3 don't exist). rtpbin also checks that the aux element has no parent, etc. For any failure, it return NULL as the request pad. 5. rtpbin connects the aux element between the requested send_rtcp_src_%u and the rtpsession, emits new sometimes pad for this new session. 6. rtpbin creates a new rtpsession for each extra pad on the aux element and emits sometimes pads with matching numbers 7. rtpbin returns the pad requested in step 2 If the application tries to request the send_rtcp_sink_%u for any aux session, NULL is returned. For the receiver: 1. The application creates a sender aux element and configures it by requesting the "sink_%u" pads and setting properties 2. the application requests a recv_rtp_sink_%u pad 3. rtpbin emits a "request-aux-receiver" signal, the paremeter of the signal would be the session number, the return value is the aux elements from step 1, one of the pads of this aux pad must have a number corresponding to the session number requested in step 2 4. rtpbin checks that no session currently exist with numbers matching the pads from the aux element (so if it has pads sink_1, sink_2, sink_3, it checks that sessions 1, 2 and 3 don't exist). rtpbin also checks that the aux element has no parent, etc. For any failure, it return NULL as the request pad. 5. rtpbin connects the aux element between rtpsession and ssrcdemux 6. rtpbin returns the pad requested in step 2 7. the application requested another recv_rtp_sink_%u pad for the aux session configured on the aux element in step 1 8. rtpbin emits a "request-aux-receiver" signal, the paremeter of the signal would be the session number, the return value is the aux elements from step 1 9. if the element is not the same, request_pad returns NULL 10. if the element is the same, check that it has a sink_%u pad matching the one requested in step 7, otherwise also return NULL 11. create a new rtpsession, connect it to the aux element 12. return the request pad 13. for any other pad, goto 7 This API has the advantage that it's thread safe (two thread can request different sessions and just try again if it returns NULL, there can be no clash), etc.
Implemented like that now: commit c83ed4f61e882b0170d4c5a4b38d1883aebc6551 Author: Wim Taymans <wtaymans@redhat.com> Date: Tue Dec 31 14:57:27 2013 +0100 tests: add AUX receiver unit test commit b91e0096b7a4adfa162172542d3b3ef3d11a08d0 Author: Wim Taymans <wtaymans@redhat.com> Date: Tue Dec 31 13:20:01 2013 +0100 tests: improve rtpbin test commit bb2d37b11d68b788d1ddaeecdc90ae4985cab764 Author: Wim Taymans <wtaymans@redhat.com> Date: Tue Dec 31 13:16:46 2013 +0100 rtpbin: add some docs about AUX elements commit 3e83e6a33d0430bdfdf4cc754e44bace80cb56cc Author: Wim Taymans <wtaymans@redhat.com> Date: Tue Dec 31 13:01:22 2013 +0100 tests: add AUX sender unit test commit d08e05b4ef8dae23fd8b5f551c6514db5fe222cb Author: Wim Taymans <wtaymans@redhat.com> Date: Tue Dec 31 12:31:25 2013 +0100 rtpbin: add support for AUX sender and receiver AUX elements are elements that can be inserted into the rtpbin pipeline right before or after 1 or more session elements. The AUX elements are essential for implementing functionality such as error correction (FEC) and retransmission (RTX). Fixes https://bugzilla.gnome.org/show_bug.cgi?id=711087