GNOME Bugzilla – Bug 574408
audiomixer: Need an element to do synchronized audio mixing
Last modified: 2014-12-26 18:34:24 UTC
Please describe the problem: Adder resets the timestamp of buffers to zero on state change from ready to pause. It disregards the timestamp of the incoming buffers. This causes synchronisation problems when downstream elements (eg. alsasink) sync on the clock. It results in having no audio. Steps to reproduce: 1. Place an adder in a sink bin:= queue2 ! adder ! alsasink. 2. Add this sink bin to a playing pipeline. 3. Link a playing source to this sink bin. Actual results: There is no audio written out to the soundcard, unless the "sync" property of alsasink is set to false. If false, audio is output with a substantial latency. Expected results: Adder should be able to produce a stream that is playable by syncing elements downstream. Audio is output when the adder is removed from the sink bin, the same behavior is expected when the adder element is a part of the bin. Does this happen every time? Yes. Other information: This is because the timestamp of the input buffers are not considered in computing the timestamp of the output buffer. Timestamp is set to zero everytime the element changes state from READY to PAUSED.
Having a test in adders unit-test suite would be great (gst-plugins-base/tests/check/elements/adder.c). Balachandran, could you try writing one and attach it here?
Now thinking more about it, I think the problem is the one in bug #474016. tee is not sending a newsegment to the new pad and thus adder starts from scratch.
Almost all elements reset their timestamps to zero when going to READY so this is not so much different from, say, audiotestsrc. If adder is supposed to preserve incomming timestamps it needs to be configured in sync mode (not implemented yet) where it would only mix incomming samples with the same timestamps to produce an outgoing stream with those timestamps. If streams that don't need sync need to be mixed with streams that need sync, then 2 adders should be used, one configured with sync and the other without. I think that would make most sense.
liveadder does that... but I guess the real fix should be in GstCollectPads2 or something about which there is an open bug somewhere
I started working on this and have a somewhat working element called audiomixer in a local branch. Hope I have some time to finish it :)
I would love to know the status of this ? I have built a pipeline with 6 samples added with addr, when i set the pipeline playing they all play perfectly in sync. If i stop 3 of them setting there state to ready then trigger the pipeline back to playing they all start but not in sync i can hear them starting slightly one after the other sounds like it the same issue that others have experienced here.
I'm on it, I hope to be able to finish it in the next 1-2 weeks and put it in gst-plugins-bad then. If there's something to test I'll write it on the bug here :)
sounds great keep us posted :)
Took longer than expected, but here it is. New "audiomixer" element in gst-plugins-bad: commit 69525a147771bcd95d1e24df3e501dad009bb126 Author: Sebastian Dröge <sebastian@centricular.com> Date: Wed Nov 6 15:31:31 2013 +0100 audiomixer: Add test to the .gitignore file commit e2e4e8e7414b0a92922fb2f6e08aecaa8b11ea95 Author: Sebastian Dröge <sebastian@centricular.com> Date: Wed Nov 6 15:18:58 2013 +0100 audiomixer: Add simply synchronization test commit caeaade7008b9716e4801afeb8a37d9ed240ccaf Author: Sebastian Dröge <sebastian@centricular.com> Date: Wed Nov 6 15:18:50 2013 +0100 audiomixer: Add new element based on adder that does synchronized audio mixing
Nice. Some comments about this: I had to remove the special channel-mask handling. Otherwise negotiation failed because alsa sets it for channels=2. Note: I had to backport your patches to 1.0.x so this might have changed. I mix a alsasrc with something unreliable (e.g. some kind of network source). I'd like to always get the audio from alsa, even when the other source stops providing new buffers. Right now audiomixer just stalls. What's the correct way to handle this?
(In reply to comment #11) > Nice. Some comments about this: > I had to remove the special channel-mask handling. Otherwise negotiation failed > because alsa sets it for channels=2. Note: I had to backport your patches to > 1.0.x so this might have changed. That should be fixed already, I fixed that soon after pushing the initial version > I mix a alsasrc with something unreliable (e.g. some kind of network source). > I'd like to always get the audio from alsa, even when the other source stops > providing new buffers. Right now audiomixer just stalls. What's the correct way > to handle this? Proper live support in collectpads and the mixers/muxers (i.e. not wait forever until buffers arrive but time out to properly continue mixing). I'm currently working on that whenever I have some time.