Does the video at the first buffer represents the 0.3s or is it for the 0s but is being pushed late (latency)? In another words, are the timestamp on the video buffers correct?

The edit list in qtmux is used, among other things, to represent late streams. Can you share a problematic file that you muxed for further investigation?

(In reply to Thiago Sousa Santos from comment #1)
> Does the video at the first buffer represents the 0.3s or is it for the 0s
> but is being pushed late (latency)? In another words, are the timestamp on
> the video buffers correct?

The video corresponds to the 0 but is timestamped at 0.3 because of start latency.

> 
> The edit list in qtmux is used, among other things, to represent late
> streams. Can you share a problematic file that you muxed for further
> investigation?

Will try to do that tomorrow. Any preference on where to store it ?

(In reply to Jean-Michel Hautbois (jmleo) from comment #2)
> (In reply to Thiago Sousa Santos from comment #1)
> > Does the video at the first buffer represents the 0.3s or is it for the 0s
> > but is being pushed late (latency)? In another words, are the timestamp on
> > the video buffers correct?
> 
> The video corresponds to the 0 but is timestamped at 0.3 because of start
> latency.

I guess this is the problem, it should have been timestamped as 0, there is no way from the muxer point of view to differentiate that the 0.3 should be mapped to 0 or that it is indeed to 0.3.

> 
> > 
> > The edit list in qtmux is used, among other things, to represent late
> > streams. Can you share a problematic file that you muxed for further
> > investigation?
> 
> Will try to do that tomorrow. Any preference on where to store it ?

No, just store wherever you can provide a link to us.
A GST_DEBUG=6 log would also help understanding the problem.

Here is the mp4 file :
http://we.tl/Q44fB3qn41

How are you playing the file? gstreamer plays it just fine here

mplayer or vlc don't play it correctly. gstreamer can correct the issue indeed, if you take the file, qtdemux it, and remux it in mkv, it plays nicely. And if you do a qtdmux->mp4mux I am pretty sure (will verify) it corrects the issue too...

Looks like what the format support, and what the players can do differ. The gap in video is played correctly in git master. Ideally, we'd need an element that drops until it get data from all the sources. So everything is aligned in the recording. (can't be done by the muxer, you need to to that at raw level)

Created attachment 305560 [details] [review]
multisync: create new multisync plugin

Here is a proposal for such a plugin.

Waiting for your comments! I know it is not perfect at all ;)

Review of attachment 305560 [details] [review]:

This is just a drop toward what comes to my mind. The sycnhronization in that code is base on PTS instead of running-time PTS. This means that unless you have sources that starts at PTS 0, you won't have synchronized output. It is interesting that this implementation is non-blocking. While this creates simplicity, it reduce the ability to make the right decision upon which buffer should be kept (and clipped) or pushed downstream. A very complete solution in my opinion would actually queue up to two buffers so we can ensure to always keep the best buffer.

* Each pad would queue up to two buffers (will all in-between serialized events) and block.
* When a buffer is received, look for a start "running-time" on all pads.
* When that is found, for each pad, lookup for buffers that don't fit (the start running time isn't between buffer start and stop (start + duration) and drop.
* In case you have dropped something, unblock that pad, wait for the next buffer.
* In case you didn't drop anything, update segments event for each pads (they can be all different), drain all queued, unclock all pads (basically passthrough afterward).

::: gst/multisync/gstmultisync.c
@@ +43,3 @@
+
+/**
+ * SECTION:element-multi_sync

Sections need to be added docs/plugins/gst-plugins-bad-plugins-sections.txt

@@ +98,3 @@
+#define GST_MULTI_SYNC_MUTEX_UNLOCK(m) G_STMT_START {                        \
+  g_mutex_unlock (&m->lock);                                            \
+} G_STMT_END

I think you can use the GST_OBJECT_LOCK() instead of creating a new one.  For the future, add extra parenthesis when using macro member. The member could be an expression which may lead to an ambiguity.

@@ +108,3 @@
+    GST_PAD_SRC,
+    GST_PAD_SOMETIMES,
+    GST_STATIC_CAPS_ANY);

I don't thinks this is media agnostic as you don't deal with deltas, neither you add decode only flags. You should only mark formats that you are use won't break.

@@ +309,3 @@
+    gst_buffer_unref (sstream->last_buffer);
+  }
+  g_free(sstream);

I think you need to do something more. If all pads, except the this one, had data, the streaming will be stalled.

@@ +376,3 @@
+  GST_DEBUG_OBJECT (pad, "Current buffer TS is: %" GST_TIME_FORMAT, GST_TIME_ARGS (GST_BUFFER_PTS (buffer)));
+
+  if (GST_BUFFER_PTS (buffer) < msync->start_timestamp) {

This is just an example where this cannot work for let's say H264 with reordering. The delta chain will be broken + the PTS are not monotonic in that case.

One of the issue with this, is that you blindly drop base on the TS. You don't consider the duration of the buffer. This mostly reduce accuracy, but you don't seem to break a/v sync. This will have side effects if you have buffer with super long duration in one stream, or as I already seen, a single buffer in a video stream and then audio.

@@ +382,3 @@
+  }
+
+  GST_BUFFER_PTS (buffer) -= msync->start_timestamp;

Don't re-timestamp. Send a segment event instead (to bring back the running time of the PTS).

@@ +413,3 @@
+
+    GST_DEBUG_OBJECT (pad, "Trying to determine start TS");
+    msync->start_timestamp = gst_multi_sync_find_start_timestamp (msync);

This is fundamentally broken because timestamp alone don't allow synchronization. As what you want is to synchosize, you need to convert the TS base on the received segment to a common time base. In this case, you want to convert the PTS to running time.

Note, I found the the muxer was generating edit list using the wrong timescale. That may help here.

I have tested using upstream master, but it does not help for my case.
The problem still persists.

We found more problems yesterday. When the fix land, we think AV sync on at least VLC shall be right. For the record, the duration of the first buffer is as large as the gap, which mean the gap duration get added with the first duration causing video to be displayed way after the audio.

Would be nice to retest against master. This type of recording is fixed through bug 751361. This is known to fix lib sync issue when playing in VLC (and libsync issue when playing correctly encoder mp4 with empty elst in gst). The proposed element would still be requires for players that don't support gaps (like mplayer).

I tried the new master. It indeed fixes the issue when using VLC. Thanks!
For information, I have tried some other players:

vlc (linux & win) OK
mplayer (linux) NOK
chromium (linux) OK
chrome (win) OK
firefox (win) NOK
windows media player NOK

I think, this is not enough for us, so I will continue with the multisync element.

Then it's fair to continue this work. Shall we expect a updated patch soon ?

Not soon, since I'm leaving for holiday.
I might be working on it later in the summer.

Here stands our work-in-progress branch (not much work for now):
https://gitlab.com/veo-labs/gst-plugins-bad/commits/multisync

-- GitLab Migration Automatic Message --

This bug has been migrated to freedesktop.org's GitLab instance and has been closed from further activity.

You can subscribe and participate further through the new bug through this link to our GitLab instance: https://gitlab.freedesktop.org/gstreamer/gst-plugins-good/issues/191.