In v4l2src, we first try to trust the timestamp provided by the driver, which should have been deduced from the firmware timestamp. Those timestamp are extremely bogus and the UVC driver isn't so good at fixing them. The C920 is also one with a very buggy firmware (timestamp go backward sometimes, even on raw data). From my point of view, this is to be fixed at both firmware and kernel module. There remains valid things that could somehow be done to improve that.

In GST, assuming you are willing to provide patches:
- Better bad kernel timestamp detection
- Better timestamp interpolation

In Linux-media (out of scope for GST)
- Distinction between DTS and PTS

Creating timestamp from expected framerate is not acceptable, this would drift too much. A v4l2src is a life source, it is completely acceptable if the frame are not equally spaced. In fact, UVC camera behave like this. Trying to force fixed duration will break other use cases (some camera can send frames when something happen, movement detection, face detection, weapon detection etc).

If you have a plan to improve this clearly in GST, let me know, otherwise I will close as not our bug. Especially that h264 over UVC is poorly implemented buy most firmware and the Linux kernel.

I promised myself to abtain from coding GStreamer, don't know the internal well enough, and better spend my time for the project finding and document bugs. That said, let me think about it. I might look into it and see what can be done.

Can you point me to some file/functions where you do the bad time stamp detection and time stamp interpolation in v4l2src?

In my code, I read from a frame fifo at a constant frame rate and it works pretty well even out time stamps from all these buggy webcams. The encoded frames may have variable timed timestamps, but the frames them selves are captured at an steady rate (and subsequently should be palyed at a steady rate).

Anyway in uvch264src, you have the parameter 'fixed-framerate'. I would have expected that, when enabled, it would try to generate equal spaced time stamps.

True that we should not break things. However, I also believe we should in general try to correct the timestamps, when possible. It could be a parameter you can enable so other things that don't need/want it can ignore it.

True the firmware should be fixed in all these cameras, but that is not going to happen. True it should be fixed in the driver and or in V4L2 community. Don't really know anyone there so that may not happen either.

True I can compensate for it in my software, although it would be really nice to have it done earlier.

One thought. h264parse does, as I understand try to correct some TS errors. It could happen there as well. Something like 'videorate', but for a H264 stream although adding frames is most likely not possible and throwing away frames can cause problems. Admittedly I haven't thought it through. But best would be to be able to correct timestamps in v4l2src or alterntively in the uvch264src using v4l2src.

Regards
Peter

I just notice that your thread is about 1.4.5 on Beagle Bone. Have you tested recent kernel and/or recent GStreamer ? I have taken out my C920, and I cannot reproduce your result (GStreamer master and kernel 4.4). As expected, on older kernel I expect that the bad-timestamp code triggers, and timestamp endup the time when v4l2src manage to dequeue the frame. This might be a little jitterry, but it's the best interpolation you can do. Adding a queue after v4l2src usually remove the jitter. On newer kernel, the HW timestamp is ignored, hence the timestamp is the time capture right after the IRQ is triggered (which is accurate, minus the latency of course). This is live sources, so the moment the frame is capture, is the timestamp. If your camera is jittery by design, you should have jittery timestamp.

(In reply to Peter Maersk-Moller from comment #2)
> Can you point me to some file/functions where you do the bad time stamp
> detection and time stamp interpolation in v4l2src?

https://cgit.freedesktop.org/gstreamer/gst-plugins-good/tree/sys/v4l2/gstv4l2src.c#n721

We don't do interpolation from existing timestamp. My opinion is that the kernel should do that, because it gets much more accurate information. Though, that code is broken in UVC driver and was disabled temporarily in 4.2 until someone get the time to look at it. Obviously, if it turns out your firmware is broken, complain to Logitech.

> 
> In my code, I read from a frame fifo at a constant frame rate and it works
> pretty well even out time stamps from all these buggy webcams. The encoded
> frames may have variable timed timestamps, but the frames them selves are
> captured at an steady rate (and subsequently should be palyed at a steady
> rate).

Not sure what your code is doing (share code if you want to keep that bug open without providing solutions). I do have deployed code using the current v4l2src (and also uvch264src) and it worked fine.

> 
> Anyway in uvch264src, you have the parameter 'fixed-framerate'. I would have
> expected that, when enabled, it would try to generate equal spaced time
> stamps.

This property will send UVC_H264_RATECONTROL_FIXED_FRM_FLG to the camera. If your camera ignores it, complain to your vendor. I do remember Youness (this element author) reporting that it only worked if you use H264 muxed into JPEG (anything with a preview feed).

> 
> True that we should not break things. However, I also believe we should in
> general try to correct the timestamps, when possible. It could be a
> parameter you can enable so other things that don't need/want it can ignore
> it.

The only completely incorrect timestamp are those that go backward. GStreamer/V4L2 does not let timestamp go backward for capture since 1.6, so if this is the problem you see, update GStreamer, or backport the fixes.

> 
> True the firmware should be fixed in all these cameras, but that is not
> going to happen. True it should be fixed in the driver and or in V4L2
> community. Don't really know anyone there so that may not happen either.
> 
> True I can compensate for it in my software, although it would be really
> nice to have it done earlier.
> 
> One thought. h264parse does, as I understand try to correct some TS errors.
> It could happen there as well. Something like 'videorate', but for a H264
> stream although adding frames is most likely not possible and throwing away
> frames can cause problems. Admittedly I haven't thought it through. But best
> would be to be able to correct timestamps in v4l2src or alterntively in the
> uvch264src using v4l2src.

You need at least one valid timestamp (DTS or PTS) to fix the timestamp in h264parse. At the moment, you seem to state that we have none, so this won't work. Again, smoothing timestamp can be wrong, as it may not be representative to the capture time. There is load of people that do cares about capture time accuracy.

Hi Nicolas.

> (In reply to Nicolas Dufresne (stormer) from comment #3)
> I just notice that your thread is about 1.4.5 on Beagle Bone. Have you
> tested recent kernel and/or recent GStreamer ?

I thought I had tested the timestamps on my laptop (x86) with GStreamer 1.8.1. But apparently not ... or the C920 I have here has a better firmware. I will within some days test on various platforms and with a selection of cameras.
The Beaglebone has an older kernel and the GStreamer 1.4.5. I will try to update both, however this is a process that will take some time.

> I have taken out my C920, and
> I cannot reproduce your result (GStreamer master and kernel 4.4). As
> expected, on older kernel I expect that the bad-timestamp code triggers, and
> timestamp endup the time when v4l2src manage to dequeue the frame. This
> might be a little jitterry, but it's the best interpolation you can do.
> Adding a queue after v4l2src usually remove the jitter. On newer kernel, the
> HW timestamp is ignored, hence the timestamp is the time capture right after
> the IRQ is triggered (which is accurate, minus the latency of course). This
> is live sources, so the moment the frame is capture, is the timestamp. If
> your camera is jittery by design, you should have jittery timestamp.

The timestamps I have measured can be due to older kernel and/or older GStreamer. However this could, at least theoretically be due to effects more visible when running on a platform as slow (compared to a x86 laptop) as the BeagleBone. These effect, in theory could be caused in kernel as well in GStreamer (theoretically, with the little knowledge I yet have on the timestamping code of v4l2src). If the timestamps are due to effects caused by a slower platform, then the issue is worth looking into. If not, then the matter can be closed as no longer relevant.


> (In reply to Peter Maersk-Moller from comment #2)
> > Can you point me to some file/functions where you do the bad time stamp
> > detection and time stamp interpolation in v4l2src?
> https://cgit.freedesktop.org/gstreamer/gst-plugins-good/tree/sys/v4l2/
> gstv4l2src.c#n721

Will look into it. Thanks. Please have patience as it will take some time and I unfortunately can't spend work time on it, only my little spare time.

> We don't do interpolation from existing timestamp. My opinion is that the
> kernel should do that, because it gets much more accurate information.
> Though, that code is broken in UVC driver and was disabled temporarily in
> 4.2 until someone get the time to look at it. Obviously, if it turns out
> your firmware is broken, complain to Logitech.

Obviously, except the result will be obvious.

> > In my code, I read from a frame fifo at a constant frame rate and it works
> > pretty well even out time stamps from all these buggy webcams. The encoded
> > frames may have variable timed timestamps, but the frames them selves are
> > captured at an steady rate (and subsequently should be palyed at a steady
> > rate).
> 
> Not sure what your code is doing (share code if you want to keep that bug
> open without providing solutions). I do have deployed code using the current
> v4l2src (and also uvch264src) and it worked fine.

The the code is freely available in Snowmix feeds, but is as code not relevant for GStreamer, since it uses a different technique for mixing and syncing various sources. It is just a bunch fifos supplied by Gstreamer and read/used at a steady framerate by Snowmix. Nothing fancy or worth looking into, but using this, I can see movements between frames appear uniform smooth when I disregard the variations in timestamps and read at a steady framerate.

> > Anyway in uvch264src, you have the parameter 'fixed-framerate'. I would have
> > expected that, when enabled, it would try to generate equal spaced time
> > stamps.
> 
> This property will send UVC_H264_RATECONTROL_FIXED_FRM_FLG to the camera. If
> your camera ignores it, complain to your vendor. I do remember Youness (this
> element author) reporting that it only worked if you use H264 muxed into
> JPEG (anything with a preview feed).

Ok, thanks, Yes I can't see any difference whether it is enabled or not.

> > True that we should not break things. However, I also believe we should in
> > general try to correct the timestamps, when possible. It could be a
> > parameter you can enable so other things that don't need/want it can ignore
> > it.
> The only completely incorrect timestamp are those that go backward.
> GStreamer/V4L2 does not let timestamp go backward for capture since 1.6, so
> if this is the problem you see, update GStreamer, or backport the fixes.

Ok.

> > True the firmware should be fixed in all these cameras, but that is not
> > going to happen. True it should be fixed in the driver and or in V4L2
> > community. Don't really know anyone there so that may not happen either.
> > True I can compensate for it in my software, although it would be really
> > nice to have it done earlier.
> > One thought. h264parse does, as I understand try to correct some TS errors.
> > It could happen there as well. Something like 'videorate', but for a H264
> > stream although adding frames is most likely not possible and throwing away
> > frames can cause problems. Admittedly I haven't thought it through. But best
> > would be to be able to correct timestamps in v4l2src or alterntively in the
> > uvch264src using v4l2src.
> You need at least one valid timestamp (DTS or PTS) to fix the timestamp in
> h264parse. At the moment, you seem to state that we have none, so this won't
> work.

Ok, thanks. Yes the timestamps observed are incorrect, so I wouldn't have correct timestamps to work from. However what I suggest is a mechanism like the videorate element, but an element that would be able to adaptively generate approximated even spaced timestamps. Right now, if I wanted to even out the space in timestamps, I would have to decode the stream, send it through videorate and then encode the stream again. And that is obviously not attractive. However there is a chance that what I write here is wrong in case I have misunderstood what module videorate can do.

> Again, smoothing timestamp can be wrong, as it may not be
> representative to the capture time. There is load of people that do cares
> about capture time accuracy.

So do I. Definitely, or at least here I care about the relative values and they happen to be wrong - for all the reasons mentioned earlier (kernel/driver/firmware and possible GStreamer older version).
 
You can choose to close the ticket. It can always be reopnened if needed. I'll look into the issues mentioned and report back if I find anything useful.

That said, a way in GStreamer to even space between timestamps without decoding could be a useful features. So much software and hardware have a hard time doing timestamps right and being able to use GStreamer as a tool to fix some of it, could be worth it. But that is just me being oppinionated.

Thanks for help, explanation and tips.

Peter

I'll close it for now, let us know if you find something to address. I agree an element like this could be useful, but it's not related to V4L2 here.