it's also leak without queue:

gst-launch videotestsrc pattern=18 !
video/x-raw-yuv,width=720,height=576,framerate=25/1 ! x264enc pass=0 !
h264parse access-unit=true ! matroskamux ! filesink
location=channel0.mkv

I didn't have enough cpu/time to track it more, but that pipeline without encoding/parsing grows by 1MB for 30 000 frames (the equivalent of 20mins of *realtime* 25fps encoding).

That growth is due to the index in matroskamux growing.

Isn't that the 'leak' you're seeing ?

If you set videotestsrc is-live=True , do you still get the same leak ?

> it's also leak without queue:
> 
> gst-launch videotestsrc pattern=18 !
> video/x-raw-yuv,width=720,height=576,framerate=25/1 ! x264enc pass=0 !
> h264parse access-unit=true ! matroskamux ! filesink
> location=channel0.mkv

It would be helpful if you could try and come up with the *minimal* pipeline that still leaks, to narrow down the issue. In this case, in order:

 - does it happen without the pattern=18 parameter to videotestsrc?

 - does it happen with .. ! fakesink instead of ... ! filesink?

 - does it happen with .. ! h264parse access-unit=true ! fakesink ?

 - does it happen with .. ! x264enc pass=0 ! fakesink ?

 - does it happen without the pass=0 parameter?

that's what boci doing in the last 2 days (starting from a much more complicated pipeline with our own element). we're still investigating and boci will report his result here

gst-launch videotestsrc pattern=18 ! video/x-raw-yuv,width=720,height=576,framerate=25/1 ! x264enc pass=0 ! h264parse access-unit=true ! matroskamux ! filesink location=channel0.mkv   6MB/20min

gst-launch videotestsrc ! video/x-raw-yuv,width=720,height=576,framerate=25/1 ! x264enc pass=0 ! h264parse access-unit=true !  matroskamux !  filesink location=channel0.mkv 6MB/20min

gst-launch videotestsrc ! video/x-raw-yuv,width=720,height=576,framerate=25/1 ! x264enc pass=0 ! h264parse access-unit=true !  matroskamux !  fakesink 6MB/20min

gst-launch videotestsrc ! video/x-raw-yuv,width=720,height=576,framerate=25/1 ! x264enc pass=0 ! h264parse access-unit=true ! fakesink  NO MEMORY EAT

The last 2 test will be soon ...

gst-launch videotestsrc ! video/x-raw-yuv,width=720,height=576,framerate=25/1 ! x264enc pass=0 ! fakesink NO MEMORY EAT

gst-launch videotestsrc ! video/x-raw-yuv,width=720,height=576,framerate=25/1 ! x264enc ! fakesink NO MEMORY EAT

hi

As i see the problem is in matroskamux filter. I tired the streamable property to set true, but the result was the same about 6MB/20 minute. If we can help in enything, please feel free to contact us.

thanks

is there any chance that a fix for this leak can be put into this release (10.33)?

> is there any chance that a fix for this leak can be put into this release
> (10.33)?

This release = 0.10.29 for -good (and 0.10.33 for core/base).

If there is in fact a leak (in GStreamer git/pre-releases), and you come up with a non-intrusive patch that's obviously correct, sure we can put it in.

Maybe you could provide a valgrind trace of the leak, or find out with massif where it's accumulating data. Edward's suggestion seems a likely candidate for investigation.

we also test with videotestsrc is-live=True and it's also leak in that case too:-(

> we also test with videotestsrc is-live=True and it's also leak in that case
> too:-(

Uh? What does? I thought you already narrowed it down to matroskamux?

i mean even if we run:
gst-launch videotestsrc pattern=18 is-live=true ! video/x-raw-yuv,width=720,height=576,framerate=25/1 ! x264enc pass=0 ! h264parse access-unit=true !  matroskamux !  filesink location=channel0.mkv

it's still leak the memory.

until you show us it's more than just the index creation taking up memory ... nothing is going to happen.

As for leaking in matroskamux in streaming-mode=True, it's not 'really' a leak. It's just that it's creating the index which isn't really needed. That should be trivial to fix (the index is only created/grown in one place).

ok here is a very basic setup. i hope it'll clear everything. 
the memory do NOT grow with this pipeline:

gst-launch videotestsrc ! video/x-raw-yuv,width=720,height=576,framerate=25/1 ! x264enc pass=0 ! matroskamux ! fakesink

while this pipeline the memory IS constantly grow:

gst-launch videotestsrc ! video/x-raw-yuv,width=720,height=576,framerate=25/1 ! x264enc pass=0 ! h264parse access-unit=true ! matroskamux ! fakesink

you can simple test by running both pipeline about 30 minutes. about after 5 minutes the memory usage keeps a certain level and if you watch longer in the first case it's not increase while in the second case it's continuously growing. 

both of the above pipeline contains only basic gstreamer's elements and both contains matroskamux. so if matroskamux index creation taking up memory, than why not doing the same in the first case?

Removing the muxer and looking at the output from gst-launch -v, it quickly shows that the parser is unsetting the buffer flags. I guess that makes the muxer believe that all frames are key frames (delta unit flag is dropped), which could explain why the index becomes an order of magnitude larger. I would assume the file is pretty much broken anyways. The parser should probably push correctly flagged buffers, but is the parser even needed in this case?

we've a much more complicated pipeline where the source is an rtsp h264 stream. in this case the parser needed. so this pipeline is just a demo pipeline, but imho should have to work properly. we just try to reduce the pipeline as simple as possible but the leak still happened.
so do you assume it's a bug in h264parse?
anyway in the first case the memory usage is NOT increase even if we run for 24 hours. so for us it seem this is not just a magnitude difference.

The delta-flag issue mentioned in Comment #14 does indeed occur when using legacyh264parse (which is the one having an access-unit property), but it does not with the new h264parse.  It would not occur with the older one if also setting split-packetized=true (and not doing so is messing up its parsing, or lack thereof for packetized input).

As for Comment #15, not seeing a memory build-up with first pipeline only shows that whatever method is used to measure is less precise than that matroskamux is efficient in storing index data, since even with the first pipeline it should most definitely be accumulating an index that should show up somewhere/somehow.
As mentioned, the delta-flag issue may affect the magnitude in such a way that it is brought in measurable range/granularity.

And fwiw, valgrind does not show anything for those pipelines.

Created attachment 187175 [details] [review]
matroskademux: avoid building index when streamable

Trivial patch to address issue mentioned in Comment #12.

Comment on attachment 187175 [details] [review]
matroskademux: avoid building index when streamable

Feel free to push that for the release, but it looks like there's a word missing in your commit message? (a reference to the bug would be nice as well)

Dare I say patch fixes this ... ?

commit fc94b78fe1421214ed087e1eae83f356021f9cc3
Author: Mark Nauwelaerts <mark.nauwelaerts@collabora.co.uk>
Date:   Wed May 4 12:04:15 2011 +0200

    matroskamux: avoid building index when streamable
    
    ... as it will not be written anyway.
    
    Fixes #648937 (?).

(In reply to comment #16)
> The delta-flag issue mentioned in Comment #14 does indeed occur when using
> legacyh264parse (which is the one having an access-unit property), but it does
> not with the new h264parse.  It would not occur with the older one if also
> setting split-packetized=true (and not doing so is messing up its parsing, or
> lack thereof for packetized input).

what do you mean by legacyh264parse? in 0.10.21's 
gst-inspect h264parse
show me access-unit!?

mark?

A more recent -bad is needed (such as git or latest release 0.10.22) for the legacyh264parse versus new h264parse as mentioned above.

Assuming this to be FIXED as per comment #19. Please reopen if this is not the case.