I can confirm that pitivi tends to have big/unrestrained memory usage. I never bothered to report it because of lack of time and because there were more pressing bugs for me at the time (I'm also guilty of having 4 Gb of ram, so I was less prone to be affected by this).

I spent some hours looking at massif outputs of memory consumption during rendering (start pitivi, load a project, render it).

The conclusion I come to is that the memory increases in steps, those steps being whenever a new source is activated in the composition. Therefore there's a good news and a bad news.

The good news... we're not leaking buffers (yay!) and it's not buffers taking up all this leaked memory (i.e. there's no 'stale' buffers being held up in memory after they've been encoded/transformed).

The bad news... The problem is the memory that the actual gstreamer elements consume once initialized (i.e. in PAUSED). An avi demuxer can consume several megabytes alone just to store the index :(

This is due to the way GnlComposition works. It sets the internal elements (sources/operations and their contents) state to PAUSED/PLAYING on a when-needed basis ... but then doesn't reset the state of those elements back to NULL/READY until the composition itself is set back to NULL/READY.
The reason for this is to provide fast/smooth switching internally. You can notice this when loading a project, then seeking to the second/third source in your timeline. You'll notice it takes slighty longer to display the content of that source the first time you go into it.

In theory, if you were to play back the whole timeline (not rendering it, but just playing it back), you should end up with the same final memory usage.

For info, I was trying with a sequence of 5 short (15s) mov/mjpeg/raw-audio clips without thumbnails/waveforms activated. Each new clip would consume an extra 20-30kbytes. Might not seem like much... but those clips are the most lightweight clips you can find (qtdemux is pretty optimized, the index is small, etc...).

So... several things to improve the memory consumption (by increasing order of priority and decreasing order of complexity):
* Figure out why the timeline mentionned above (60 10MPixel jpeg photos + 2 mp3) consumes 1.2GByte which is definitely excessive :) That one definitely smells like buffers are being leaked, maybe in the imagefreeze element (i.e. not releasing the buffer when it receives a flush event).
* Figure out if some gstreamer elements couldn't use some loving to reduce the memory consumption in PAUSED.
* Come up with a smarter way to handle the state of the sources/operations in a composition. That one's *really* tricky and smells like there's no magic way to do it.

For pictures, due to the way imagefreeze works, we always keep a copy of the decoded picture (yup, that's 30Mbytes per source in your case).

We could make imagefreeze release the buffer whenever possible (basically when it receives a flush)...
... but that will mean having to decode the image every time we seek which can be considerably slow. The advantage would be that the maximum extra memory consumption would be one full resolution raw frame (instead of one per source) in the worst case scenario (i.e. when actually viewing a picture source).

Still pondering what to do with this...

So what you want is the best of both worlds. 

When there is heaps of spare memory around then keeping the pictures it in memory is preferred so seeking can be fast.

However when memory starts to run out, keeping everything in memory is no longer a valid option...  I would rather it was slow then that it crashed.

I'm not familure with python but in C# it would be something like this
try
{
   //allocate memory
}
catch(OutOfMemoryException)
{
   //O crap that Daniel guy must be using his huge project again
   //lets free up some memory...
}

Also the Pictures are much larger then the resolution of the output. The whole high res picture does not need to be stored in memory just a much smaller version that matches the output resolution.... again I'm not an expert so could be wrong.

Finally I'm still confused about why we are running out of memory when System Monitor still shows a large amount of unused memory, and I can still run and use other programs(which don't run out of memory). Is this some kind of python memory allocation limitation or something?

(In reply to comment #4)
> So what you want is the best of both worlds. 
> 
> When there is heaps of spare memory around then keeping the pictures it in
> memory is preferred so seeking can be fast.

  There's one thing to take into account here to have fast seeking:
  * next chunk of data to play is available straight away (cache the image), and if that is not possible/desirable (because we don't want to always cache that image):
  * make the availabality of that data as fast as possible (optimize decoding as much as possible).

  Regarding the second point, I committed a minor change yesterday to pitivi that speeds up any push-based source decoding (by reading bigger chunks at once (512k buffers instead of the default 4k buffers)),  which includes jpeg pictures and to a lesser extent mpeg-ts files.
  The jpeg decoder element (gstjpegdec) could also do with some loving to speed things up, I committed such a fix yesterday to that element, but more could be done.

> 
> However when memory starts to run out, keeping everything in memory is no
> longer a valid option...  I would rather it was slow then that it crashed.
> 
> I'm not familure with python but in C# it would be something like this
> try
> {
>    //allocate memory
> }
> catch(OutOfMemoryException)
> {
>    //O crap that Daniel guy must be using his huge project again
>    //lets free up some memory...
> }

  Noble idea... if we were a monolithic application :) Unfortunately this memory allocation is buried deep in gstreamer elements.

  The 3rd proposal at the bottom of comment #2 is the only way we could avoid high memory consumption. One such idea would be to only activate sources/operations that are just before/after the currently playing segment, resulting in:
  * fast switching (the data is prerolled if we go forward/back)
  * lower memory usage (not all sources/operations are activated but only those in the current segment vicinity).

  It's when elements are activated (i.e. in PAUSED/PLAYING) that they allocate memory. When they're not (in NULL/READY), the memory usage is at it's lowest.

> 
> Also the Pictures are much larger then the resolution of the output. The whole
> high res picture does not need to be stored in memory just a much smaller
> version that matches the output resolution.... again I'm not an expert so could
> be wrong.

  Yes, scaling down the video at the source level is one possible option. Right now we could just do that because we don't allow any complex compositing.
  But the problem that might rise in the future is if we start doing *simple* effects like a Ken-Burns effect. We then wouldn't know how much of the input video/picture we want and would still require caching the full decoded image.

> 
> Finally I'm still confused about why we are running out of memory when System
> Monitor still shows a large amount of unused memory, and I can still run and
> use other programs(which don't run out of memory). Is this some kind of python
> memory allocation limitation or something?

  Python is definitely not the culprit here :) Massif shows no memory usage increase at the python level once the timeline is loaded.
  I'm not saying the memory usage at the python level (i.e. pure pitivi objects) can't be improved, but it's several orders of magnitude lower than the memory consumption of the actual multimedia processing (i.e. gstreamer elements).

One question I have is whether or not fast switching is relevant during rendering, since at that point we're writing to disk and the pipeline isn't even synced to a clock. Could there be a render-specific pipeline mode in which only the currently playing source is set to paused?

agreed... we could make it such that unused objects are set to READY/NULL when rendering... but the memory consumption will still be high when previewing/working.

Fast switching is awesome and I applaud you for making it as fast as possible. But  we are living in a resource constrained world,
everything has a cost, 
everything is a trade off.

From my perspective I am willing to sacrifice fast switching during previewing/working in order to gain the ability to render my project.

I can not talk for all users, and you must way up the needs of your user base as a whole. I'm guessing most users don't have this many large photos in their projects, However I'm also guessing most users don't have as much ram as I do either.

If you made it a configurable option, like a check box or a command line arguments which turned on or off fast switching. It would at least allow me to render my project.

Keep up the good work guys...

*** Bug 582072 has been marked as a duplicate of this bug. ***

This bug has been migrated to https://phabricator.freedesktop.org/T2335.

Please use the Phabricator interface to report further bugs by creating a task and associating it with Project: Pitivi.

See http://wiki.pitivi.org/wiki/Bug_reporting for details.