Can you get a debug log? Also what exactly is complaining about too many clients?

Are you reproducing the same problem with gst-launch-1.0? How are you switching exactly between cameras?

1. What is saying: "Maximum number of clients reached".  It sounds like the server is saying that.
2. Does the leak occur in gst-launch instead of your application?
3. Does removing elements by using fakesink's remove the leak.  Which elements?
4. What about replacing elements like vaapidecode with avdec_h264 or the gl part with xvimagesink?
5. Where does valgrind say the memory leak comes from?

Essentially where is the leak?

Here http://pastebin.com/m0bi4xZ0 is a link to debug log with level 4. There's one camera change.

I'm not sure what is complaining. I just get that message to my QtCreator application output window. Also when the app reaches this state using xwininfo refuses to show output. I reports the same error.

I haven't tried with gst-launch to reproduce the error, it takes about 100 camera source changes (I have 1 to 6 cameras so with 6 cameras I launch 6 pipelines) and therefore is quite a pain in the ass to start launching that many terminals. Also the cameras couldn't take that many connections.

I counted at some point that the error comes approximately by the time there would be 255 x server connections.

I change cameras by removing the widget from the gui that owns the stream objects. I believe they are parented in the correct way now (I had also problems with open gst connections, not anymore though). Then I create a new widget and add the stream object widgets to its gridLayout. There are situations where I might be having nested gridLayouts.

Did this answer to your questions? 

I noticed the debug level 4 is quite thorough. Would level 3 be better?

Thanks for awesome response time! :)

Okay,

so I started looking deeper into this and run my program to a point where it starts to complain about this. There's this line in the debug log:

0:34:08.555145926  3597      0x21c68c0 WARN             glimagesink gstglimagesink.c:833:_ensure_gl_setup:<sink> error: Failed to connect to X display server

So it would seem that glimagesink is having troubles. I can also put a part of the log in pastebin.

I'll try next replacing glimagesink with fakesink to see if it makes any difference. I'll also try using avdec_h264 instead of vaapidecode. Haven't tried valgrind yet.

http://pastebin.com/1CykidFq

There's the debug log from the point of error.

Okay,

so it might be that the problem is not in the gl elements after all. I've tried removing them and it has no effect. I'm seeing these debug messages atm and I feel that it might well be the source of the problems. The messages are:

0:03:36.209567054 18525 0x7f442c002140 FIXME                default gstutils.c:3764:gst_pad_create_stream_id_internal:<fakesrc111:src> Creating random stream-id, consider implementing a deterministic way of creating a stream-id

and

0:03:40.537169554 18525 0x7f43b4002190 WARN                 rtspsrc gstrtspsrc.c:5477:gst_rtspsrc_try_send:<rtspsrc> receive interrupted
0:03:40.537179266 18525 0x7f43b4002190 WARN                 rtspsrc gstrtspsrc.c:7500:gst_rtspsrc_pause:<rtspsrc> PAUSE interrupted
0:03:40.539637673 18525 0x7f43b4002190 WARN                 rtspsrc gstrtspsrc.c:5477:gst_rtspsrc_try_send:<rtspsrc> receive interrupted
0:03:40.539659195 18525 0x7f43b4002190 WARN                 rtspsrc gstrtspsrc.c:6971:gst_rtspsrc_close:<rtspsrc> TEARDOWN interrupted

and I get these when ever I change to another camera source.

Can you test without gstreamer-vaapi (and maybe also fakesink, but the gl elements should handle this just fine) and let us know if that helps anything?

I tested that just now and changing vaapidecode to avdec_h264 has no effect.

 <<< Closing player  "Axis_Q3709-PVE_Center_Stream"
0:00:47.725437202  6261 0x7efdf8002cf0 WARN                 rtspsrc gstrtspsrc.c:5477:gst_rtspsrc_try_send:<rtspsrc17> receive interrupted
0:00:47.725447039  6261 0x7efdf8002cf0 WARN                 rtspsrc gstrtspsrc.c:7500:gst_rtspsrc_pause:<rtspsrc17> PAUSE interrupted
0:00:47.726991959  6261 0x7efdf8002cf0 WARN                 rtspsrc gstrtspsrc.c:5477:gst_rtspsrc_try_send:<rtspsrc17> receive interrupted
0:00:47.727000353  6261 0x7efdf8002cf0 WARN                 rtspsrc gstrtspsrc.c:6971:gst_rtspsrc_close:<rtspsrc17> TEARDOWN interrupted
 <<< Closing player  "Axis_Q3709-PVE_Center_Stream"
0:00:47.729601257  6261 0x7efe30002680 WARN                 rtspsrc gstrtspsrc.c:5477:gst_rtspsrc_try_send:<rtspsrc18> receive interrupted
0:00:47.729619725  6261 0x7efe30002680 WARN                 rtspsrc gstrtspsrc.c:7500:gst_rtspsrc_pause:<rtspsrc18> PAUSE interrupted
0:00:47.734494498  6261 0x7efe30002680 WARN                 rtspsrc gstrtspsrc.c:5477:gst_rtspsrc_try_send:<rtspsrc18> receive interrupted
0:00:47.734509126  6261 0x7efe30002680 WARN                 rtspsrc gstrtspsrc.c:6971:gst_rtspsrc_close:<rtspsrc18> TEARDOWN interrupted
 <<< Closing player  "Axis_Q3709-PVE_Center_Stream"
0:00:47.738842101  6261 0x7efe500022d0 WARN                 rtspsrc gstrtspsrc.c:5477:gst_rtspsrc_try_send:<rtspsrc19> receive interrupted
0:00:47.738859005  6261 0x7efe500022d0 WARN                 rtspsrc gstrtspsrc.c:7500:gst_rtspsrc_pause:<rtspsrc19> PAUSE interrupted
0:00:47.740214863  6261 0x7efe500022d0 WARN                 rtspsrc gstrtspsrc.c:5477:gst_rtspsrc_try_send:<rtspsrc19> receive interrupted
0:00:47.740222776  6261 0x7efe500022d0 WARN                 rtspsrc gstrtspsrc.c:6971:gst_rtspsrc_close:<rtspsrc19> TEARDOWN interrupted
 <<< Closing player  "Axis_Q3709-PVE_Center_Stream"
0:00:47.743138268  6261 0x7efeac038370 WARN                 rtspsrc gstrtspsrc.c:5477:gst_rtspsrc_try_send:<rtspsrc20> receive interrupted
0:00:47.743149911  6261 0x7efeac038370 WARN                 rtspsrc gstrtspsrc.c:7500:gst_rtspsrc_pause:<rtspsrc20> PAUSE interrupted
0:00:47.744779745  6261 0x7efeac038370 WARN                 rtspsrc gstrtspsrc.c:5477:gst_rtspsrc_try_send:<rtspsrc20> receive interrupted
0:00:47.744791132  6261 0x7efeac038370 WARN                 rtspsrc gstrtspsrc.c:6971:gst_rtspsrc_close:<rtspsrc20> TEARDOWN interrupted


Thos are the last debug messages I get before it crashes. So I'm able to make 20 new pipelines before it goes down.

There are situations where I'm having two streams from the same camera playing at the same time.

Could it be that the rtspsrc elements don't destroy when I delete the pipeline?

Here's my player class destructor:

Player::~Player()
{
    if (_pipeline) {
        qDebug() << " <<< Closing player " << this->name;
        this->stop();
        _source->unlink(_depay);
        stopPipelineWatch();
        releaseVideoSink();
        _pipeline->setState(QGst::StateNull);
    }
}

It is inherited from QWidget and QGst::Ui::VideoWidget.

The source and rtph264depay are linked in the same function with all the other elements, but they are linked again in onPadAdded callback. Can that have something to do with this?

And btw, I'm not seeing that "maximum number of clients reached" anymore. I did something to the way I was removing objects and that error message changed to what I'm getting now.

This is how I remove the pipelines/videowidgets:

 // First close all existing streams
    if (_players->length() > 0)     //_players is a QVector
    {
        Player *p;
        for (int i = 0; i < _players->length(); ++i)
        {
//            _players->at(i)->stop();
            p = _players->at(i);
            delete p;
            p = NULL;
        }
        _players->clear();

        QLayoutItem *child;
        while ((child = _parentLayout->takeAt(0)) != 0)
        {
            QWidget *asd = child->widget();
            asd->deleteLater();

        }

    }

Can you provide a (compileable & runnable) testcase for this problem? If I understand this corrctly it should also be easily reproducible with a gst-rtsp-server based server, the server doesn't seem to matter. Right?

The server doesn't matter, as long as it is rtsp server. I'm using Axis cameras and based on the rtspsrc debug messages they might have a gst-rtsp-server in them. Not sure if that makes any difference. But if they had an old version of it with some bug (hypothetically) then perhaps that might explain it.

I'll try to make you a testcase. Is it okay if it depends on Qt5 since that's what I'm using atm?

Sure but just some simple C code that only uses GStreamer and is a commandline application would be better. Keep it as minimal and simple as possible to reduce all noise and additional cognitive overhead that would be needed to do something with it.

It might take a while for me to make that testcase, since I just realized that linking rtspsrc and rtph264depay is not that straight forward as linking other elements.

That got me thinking about my current QtGStreamer onPadAdded callback. It looks like the following:

void Player::onPadAdded(const QGst::PadPtr &pad)
{
    Q_UNUSED(pad);
    if (_depay) {
        _source->link(_depay, "sink");
    } else if (_demux) {
        _demux->link(_capsfilter, "sink");
    }
}

I've got a feeling that's not quite what it's supposed to be...

You will have to look at the pad you get there, and then decide based on the caps on the pad what to link it to. You could also get multiple pads btw, i.e. have your callback called multiple times. This is also something that you would ideally handle gracefully.

Okay, I could look into that pad-adding first to see if fixing that makes a difference. Is it possible that the way I'm linking them causes the problems?

Sure, you'll have to check that :)

Thanks!

A sidenote on the matter: I used to have the following environment set up:
GST_GL_WINDOW=x11
GST_GL_API=egl
GST_GL_PLATFORM=gles

That was due to some forum that suggested having that combination of environment variable. This caused my program to crush in random situations; sometimes my program ran fine for several minutes, but sometimes it crashed after a couple of tens of seconds. Also sometimes when changing to another camera source the program complained about EGL_BAD_ALLOC. 

Well, now I tried the following combination:
GST_GL_WINDOW=x11
GST_GL_API=glx
GST_GL_PLATFORM=opengl

with addition of XInitThreads(). This fixed my problem. More precisely some of it. My program doesn't crash anymore, but the memory still slowly accumulates and I keep getting these messages in the debug log:

0:12:21.238679233  2762 0x7f38642ffd90 WARN  rtspsrc gstrtspsrc.c:5477:gst_rtspsrc_try_send:<rtspsrc388> receive interrupted
0:12:21.238687020  2762 0x7f38642ffd90 WARN  rtspsrc gstrtspsrc.c:7500:gst_rtspsrc_pause:<rtspsrc388> PAUSE interrupted
0:12:21.245145826  2762 0x7f38642ffd90 WARN  rtspsrc gstrtspsrc.c:5477:gst_rtspsrc_try_send:<rtspsrc388> receive interrupted
0:12:21.245156453  2762 0x7f38642ffd90 WARN  rtspsrc gstrtspsrc.c:6971:gst_rtspsrc_close:<rtspsrc388> TEARDOWN interrupted

I believe the number after rtspsrc388 refers to the index of the connection. So there are a lot of connections, most probably because the elements are poorly linked, therefore not released.

That means that a new rtspsrc is created for each of your runs. They might be leaked in your code then, hard to say without more details or an actual testcase.

Hi,

I'm coming back to this since I just noticed that when I'm running my pipeline from terminal and I have GST_DEBUG=3 I get the same warnings about PAUSE and TEARDOWN interruption. I had only used the -v flag on the gst-launch and those messages didn't appear then. 

gst-launch-1.0 rtspsrc location=rtsp://root:root@10.128.1.82/axis-media/media.amp latency=0 ! rtph264depay ! h264parse ! queue ! vaapidecode ! videoconvert ! videocrop ! glimagesink
Setting pipeline to PAUSED ...
libva info: VA-API version 0.39.0
libva info: va_getDriverName() returns 0
libva info: Trying to open /usr/local/lib/dri/i965_drv_video.so
libva info: Found init function __vaDriverInit_0_39
libva info: va_openDriver() returns 0
Pipeline is live and does not need PREROLL ...
Got context from element 'sink': gst.gl.GLDisplay=context, gst.gl.GLDisplay=(GstGLDisplay)"\(GstGLDisplayX11\)\ gldisplayx11-0";
0:00:00.136132553  6985      0x11ce640 WARN               structure gststructure.c:1935:priv_gst_structure_append_to_gstring: No value transform to serialize field 'gst.vaapi.Display' of type 'GstVaapiDisplay'
Got context from element 'vaapidecode0': gst.vaapi.Display=context, gst.vaapi.Display=(GstVaapiDisplay)NULL;
Progress: (open) Opening Stream
Progress: (connect) Connecting to rtsp://root:root@10.128.1.82/axis-media/media.amp
Progress: (open) Retrieving server options
Progress: (open) Retrieving media info
Progress: (request) SETUP stream 0
Progress: (open) Opened Stream
Setting pipeline to PLAYING ...
New clock: GstSystemClock
Progress: (request) Sending PLAY request
Progress: (request) Sending PLAY request
0:00:00.527462710  6985 0x7f69d8021e80 FIXME                default gstutils.c:3764:gst_pad_create_stream_id_internal:<fakesrc0:src> Creating random stream-id, consider implementing a deterministic way of creating a stream-id
Progress: (request) Sent PLAY request
Redistribute latency...
Redistribute latency...
0:00:03.079206448  6985      0x119cb70 WARN                 default gstglutils.c:470:gst_gl_context_set_error: Output window was closed
0:00:03.102938067  6985      0x11c9990 WARN             glimagesink gstglimagesink.c:1552:gst_glimage_sink_show_frame:<sink> error: Output window was closed
ERROR: from element /GstPipeline:pipeline0/GstGLImageSinkBin:glimagesinkbin0/GstGLImageSink:sink: Output window was closed
Additional debug info:
gstglimagesink.c(1552): gst_glimage_sink_show_frame (): /GstPipeline:pipeline0/GstGLImageSinkBin:glimagesinkbin0/GstGLImageSink:sink
Execution ended after 0:00:02.575881824
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
Setting pipeline to NULL ...
0:00:03.103959581  6985      0x11c98f0 WARN                 rtspsrc gstrtspsrc.c:5477:gst_rtspsrc_try_send:<rtspsrc0> receive interrupted
0:00:03.103982304  6985      0x11c98f0 WARN                 rtspsrc gstrtspsrc.c:7500:gst_rtspsrc_pause:<rtspsrc0> PAUSE interrupted
0:00:03.108056070  6985      0x11c98f0 WARN                 rtspsrc gstrtspsrc.c:5477:gst_rtspsrc_try_send:<rtspsrc0> receive interrupted
0:00:03.108092328  6985      0x11c98f0 WARN                 rtspsrc gstrtspsrc.c:6971:gst_rtspsrc_close:<rtspsrc0> TEARDOWN interrupted
Freeing pipeline ...


Above log comes from launcing the pipeline and closing it after streaming begins. Can this be a issue in the RTSP server or is it still my element linking that causes this? I'm yet to fix the onPadAdded() callback...

-Joona

I've now got the C version of the pipeline construction, but I'm up against an issue with how am I going to set the QVideoWidget to display the frames in glimagesink, since I cannot use QGst::Ui::VideoWidget::setVideoSink(const ElementPtr &sink) when my sink is now a GstElement *?


Thanks!

-Joona

Just don't have any GUI in your testcase and let glimagesink create its own window as needed

Okay,

I managed to make a testcase in C and it shows a videostream from rtsp server. The rtsp server is on an Axis IP camera, the GStreamer version of the server is 1.4.2, plugins are 1.4.5. I'm using version 1.8.1 on my pipelines. Here's the code:

#include <gst/gst.h>

static void onPadAdded(GstElement *element, GstPad *pad, gpointer data)
{
    gchar *name;

    name = gst_pad_get_name(pad);
    g_print("A new pad %s was created\n", name);

    // here, you would setup a new pad link for the newly created pad
    // sooo, now find that rtph264depay is needed and link them?
    GstCaps * p_caps = gst_pad_get_pad_template_caps (pad);

    gchar * description = gst_caps_to_string(p_caps);
//    std::cout << p_caps << ", " << description << std::endl;
    g_free(description);

    GstElement *depay = GST_ELEMENT(data);

    // try to link the pads then ...
    if(gst_element_link_pads(element, name, depay, "sink") == 0)
    {
        g_print("cb_new_rtspsrc_pad : failed to link elements \n");
    }

    g_free(name);
}


static gboolean
bus_call (GstBus     *bus,
	  GstMessage *msg,
	  gpointer    data)
{
  GMainLoop *loop = data;

  switch (GST_MESSAGE_TYPE (msg)) {
    case GST_MESSAGE_EOS:
      g_print ("End-of-stream\n");
      g_main_loop_quit (loop);
      break;
    case GST_MESSAGE_ERROR: {
      gchar *debug = NULL;
      GError *err = NULL;

      gst_message_parse_error (msg, &err, &debug);

      g_print ("Error: %s\n", err->message);
      g_error_free (err);

      if (debug) {
        g_print ("Debug details: %s\n", debug);
        g_free (debug);
      }

      g_main_loop_quit (loop);
      break;
    }
    default:
      break;
  }

  return TRUE;
}

gint
main (gint   argc, gchar *argv[])
{
  GstStateChangeReturn ret;
  GstElement *pipeline, *rtspsrc, *depayer, *parser, *queue, *decoder, *filter, *sink;
  GMainLoop *loop;
  GstBus *bus;
  guint watch_id;

  char *uri1 = "rtsp://root:root@10.128.1.88/axis-media/media.amp";
//  char *uri2 = "rtsp://root:root@10.128.1.82/axis-media/media.amp";

  /* initialization */
  gst_init (&argc, &argv);
  loop = g_main_loop_new (NULL, FALSE);

  /* create elements */
  pipeline = gst_pipeline_new ("my_pipeline");

  /* watch for messages on the pipeline's bus (note that this will only
   * work like this when a GLib main loop is running) */
  bus = gst_pipeline_get_bus (GST_PIPELINE (pipeline));
  watch_id = gst_bus_add_watch (bus, bus_call, loop);
  gst_object_unref (bus);

  rtspsrc = gst_element_factory_make ("rtspsrc", "my_source");
  depayer = gst_element_factory_make ("rtph264depay", "my_depayer");
  parser = gst_element_factory_make ("h264parse", "my_parser");
  queue =  gst_element_factory_make ("queue", "my_queue");
  decoder = gst_element_factory_make ("vaapidecode", "my_decoder");
  filter = gst_element_factory_make ("videoconvert", "my_filter");
  sink = gst_element_factory_make ("glimagesink", "my_sink");


  if (!sink || !decoder) {
    g_print ("Decoder or output could not be found - check your install\n");
    return -1;
  } else if (!depayer || !parser || !queue) {
    g_print ("Could not create audioconvert or audioresample element, "
             "check your installation\n");
    return -1;
  } else if (!filter) {
    g_print ("Your self-written filter could not be found. Make sure it "
             "is installed correctly in $(libdir)/gstreamer-1.0/ or "
             "~/.gstreamer-1.0/plugins/ and that gst-inspect-1.0 lists it. "
             "If it doesn't, check with 'GST_DEBUG=*:2 gst-inspect-1.0' for "
             "the reason why it is not being loaded.");
    return -1;
  }

  g_object_set (G_OBJECT (rtspsrc), "location", uri1, NULL);
  g_object_set (G_OBJECT (rtspsrc), "latency", 0, NULL);

  g_signal_connect(G_OBJECT(rtspsrc), "pad-added", G_CALLBACK(onPadAdded), depayer);
  
  gst_bin_add_many (GST_BIN (pipeline), rtspsrc, depayer, parser, queue, decoder, filter, sink, NULL);

  /* link everything together */
  if (!gst_element_link_many (depayer, parser, queue, decoder, filter, sink, NULL)) {
    g_print ("Failed to link one or more elements!\n");
    return -1;
  }

  /* run */
  ret = gst_element_set_state (pipeline, GST_STATE_PLAYING);
  if (ret == GST_STATE_CHANGE_FAILURE) {
    GstMessage *msg;

    g_print ("Failed to start up pipeline!\n");

    /* check if there is an error message with details on the bus */
    msg = gst_bus_poll (bus, GST_MESSAGE_ERROR, 0);
    if (msg) {
      GError *err = NULL;

      gst_message_parse_error (msg, &err, NULL);
      g_print ("ERROR: %s\n", err->message);
      g_error_free (err);
      gst_message_unref (msg);
    }
    return -1;
  }

  g_main_loop_run (loop);

  /* clean up */
  gst_element_set_state (pipeline, GST_STATE_NULL);
  gst_object_unref (pipeline);
  g_source_remove (watch_id);
  g_main_loop_unref (loop);

  return 0;
}

When I close the video I still get the same error messages (GST_DEBUG=3) about GStreamer not being able to pause and teardown the stream. 

What am I doing wrong?


Cheers,

Joona

Sounds like it might be the same as bug #757624.

I don't think we will be able to help without you digging more into details to narrow down the actual problem, sorry.

For memory leak analysis perhaps try valgrind's --leak-check=yes or --tool=massif.

*** This bug has been marked as a duplicate of bug 757624 ***