I have an experimental plugin.

Playback of sound files works.
Connecting to 500@ekiga.net crashes Ekiga in some GStreamer function.

Just put the 3 files in (please adapt to better paths) :
* /home/damien/CVS/Phobos/pwlib/plugins/sound_gstreamer/
* export PWLIBDIR=/home/damien/CVS/Phobos/pwlib/
* make opt
* sudo cp ../pwlib/device/sound/gstreamer_pwplugin.so /usr/lib/pwlib/device/sound/

(device or devices).

Created attachment 80816 [details]
CXX file

Created attachment 80817 [details]
header file

Created attachment 80818 [details]
Makefile

Some remarks :
- neat !
- it doesn't detect the various sink/src devices : see how I do for the video experimental code ;
- does it have CPU issues like my video code?

It doesn't have specific CPU issues.

However, in your code, I think the problem is that GetFrameDataNoDelay is called too often while the handoff callback has not fed data yet.

As far as I know, GetFrameDataNoDelay is only GetFrameData calls GetFrameDataNoDelay, and it does it with a proper delay.

I have had an idea this week-end, but have since lacked the time to test it.

I found time to make some tests, and I can confirm that GetFrameData and GetFrameDataNoDelay are called as much as needed, and not more.


OnHandoff is called way too often, but that isn't my function causing the harm : if I don't really start the pipeline, the problem disappears, and if I launch the pipeline but don't ask to signal handoffs, then the problem is there.

I really have the impression that for some reason my pipeline works way too fast...

Eh, if I comment about video input issues on the bug about the audio issues, no wonder I had no answer and couldn't find my questions... *sigh*

I'll try to have a look at your plugin some time soon.

Ok, I found the time to have a look at the code :
* I had to add a #include <ptlib/sound.h> to make it compile ;
* the handling of filtered caps is much better than in my video plugin, where I use a fixed string ;
* it does only playing ;
* it doesn't detect devices yet.

Looks like mixing ideas from both our plugins will make something good :-)

Created attachment 81783 [details]
Snark's version

Well, this version has better detection and better crash.

I have the impression it doesn't work because the fakesink doesn't sync -- again ;-)

Well, I'll try to find what's wrong in my code later.

Created attachment 81853 [details]
Snark's version (header)

I changed it a little.

Created attachment 81854 [details]
Snark's version (implementation)

Ok, this one makes shitty sound, but records and play.

The reason why it makes shitty sound is probably because of those warnings : 
(ekiga-snapshot:7936): GStreamer-CRITICAL **: gst_segment_clip: assertion `segment->format == format' failed

I couldn't make device detection work properly... do the alsasrc and alsasink really implement GstPropertyProbe ?

It's starting to make a little more sense, though.

I forgot to say I compile it using my code from bug #332442.

Ok, I made patches to improve the GstPropertyProbe of the alsasrc&alsasink elements : it works quite well in my sample code.

Unfortunately, it seems today my plugin won't accept to find the alsasrc and alsasink elements, hence I can't test properly :-(

Created attachment 82102 [details]
Snark's version (header)

Changed the Construct method to be static, private and named InitializeGStreamer.

Created attachment 82103 [details]
Snark's version (implementation)

This version does the detection of devices right. And if you patched according to the bugs about GstPropertyProbe in alsasrc&alsasink (bug #405020 and bug #405024), then the device names look as neat as possible.

There is still the problem of the warnings :
(ekiga-snapshot:30243): GStreamer-CRITICAL **: gst_segment_clip: assertion `segment->format == format' failed
those kill the performance and give shitty sound!

Can someone lend a hand to fix them? A hint might be enough...

Created attachment 82265 [details]
Snark's version (implementation)

This one works quite nicely. But notice the hack it is based on : it may stop working any day, we have been warned. If I understand the danger correctly, fakesrc is really a GstBaseAudioSrc, but it is just an implementation detail, it may change.

We can still test if we can get gstreamer to do nice things for us, then fix it afterwards : it will probably just mean implement a GstBaseAudioSrc, which will do the right magic on itself.

Forgot to say I compile it with :
g++ -shared -DPTRACING -I../pwlib_PNoCollisionDictionary `pkg-config --cflags --libs gstreamer-plugins-base-0.10` -lgstinterfaces-0.10 -lgstbase-0.10 sound_gstreamer.cxx -o gst_pwplugin.so

(where pwlib_PNoCollisionDictionary is my code from bug #332442)

Gasp... it crashes during calls... time for some kind debugging, to understand why the audio test works but the calls fail.

Here is a trace :

+ Trace 109902

There is no crash in that trace.

There is : in Read. I modified it to look like this :

BOOL PSoundChannelGSTREAMER::Read (void *buf, PINDEX len)
{
  int bufPos = 0;
  char *outgoingBuffer = (char *) buf;

  if (len > GST_BUFFER_SIZE (myBuffer))
    cout << "Dying!\n";

  while (bufPos < len) {

and now I see before the crash :
Dying!
Dying!

So the crash is because opal wants to read data, and I don't have that many in the buffer... but still try to read it!

Which leads to a question :
- should I try to wait until a new buffer is ready ?
- or should I put whatever data I have, set the rest to \0, and be done.

Of course, the best would be if Read allowed me to return the count of actually read bytes... but that isn't an option :-/

Hmmm... no, that's not exactly the problem ; we don't read more than we can, since the code reads like this :
      memcpy ((void *)&outgoingBuffer [bufPos], GST_BUFFER_DATA (myBuffer),
	      PMIN (GST_BUFFER_SIZE (myBuffer), (len - bufPos)));
we don't write more to the buffer than we can either :-/

Here is the error message :
*** glibc detected *** free(): invalid next size (fast): 0x08b683c8 ***

Facts :
- using the ALSA plugin works
- using the GSTREAMER plugin gives that error message
=> the problem is in the GSTREAMER plugin

Facts :
- sound events work ;
- sound testing in the druid works ;
- sound in a call fails with an error message
=> I'm confused.

Here is the result of "thread apply all bt" in gdb, in case someone sees something I don't :

+ Trace 110068

I got a clue ! I got a clue !

I made the plugin print its opening options, and saw that the druit test used a sample rate of 16000 while the call used 8000.

I patched ekiga to test with 8000, and now the druid test crashes too!

So I guess we'll have to patch the druid so it makes the test correctly.

Now it should be possible to fix that plugin :-)

The default codec in Ekiga is Speex Wideband, which uses a sample rate of 16000.

So the druid is right.

Created attachment 82447 [details]
Snark's version (implementation)

Well, this version doesn't crash, passes the druid test, and plays sound events.

But this version doesn't give sound when calling 500 : only noise. And it makes ekiga dog slow (stopping the call takes ages, for example).

There is obviously something I don't get :-/

Ah, ah. I butchered ekiga to be able to choose if it will use the configured device or alsa for either playing and recording, and called 500 :
- alsa plays, gstreamer records : ok.
- alsa records, gstreamer plays : not ok.

So the problem is in the playing... how can it work for sound events and audio test and fail for calls is still a pending question.

Created attachment 83366 [details]
Snark's version (header)

The new version uses a GstBaseSrc implementation for playback. It doesn't work either :

(ekiga-snapshot:8153): GStreamer-CRITICAL **: gst_segment_clip: assertion `segment->format == format' failed

But since it's supposed to be the RightThingToDo, I guess it will evolve into something which works.

Created attachment 83367 [details]
Snark's version (implementation)

Created attachment 83368 [details]
Snark's Gst*Src declaration

Created attachment 83369 [details]
Snark's Gst*Src implementation

I just added an appsrc element to gst-plugins-bad that you might find useful.  It's a source element that also has an API, so you can tell it to push buffers, flush, create EOS events, etc.  It's off the top of my head and completely untested, but I've written similar elements before.

Note that it contains two parts, a library and a plugin.  The element is actually implemented in the library, so it can have an API, and the plugin merely calls the get_type() function in order to register the element.

Uh... David, when you wake up, could you cvs add the files ? There's just an empty app/ directory. ;-)

Hehe, the commit failed and I didn't notice.  OOps.

Ok, now it's there ; the api looks nice.

May I ask :
1) why it's bad and not good or base ?
2) what would it take to make it good or base ?

Notice that I may have the answers when I try to use it ;-)

It's in -bad because that's where we put new things.

In particular, we'll need to play around with the API a bit before it settles into something that's generally useful.

Ok, I reported bug #413418 so we can make it better ; I couldn't make my plugin work with it either, but since it looks like the sane thing to do, I'm willing to invest time on it.

After all, GstAppSrc will also help me when I'll want to use GStreamer for codecs too...

Ok, I proposed a patch for GstAppSrc : it doesn't make it fully good, but my hope is that it will give at least a hint how things can work out.

Now, my tests to use it in this plugin haven't been a success so far : apparently, I can't put my pipeline to PLAYING. The reason seems to be that the alsasink takes too long to get into PLAYING state.

Seeing that, I tried to do the switch to PLAYING in two stages : first READY, then PLAYING. The first stage works, the second doesn't.

Here is a short part of a GST_DEBUG=GST_STATES:4 log :

0:01:19.162559000  5547 0x8121dc0 DEBUG           GST_STATES gstelement.c:2195:gst_element_change_state:<pipeline> element will change state ASYNC
0:01:19.162643000  5547 0x8121dc0 DEBUG           GST_STATES gstelement.c:2148:gst_element_set_state_func:<pipeline> returned 2
0:01:19.162853000  5547 0x8121dc0 INFO            GST_STATES gstbin.c:1282:gst_bin_get_state_func:<pipeline> getting state
0:01:19.162939000  5547 0x8121dc0 INFO            GST_STATES gstbin.c:1426:gst_bin_recalc_state:<pipeline> not dirty
0:01:19.163020000  5547 0x8121dc0 DEBUG           GST_STATES gstelement.c:1655:gst_element_get_state_func:<pipeline> getting state
0:01:19.163104000  5547 0x8121dc0 INFO            GST_STATES gstelement.c:1694:gst_element_get_state_func:<pipeline> waiting for element to commit state
gst_app_src_create in
0:01:20.165862000  5547 0x8121dc0 INFO            GST_STATES gstelement.c:1698:gst_element_get_state_func:<pipeline> timed out
0:01:20.166068000  5547 0x8121dc0 DEBUG           GST_STATES gstelement.c:1730:gst_element_get_state_func:<pipeline> state current: PAUSED, pending: PLAYING, result: 2
Couldn't put the pipeline to play

(notice that I added g_print ("%s in\n", __PRETTY_FUNCTION__); at the beginning of each method of GstAppSrc for this test (and "%s out\n" at the end), so we see gst_app_src_create in -- the final line is sound_gstreamer.cxx which prints its failure).

Here is how the code looks :
  (void)gst_element_set_state (pipeline, GST_STATE_READY);

  (void)gst_element_get_state (pipeline, &current, NULL, GST_SECOND);

  if (current != GST_STATE_READY) {

    g_print ("Couldn't put the pipeline to ready\n");
    PTRACE(3, "Couldn't put the pipeline to ready");
    gst_element_set_state (pipeline, GST_STATE_NULL);
    (void)gst_element_get_state (pipeline, NULL, NULL, GST_SECOND);
    gst_object_unref (GST_OBJECT (pipeline));
    pipeline = NULL;
    return FALSE;
  }

  (void)gst_element_set_state (pipeline, GST_STATE_PLAYING);

  (void)gst_element_get_state (pipeline, &current, NULL, GST_SECOND);

  if (current != GST_STATE_PLAYING) {

    g_print ("Couldn't put the pipeline to play\n");
    PTRACE(3, "Couldn't put the pipeline to play");
    gst_element_set_state (pipeline, GST_STATE_NULL);
    (void)gst_element_get_state (pipeline, NULL, NULL, GST_SECOND);
    gst_object_unref (GST_OBJECT (pipeline));
    pipeline = NULL;
    return FALSE;
  }

Any input as to why that isn't enough is welcome!

Created attachment 83816 [details]
Snark's version (implementation)

David Schleef's work on GstAppSrc makes my own implementation of a special source element obsolete. Thanks!

This version of the plugin makes use of GstAppSrc/GstAppBuffer as found in gst-plugins-bad/gst-libs/gst/app in cvs.

Here is what I use to compile :
gcc -g -c gstappbuffer.c `pkg-config --cflags gstreamer-plugins-base-0.10`
gcc -g -c gstappsrc.c `pkg-config --cflags gstreamer-plugins-base-0.10`
g++ -g -c -DPTRACING -I../pwlib_PNoCollisionDictionary `pkg-config --cflags gstreamer-plugins-base-0.10` sound_gstreamer.cxx
g++ -g -shared sound_gstreamer.o gstappsrc.o gstappbuffer.o -o gst_pwplugin.so `pkg-config --libs gstreamer-plugins-base-0.10` -lgstinterfaces-0.10 -lgstbase-0.10 -lgstaudio-0.10

I made some tests the recording indeed works, but it's not as good as our old alsa plugin.

I made some tests : the playing doesn't indeed work ; here is a sample error message :
(ekiga-snapshot:3999): GStreamer-CRITICAL **: gst_segment_clip: assertion `segment->format == format' failed

It's the very same error message I had when I was using a GstFakeSrc and fed the buffers violently to its pad!

The GstAppSrc solution is cleaner of course, but I can't help to think there's something I have overlooked.

Notice that a GstAppSink would perhaps be a good idea too : cleaner.

Created attachment 84269 [details]
Snark's version (header)

Created attachment 84271 [details]
Snark's version (implementation)

Ok, this version does :
- record perfectly ;
- play sound events almost correctly ;
- but doesn't work in the audio test.

The two problems left are :
(1) since I don't know how to compute the duration of a PSound and gstreamer doesn't make it really possible to know when it has finished swallowing data, then I'm using a sleep (2) to make sound events heard (ekiga would close immediately and we wouldn't hear anything otherwise).
(2) I have the impression that playing taxes my box very very hard, so probably there's a sync issue somewhere -- this is why playing a short sound is ok, but playing back a longer recording is problematic.

Comments and help welcome.

I could reproduce the second problem with a simple gstreamer test application : pushing the issue to bug #413418.

GNOME (control-center) has a sound preferences dialog where you specify the devices to be used for audio conferencing (capture and playback). Shouldn't ekiga just use those settings instead of enumerating all available devices?

Well, there are a few nitpicks :
(1) ekiga doesn't just work on gnome desktops (be they GNU/Linux or *BSD), but also on kde, xfce, enlightenment, win32, and hopefully MOSX ;
(2) this plugin would be used by ekiga, but through pwlib, which isn't part of gnome.

Ok, I committed some gstreamer code in ekiga, which supposedly does audio in/out, and which I'll iron out now.

The sources need to be configured with --enable-gstreamer, and the relevant headers/libs have to be installed.

Any help to debug would be greatly appreciated.