GNOME Bugzilla – Bug 345181
Allow plugins to add gstreamer element to the playback pipeline
Last modified: 2007-03-02 12:05:35 UTC
It would be nice if plugins could add GStreamer elements to the pipeline. I can think of two main types: filtering elements (e.g. for an equaliser) that change the output audio, and tee elements (e.g. for visualisation or gshrooms) which get a copy of the audio.
Created attachment 67537 [details] [review] start of a patch This is an initial implementation which supports adding tee elements. You can test it, creating a poor-mans-visualisation, by entering the following in the python console. Visualisation is probably best done with playbin's support, but it works as a test. import gst b = gst.Bin() goom = gst.element_factory_make ("goom") sink = gst.element_factory_make ("ximagesink") colour = gst.element_factory_make ("ffmpegcolorspace") b.add (goom, colour, sink) goom.link(colour) colour.link(sink) b.add_pad(gst.GhostPad("sink", goom.get_pad("sink"))) shell.get_player().props.player.add_tee_element(b) Known issues: * elements will get removed if playback is stopped completely (not just paused) * filtering elements are not supported * things sometimes don't work, for a reason I haven't figured out
I've been working on a different approach for visualisation and filtering elements, making the player emit signals. 'create-filter' to create a set of elements to insert before the audio sink, 'mutate-playbin' to do arbitrary other stuff to playbin. This is probably a better approach for filter elements, as it allows them to be removed.
Created attachment 76148 [details] [review] mroe of a patch This makes some fixes and has code for adding filters. The filter-adding bit doesn't quite work yet, it hangs when you try to use it, but the basic structure is there. Something like the follow could (if it worked) be used to test: import gst e = gst.element_factory_make("lpwsinc") shell.props.shell_player.props.player.add_filter(e)
Created attachment 76372 [details] [review] better patch Adding filter elements now works (removing them doesn't yet). I've also split up the interfaces into one for Filters, one for Tees and one for an as yet unimplemented Data Tee (which would get the undecoded data). This is more easily tested by using a "volume" element instead of my previous example, as you can modify the volume with "e.props.volume = 0.1" in the python console.
I'm trying this patch to go on with the equalizer stuff I started a while ago, but it seems to not work with latest SVN. The patch fail to apply: patching file backends/Makefile.am patching file backends/rb-player.c patching file backends/rb-player.h patching file backends/gstreamer/rb-player-gst.c Hunk #1 succeeded at 36 with fuzz 2. Hunk #2 succeeded at 73 (offset 4 lines). Hunk #3 succeeded at 112 (offset 10 lines). Hunk #4 succeeded at 250 (offset 37 lines). Hunk #5 FAILED at 664. Hunk #6 succeeded at 1231 (offset 63 lines). 1 out of 6 hunks FAILED -- saving rejects to file backends/gstreamer/rb-player-gst.c.rej patching file bindings/python/Makefile.am patching file bindings/python/rb.defs Hunk #3 succeeded at 1763 (offset 61 lines). patching file bindings/python/rb.override patching file shell/Makefile.am Hunk #1 FAILED at 100. Hunk #2 succeeded at 141 (offset -4 lines). 1 out of 2 hunks FAILED -- saving rejects to file shell/Makefile.am.rej patching file backends/rb-player-gst-tee.h patching file backends/rb-player-gst-tee.c patching file backends/rb-player-gst-data-tee.h patching file backends/rb-player-gst-data-tee.c patching file backends/rb-player-gst-filter.h patching file backends/rb-player-gst-filter.c can you please update the patch ?
Created attachment 81897 [details] [review] updated for svn Updated. From memory adding filters worked, but I think there was still a nasty bug when you try to remove them.
Thanks for quick reply. Some feedback: * The last patch seems to miss some files... but combining it with the last one marked obsolete, I managed to get something working. * I jumped to what I'm interested in: inserting an equalizer filter. AFAIK the only ones are from the ladspa plugin: ladspa-Eq and ladspa-tap-equalizer. The problem is that add_filter() assumes that the sink pad of the element is actually named "sink" but those nasty ladspa filters don't follow this convention: for ladspa-Eq it's "in"/"out" and for ladspa-tap-equalizer it's "Input"/"Output". I changed the relevant line to make ladspa-Eq work and this >> import gst >> eq = gst.element_factory_make ("ladspa-Eq") >> shell.props.shell_player.props.player.add_filter(eq) >> eg.set_property ("param-31-Hz", 30); actually produce a bass boosted sound. As a proper solution, the code could scan all pads of the inserted element and pick the first one that as direction GST_PAD_SINK. * the element to be inserted is wrapped in a bin with an "audioconvert" element before. Is another "audioresample" not necessary too ?
Just to add that those elements seem quite fragile actually (they are now in gst-bad): trying to seek or change song crashes the pipeline: ** (rhythmbox:19225): CRITICAL **: gst_signal_processor_process: assertion `self->pending_out == 0' failed ** (rhythmbox:19225): WARNING **: Unexpectedly empty buffer pen for pad ladspa-eq0:out ** (rhythmbox:19225): CRITICAL **: Something wierd happened...
(In reply to comment #7) > Thanks for quick reply. Some feedback: > > * The last patch seems to miss some files... but combining it with the last one > marked obsolete, I managed to get something working. Oops, I've fix that when I upload a new version with a couple of minor improvements a bit later tonight. > * I jumped to what I'm interested in: inserting an equalizer filter. AFAIK the > only ones are from the ladspa plugin: ladspa-Eq and ladspa-tap-equalizer. GStreamer 0.8 had an "equalizer" plugin, which I ported to 0.10 a few days ago. It's in gst-plugins-bad now, so you could use that if you build GStreamer and the plugin from CVS. Note that I haven't done any testing that it actually works, but it runs okay with the default values of no (noticable) audio changes. > The problem is that add_filter() assumes that the sink pad of the element is > actually named "sink" but those nasty ladspa filters don't follow this > convention: for ladspa-Eq it's "in"/"out" and for ladspa-tap-equalizer it's > "Input"/"Output". As you said below, probably the best thing to do is to look for the pads "sink" and "src", and then fallback to the first unconnected pad of the right direction if those don't exist. if you want to post the code fragment that you changed to make that work, I'll put it in the main patch. > I changed the relevant line to make ladspa-Eq work and this > > >> import gst > >> eq = gst.element_factory_make ("ladspa-Eq") > >> shell.props.shell_player.props.player.add_filter(eq) > >> eg.set_property ("param-31-Hz", 30); > > actually produce a bass boosted sound. Nice. If the pipeline is using push mode (which is probably is), they'll be lag when you change the settings, but there isn't a whole lot we can do about that. > * the element to be inserted is wrapped in a bin with an "audioconvert" element > before. Is another "audioresample" not necessary too ? That would be necessary if the elements to be inserted only work on fixed sample rates. I'm hoping that most elements would work with a variety of sample rates - if that is true in the general case but not for specific elements, then you could create a bin with "audioresample ! element" (or is it the other way?) when inserting your filter.
> GStreamer 0.8 had an "equalizer" plugin, which I ported to 0.10 a few days ago. > It's in gst-plugins-bad now, so you could use that if you build GStreamer and > the plugin from CVS. Note that I haven't done any testing that it actually > works, but it runs okay with the default values of no (noticable) audio > changes. > cool. I installed gstreamer CVS. The equalizer plugin is segfaulting, I can't even gst-inspect it. Note that it could really be me having screwed up the installation. (for the record, here is the warning just before segfault: GLib-GObject-WARNING **: invalid cast from `GstIirEqualizer' to `GstBaseTransform' ) For the moment, I'll use one of te ladspa plugins and try to make things work with several equalizer plugins. > Nice. If the pipeline is using push mode (which is probably is), they'll be lag > when you change the settings, but there isn't a whole lot we can do about that. > Indeed.
Created attachment 82190 [details] [review] patch to rb-player-gst.c Change add_filter() to scan throught the sink pads of the added element, and taking the first one not yet linked. (apply in backends/gstreamer/ directory)
(In reply to comment #10) > I installed gstreamer CVS. The equalizer plugin is segfaulting, I can't even > gst-inspect it. Note that it could really be me having screwed up the > installation. > (for the record, here is the warning just before segfault: > GLib-GObject-WARNING **: invalid cast from `GstIirEqualizer' to > `GstBaseTransform' > ) Did you replace your normal GStreamer installation, or just install it somewhere else? What you're seeing could easily be caused if you tried to run the equaliser element from CVS against a non-CVS version of GStreamer. If you installed it somewhere else, make sure you run RB with "LD_LIBRARY_PATH=/path/to/gstreamer/lib rhythmbox".
Also, generating patches with "-u" is generally better, as it makes them much easier to read.
Is there any chance you could re-generate the patch with -u? This one doesn't apply cleanly, and due to the patch format it's difficult to fix.
Created attachment 82788 [details] [review] corret patch Oh, sorry, I didn't realized that was blocking. Attaching both the patch and the new file, just in case.
Created attachment 82789 [details] modified rb-player-gst.c
Created attachment 83292 [details] [review] merged and updated Merges together the changes and updated for SVN. I think this look okay to commit, and we can think about implementing RBPlayerGstDataTee later.
Created attachment 83302 [details] [review] auto sufficient patch previous patch misses some files (rb-player-gst-* interfaces ni backends/), so here is a more complete one. Tested, works fine for me, even removing a filter from the pipeline.
> Tested, works fine for me, even removing a filter from the pipeline. Well, removing a filter works, but adding then removing it too fast make RB hangs. The precise call that make it hangs is: if (gst_element_set_state (bin, GST_STATE_NULL) == GST_STATE_CHANGE_ASYNC) { ... } in function rb_player_gst_remove_filter() I'm afraid I don't know well enough how gstreamer is working to fix this, especially if no warning is printed before the hang. Also the rb_player_gst_remove_filter() somewhat unref the element to remove, so an application wanting to hold it (e.g. for further re-adding), should gst_object_ref on it. That's not a problem but maybe it should be advertised.
Created attachment 83535 [details] [review] fix pad leaks unref pads where necessary. still seems to work afterwards.
Patch committed to trunk.