GNOME Bugzilla – Bug 767856
QA: run tests using leaks tracer with Jenkins
Last modified: 2018-11-03 12:35:05 UTC
Now that the leaks tracer has been merged (bug #765052) we could consider adding a Jenkins job on https://ci.gstreamer.net/ running tests with the leaks tracer to prevent memory leaks regressions. To do so we first should make sure that all tests are currently not reported as leaking. I'll mark those as blockers.
Please also look make sure that the tests pass reliably in all modules with the leak checker. For me the core tests just don't pass reliably, for example. There are some tests which sometimes leak (also in valgrdind), those need to be tracked down first.
Are there any others left after the patches I merged now?
(In reply to Sebastian Dröge (slomo) from comment #2) > Are there any others left after the patches I merged now? Nope, I think we're good for now. Thanks!
(In reply to Tim-Philipp Müller from comment #1) > For me the core tests just don't pass reliably, for > example. Which ones are those? It seems all good here.
(In reply to Guillaume Desmottes from comment #4) > (In reply to Tim-Philipp Müller from comment #1) > > For me the core tests just don't pass reliably, for > > example. > > Which ones are those? It seems all good here. Ok I managed to reproduce some when turning on the stack trace logging feature from bug #767862). They are failing because the tracer is slowing tests too much and we reach the check timeout. An easy way to fix this is to use "CK_TIMEOUT_MULTIPLIER=4" for example. So I think we can just document it and use this when running tests with QA bots.
That was before the stack tracing landed. I'll re-check later.
(In reply to Tim-Philipp Müller from comment #6) > That was before the stack tracing landed. I'll re-check later. It's not landed yet. But if it's just the tests becoming too slow it depends of the speed of the hardware running the tests. Please tests with CK_TIMEOUT_MULTIPLIER=4 and let me know how it goes for you.
The failure wasn't a timeout, it was leaks :)
These all appear to be racy / timing-dependent, none of them fails every time consistently: FAIL: gst/gstbus stresstest:test_custom_main_context:0: Unexpected critical/warning: Leaks detected FAIL: gst/gstghostpad ghost pad tests:test_ghost_pads:0: Unexpected critical/warning: Leaks detected FAIL: gst/gstpipeline pipeline tests:test_async_state_change_fake:0: Unexpected critical/warning: Leaks detected FAIL: gst/gstutils general:test_element_link_with_ghost_pads:0: Unexpected critical/warning: Leaks detected FAIL: pipelines/simple-launch-lines linear:test_2_elements:0: Unexpected critical/warning: Leaks detected FAIL: libs/collectpads pipeline:test_linear_pipeline:0: Unexpected critical/warning: Leaks detected pipeline:test_branched_pipeline:0: Unexpected critical/warning: Leaks detected Could all be the same issue of course.
I'm running those in loop and can't reproduce so far. :\ Can you please provide GST_DEBUG="GST_TRACER:7" logs to check what's leaked? Extra pingouin points if you can even reproduce with this branch https://git.collabora.com/cgit/user/cassidy/gstreamer/gstreamer.git/log/?h=leak-tracer and run it with GST_LEAKS_TRACER_STACK_TRACE=1 Thanks!
Ah! Just managed to reproduce the pipelines/simple-launch-lines one! 0:00:01.698917222 7106 0x10fb260 TRACE GST_TRACER :0:: object-alive, type-name=(string)GstFakeSink, address=(gpointer)0x1116610, description=(string)<fakesink4>, ref-count=(uint)2, trace=(string)handle_object_created.part.0 gst_object_constructed g_object_unref g_object_newv gst_element_factory_create gst_element_factory_make priv_gst_parse_yyparse priv_gst_parse_launch gst_parse_launch_full setup_pipeline test_2_elements srunner_run gst_check_run_suite main __libc_start_main _start ; 0:00:01.698943857 7106 0x10fb260 TRACE GST_TRACER :0:: object-alive, type-name=(string)GstFakeSrc, address=(gpointer)0x110ac60, description=(string)<fakesrc4>, ref-count=(uint)4, trace=(string)handle_object_created.part.0 gst_object_constructed g_object_unref g_object_newv gst_element_factory_create gst_element_factory_make priv_gst_parse_yyparse priv_gst_parse_launch gst_parse_launch_full setup_pipeline test_2_elements srunner_run gst_check_run_suite main __libc_start_main _start ; 0:00:01.698956877 7106 0x10fb260 TRACE GST_TRACER :0:: object-alive, type-name=(string)GstMessage, address=(gpointer)0x7f3b280072c0, description=(string)structure-change message: 0x7f3b280072c0, time 99:99:99.999999999, seq-num 168, element 'sink', GstMessageStructureChange, type=(GstStructureChangeType)GST_STRUCTURE_CHANGE_TYPE_PAD_UNLINK, owner=(GstElement)"\(GstFakeSrc\)\ fakesrc4", busy=(boolean)true;, ref-count=(uint)1, trace=(string)handle_object_created.part.0 gst_mini_object_init gst_message_init gst_message_new_custom gst_pad_unlink unlink_pads foreach_fold_func gst_iterator_fold gst_iterator_foreach gst_bin_remove_func gst_bin_remove gst_bin_dispose g_object_unref gst_element_call_async_func g_thread_pool_new g_test_get_filename start_thread clone ; 0:00:01.699159803 7106 0x10fb260 TRACE GST_TRACER :0:: object-alive, type-name=(string)GstPad, address=(gpointer)0x110c2a0, description=(string)<fakesink4:sink>, ref-count=(uint)3, trace=(string)handle_object_created.part.0 gst_object_constructed g_object_unref g_object_new_valist g_object_new gst_pad_new_from_template gst_base_sink_init g_type_create_instance g_object_unref g_object_newv gst_element_factory_create gst_element_factory_make priv_gst_parse_yyparse priv_gst_parse_launch gst_parse_launch_full setup_pipeline test_2_elements srunner_run gst_check_run_suite main __libc_start_main _start ; 0:00:01.699179819 7106 0x10fb260 TRACE GST_TRACER :0:: object-alive, type-name=(string)GstPad, address=(gpointer)0x110c060, description=(string)<fakesrc4:src>, ref-count=(uint)2, trace=(string)handle_object_created.part.0 gst_object_constructed g_object_unref g_object_new_valist g_object_new gst_pad_new_from_template gst_base_src_init g_type_create_instance g_object_unref g_object_newv gst_element_factory_create gst_element_factory_make priv_gst_parse_yyparse priv_gst_parse_launch gst_parse_launch_full setup_pipeline test_2_elements srunner_run gst_check_run_suite main __libc_start_main _start ; 0:00:01.699192156 7106 0x10fb260 TRACE GST_TRACER :0:: object-alive, type-name=(string)GstPipeline, address=(gpointer)0x1112710, description=(string)<pipeline4>, ref-count=(uint)1, trace=(string)handle_object_created.part.0 gst_object_constructed g_object_unref g_object_newv gst_element_factory_create gst_element_factory_make priv_gst_parse_launch gst_parse_launch_full setup_pipeline test_2_elements srunner_run gst_check_run_suite main __libc_start_main _start ;
For the record, this race is because of bug#768577
(In reply to Tim-Philipp Müller from comment #9) > These all appear to be racy / timing-dependent, none of them fails every > time consistently: Can you still reproduce those now that bug #768577 has been fixed?
No, seems to be all good now, thanks! (only checked core though)
Great! I runned the core tests in a loop for a while and didn't experience any issue either. So I think we can now consider adding a build bot running core tests with the leaks detector enabled. I'm also interested seeing test results on non-Linux systems. I have no experience with Jenkins or our QA system in general. What would be the next step here?
With my latest patches (bug #778279 and bug #768762) I can now successfully run the core, base, good, ugly and devtools meson tests (269) with the leaks tracer enabled. I take less than 4 minutes to run on my laptop. I think now would be a good time to add one job running those to https://ci.gstreamer.net/ I'm happy to help doing it but would need some pointers about how to do so.
Created attachment 345269 [details] [review] validate:launcher: Add a way to specify a set of tests to run under the leak tracer
Proposing that patch so that we can just run the unit tests with gst-validate-launcher --gst-check-leak-trace-testnames='known-not-leaky' check and the leaks tracer will be activated on all the tests know to be not leaky (and we easily add up to the list as needed). (we could maybe even activate that by default?)
Re-assigning to core for now (until we have a ci module on gitlab). Just to confirm, the only thing required is to run make check with GST_TRACERS=leak, right ?
It's GST_TRACERS=leaks but yes. The tracer will raise a g_warning if it detects leaks. We could add extra flags to get proper stacks trace and make it easier to spot the leaks just by the looking at the logs. But they will improve the memory usage a lot so I wouldn't recommend it, I think. We'll have to reproduce locally to fix the regression anyway so we may as well do it when debugging.
-- GitLab Migration Automatic Message -- This bug has been migrated to freedesktop.org's GitLab instance and has been closed from further activity. You can subscribe and participate further through the new bug through this link to our GitLab instance: https://gitlab.freedesktop.org/gstreamer/gstreamer/issues/176.