GNOME Bugzilla – Bug 797159
Significant memory leak in (at least) omxh264enc and omxmjpegdec on Raspberry Pi
Last modified: 2018-10-05 21:36:41 UTC
I am observing a significant memory leak in both omxh264enc and omxmjpegdec (other components may also exhibit the leak, but I haven't tested them) on a Raspberry Pi 3. This can be reproduced with as simple a pipeline as: videotestsrc ! omxh264enc ! fakesink This particular example fills up the 1 GB of RAM on the Raspberry Pi within a few minutes (my real world examples take a little longer because they run at lower frame rates). I am running gstreamer 1.14.0, and have tested gst-omx versions 1.14.0, 1.14.3 and git master (all with gstreamer 1.14.0). I can update gstreamer to 1.14.3 or even git master if necessary. The Raspberry Pi 3 is running a 64-bit kernel with 32-bit gstreamer (OpenMAX only works with 32-bit userspace libraries on the Raspberry Pi). I attempted to use valgrind to find the location of the leak, but I have not been able to get it to give me any useful information (it detects the leak, but gives no stack traces; this might be a problem with my setup).
To valgrind, add --leak-check=full to get stack traces. Make sure to install debug symbols first. You should also use GStreamer suppression file to remove all the false positive. https://cgit.freedesktop.org/gstreamer/common/plain/gst.supp If you find how, you should test with older firmware, as we believe there is regression in recent update by the RPi foundation.
I used that option and (I'm pretty sure) I have debug symbols setup correctly. My stack traces only show malloc, not any of the calling functions, for example: ==31538== 941,184 (908,160 direct, 33,024 indirect) bytes in 220 blocks are definitely lost in loss record 14 of 14 ==31538== at 0x4847AD0: malloc (in /nix/store/ikv9hdyn5093s379nysgqxx6kv7chx76-valgrind-3.13.0/lib/valgrind/vgpreload_memcheck-arm-linux.so) ==31538== Even without debug symbols, the addresses and symbol names should be shown AFAIK. I tried using massif as well and it shows an address of 0xFFFFFFFF. The stack trace problem is not specific to gstreamer, which makes me suspect it is something to do with how valgrind is setup on my system. I am using firmware version 1.20180619 and commit a343dcad1dae4e93f4bfb99496697e207f91027e (from February) of the userspace libraries (https://github.com/raspberrypi/userland, compiled from source, not using the precompiled versions in the firmware repository). I didn't think to try different versions of these packages, so I'll try that now. Is there a specific version you think should work?
I figured out what was wrong with my valgrind setup and was able to get this leak report (note that the pipeline was only running for a short time, so it didn't leak that much memory): ==14223== Process terminating with default action of signal 2 (SIGINT) ==14223== at 0x4BCFE34: _Exit (in /nix/store/b6b0lfsn1wqhsinvy078i1b1vhi96fqs-glibc-2.27/lib/libc-2.27.so) ==14223== ==14223== HEAP SUMMARY: ==14223== in use at exit: 2,315,683 bytes in 20,885 blocks ==14223== total heap usage: 101,153 allocs, 80,268 frees, 230,807,468 bytes allocated ==14223== ==14223== Thread 1: ==14223== 100 bytes in 1 blocks are possibly lost in loss record 7 of 31 ==14223== at 0x4847AD0: malloc (vg_replace_malloc.c:299) ==14223== by 0x649C813: vcos_generic_mem_alloc_aligned (vcos_mem_from_malloc.c:56) ==14223== by 0x64646B7: vcos_malloc_aligned (vcos_mem_from_malloc.h:68) ==14223== by 0x64646B7: vc_vchi_cec_init (vc_vchi_cecservice.c:236) ==14223== by 0x645B473: bcm_host_init (bcm_host.c:115) ==14223== by 0x577485B: gst_omx_core_acquire (gstomx.c:129) ==14223== by 0x577B8FF: gst_omx_component_new (gstomx.c:781) ==14223== by 0x578EDD7: gst_omx_video_enc_open (gstomxvideoenc.c:749) ==14223== by 0x5573DDB: gst_video_encoder_change_state (in /nix/store/wx5wjmlkfp8k32hqq26wby5i2jigwzbf-gst-plugins-base-1.14.3/lib/libgstvideo-1.0.so.0.1403.0) ==14223== ==14223== 108 bytes in 14 blocks are possibly lost in loss record 8 of 31 ==14223== at 0x48479D8: malloc (vg_replace_malloc.c:298) ==14223== by 0x484A4DF: realloc (vg_replace_malloc.c:785) ==14223== by 0x49F5943: g_realloc (in /nix/store/l8wanykgzg5h18axlvv0nnicc0a40las-glib-2.56.0/lib/libglib-2.0.so.0.5600.0) ==14223== ==14223== 240 bytes in 1 blocks are possibly lost in loss record 9 of 31 ==14223== at 0x4847AD0: malloc (vg_replace_malloc.c:299) ==14223== by 0x649C813: vcos_generic_mem_alloc_aligned (vcos_mem_from_malloc.c:56) ==14223== by 0x649BC9F: vcos_malloc (vcos_mem_from_malloc.h:53) ==14223== by 0x649BC9F: vcos_dummy_thread_create (vcos_pthreads.c:637) ==14223== by 0x84BA03B: vcos_thread_current (vcos_platform.h:370) ==14223== by 0x84BA03B: ilcs_execute_function_ex (vcilcs.c:800) ==14223== by 0x84BAE53: ilcs_execute_function (vcilcs.c:833) ==14223== by 0x84BBB7B: vcil_out_get (vcilcs_out.c:239) ==14223== by 0x5775A23: gst_omx_component_get_parameter (gstomx.c:1133) ==14223== by 0x5776BFB: gst_omx_port_update_port_definition (gstomx.c:1328) ==14223== by 0x5776C9B: gst_omx_port_is_enabled (gstomx.c:2508) ==14223== by 0x578218B: gst_omx_video_dec_loop (gstomxvideodec.c:1569) ==14223== by 0x49343F3: gst_task_func (in /nix/store/iqw3lnskbqrhg8l0sdpj9wimsl4ac8zp-gstreamer-1.14.3/lib/libgstreamer-1.0.so.0.1403.0) ==14223== ==14223== 492 bytes in 4 blocks are possibly lost in loss record 13 of 31 ==14223== at 0x484A490: realloc (vg_replace_malloc.c:785) ==14223== by 0x49F5943: g_realloc (in /nix/store/l8wanykgzg5h18axlvv0nnicc0a40las-glib-2.56.0/lib/libglib-2.0.so.0.5600.0) ==14223== ==14223== 1,008 bytes in 6 blocks are possibly lost in loss record 15 of 31 ==14223== at 0x484A228: calloc (vg_replace_malloc.c:711) ==14223== by 0x40121DF: allocate_dtv (in /nix/store/b6b0lfsn1wqhsinvy078i1b1vhi96fqs-glibc-2.27/lib/ld-2.27.so) ==14223== ==14223== 1,628 bytes in 43 blocks are possibly lost in loss record 17 of 31 ==14223== at 0x484A228: calloc (vg_replace_malloc.c:711) ==14223== by 0x49F58D3: g_malloc0 (in /nix/store/l8wanykgzg5h18axlvv0nnicc0a40las-glib-2.56.0/lib/libglib-2.0.so.0.5600.0) ==14223== ==14223== 210,528 bytes in 51 blocks are possibly lost in loss record 28 of 31 ==14223== at 0x4847AD0: malloc (vg_replace_malloc.c:299) ==14223== by 0x649C813: vcos_generic_mem_alloc_aligned (vcos_mem_from_malloc.c:56) ==14223== by 0x648213F: vcos_malloc (vcos_mem_from_malloc.h:53) ==14223== by 0x648213F: alloc_msgbuf (vchiq_lib.c:1753) ==14223== by 0x648213F: completion_thread (vchiq_lib.c:1528) ==14223== by 0x649B357: vcos_thread_entry (vcos_pthreads.c:144) ==14223== by 0x4B0EFE3: start_thread (in /nix/store/b6b0lfsn1wqhsinvy078i1b1vhi96fqs-glibc-2.27/lib/libpthread-2.27.so) ==14223== ==14223== 1,110,432 bytes in 269 blocks are definitely lost in loss record 31 of 31 ==14223== at 0x4847AD0: malloc (vg_replace_malloc.c:299) ==14223== by 0x649C813: vcos_generic_mem_alloc_aligned (vcos_mem_from_malloc.c:56) ==14223== by 0x648213F: vcos_malloc (vcos_mem_from_malloc.h:53) ==14223== by 0x648213F: alloc_msgbuf (vchiq_lib.c:1753) ==14223== by 0x648213F: completion_thread (vchiq_lib.c:1528) ==14223== by 0x649B357: vcos_thread_entry (vcos_pthreads.c:144) ==14223== by 0x4B0EFE3: start_thread (in /nix/store/b6b0lfsn1wqhsinvy078i1b1vhi96fqs-glibc-2.27/lib/libpthread-2.27.so) ==14223== ==14223== LEAK SUMMARY: ==14223== definitely lost: 1,110,432 bytes in 269 blocks ==14223== indirectly lost: 0 bytes in 0 blocks ==14223== possibly lost: 214,104 bytes in 120 blocks ==14223== still reachable: 979,155 bytes in 20,355 blocks ==14223== of which reachable via heuristic: ==14223== newarray : 1,736 bytes in 63 blocks ==14223== suppressed: 0 bytes in 0 blocks ==14223== Reachable blocks (those to which a pointer was found) are not shown.
(In reply to benwolsieffer from comment #3) > I figured out what was wrong with my valgrind setup and was able to get this > leak report (note that the pipeline was only running for a short time, so it > didn't leak that much memory): > > ==14223== Process terminating with default action of signal 2 (SIGINT) > ==14223== at 0x4BCFE34: _Exit (in > /nix/store/b6b0lfsn1wqhsinvy078i1b1vhi96fqs-glibc-2.27/lib/libc-2.27.so) > ==14223== > ==14223== HEAP SUMMARY: > ==14223== in use at exit: 2,315,683 bytes in 20,885 blocks > ==14223== total heap usage: 101,153 allocs, 80,268 frees, 230,807,468 > bytes allocated > ==14223== > ==14223== Thread 1: > ==14223== 100 bytes in 1 blocks are possibly lost in loss record 7 of 31 > ==14223== at 0x4847AD0: malloc (vg_replace_malloc.c:299) > ==14223== by 0x649C813: vcos_generic_mem_alloc_aligned > (vcos_mem_from_malloc.c:56) > ==14223== by 0x64646B7: vcos_malloc_aligned (vcos_mem_from_malloc.h:68) > ==14223== by 0x64646B7: vc_vchi_cec_init (vc_vchi_cecservice.c:236) > ==14223== by 0x645B473: bcm_host_init (bcm_host.c:115) > ==14223== by 0x577485B: gst_omx_core_acquire (gstomx.c:129) > ==14223== by 0x577B8FF: gst_omx_component_new (gstomx.c:781) > ==14223== by 0x578EDD7: gst_omx_video_enc_open (gstomxvideoenc.c:749) > ==14223== by 0x5573DDB: gst_video_encoder_change_state (in > /nix/store/wx5wjmlkfp8k32hqq26wby5i2jigwzbf-gst-plugins-base-1.14.3/lib/ > libgstvideo-1.0.so.0.1403.0) This one can be ignored, it's system setup, ran once. > ==14223== > ==14223== 108 bytes in 14 blocks are possibly lost in loss record 8 of 31 > ==14223== at 0x48479D8: malloc (vg_replace_malloc.c:298) > ==14223== by 0x484A4DF: realloc (vg_replace_malloc.c:785) > ==14223== by 0x49F5943: g_realloc (in > /nix/store/l8wanykgzg5h18axlvv0nnicc0a40las-glib-2.56.0/lib/libglib-2.0.so.0. > 5600.0) I'm guessing you are missing glib symbols, or the glib is build with -fomit-frame-pointer, this stack trace give no information. Pretty small though. > ==14223== > ==14223== 240 bytes in 1 blocks are possibly lost in loss record 9 of 31 > ==14223== at 0x4847AD0: malloc (vg_replace_malloc.c:299) > ==14223== by 0x649C813: vcos_generic_mem_alloc_aligned > (vcos_mem_from_malloc.c:56) > ==14223== by 0x649BC9F: vcos_malloc (vcos_mem_from_malloc.h:53) > ==14223== by 0x649BC9F: vcos_dummy_thread_create (vcos_pthreads.c:637) > ==14223== by 0x84BA03B: vcos_thread_current (vcos_platform.h:370) > ==14223== by 0x84BA03B: ilcs_execute_function_ex (vcilcs.c:800) > ==14223== by 0x84BAE53: ilcs_execute_function (vcilcs.c:833) > ==14223== by 0x84BBB7B: vcil_out_get (vcilcs_out.c:239) > ==14223== by 0x5775A23: gst_omx_component_get_parameter (gstomx.c:1133) > ==14223== by 0x5776BFB: gst_omx_port_update_port_definition > (gstomx.c:1328) > ==14223== by 0x5776C9B: gst_omx_port_is_enabled (gstomx.c:2508) > ==14223== by 0x578218B: gst_omx_video_dec_loop (gstomxvideodec.c:1569) > ==14223== by 0x49343F3: gst_task_func (in > /nix/store/iqw3lnskbqrhg8l0sdpj9wimsl4ac8zp-gstreamer-1.14.3/lib/ > libgstreamer-1.0.so.0.1403.0) Would need to study vcos code, also small. > ==14223== > ==14223== 492 bytes in 4 blocks are possibly lost in loss record 13 of 31 > ==14223== at 0x484A490: realloc (vg_replace_malloc.c:785) > ==14223== by 0x49F5943: g_realloc (in > /nix/store/l8wanykgzg5h18axlvv0nnicc0a40las-glib-2.56.0/lib/libglib-2.0.so.0. > 5600.0) > ==14223== > ==14223== 1,008 bytes in 6 blocks are possibly lost in loss record 15 of 31 > ==14223== at 0x484A228: calloc (vg_replace_malloc.c:711) > ==14223== by 0x40121DF: allocate_dtv (in > /nix/store/b6b0lfsn1wqhsinvy078i1b1vhi96fqs-glibc-2.27/lib/ld-2.27.so) > ==14223== > ==14223== 1,628 bytes in 43 blocks are possibly lost in loss record 17 of 31 > ==14223== at 0x484A228: calloc (vg_replace_malloc.c:711) > ==14223== by 0x49F58D3: g_malloc0 (in > /nix/store/l8wanykgzg5h18axlvv0nnicc0a40las-glib-2.56.0/lib/libglib-2.0.so.0. > 5600.0) > ==14223== > ==14223== 210,528 bytes in 51 blocks are possibly lost in loss record 28 of > 31 > ==14223== at 0x4847AD0: malloc (vg_replace_malloc.c:299) > ==14223== by 0x649C813: vcos_generic_mem_alloc_aligned > (vcos_mem_from_malloc.c:56) > ==14223== by 0x648213F: vcos_malloc (vcos_mem_from_malloc.h:53) > ==14223== by 0x648213F: alloc_msgbuf (vchiq_lib.c:1753) > ==14223== by 0x648213F: completion_thread (vchiq_lib.c:1528) > ==14223== by 0x649B357: vcos_thread_entry (vcos_pthreads.c:144) > ==14223== by 0x4B0EFE3: start_thread (in > /nix/store/b6b0lfsn1wqhsinvy078i1b1vhi96fqs-glibc-2.27/lib/libpthread-2.27. > so) Vcos thread. > ==14223== > ==14223== 1,110,432 bytes in 269 blocks are definitely lost in loss record > 31 of 31 > ==14223== at 0x4847AD0: malloc (vg_replace_malloc.c:299) > ==14223== by 0x649C813: vcos_generic_mem_alloc_aligned > (vcos_mem_from_malloc.c:56) > ==14223== by 0x648213F: vcos_malloc (vcos_mem_from_malloc.h:53) > ==14223== by 0x648213F: alloc_msgbuf (vchiq_lib.c:1753) > ==14223== by 0x648213F: completion_thread (vchiq_lib.c:1528) > ==14223== by 0x649B357: vcos_thread_entry (vcos_pthreads.c:144) > ==14223== by 0x4B0EFE3: start_thread (in > /nix/store/b6b0lfsn1wqhsinvy078i1b1vhi96fqs-glibc-2.27/lib/libpthread-2.27. > so) Another vcos thread, with a bigger leak, maybe the one. So far, it looks like you'll have to file a bug against vcos.
It turns out that this is a kernel bug, see: https://github.com/raspberrypi/linux/pull/2703