GNOME Bugzilla – Bug 790989
compositor performance issue with gst-build (meson)
Last modified: 2017-12-06 11:38:23 UTC
I noticed that master (55823ae92b204c5e0dc326f8309b0a10b260cbb2) induces a massive performance regression on compositor. gst-launch-1.0 filesrc location=/dev/zero blocksize=3110400 num-buffers=900 ! rawvideoparse width=1920 height=1080 framerate=30 format=nv12 ! queue ! compositor ! "video/x-raw, format=(string)NV12, width=(int)3840, height=(int)2160, framerate=(fraction)30, colorimetry=(string)bt709" ! fakesink -v Numbers below are in seconds, running the same pipeline in 1.12.3 and master +------+------+--------+ | Csp | 1.12 | master | +------+------+--------+ | NV12 | 6.69 | 15.36 | | YUY2 | 4.57 | 9.12 | | I420 | 6.97 | 15.66 | +------+------+--------+ I don't see any changes in the caps between the two versions, it seems to be internal to compositor. Might be related to https://bugzilla.gnome.org/show_bug.cgi?id=786078
I'm not getting these results at all here, timing your example pipeline gives me 15.5 seconds both on 1.12.3 and master. Are you sure the environment is the same in both cases (optimization flags / orc available)?
Also, can you maybe bisect to determine the problematic commit? I tried reverting my performance improvement commits and I got the same results.
I am using gst-build, which is the main difference. I don't have the issue with stable (my distro packages), or the "legacy" gst-uninstalled way. All libgstcompositor.so link against /usr/lib/liborc-0.4.so.0 Here is the meson build output; are you noticing anything shady ? ~/gst/gst-build/gst-plugins-bad$ (master) meson build The Meson build system Version: 0.43.0 Source dir: /home/fthiery/gst/gst-build/subprojects/gst-plugins-bad Build dir: /home/fthiery/gst/gst-build/subprojects/gst-plugins-bad/build Build type: native build Project name: gst-plugins-bad Native C compiler: cc (gcc 7.2.0) Native C++ compiler: c++ (gcc 7.2.0) Build machine cpu family: x86_64 Build machine cpu: x86_64 Compiler for C++ supports argument -Wno-non-virtual-dtor: YES Compiler for C supports argument -fvisibility=hidden: YES Compiler for C++ supports argument -fvisibility=hidden: YES Has header "dlfcn.h": YES Has header "fcntl.h": YES Has header "highgui.h": NO Has header "inttypes.h": YES Has header "memory.h": YES Has header "msacm.h": NO Has header "netinet/in.h": YES Has header "netinet/ip.h": YES Has header "netinet/tcp.h": YES Has header "opencv2/highgui/highgui_c.h": YES Has header "pthread.h": YES Has header "stdint.h": YES Has header "stdlib.h": YES Has header "strings.h": YES Has header "string.h": YES Has header "sys/param.h": YES Has header "sys/socket.h": YES Has header "sys/stat.h": YES Has header "sys/time.h": YES Has header "sys/types.h": YES Has header "sys/utsname.h": YES Has header "unistd.h": YES Has header "windows.h": NO Has header "winsock2.h": NO Has header "ws2tcpip.h": NO Checking for function "dcgettext": YES Checking for function "getpagesize": YES Checking for function "gmtime_r": YES Checking for function "mmap": YES Checking for function "pipe2": YES Checking for size of "char": 1 Checking for size of "int": 4 Checking for size of "long": 8 Checking for size of "short": 2 Checking for size of "void*": 8 Found pkg-config: /usr/bin/pkg-config (0.29.2) Native dependency gstreamer-1.0 found: YES 1.13.0.1 Native dependency gstreamer-base-1.0 found: YES 1.13.0.1 Native dependency gstreamer-net-1.0 found: YES 1.13.0.1 Native dependency gstreamer-controller-1.0 found: YES 1.13.0.1 Native dependency gstreamer-pbutils-1.0 found: YES 1.13.0.1 Native dependency gstreamer-allocators-1.0 found: YES 1.13.0.1 Native dependency gstreamer-app-1.0 found: YES 1.13.0.1 Native dependency gstreamer-audio-1.0 found: YES 1.13.0.1 Native dependency gstreamer-fft-1.0 found: YES 1.13.0.1 Native dependency gstreamer-riff-1.0 found: YES 1.13.0.1 Native dependency gstreamer-rtp-1.0 found: YES 1.13.0.1 Native dependency gstreamer-rtsp-1.0 found: YES 1.13.0.1 Native dependency gstreamer-sdp-1.0 found: YES 1.13.0.1 Native dependency gstreamer-tag-1.0 found: YES 1.13.0.1 Native dependency gstreamer-video-1.0 found: YES 1.13.0.1 Native dependency gstreamer-check-1.0 found: YES 1.13.0.1 Library m found: YES Native dependency glib-2.0 found: YES 2.54.0 Native dependency gmodule-2.0 found: YES 2.54.0 Native dependency gio-2.0 found: YES 2.54.0 Native dependency x11 found: YES 1.6.5 Native dependency openssl found: YES 1.1.0g Native dependency mjpegtools found: YES 2.1.0 Library m found: YES Native dependency orc-0.4 found: YES 0.4.28 Program orcc found: YES (/usr/bin/orcc) Message: GStreamer debug system is enabled Configuring config.h using configuration Program glib-mkenums found: YES (/usr/bin/glib-mkenums) Program g-ir-scanner found: YES (/usr/bin/g-ir-scanner) Program g-ir-scanner found: YES (/usr/bin/g-ir-scanner) Program g-ir-compiler found: YES (/usr/bin/g-ir-compiler) Native dependency gobject-introspection-1.0 found: YES 1.54.1 Program build_mkenum.py found: YES (/home/fthiery/gst/gst-build/subprojects/gst-plugins-bad/gst-libs/gst/interfaces/build_mkenum.py) Program mpegts_enum.py found: YES (/home/fthiery/gst/gst-build/subprojects/gst-plugins-bad/gst-libs/gst/mpegts/mpegts_enum.py) Native dependency opencv found: YES 3.3.1 Native dependency wayland-client found: YES 1.14.0 Native dependency libdrm found: YES 2.4.88 Native dependency wayland-protocols found: YES 1.11 Program wayland-scanner found: YES (/usr/bin/wayland-scanner) Native dependency gmodule-no-export-2.0 found: YES 2.54.0 Dependency found: NO Native dependency gl found: YES 17.2.6 Native dependency glesv2 found: YES 17.2.6 Has header "GLES3/gl3.h": YES Has header "GLES3/gl3ext.h": YES Native dependency egl found: YES 17.2.6 Has header "libdrm/drm_fourcc.h": YES Native dependency wayland-cursor found: YES 1.14.0 Native dependency wayland-egl found: YES 17.2.6 Native dependency x11-xcb found: YES 1.6.5 Dependency bcm_host found: NO Library bcm_host found: NO Library gdi32 found: NO Header <GL/wglext.h> has symbol "WGL_WGLEXT_VERSION": NO Checking for function "fbGetDisplay": NO Checking for type "GLeglImageOES": YES Checking for type "GLchar": YES Checking for type "GLsizeiptr": YES Checking for type "GLintptr": YES Checking for type "GLsync": YES Checking for type "GLuint64": YES Checking for type "GLint64": YES Checking for type "EGLAttrib": YES Message: Building libgstgl with GL api: gles2 opengl Message: Building libgstgl with GL platform: egl glx Message: Building libgstgl with GL winsys: wayland x11 Configuring gstglconfig.h using configuration Native dependency bluez found: YES 5.47 Native dependency gio-unix-2.0 found: YES 2.54.0 Program gdbus-codegen found: YES (/usr/bin/gdbus-codegen) Library dl found: YES Dependency threads found: YES Message: Building decklink plugin Has header "linux/fb.h": YES Has header "sys/socket.h": YES Checking for function "pipe": YES Checking for function "socketpair": YES Library rt found: YES Has header "sys/socket.h": YES Native dependency gudev-1.0 found: YES 232 Native dependency libusb-1.0 found: YES 1.0.21 Has header "linux/uvcvideo.h": YES Native dependency libass found: YES 0.14.0 Library bz2 found: YES Header <bzlib.h> has symbol "BZ2_bzlibVersion": YES Native dependency libchromaprint found: YES 1.4.2 Native dependency libcurl found: YES 7.57.0 Native dependency libxml-2.0 found: YES 2.9.7 Dependency directfb found: NO Native dependency libcrypto found: YES 1.1.0g Native dependency libdca found: YES 0.0.5 Library faac found: YES Header <faac.h> has symbol "faacEncOpen": YES Header <neaacdec.h> has symbol "NeAACDecOpen": YES Header <neaacdec.h> has symbol "LATM": YES Library faad found: YES Native dependency fdk-aac found: YES 0.1.5 Library flite found: NO Library flite_cmu_us_kal found: NO Library flite_usenglish found: NO Library flite_cmulex found: NO Native dependency fluidsynth found: YES 1.1.6 Native dependency graphene-1.0 found: YES 1.6.0 Native dependency libpng found: YES 1.6.34 Library jpeg-mmx found: NO Library jpeg found: YES Has header "libdrm/drm_fourcc.h": YES Library gsm found: YES Header <gsm.h> has symbol "gsm_create": NO Header <gsm/smg.h> has symbol "gsm_create": NO Native dependency gtk+-3.0 found: YES 3.22.26 Native dependency gtk+-x11-3.0 found: YES 3.22.26 Native dependency gtk+-wayland-3.0 found: YES 3.22.26 Native dependency nettle found: YES 3.4 Dependency dssim found: NO Also couldn't find a fallback subproject in subprojects/dssim for the dependency dssim Native dependency kate found: YES 0.4.1 Dependency libde265 found: NO Native dependency libmms found: YES 0.6.4 Library mpeg2encpp found: YES Library mplex2 found: YES Native dependency opencv found: NO found '3.3.1' but need: '<= 3.3.0' ; matched: '>= 2.3.0' Dependency opencv found: NO Native dependency openh264 found: YES 1.7.0 Native dependency libopenjp2 found: YES 2.3.0 Native dependency opus found: YES 1.2.1 Native dependency dvdnav found: YES 5.0.3 Native dependency dvdread found: YES 5.0.4 Native dependency librsvg-2.0 found: YES 2.40.19 Native dependency librtmp found: YES v2.4 Native dependency sbc found: YES 1.3 Native dependency schroedinger-1.0 found: YES 1.0.11 Native dependency soundtouch found: YES 2.0.0 Native dependency spandsp found: YES 0.0.6 Dependency libsrt found: NO Header <srt/srt.h> has symbol "srt_startup": NO Native dependency libsrtp found: YES 1.5.4 Program srtp_mkenum.py found: YES (/home/fthiery/gst/gst-build/subprojects/gst-plugins-bad/ext/srtp/srtp_mkenum.py) Native dependency pango found: YES 1.40.13 Native dependency cairo found: YES 1.15.8 Native dependency pangocairo found: YES 1.40.13 Library m found: YES Dependency vo-aacenc found: NO Library vulkan found: YES Has header "vulkan/vulkan.h": YES Native dependency xcb found: YES 1.12 Configuring vkconfig.h using configuration Native dependency webrtc-audio-processing found: YES 0.3 Native dependency libwebp found: YES 0.6.0 Native dependency x265 found: YES 2.5 Native dependency zbar found: YES 0.10 Native dependency libexif found: YES 0.6.21 Header <faac.h> has symbol "faacEncOpen": YES Native dependency gstreamer-plugins-base-1.0 found: YES 1.13.0.1 Native dependency sdl found: YES 1.2.15 Configuring gstreamer-bad-audio-1.0.pc using configuration Configuring gstreamer-bad-audio-1.0-uninstalled.pc using configuration Configuring gstreamer-bad-base-1.0.pc using configuration Configuring gstreamer-bad-base-1.0-uninstalled.pc using configuration Configuring gstreamer-bad-video-1.0.pc using configuration Configuring gstreamer-bad-video-1.0-uninstalled.pc using configuration Configuring gstreamer-codecparsers-1.0.pc using configuration Configuring gstreamer-codecparsers-1.0-uninstalled.pc using configuration Configuring gstreamer-insertbin-1.0.pc using configuration Configuring gstreamer-insertbin-1.0-uninstalled.pc using configuration Configuring gstreamer-mpegts-1.0.pc using configuration Configuring gstreamer-mpegts-1.0-uninstalled.pc using configuration Configuring gstreamer-player-1.0.pc using configuration Configuring gstreamer-player-1.0-uninstalled.pc using configuration Configuring gstreamer-plugins-bad-1.0.pc using configuration Configuring gstreamer-plugins-bad-1.0-uninstalled.pc using configuration WARNING: The variable(s) 'waylandlibdir' in the input file pkgconfig/gstreamer-plugins-bad-uninstalled.pc.in are not present in the given configuration data Configuring gstreamer-gl-1.0.pc using configuration WARNING: The variable(s) 'GL_CFLAGS' in the input file pkgconfig/gstreamer-gl.pc.in are not present in the given configuration data Configuring gstreamer-gl-1.0-uninstalled.pc using configuration WARNING: The variable(s) 'GL_CFLAGS' in the input file pkgconfig/gstreamer-gl-uninstalled.pc.in are not present in the given configuration data Build targets in project: 202 Found ninja-1.8.2 at /usr/bin/ninja
(In reply to Florent Thiéry from comment #3) > I am using gst-build, which is the main difference. I don't have the issue > with stable (my distro packages), or the "legacy" gst-uninstalled way. > > All libgstcompositor.so link against /usr/lib/liborc-0.4.so.0 > Okay, thanks for checking > Here is the meson build output; are you noticing anything shady ? Not really, can you try bisecting with 1.12 as the first good commit?
I've also tested here with 1.12 and master, same result on X86 64, just like Mathieu. In fact, ORC_CODE=backup/emulate is in fact slightly faster so ORC is not the problem (it's making it slighly worst). You could also run with perf to identify the bottleneck, or try and GST_DEBUG=CAT_PERFORMANCE:5 (or 7).
> Not really, can you try bisecting with 1.12 as the first good commit? Well, gst-uninstalled does not exhibit the problem on the latest commit, so it looks more meson-related in my opinion. I'm running Arch, anyone else running it ? > GST_DEBUG=CAT_PERFORMANCE:5 Not seeing anything different with this enabled (5 or 7). perf record gst-launch-1.0 filesrc location=/dev/zero blocksize=3110400 num-buffers=900 ! queue name=vparse ! rawvideoparse width=1920 height=1080 framerate=30 format=nv12 ! queue name=compin ! compositor ! "video/x-raw, format=(string)NV12, width=(int)3840, height=(int)2160, framerate=(fraction)30, colorimetry=(string)bt709" ! queue name=fsink ! fakesink -v perf report shows: For gst-build: 25,66% vparse:src libc-2.26.so [.] __memmove_avx_unaligned_erms 18,29% compositor0:src libgstcompositor.so [.] fill_checker_nv12 10,91% gst-launch-1.0 libpthread-2.26.so [.] __pthread_rwlock_unlock 10,91% fsink:src libgstreamer-1.0.so.0.1300.0 [.] gst_memory_get_sizes 6,42% vparse:src [kernel.vmlinux] [.] native_irq_return_iret 5,77% fsink:src libgstbase-1.0.so.0.1300.0 [.] gst_base_sink_get_sync_times 5,13% filesrc0:src libgstcoreelements.so [.] gst_buffer_unmap@plt 4,49% compin:src libglib-2.0.so.0.5400.0 [.] g_cond_wait 4,49% compin:src libgstcoreelements.so [.] apply_buffer 3,85% filesrc0:src libc-2.26.so [.] _int_malloc 3,44% filesrc0:src libgstbase-1.0.so.0.1300.0 [.] gst_base_src_get_range 0,64% compositor0:src libc-2.26.so [.] __memset_avx2_erms For stable: 73,17% fsink:src libglib-2.0.so.0.5400.0 [.] 0x0000000000020353 6,86% vparse:src [kernel.vmlinux] [.] native_irq_return_iret 5,55% compositor0:src libgstcompositor.so [.] 0x00000000000038aa 3,27% gst-launch-1.0 libglib-2.0.so.0.5400.0 [.] g_mutex_lock 2,94% filesrc0:src libgstreamer-1.0.so.0.1203.0 [.] 0x00000000000a90b3 2,29% compin:src libgstbase-1.0.so.0.1203.0 [.] gst_queue_array_peek_head_struct 1,96% filesrc0:src libpthread-2.26.so [.] __pthread_mutex_lock 1,75% filesrc0:src libglib-2.0.so.0.5400.0 [.] g_mutex_lock 0,58% compositor0:src libc-2.26.so [.] __memmove_avx_unaligned_erms For gst-uninstalled: 41,53% fsink:src libgstbase-1.0.so.0.1300.0 [.] gst_base_sink_chain_unlocked.isr 38,89% compositor0:src libgstcompositor.so [.] fill_checker_nv12 4,37% filesrc0:src libgstreamer-1.0.so.0.1300.0 [.] gst_buffer_get_sizes_range 4,01% vparse:src libc-2.26.so [.] __memmove_avx_unaligned_erms 3,64% vparse:src libgstreamer-1.0.so.0.1300.0 [.] gst_mini_object_unlock 2,55% lt-gst-launch-1 libgstreamer-1.0.so.0.1300.0 [.] gst_registry_scan_path_level 2,55% filesrc0:src libpthread-2.26.so [.] __libc_read 2,19% filesrc0:src libglib-2.0.so.0.5400.0 [.] g_malloc This seems to indicate that the bottleneck is rawvideoparse instead of compositor, right ? But why does it only happen on gst-build (and not gst-uninstalled)?
From now on i'll be only comparing gst-uninstalled and gst-build (not stable). Removing compositor does not exhibit the problem (getting the same time in both): gst-launch-1.0 filesrc location=/dev/zero blocksize=3110400 num-buffers=9000 ! queue name=vparse ! rawvideoparse width=1920 height=1080 framerate=30 format=nv12 ! queue name=compin ! fakesink -v Even this reproduces the problem: gst-launch-1.0 videotestsrc num-buffers=100000 pattern=black ! compositor ! fakesink -v perf shows the following difference (see my attached png file, top is gst-build, bottom is gst-uninstalled): gst-build (18s): 10,20% compositor0:src libc-2.26.so [.] __memset_avx2_unaligned_erms gst-uninstalled (8s): 0,73% compositor0:src libc-2.26.so [.] __memset_avx2_unaligned_erms
Created attachment 365025 [details] perf output
When building, you can set V=1 as make argument. My impression is that -ON is different, where N is the delta. Meson always pick -O2. You could also test forcing -O2 for both.
Seems to be -O2 for both cases: make V=1 /usr/bin/orcc --compat 0.4.17 --implementation --include glib.h -o tmp-orc.c ./compositororc.orc libtool: compile: gcc -DHAVE_CONFIG_H -I. -I../.. -I../../gst-libs -I../../gst-libs -pthread -I/home/fthiery/gst/master/gstreamer -I/home/fthiery/gst/master/gstreamer/libs -I/home/fthiery/gst/master/gstreamer -I/home/fthiery/gst/master/gstreamer/libs -I/home/fthiery/gst/master/gst-plugins-base/gst-libs -I/usr/include/glib-2.0 -I/usr/lib/glib-2.0/include -pthread -I/home/fthiery/gst/master/gstreamer/libs -I/home/fthiery/gst/master/gstreamer -I/home/fthiery/gst/master/gstreamer/libs -I/home/fthiery/gst/master/gstreamer -I/home/fthiery/gst/master/gstreamer/libs -I/usr/include/glib-2.0 -I/usr/lib/glib-2.0/include -pthread -I/home/fthiery/gst/master/gstreamer -I/home/fthiery/gst/master/gstreamer/libs -I/home/fthiery/gst/master/gstreamer -I/home/fthiery/gst/master/gstreamer/libs -I/usr/include/glib-2.0 -I/usr/lib/glib-2.0/include -DGST_USE_UNSTABLE_API -DG_THREADS_MANDATORY -DG_DISABLE_DEPRECATED -Wall -Wdeclaration-after-statement -Wvla -Wpointer-arith -Wmissing-declarations -Wmissing-prototypes -Wredundant-decls -Wwrite-strings -Wformat-security -Wold-style-definition -Winit-self -Wmissing-include-dirs -Waddress -Wno-multichar -Wnested-externs -Werror -g -DGST_DISABLE_DEPRECATED -fvisibility=hidden -I/usr/include/orc-0.4 -g -O2 -MT libgstcompositor_la-compositor.lo -MD -MP -MF .deps/libgstcompositor_la-compositor.Tpo -c compositor.c -fPIC -DPIC -o .libs/libgstcompositor_la-compositor.o With meson: [84/828] /usr/bin/orcc --include glib.h --implementation -o gst/compositor/compositororc.c ../gst/compositor/compositororc.orc ... [185/828] cc -Igst/compositor/gstcompositor@sha -Igst/compositor -I../gst/compositor -I. -I../ -Igst-libs -I../gst-libs -I/home/fthiery/gst/gst-build/subprojects/gst-plugins-base/pkgconfig/../gst-libs -I/home/fthiery/gst/gst-build/build/subprojects/gst-plugins-base/pkgconfig/../gst-libs -I/home/fthiery/gst/gst-build/subprojects/gstreamer/pkgconfig/.. -I/home/fthiery/gst/gst-build/subprojects/gstreamer/pkgconfig/../libs -I/home/fthiery/gst/gst-build/build/subprojects/gstreamer/pkgconfig/.. -I/home/fthiery/gst/gst-build/build/subprojects/gstreamer/pkgconfig/../libs -I/usr/include/glib-2.0 -I/usr/lib/glib-2.0/include -I/usr/include/orc-0.4 -fdiagnostics-color=always -pipe -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -O2 -g -fvisibility=hidden -fPIC -pthread -DHAVE_CONFIG_H -DGST_USE_UNSTABLE_API -MMD -MQ 'gst/compositor/gstcompositor@sha/meson-generated_.._compositororc.c.o' -MF 'gst/compositor/gstcompositor@sha/meson-generated_.._composito:
Created attachment 365035 [details] meson verbose output
Created attachment 365036 [details] autotools verbose output
I can confirm this here on Debian/unstable: 7s vs. 15s.
Happens because meson by default does a debug build with -O0, while gst-uninstalled/autotools do -O2 by default.
I'm no expert but i see -O2 with ninja -v
I've pushed this fwiw: commit 8ff8df21121985d95a51585525cdfde5c8196871 Author: Tim-Philipp Müller <tim@centricular.com> Date: Wed Dec 6 10:58:23 2017 +0000 Build with buildtype debugoptimized by default https://bugzilla.gnome.org/show_bug.cgi?id=790989
Just built with: meson build -Denable_python=true -Ddisable_gstreamer_sharp=true -Ddisable_rtsp_server=true -Dbuildtype=debugoptimized It's even faster than stable now, thanks !
Correction, -Dbuildtype=debugoptimized does not work, the following does: meson --buildtype=debugoptimized -Denable_python=true -Ddisable_gstreamer_sharp=true -Ddisable_rtsp_server=true -Ddisable_gst_editing_services=true build