Bug 778090 – PNG extractor decompresses images in memory

After an evaluation, GNOME has moved from Bugzilla to GitLab. Learn more about GitLab.
No new issues can be reported in GNOME Bugzilla anymore.
To report an issue in a GNOME project, go to GNOME GitLab.
Do not go to GNOME Gitlab for: Bluefish, Doxygen, GnuCash, GStreamer, java-gnome, LDTP, NetworkManager, Tomboy.

Bug 778090 - PNG extractor decompresses images in memory


Summary:	PNG extractor decompresses images in memory


Status:	RESOLVED FIXED

Product:	tracker
Classification:	Core
Component:	Extractor
Version:	unspecified
Hardware:	Other Linux

Importance:	Normal normal
Target Milestone:	---
Assigned To:	tracker-extractor
QA Contact:	tracker-extractor

URL:
Whiteboard:

Depends on:
Blocks:

Reported:	2017-02-02 17:33 UTC by Mikhail
Modified:	2017-03-08 10:41 UTC

See Also:
GNOME target:	---
GNOME version:	---

Attachments
htop screenshot (326.34 KB, image/png) 2017-02-02 17:33 UTC, Mikhail		Details
tracker extract backtrace (25.71 KB, text/plain) 2017-02-02 17:36 UTC, Mikhail		Details
tracker-extract: Avoid allocating full image in memory at once (1.78 KB, patch) 2017-02-28 18:38 UTC, Carlos Garnacho	committed	Details \| Review
tracker-extract backtrace (24.94 KB, text/plain) 2017-03-08 08:58 UTC, Mikhail		Details

Description Mikhail 2017-02-02 17:33:56 UTC

Created attachment 344798 [details]
htop screenshot

tracker-extract process DDOS my system

(gdb) thread apply all bt

+ Trace 237104

Thread 28 (Thread 0x7f7c7e169700 (LWP 12217))

#0 syscall
at ../sysdeps/unix/sysv/linux/x86_64/syscall.S line 38
#1 g_cond_wait
at gthread-posix.c line 1395
#2 g_async_queue_pop_intern_unlocked
at gasyncqueue.c line 419
#3 g_async_queue_pop
at gasyncqueue.c line 453
#4 single_thread_get_metadata
at tracker-extract.c line 533
#5 g_thread_proxy
at gthread.c line 784
#6 start_thread
at pthread_create.c line 333
#7 clone
at ../sysdeps/unix/sysv/linux/x86_64/clone.S line 105

Thread 27 (Thread 0x7f7c813ab700 (LWP 12158))

#0 syscall
at ../sysdeps/unix/sysv/linux/x86_64/syscall.S line 38
#1 g_cond_wait
at gthread-posix.c line 1395
#2 g_async_queue_pop_intern_unlocked
at gasyncqueue.c line 419
#3 g_async_queue_pop
at gasyncqueue.c line 453
#4 single_thread_get_metadata
at tracker-extract.c line 533
#5 g_thread_proxy
at gthread.c line 784
#6 start_thread
at pthread_create.c line 333
#7 clone
at ../sysdeps/unix/sysv/linux/x86_64/clone.S line 105

Thread 26 (Thread 0x7f7c82ffd700 (LWP 12155))

#0 syscall
at ../sysdeps/unix/sysv/linux/x86_64/syscall.S line 38
#1 g_cond_wait
at gthread-posix.c line 1395
#2 g_async_queue_pop_intern_unlocked
at gasyncqueue.c line 419
#3 g_async_queue_pop
at gasyncqueue.c line 453
#4 single_thread_get_metadata
at tracker-extract.c line 533
#5 g_thread_proxy
at gthread.c line 784
#6 start_thread
at pthread_create.c line 333
#7 clone
at ../sysdeps/unix/sysv/linux/x86_64/clone.S line 105

Thread 25 (Thread 0x7f7c83fff700 (LWP 12137))

#0 syscall
at ../sysdeps/unix/sysv/linux/x86_64/syscall.S line 38
#1 g_cond_wait
at gthread-posix.c line 1395
#2 g_async_queue_pop_intern_unlocked
at gasyncqueue.c line 419
#3 g_async_queue_pop
at gasyncqueue.c line 453
#4 single_thread_get_metadata
at tracker-extract.c line 533
#5 g_thread_proxy
at gthread.c line 784
#6 start_thread
at pthread_create.c line 333
#7 clone
at ../sysdeps/unix/sysv/linux/x86_64/clone.S line 105

Thread 24 (Thread 0x7f7cb10e3700 (LWP 12134))

#0 syscall
at ../sysdeps/unix/sysv/linux/x86_64/syscall.S line 38
#1 g_cond_wait
at gthread-posix.c line 1395
#2 g_async_queue_pop_intern_unlocked
at gasyncqueue.c line 419
#3 g_async_queue_pop
at gasyncqueue.c line 453
#4 single_thread_get_metadata
at tracker-extract.c line 533
#5 g_thread_proxy
at gthread.c line 784
#6 start_thread
at pthread_create.c line 333
#7 clone
at ../sysdeps/unix/sysv/linux/x86_64/clone.S line 105

Thread 23 (Thread 0x7f7cb290c700 (LWP 12133))

#0 syscall
at ../sysdeps/unix/sysv/linux/x86_64/syscall.S line 38
#1 g_cond_wait
at gthread-posix.c line 1395
#2 g_async_queue_pop_intern_unlocked
at gasyncqueue.c line 419
#3 g_async_queue_pop
at gasyncqueue.c line 453
#4 single_thread_get_metadata
at tracker-extract.c line 533
#5 g_thread_proxy
at gthread.c line 784
#6 start_thread
at pthread_create.c line 333
#7 clone
at ../sysdeps/unix/sysv/linux/x86_64/clone.S line 105

Thread 22 (Thread 0x7f7cb3737700 (LWP 12132))

#0 syscall
at ../sysdeps/unix/sysv/linux/x86_64/syscall.S line 38
#1 g_cond_wait
at gthread-posix.c line 1395
#2 g_async_queue_pop_intern_unlocked
at gasyncqueue.c line 419
#3 g_async_queue_pop
at gasyncqueue.c line 453
#4 single_thread_get_metadata
at tracker-extract.c line 533
#5 g_thread_proxy
at gthread.c line 784
#6 start_thread
at pthread_create.c line 333
#7 clone
at ../sysdeps/unix/sysv/linux/x86_64/clone.S line 105

Thread 21 (Thread 0x7f7cd2140700 (LWP 12131))

#0 syscall
at ../sysdeps/unix/sysv/linux/x86_64/syscall.S line 38
#1 g_cond_wait
at gthread-posix.c line 1395
#2 g_async_queue_pop_intern_unlocked
at gasyncqueue.c line 419
#3 g_async_queue_pop
at gasyncqueue.c line 453
#4 single_thread_get_metadata
at tracker-extract.c line 533
#5 g_thread_proxy
at gthread.c line 784
#6 start_thread
at pthread_create.c line 333
#7 clone
at ../sysdeps/unix/sysv/linux/x86_64/clone.S line 105

Thread 20 (Thread 0x7f7cb18e4700 (LWP 12115))

#0 syscall
at ../sysdeps/unix/sysv/linux/x86_64/syscall.S line 38
#1 g_cond_wait
at gthread-posix.c line 1395
#2 g_async_queue_pop_intern_unlocked
at gasyncqueue.c line 419
#3 g_async_queue_pop
at gasyncqueue.c line 453
#4 single_thread_get_metadata
at tracker-extract.c line 533
#5 g_thread_proxy
at gthread.c line 784
#6 start_thread
at pthread_create.c line 333
#7 clone
at ../sysdeps/unix/sysv/linux/x86_64/clone.S line 105

Thread 19 (Thread 0x7f7cb20e5700 (LWP 12100))

#0 syscall
at ../sysdeps/unix/sysv/linux/x86_64/syscall.S line 38
#1 g_cond_wait
at gthread-posix.c line 1395
#2 g_async_queue_pop_intern_unlocked
at gasyncqueue.c line 419
#3 g_async_queue_pop
at gasyncqueue.c line 453
#4 single_thread_get_metadata
at tracker-extract.c line 533
#5 g_thread_proxy
at gthread.c line 784
#6 start_thread
at pthread_create.c line 333
#7 clone
at ../sysdeps/unix/sysv/linux/x86_64/clone.S line 105

Thread 18 (Thread 0x7f7cb673d700 (LWP 12099))

#0 syscall
at ../sysdeps/unix/sysv/linux/x86_64/syscall.S line 38
#1 g_cond_wait
at gthread-posix.c line 1395
#2 g_async_queue_pop_intern_unlocked
at gasyncqueue.c line 419
#3 g_async_queue_pop
at gasyncqueue.c line 453
#4 single_thread_get_metadata
at tracker-extract.c line 533
#5 g_thread_proxy
at gthread.c line 784
#6 start_thread
at pthread_create.c line 333
#7 clone
at ../sysdeps/unix/sysv/linux/x86_64/clone.S line 105

Thread 17 (Thread 0x7f7cb5f3c700 (LWP 12098))

#0 _int_malloc
at malloc.c line 3786
#1 __GI___libc_malloc
at malloc.c line 2925
#2 png_malloc
at pngmem.c line 179
#3 tracker_extract_get_metadata
at tracker-extract-png.c line 692
#4 get_file_metadata
at tracker-extract.c line 326
#5 get_metadata
at tracker-extract.c line 510
#6 single_thread_get_metadata
at tracker-extract.c line 538
#7 g_thread_proxy
at gthread.c line 784
#8 start_thread
at pthread_create.c line 333
#9 clone
at ../sysdeps/unix/sysv/linux/x86_64/clone.S line 105

Thread 16 (Thread 0x7f7cd3fff700 (LWP 12054))

#0 syscall
at ../sysdeps/unix/sysv/linux/x86_64/syscall.S line 38
#1 g_cond_wait
at gthread-posix.c line 1395
#2 g_async_queue_pop_intern_unlocked
at gasyncqueue.c line 419
#3 g_async_queue_pop
at gasyncqueue.c line 453
#4 single_thread_get_metadata
at tracker-extract.c line 533
#5 g_thread_proxy
at gthread.c line 784
#6 start_thread
at pthread_create.c line 333
#7 clone
at ../sysdeps/unix/sysv/linux/x86_64/clone.S line 105

Thread 15 (Thread 0x7f7ce1ffb700 (LWP 12053))

#0 syscall
at ../sysdeps/unix/sysv/linux/x86_64/syscall.S line 38
#1 g_cond_wait
at gthread-posix.c line 1395
#2 g_async_queue_pop_intern_unlocked
at gasyncqueue.c line 419
#3 g_async_queue_pop
at gasyncqueue.c line 453
#4 single_thread_get_metadata
at tracker-extract.c line 533
#5 g_thread_proxy
at gthread.c line 784
#6 start_thread
at pthread_create.c line 333
#7 clone
at ../sysdeps/unix/sysv/linux/x86_64/clone.S line 105

Thread 14 (Thread 0x7f7ce2ffd700 (LWP 12046))

#0 syscall
at ../sysdeps/unix/sysv/linux/x86_64/syscall.S line 38
#1 g_cond_wait
at gthread-posix.c line 1395
#2 g_async_queue_pop_intern_unlocked
at gasyncqueue.c line 419
#3 g_async_queue_pop_unlocked
at gasyncqueue.c line 475
#4 g_thread_pool_wait_for_new_task
at gthreadpool.c line 245
#5 g_thread_pool_thread_proxy
at gthreadpool.c line 296
#6 g_thread_proxy
at gthread.c line 784
#7 start_thread
at pthread_create.c line 333
#8 clone
at ../sysdeps/unix/sysv/linux/x86_64/clone.S line 105

Thread 13 (Thread 0x7f7ce37fe700 (LWP 12045))

#0 syscall
at ../sysdeps/unix/sysv/linux/x86_64/syscall.S line 38
#1 g_cond_wait
at gthread-posix.c line 1395
#2 g_async_queue_pop_intern_unlocked
at gasyncqueue.c line 419
#3 g_async_queue_pop_unlocked
at gasyncqueue.c line 475
#4 g_thread_pool_wait_for_new_task
at gthreadpool.c line 245
#5 g_thread_pool_thread_proxy
at gthreadpool.c line 296
#6 g_thread_proxy
at gthread.c line 784
#7 start_thread
at pthread_create.c line 333
#8 clone
at ../sysdeps/unix/sysv/linux/x86_64/clone.S line 105

Thread 12 (Thread 0x7f7ce3fff700 (LWP 12044))

#0 syscall
at ../sysdeps/unix/sysv/linux/x86_64/syscall.S line 38
#1 g_cond_wait
at gthread-posix.c line 1395
#2 g_async_queue_pop_intern_unlocked
at gasyncqueue.c line 419
#3 g_async_queue_pop_unlocked
at gasyncqueue.c line 475
#4 g_thread_pool_wait_for_new_task
at gthreadpool.c line 245
#5 g_thread_pool_thread_proxy
at gthreadpool.c line 296
#6 g_thread_proxy
at gthread.c line 784
#7 start_thread
at pthread_create.c line 333
#8 clone
at ../sysdeps/unix/sysv/linux/x86_64/clone.S line 105

Thread 11 (Thread 0x7f7ce8ff9700 (LWP 12043))

#0 syscall
at ../sysdeps/unix/sysv/linux/x86_64/syscall.S line 38
#1 g_cond_wait
at gthread-posix.c line 1395
#2 g_async_queue_pop_intern_unlocked
at gasyncqueue.c line 419
#3 g_async_queue_pop_unlocked
at gasyncqueue.c line 475
#4 g_thread_pool_wait_for_new_task
at gthreadpool.c line 245
#5 g_thread_pool_thread_proxy
at gthreadpool.c line 296
#6 g_thread_proxy
at gthread.c line 784
#7 start_thread
at pthread_create.c line 333
#8 clone
at ../sysdeps/unix/sysv/linux/x86_64/clone.S line 105

Thread 10 (Thread 0x7f7ce97fa700 (LWP 12042))

#0 syscall
at ../sysdeps/unix/sysv/linux/x86_64/syscall.S line 38
#1 g_cond_wait
at gthread-posix.c line 1395
#2 g_async_queue_pop_intern_unlocked
at gasyncqueue.c line 419
#3 g_async_queue_pop_unlocked
at gasyncqueue.c line 475
#4 g_thread_pool_wait_for_new_task
at gthreadpool.c line 245
#5 g_thread_pool_thread_proxy
at gthreadpool.c line 296
#6 g_thread_proxy
at gthread.c line 784
#7 start_thread
at pthread_create.c line 333
#8 clone
at ../sysdeps/unix/sysv/linux/x86_64/clone.S line 105

Thread 9 (Thread 0x7f7ce9ffb700 (LWP 12041))

#0 syscall
at ../sysdeps/unix/sysv/linux/x86_64/syscall.S line 38
#1 g_cond_wait
at gthread-posix.c line 1395
#2 g_async_queue_pop_intern_unlocked
at gasyncqueue.c line 419
#3 g_async_queue_pop_unlocked
at gasyncqueue.c line 475
#4 g_thread_pool_wait_for_new_task
at gthreadpool.c line 245
#5 g_thread_pool_thread_proxy
at gthreadpool.c line 296
#6 g_thread_proxy
at gthread.c line 784
#7 start_thread
at pthread_create.c line 333
#8 clone
at ../sysdeps/unix/sysv/linux/x86_64/clone.S line 105

Thread 8 (Thread 0x7f7cea7fc700 (LWP 12040))

#0 syscall
at ../sysdeps/unix/sysv/linux/x86_64/syscall.S line 38
#1 g_cond_wait
at gthread-posix.c line 1395
#2 g_async_queue_pop_intern_unlocked
at gasyncqueue.c line 419
#3 g_async_queue_pop_unlocked
at gasyncqueue.c line 475
#4 g_thread_pool_wait_for_new_task
at gthreadpool.c line 245
#5 g_thread_pool_thread_proxy
at gthreadpool.c line 296
#6 g_thread_proxy
at gthread.c line 784
#7 start_thread
at pthread_create.c line 333
#8 clone
at ../sysdeps/unix/sysv/linux/x86_64/clone.S line 105

Thread 7 (Thread 0x7f7ceaffd700 (LWP 12039))

#0 syscall
at ../sysdeps/unix/sysv/linux/x86_64/syscall.S line 38
#1 g_cond_wait
at gthread-posix.c line 1395
#2 g_async_queue_pop_intern_unlocked
at gasyncqueue.c line 419
#3 g_async_queue_pop_unlocked
at gasyncqueue.c line 475
#4 g_thread_pool_wait_for_new_task
at gthreadpool.c line 245
#5 g_thread_pool_thread_proxy
at gthreadpool.c line 296
#6 g_thread_proxy
at gthread.c line 784
#7 start_thread
at pthread_create.c line 333
#8 clone
at ../sysdeps/unix/sysv/linux/x86_64/clone.S line 105

Thread 6 (Thread 0x7f7ceb7fe700 (LWP 12038))

#0 syscall
at ../sysdeps/unix/sysv/linux/x86_64/syscall.S line 38
#1 g_cond_wait
at gthread-posix.c line 1395
#2 g_async_queue_pop_intern_unlocked
at gasyncqueue.c line 419
#3 g_async_queue_pop_unlocked
at gasyncqueue.c line 475
#4 g_thread_pool_wait_for_new_task
at gthreadpool.c line 245
#5 g_thread_pool_thread_proxy
at gthreadpool.c line 296
#6 g_thread_proxy
at gthread.c line 784
#7 start_thread
at pthread_create.c line 333
#8 clone
at ../sysdeps/unix/sysv/linux/x86_64/clone.S line 105

Thread 5 (Thread 0x7f7cebfff700 (LWP 12037))

#0 syscall
at ../sysdeps/unix/sysv/linux/x86_64/syscall.S line 38
#1 g_cond_wait
at gthread-posix.c line 1395
#2 g_async_queue_pop_intern_unlocked
at gasyncqueue.c line 419
#3 g_async_queue_pop_unlocked
at gasyncqueue.c line 475
#4 g_thread_pool_wait_for_new_task
at gthreadpool.c line 245
#5 g_thread_pool_thread_proxy
at gthreadpool.c line 296
#6 g_thread_proxy
at gthread.c line 784
#7 start_thread
at pthread_create.c line 333
#8 clone
at ../sysdeps/unix/sysv/linux/x86_64/clone.S line 105

(gdb) 
(gdb)

Comment 1 Mikhail 2017-02-02 17:36:29 UTC

Created attachment 344799 [details]
tracker extract backtrace

Comment 2 Mikhail 2017-02-02 17:45:08 UTC

$ tracker --version
Tracker 1.10.4

Comment 3 Carlos Garnacho 2017-02-28 16:52:15 UTC

Thanks for the bug report, and the backtrace.

Thread 17 is the interesting one here. That path might indeed result expensive with huge PNGs, as we have to decompress the image in memory in order to access the metadata at the end of the file. I wonder if we can serialize the parsing somehow so we don't need allocating that much memory at once.

Comment 4 Carlos Garnacho 2017-02-28 18:38:19 UTC

Pushed a fix to git master.

The following fix has been pushed:
d54665c tracker-extract: Avoid allocating full image in memory at once

Comment 5 Carlos Garnacho 2017-02-28 18:38:25 UTC

Created attachment 346928 [details] [review]
tracker-extract: Avoid allocating full image in memory at once

Just read it line by line reusing the same buffer, it will still
let us to fetch metadata at the end of the file.

Comment 6 Mikhail 2017-03-08 08:58:44 UTC

Created attachment 347449 [details]
tracker-extract backtrace

Comment 7 Mikhail 2017-03-08 08:58:53 UTC

 Carlos, thanks. Can you also look at fresh backtrace. It same issue or nor?

Comment 8 Carlos Garnacho 2017-03-08 10:41:32 UTC

Yes, this seems the same issue. The backtrace is captured at a slightly later time, after it allocated memory for all image rows, while it is decompressing the image data in these.

I think at that point not much extra memory is needed, although this step could be cpu intensive with huge images.