GNOME Bugzilla – Bug 736176
pngparse: re-assembling image from small blocks may be optimized
Last modified: 2014-11-09 23:41:19 UTC
$ rpm -q rhythmbox rhythmbox-3.0.3-4.fc21.x86_64 ^C Program received signal SIGINT, Interrupt. 0x00007fa40c4748a1 in __GI_ppoll (fds=0x15e6590, nfds=1, timeout=<optimized out>, sigmask=0x0) at ../sysdeps/unix/sysv/linux/ppoll.c:56 56 result = INLINE_SYSCALL (ppoll, 5, fds, nfds, timeout, sigmask, (gdb) thread apply all bt
+ Trace 234059
Thread 7 (Thread 0x7fa402ce2700 (LWP 8576))
Thread 6 (Thread 0x7fa3f2f0a700 (LWP 8522))
Thread 2 (Thread 0x7fa401e62700 (LWP 8470))
(gdb) (gdb) (gdb)
^C Program received signal SIGINT, Interrupt. 0x00007fa40c4748a1 in __GI_ppoll (fds=0x15f55b0, nfds=1, timeout=<optimized out>, sigmask=0x0) at ../sysdeps/unix/sysv/linux/ppoll.c:56 56 result = INLINE_SYSCALL (ppoll, 5, fds, nfds, timeout, sigmask, (gdb) thread apply all bt
+ Trace 234060
Thread 4 (Thread 0x7fa402ce2700 (LWP 7569))
Got object file from memory but can't read symbols: File truncated. 0x00007fa40c4748a1 in __GI_ppoll (fds=0x15f8fe0, nfds=1, timeout=<optimized out>, sigmask=0x0) at ../sysdeps/unix/sysv/linux/ppoll.c:56 56 result = INLINE_SYSCALL (ppoll, 5, fds, nfds, timeout, sigmask, (gdb) thread apply all bt
+ Trace 234061
Thread 3 (Thread 0x7fa3f2f0a700 (LWP 8408))
This is not a useful bug report. What am I supposed to do with these stack traces?
Ok, how make this report useful? What information you need? I hope what in stack trace you see why CPU high load. (gdb) thread apply all bt
+ Trace 234062
Thread 96 (Thread 0x7fff8b1dc700 (LWP 10140))
Thread 62 (Thread 0x7fffd8969700 (LWP 9967))
Thread 30 (Thread 0x7fff8b9dd700 (LWP 9908))
Thread 29 (Thread 0x7fffc9a81700 (LWP 9907))
Thread 27 (Thread 0x7fffa3fff700 (LWP 9905))
(gdb) (gdb)
Created attachment 285552 [details] screenshot of htop
(In reply to comment #4) > Ok, how make this report useful? What information you need? You could start by describing the problem. What does 'CPU high load' mean? Why do you think it's a problem rather than the normal state? Is it higher than it was in a previous version, and if so, what version? Does it take longer to import a given number of tracks than a previous version? Your stack traces seem to be from the metadata helper process. Is that the one that's using all the cpu time? How much is the main process using? > I hope what in stack trace you see why CPU high load. No, stack traces are not useful for this.
(In reply to comment #6) > You could start by describing the problem. What does 'CPU high load' mean? > Why do you think it's a problem rather than the normal state? Is it higher > than it was in a previous version, and if so, what version? Does it take > longer to import a given number of tracks than a previous version? I am attach htop screenshot. It demonstrate why I thought what CPU load is high. I see high cpu using by kernel (red bar) it's not normal.
Your htop screenshot shows CPU usage for the previous two seconds. Is it like that for the entire import process? Please answer the rest of my questions too.
(In reply to comment #8) > Your htop screenshot shows CPU usage for the previous two seconds. Is it like > that for the entire import process? Yes. I see what import process stuck on each png file, and CPU became high load. > Please answer the rest of my questions too. > Is it higher than it was in a previous version, and if so, what version? $ rpm -q rhythmbox rhythmbox-3.0.3-4.fc21.x86_64 I don't know about previous version because not use rhythmbox before.
Created attachment 285556 [details] htop demonstrate - rhythmbox normal import process
pngparse stuck import process maybe more informative.
Created attachment 285557 [details] htop - pngparse high load CPU
Screenshots from htop really aren't very helpful. (In reply to comment #9) > I see what import process stuck on each png file, and CPU became high load. Why is this information in comment 9 rather than the initial report?
(In reply to comment #13) > Screenshots from htop really aren't very helpful. > > (In reply to comment #9) > > I see what import process stuck on each png file, and CPU became high load. > > Why is this information in comment 9 rather than the initial report? Sorry, I hurried to capture the stack trace.
Can you find one of the problematic png files and run 'GST_DEBUG=*:3 GST_DEBUG_NO_COLOR=1 gst-discoverer-1.0 /path/to/file.png' on it, and attach the output here? (In reply to comment #14) > > Sorry, I hurried to capture the stack trace. It's not a race.
(In reply to comment #15) > Can you find one of the problematic png files and run 'GST_DEBUG=*:3 > GST_DEBUG_NO_COLOR=1 gst-discoverer-1.0 /path/to/file.png' on it, and attach > the output here? > $ GST_DEBUG_NO_COLOR=1 gst-discoverer-1.0 /home/mikhail/Music/Untitled\ Folder/Pet\ Shop\ Boys\ \(Japanese\ Press\)/1990\ -\ Behaviour/Scans/booklet2-3.png Analyzing file:///home/mikhail/Music/Untitled%20Folder/Pet%20Shop%20Boys%20(Japanese%20Press)/1990%20-%20Behaviour/Scans/booklet2-3.png Done discovering file:///home/mikhail/Music/Untitled%20Folder/Pet%20Shop%20Boys%20(Japanese%20Press)/1990%20-%20Behaviour/Scans/booklet2-3.png Analyzing URI timed out
https://mega.co.nz/#!v4lUwCxS!3ItNzCWzLXlpwIdwH388es1NVEiYFkZ8aSK3ECvN0Yg
So you have 4125x5490 PNG images and you're surprised it takes a while to read them?
(In reply to comment #18) > So you have 4125x5490 PNG images and you're surprised it takes a while to read > them? Shotwell and GNOME image viewer works with this image very fast.
reassigning to gstreamer in case they feel like looking at this. What versions of gstreamer, gst-plugins-base and gst-plugins-good do you have?
I confirm pngparse takes around 30 seconds, and pngdec around 10s. time gst-launch-1.0 filesrc location=my.png ! pngparse ! fakesink time gst-launch-1.0 filesrc location=my.png ! pngdec ! fakesink Though, if you increase the block size significantly, it's similar to other software: time gst-launch-1.0 filesrc blocksize=100000000 ! pngparse ! fakesink I think there is room for improvement for reassembling png from small blocks, though this seems rather low priority. The old 4K default block size is a bit out of date, application should most likely use large blocks.
(In reply to comment #20) > reassigning to gstreamer in case they feel like looking at this. What versions > of gstreamer, gst-plugins-base and gst-plugins-good do you have? [mikhail@localhost ~]$ rpm -q gstreamer gstreamer-0.10.36-10.fc21.x86_64 gstreamer-0.10.36-10.fc21.i686 [mikhail@localhost ~]$ rpm -q gstreamer-plugins-good gstreamer-plugins-good-0.10.31-13.fc21.x86_64 [mikhail@localhost ~]$ rpm -q gstreamer-plugins-base gstreamer-plugins-base-0.10.36-11.fc21.x86_64 gstreamer-plugins-base-0.10.36-11.fc21.i686
(In reply to comment #22) > (In reply to comment #20) > > reassigning to gstreamer in case they feel like looking at this. What versions > > of gstreamer, gst-plugins-base and gst-plugins-good do you have? > > [mikhail@localhost ~]$ rpm -q gstreamer > gstreamer-0.10.36-10.fc21.x86_64 > gstreamer-0.10.36-10.fc21.i686 > [mikhail@localhost ~]$ rpm -q gstreamer-plugins-good > gstreamer-plugins-good-0.10.31-13.fc21.x86_64 > [mikhail@localhost ~]$ rpm -q gstreamer-plugins-base > gstreamer-plugins-base-0.10.36-11.fc21.x86_64 > gstreamer-plugins-base-0.10.36-11.fc21.i686 This version will remain the way it is, it's dead/unmaintained. Rhythmbox uses gstreamer 1.X, though no need to provide us with the version you are running on Rawhide, we can figure-it out by ourself. Nevertheless, thanks for reporting, we'll try and improve this in the feature, even though this isn't high priority. It is up to you to try and convince rhythmbox maintainer to increase the filesrc block size.
Rhythmbox is just using GstDiscoverer at the point where this is happening. If setting a larger block size makes it work better in general, then GstDiscoverer should do that itself.
(In reply to comment #24) > Rhythmbox is just using GstDiscoverer at the point where this is happening. If > setting a larger block size makes it work better in general, then GstDiscoverer > should do that itself. This or fix this bug, whoever provide a patch first I suppose.
I have a patch for this.
Is there any chance that pngdec got improved with the patch in below bug? https://bugzilla.gnome.org/show_bug.cgi?id=737708
commit 6e3518dfd68d775d530bb6c4835b3e0b0e224e16 Author: Tim-Philipp Müller <tim@centricular.com> Date: Sun Nov 9 20:53:34 2014 +0000 pngparse: optimise reading of png files Read PNG data chunk in one go by letting the parser base class know the size we need, so that it doesn't drip-feed us small chunks of data (causing a lot of reallocs and memcpy in the process) until we have everything. Improves parsing performance of very large PNG files (65MB) from ~13 seconds to a couple of millisecs. https://bugzilla.gnome.org/show_bug.cgi?id=736176