GNOME Bugzilla – Bug 767051
image import must be much faster
Last modified: 2021-05-19 14:50:41 UTC
Ok, its not a really good bug report, but for me in order to be able to use shotwell the image import must be much faster. My collection right now is at 124'691 pictures ~450 GB and shotwell doesn't manage to import them even within one full day. and that on decent hardware i7 8GB ram. In addidtion it should also be able to to recognize changings / new content within my image folder fast. (lets say within 2 mins) - so i don't need to import every new folder manually. this two enhancements would make the statement: "shotwell is a programm representing my image library" two at most times (sometimes with 2 mins delay) - which is like the base i need from a programm to manage my pictures. i also see there are much more serious bugs. (things destroying the whole collection - so no worry fix that first) but i would love if my request would be the direction we want to head to. i have no idea but maybe it would also be woth thinking of rewriting the whole import function. in addition for comparison i would very much like the performance picasa has - i just don't have any idea about how they achieved it..
I'm very much concerned about startup and import performance. I consider it a part of user experience. If you could provide a sysprof profile (https://git.gnome.org/browse/sysprof/, https://blogs.gnome.org/chergert/2016/04/19/how-to-sysprof/ ) or a callgrind profile (part of valgrind, less intuitive) of your startup / the import that would be very helpful
HI ok, i installed sysprof and i pointed it to shotwell (after started) and imported one folder and the stopped. as a test, but the log file is already bigger than the upload limit here.. but anyway what exactly do you want me to to? should i delete my entire librabry (how would i do that) and then log the attempt of importing my whole library?) thanks for further advise - also about where putting the data.
Actually that would be sufficient for a first check. How big is the file? I will try to figure out a place
Did you import JPEGs or RAW files?
hey sorry for the delay, its just about 5 mb. i have also infrastructure to send larger files to you, but only in private i dont want to expose it to the world. because also this little test semed quite ok in speed. so i think i will need to generate much bigger files. to if i can reach you somehow directly would be great - or some public provider for data storage would be ok as well. i did import a mix of files. mainly jpeg, secondly video files, and some very rare raw files i own as well.
ok that was not quite correct, since the log seems to include a lot of information. and i dont' know how much about my system exactly ( at least my username) i would really much prefer to send it to you directly without posting it to the public.
did some testing: -10 first folders of my library - 2.7 GB - 1'514 Objects according to nemo - 980 fotos and videos according to shotwell - 969 pictures according to picasa (on wine) ( i guess the missing 11 are videos that picasa can't handle) picasa: ~ 40 seconds shotwell: ~ 69 seconds diffrence does not look major but it gets close to double. but also picasa did not recognize a single video. i can make more of these tests / comparisons if you like with more data. i did just delete the shotwell library in order to get a fresh one and set up a new picasa using play on linux. using picasa 3.9 and shotwell 0.23.1
Are you comfortable with testing a patch?
Hi, yess of Course :D i don't know what that would involve actually but i'm motivated to to it. i should be able to compile from source, and i should also be able to run it agains my whole library since i use btrfs snapshots i will run a comparison from before and after import so i will be able to see if it has broken/deletet/corrupted my librabry or such. ( if that might be the case)
Created attachment 331167 [details] [review] Minor changes to improve import speed The patch is more a test, it's not really suitable for committing yet, contains various ideas that need more fleshing out
Can you test this? Patch is against git master. If you need details on builing, feel free to contact me by mail directly
I think we spend way to much time decoding JPEG files here...
Yep. Half of the time of import we spend on JPEG reading, the other on image scaling.
-- GitLab Migration Automatic Message -- This bug has been migrated to GNOME's GitLab instance and has been closed from further activity. You can subscribe and participate further through the new bug through this link to our GitLab instance: https://gitlab.gnome.org/GNOME/shotwell/-/issues/4720.