GNOME Bugzilla – Bug 736961
shotwell loses photos edits
Last modified: 2021-05-19 14:34:34 UTC
I've been using shotwell for over three years. Recently I updated to Ubuntu 14.04.1 and shotwell 0.18.1. Since these upgrades, I've edited two batches of photos like I have many times in the past. Each time I've come back to shotwell later only to find that some of my edits have disappeared. This could be crop, straighten, enhance, adjust. The first time I wrote it off as a fluke, but it's just happened again. No, I can't reproduce the problem. I understand that the edits are stored in a db (mysql?), but I haven't tried to go into the database to see what's going on (or not). I have edited thousands of photos with shotwell, and this problem only started in the last month after the upgrades I mentioned. This is extremely serious for me, essentially making shotwell unusable.
Note that I am storing everything on a local hard drive where the db is also running. I have not messed with the db, the location of disks, or anything like this. Each time that I have lost edits, I've edited a batch of photos at the same time and only lost edits for *some* of the photos, not all. I am doing the same workflow that has worked for three years without any problems until now.
I've not heard anyone report this before. It would help if we had a debug log. The next time this happens, if you could close Shotwell and attach to this ticket the file at ~/.cache/shotwell/shotwell.log, that would be very helpful. (If you restart Shotwell, the file will be overwritten, so careful.) Also, is it possible your home directory drive is full or almost full? If so, that will interfere with database operations in unexpected ways.
No, my home directory is not full. I have 64G available space. I'll see if I can get a debug log. So far I've been doing very short editing sessions and haven't been able to reproduce the problem. Is it worth looking at the db and the history of a photo that I'm sure I edited twice?
Because I was curious, I went into the db and found a photo that I had to edit twice. However, I doesn't seem like I get any useful here, do I? sqlite> select filename, transformations from phototable where filename like '%24/DSCN7294%'; /home/bryan/Pictures/2014/08/24/DSCN7294.JPG|[straighten] angle=-2.0000000000000004 [crop] left=80 top=76 bottom=1704 right=2250 [adjustments] expansion={ 24, 236 } tint=0 highlights=0 shadows=0 exposure=0 temperature=0 saturation=4.38356
That looks pretty normal to me. Were those the edits you made on that picture (color adjust/enhance, crop, straighten)?
(In reply to Jim Nelson from comment #5) > That looks pretty normal to me. Were those the edits you made on that > picture (color adjust/enhance, crop, straighten)? Yes, those are the edits. Unbelievable. I essentially quit using Shotwell six months ago when this problem made shotwell unusable for me. I then forgot about the problem, started using shotwell again yesterday, edited a bunch of photos yesterday and today, and once again many of those edits are lost. Is no one else having this problem? And once again, still no debug file. :(
And now my shotwell version is 0.20.2.
Well, I don't know if this will help at all, but for some reason I have a copy of the db file from this morning: bryan@bryan-900HA:~/.local/share/shotwell/data$ ls -l total 21400 -rw-r--r-- 1 bryan bryan 10959872 Apr 6 16:06 photo.db -rw-r--r-- 1 bryan bryan 10945536 Apr 6 10:05 photo.db.bak which means I am able to see if edits that I thought I made yesterday existed in the backup from this morning. First I used the backup, and, yes, the edits are there. In the most recent version of the db those edits are gone. sqlite> .open photo.db.bak sqlite> select filename, transformations from phototable where filename like '%13/DSCN8295%'; /home/bryan/Pictures/2015/02/13/DSCN8295.JPG|[adjustments] expansion={ 29, 221 } tint=0 highlights=-3.50685 shadows=0 exposure=0 temperature=0 saturation=5.26027 sqlite> .open photo.db sqlite> select filename, transformations from phototable where filename like '%13/DSCN8295%'; /home/bryan/Pictures/2015/02/13/DSCN8295.JPG| sqlite> From your point of view, this probably isn't helpful. It's certainly frustrating. One thing I did do is add tags to the whole set of photos. This then takes some time "writing metadata to file" or something like that. I didn't wait for that progress bar to finish. I continued using shotwell. Could something get messed up if you don't wait for that writing process to finish? To me that doesn't seem possible since there shouldn't be any updates to the phototable at all...
Created attachment 301062 [details] shotwell log gile I was able to reproduce the problem. In a single event (set of photos) I made the edits (transformations) to the photos that I wanted to edit. I then closed shotwell (and backed up my db) and reopened shotwell. Then I started messing around with tags quickly, sometimes on many of the photos in the event. After a couple edits, there was a status bar that slowly moving along saying "writing metadata" or something like that. I kept writing new tags at this point. When the status bar was finished, I checked my adjustments/transformations, and 40 of them (out of 95) were gone! I have attached the debug as Jim suggested. It seems like I can reproduce this problem so if there's something further I can do to help figure this out, please let me know. I have 49G free space on my /home drive.
Created attachment 301263 [details] shotwell log file Here's another log file. The problem happened again today. I lost 20 transformations that I had just made. I lost them while I was adding tags to the photos in the set, just like last time. It does seem to be a reproducible problem.
Created attachment 301266 [details] shotwell log file -- upon only opening shotwell Sorry to keep adding the log files, but this may be helpful. After making my transformations to my photos, I made a copy of my db (using sqlite3 .backup) so that in case I lost my edits I could get them back. I did indeed lose those edits (as I reported in a previous comment) so I exited shotwell and replaced the photo.db with the one I had backed up. Before opening shotwell I checked to see that I had all the transformations in the event that I've been working on: sqlite> select count(filename) from PhotoTable where event_id = 1004 and (transformations is null or transformations = ''); 0 Indeed there were no missing transformations. I opened shotwell, waited for it to do its thing (there's an "updating library" process it goes through) and then checked the db again: sqlite> select count(filename) from PhotoTable where event_id = 1004 and (transformations is null or transformations = ''); 1 One transformation is gone. I didn't do anything except open shotwell. This log file is the one I've called "upon only opening shotwell".
It keeps getting better. To follow on my last comment, I navigated (using the tree) to the event I'm working on, and by the time I opened the event 24 (out of a total of 25 photos in the event) were missing transformations: sqlite> select count(filename) from PhotoTable where event_id = 1004 and (transformations is null or transformations = ''); 24 sqlite> select count(filename) from PhotoTable where event_id = 1004; 25 I'm not bothering to attach that log file as there seems to be no new information in it.
Bryan, where is your photo.db located? Is it on a network drive or an externally-connected drive? Are you sure you have proper read-write permissions for the file? Are you the owner of the file?
My photo.db is here: /home/bryan/.local/share/shotwell/data/photo.db No, I do not have a network drive or an externally connected drive. I am the owner of the file, and the owner has write permissions: bryan@bryan-900HA:~/.local/share/shotwell/data$ pwd /home/bryan/.local/share/shotwell/data bryan@bryan-900HA:~/.local/share/shotwell/data$ ls -l photo.db -rw-r--r-- 1 bryan bryan 10985472 Apr 10 17:23 photo.db bryan@bryan-900HA:~/.local/share/shotwell/data$ I have a pretty simple setup, just a small netbook. I am the only user though there is a rarely used guest account. I have root access that I use through sudo occasionally, but that's not necessary with shotwell.
I'm definitely finding a pattern. I was able to a few events worth of photos without any problem. When the scroll bar that says "writing metadata" was slowly scrolling across, I didn't touch shotwell. Just now I tagged about 25 photos in an event with 70. The scroll bar didn't start immediately, and I moved (with the keyboard) off the 25 photos to some other photos. However, the scroll bar (I think that's the right term) was moving then. Immediately I checked, and indeed I lost transformations. This seems to be an easy way for me to reproduce the problem.
Theory: When writing meta-data, it could be that the writing of meta-data triggers re-import which then might discard the edits.
Jens, is there anything I can do to help confirm your theory? I just had this problem happen again yesterday. I saw your theory just now as I received about this bug this morning.
Let me check if I can find any log entry that might indicate that this is happening
I'm sorry, that took a bit - if you are still interested and can rebuild shotwell yourself: Please enable the trace for directory monitoring and metadata-writing. On shotwell 0.28 and later, you can do that with meson build -Dtrace=monitoring,metadata-writer You should see a line like L 29293 2018-08-23 15:08:24 [DBG] DirectoryMonitor.vala:472: [MetadataWriter.commit_master] Blacklisting /tmp/shotwell/library/2018/08/23/foo.jpg For each modified file and a corresponding L 29293 2018-08-23 15:08:29 [DBG] DirectoryMonitor.vala:472: Blacklist for /tmp/shotwell/library/2018/08/23/foo.jpg removed This should be the last log from directory monitor. If an event happens after that, we might have a race
And it took a bit for me too :) I hadn't had this problem in a while so I thought maybe it was fixed. However, it happened again today, and it turns out that it happened in July as well so I was wrong thinking it hadn't happened in a while. However, I'm using 0.20.2 so I guess I can't do what you're suggesting here? Maybe I ought to update to the latest version. I'll see if I can do that. I'm still using ubuntu 14.04 though support for that will end in ~six months.
Hmmm, looks like with ubuntu 14.04 the best I can do is shotwell 0.20.2, the version I have, so I'm unable to build as you're suggesting here, Jens. :( I know, I know, time for an upgrade. I'm thinking about new hardware as well since I'm still using 32-bit. But why fix something that's not (really) broken? :) My wife just got a new computer and is running ubuntu 18.04 so I'm starting to feel like a dinosaur.
You can do the same with 0.20.2, you have to compile it with ./configure make USER_VALAFLAGS="-DTRACE_MONITORING -DTRACE_METADATA_WRITER"
-- GitLab Migration Automatic Message -- This bug has been migrated to GNOME's GitLab instance and has been closed from further activity. You can subscribe and participate further through the new bug through this link to our GitLab instance: https://gitlab.gnome.org/GNOME/shotwell/-/issues/4533.