I tried to reproduce this, by creating a new On This Computer book in 3.10.4, then I copied more than 300 contacts into it, which had filled various fields, aka they were not empty. Some were even contact lists and some had photos. Then I made a backup of that evolution and restored it in 3.12.10. When I entered a Contacts view the contacts were there, thus I selected all of them and pressed Ctrl+D to delete them. After I confirm the delete, all of them were deleted properly, I didn't get any SQLite error.

Would it be possible to share part of the backup privately, please?

I do not want whole backup, neither have it attached here, I'd like to have only the address book sent on my bugzilla email with a bug reference in the subject, thus I'd not miss it in my spam folder. The part I would be interested in is the .local/share/evolution/addressbook/ folder of the .tar.gz backup file, which contains all On This Computer/ addressbooks data. I do not want to share it through bugzilla due to it being a private content.

(In reply to comment #1)
> Would it be possible to share part of the backup privately, please?

Of course. 

> I do not want whole backup, neither have it attached here, I'd like to have
> only the address book sent on my bugzilla email with a bug reference in the
> subject, thus I'd not miss it in my spam folder. The part I would be interested
> in is the .local/share/evolution/addressbook/ folder of the .tar.gz backup
> file, which contains all On This Computer/ addressbooks data. I do not want to
> share it through bugzilla due to it being a private content.

Sorry for sounding stupid but is your bugzilla mail address the redhat one I see? If it is then the file is on its way and you should have it by the time you see this.

I hope it helps.

Thanks, the email was received and we had a private mail exchange as well. I wasn't able to reproduce this unfortunately, which looks odd. I was wondering about an sqlite version, I use sqlite-3.8.7.2-1.fc21.x86_64.

A user here with only 16 contacts in her address book has the same issue.  Let me know if I can provide anything to aid in the investigation.

evolution-3.12.8-1.fc21.x86_64
sqlite-3.8.8-2.fc21.x86_64

Evolution 3.12.8 is quite old, current Fedora 21 version is 3.12.11 (all core evolution packages, which counts evolution-data-server, evolution-ews and evolution-mapi).

Nonetheless, I still do not know how it happened. It seems to me like after some update, the database of contacts was updated, but something went wrong and some part of it was left in an odd state, which exhibits with this error message when trying to delete a contact. I can be wrong, of course. The thing is that Paul's contacts, which were also "to be updated" did update properly here.

Oh, right!  Evolution-data-server was the latest version, but the other ones were a bit old, apparently.

Easily fixed.  I've upgraded all the evolution packages now.  It didn't make any difference; the problem remains.

evolution-3.12.11-1.fc21.x86_64
evolution-bogofilter-3.12.11-1.fc21.x86_64
evolution-data-server-3.12.11-1.fc21.x86_64
evolution-rss-0.3.94-9.fc21.x86_64
evolution-spamassassin-3.12.11-1.fc21.x86_64

Likewise, here. Same versions, problem remains.

I'm experiencing the same problem here (https://bugzilla.redhat.com/show_bug.cgi?id=1201890). Let me know if I can help somehow.

evolution-3.12.11-1.fc21.x86_64
sqlite-3.8.8.3-1.fc21.x86_64

When I tried to reproduce it here, with a backup from one user, then it worked fine, I could delete the contacts as I wished. Then maybe if I had the actual addressbook files, already migrated with the issue, then I would be able to reproduce it here and find the cause, or at least a workaround. The local address books are at
   ~/.local/share/evolution/addressbook/
and the On This Computer/Personal is stored at
   ~/.local/share/evolution/addressbook/system/
That folder contains all the data. That also means that it cannot be simply shared publicly, because there are all your contacts (and you cannot delete any). If you do not mind, then feel free to send the files to me privately, to my bugzilla email, with a reference to this bug report, and I promise to use it for testing purposes only.

Thanks for the files, ReD, I can reproduce the error with it, in 3.12.11, but not in the current development version, 3.15.92, which will be 3.16.0 shortly. It seems it got fixed meanwhile.

Looking into the contacts.db, there is defined a constraint between folder_id_email_list and folder_id on the 'uid' field, which makes command like:
   DELETE FROM 'folder_id' WHERE uid IN ('pas-id-4EE7631900000000')
fail, because that uid is still referenced within folder_id_email_list table. There had been done quite many changes in the related code (EBookSqlite) between 3.12.11 and git master, with two version bumps and added summary fields.

None seems to be related to this issue, at least not on the first (and quick) look. I do not expect that you had running the development version and then "downgraded to the stable version, which would update the summary version to something what the previous doesn't know.

Anyway, for current workaround (to get things working again), do this:
a) store your contacts to a vCard in Evolution's UI by right-clicking
   the Personal address book and choosing "Save as vCard...". Pick
   a file name and after it's saved verify that it contains something
   (it should have the same amount of contacts as the address book contains).
   This command tells you how many contacts were saved:
   $ cat vcard.vcf | grep "BEGIN:VCARD" | wc -l
b) close evolution
c) kill evolution-addressbook-factory
d) rename system address book (do not delete it yet)
   $ cd ~/.local/share/evolution/addressbook/
   $ mv system system.bak
e) run evolution (it'll run evolution-addressbook-factory too)
f) the On This Computer/Personal will be empty
g) in menu File->Import->Single file->pick the vcard.vcf andf File Type:vCard
   then finish the wizard with the Personal address book selected
h) when the import is done, the contacts are back and then can be deleted.

We chose to not use the workaround in comment 10.  Rather we waited for evolution 3.16.

We installed it from Fedora 22α today.  It is indeed possible to remove the entries now.

It is not working quite as expected.  The contact disappears during the session, but when starting a new evolution session, they are back again.  But that's probably the subject for a separate bugzilla when we have investigated a bit closer.

Thanks for the update. I didn't notice this "they are back again after Evolution restart", mine shows the offending address book contacts deleted (I ended up with an empty address book,also after restart). One of the reason might be that the backend didn't properly notify the Evolution about the failure, thus it can be still there for you, only masked out by some code bug. I do not believe there is, but I also wasn't able to reproduce this earlier, thus I'll investigate further (I also currently run Fedora 21 packages with git master of evolution, which can be that some other update in some other packages evolution uses could cause the difference).

I did some more investigation around this and I found the problem, though not the cause. The contacts.db (usually) contains several tables:
a) 'folders' - the list of tables with some settings
b) 'folder_id' - the (main) table with all the contacts
c) 'keys' - key/value pair table for custom options for backends
d) generated additional tables with various suffixes; one default table
   is 'folder_id_email_list' which is the one with a reference to 'folder_id'
   (and the culprit with the foreign key constraint)

The list of generated tables is made only once, when the address book is created for the first time. All other times the list is computed from the content of the contacts.db. The thing with the generated additional tables is that they are found only through folders.multivalues column, which contains a list of columns which require the '_list' additional table.

The sent contacts.db file which I received has this folders.multivalues field empty, but the 'folder_id_email_list' table with the reference exists. The default value for it is "email;prefix" (quotes for clarity only). Because the field is empty, the code doesn't know about that able and when the delete happens the additional table is not cleaned up first, thus the delete in the 'folder_id' table fails.

I do not know the reason why the folders.multivalues filed was emptied yet, I suppose it was something with update. I suppose you've updated from 3.10.x to 3.12.x, or was it even earlier version of the evolution?

Even earlier.  We have updated many times keeping the evolution folders and other data.  I can't say for sure what version we used when this user created her first contact.  But it was long ago.  Checking the .local/share/evolution/addressbook/system directory, it contains BDB files called addressbook.db.old and addressbook.db.summary last changed 2010 and 2011 respectively, as well as a couple of log.* files.  When did Evolution switch from BDB to SQLite?

Would a copy of the contacts.db file be of any help?

(In reply to Göran Uddeborg from comment #11)
> It is not working quite as expected.  The contact disappears during the
> session, but when starting a new evolution session, they are back again. 
> But that's probably the subject for a separate bugzilla when we have
> investigated a bit closer.

I fixed this [1], there was a typo (a real bug) in the code, preventing the underlying error to be propagated to the caller, thus the caller thought the deletion succeeded, even it didn't.

[1] Created commit 132c16f in eds master (3.17.1+)
    Created commit aa2442a in eds gnome-3-16 (3.16.2+)

(In reply to Göran Uddeborg from comment #14)
> Even earlier.  We have updated many times keeping the evolution folders and
> other data.  I can't say for sure what version we used when this user
> created her first contact.

I'm sorry, I meant what version was used prior the update to 3.12.x. The old files are not used, they are more for a backup purpose.

> When did Evolution switch from BDB to SQLite?

Approximately in time of 3.1.5, 2011-06-26, commit 61b0957c7, thus part of 3.2.0 stable release.

> Would a copy of the contacts.db file be of any help?

Not needed to send it (once I know the problem [*]), it's enough to run this command:

   $ sqlite3 ~/.local/share/evolution/addressbook/system/contacts.db \
      "SELECT multivalues FROM folders"

to see what you've stored there. It will return an empty line, while it should return at least "email;prefix". Interesting enough, my system address book has that value empty too - I didn't know that.

To fix the bug without code touching run:

   $ sqlite3 ~/.local/share/evolution/addressbook/system/contacts.db \
      "UPDATE folders SET multivalues='email;prefix' WHERE multivalues IS NULL OR multivalues=''"

after that the previous SELECT command will return correct value (non-empty) and it'll be possible to delete the contacts again.

Ideally, the second command should be done without the evolution-addressbook-factory running, to avoid any clashes, though I think sqlite can handle this, it's only the book backend in the factory not knowing about the change until it's restarted.

[*] I looked into the code of 3.12 and I do not see why the column value would be cleared, also because it isn't cleared here, but it can be due to certain version update and the issue being there for a long time, only spotted now.

(In reply to Milan Crha from comment #16)
> > When did Evolution switch from BDB to SQLite?
> 
> Approximately in time of 3.1.5, 2011-06-26, commit 61b0957c7, thus part of
> 3.2.0 stable release.

I'm sorry, this is not accurate, there coexisted both BDB and SQLite databases together for some time, the above commit is only the first addition of the SQLite into the local address book.

The exclusive usage of SQLite was done later, with commit b66eb3c83d5 in time of 3.7.2, thus 3.8.0 stable release.

Created attachment 301517 [details] [review]
eds patch (3.12.11)

for evolution-data-server;

The migration from a previous version didn't store this default multivalue reference, thus the next backend open (not the immediate one after migration), didn't know about this table, which has a FOREIGN KEY constraint, thus an item delete caused a 'FOREIGN KEY constraint failed' error.

This patch does few things:
a) always update 'folders.multivalues' field based on current known multivalues
b) if there is no multivalues value stored, then check an existence of
   the default 'folder_id_email_list' table and when it exists, then add
   the multivalues value manually
c) do not store empty values into folders.multivalues (use NULL instead)
d) bump summary version and migrate contacts to properly populate
   the 'folder_id_email_list' table

Thus this covers both cases:
1) silently fix already broken summary references
2) store correct multivalues on database update

I was able to successfully migrate down from version 3.6 with this change.

Created commit 5103c97 in eds master (3.17.1+)
Created commit e6e167c in eds gnome-3-16 (3.16.2+)

> I meant what version was used prior the update to 3.12.x.

Ah, I misunderstood!

I'm not sure if it matters any more, you seem to have figured this out by now.  But just in case, I checked my dnf history, and it seems we have used these versions most recently:

3.10.2-2.fc20
3.12.8-1.fc21
3.12.11-1.fc21
3.16.0-1.fc22

I can dig further into the yum history if it is of any use.

Thanks for the advice on fixing the contact database.  After updating the "multivalues" column according to your instructions; removing contacts works as expected again. :-)

(In reply to Göran Uddeborg from comment #20)
> I can dig further into the yum history if it is of any use.

Thanks for the update. It's okay now, as it had been fixed meanwhile.

> Thanks for the advice on fixing the contact database.  After updating the
> "multivalues" column according to your instructions; removing contacts works
> as expected again. :-)

Nice. I also made evoltuion-data-server-3.12.11-2.f21 to have this fixed in Fedora (see the Red Hat bug cited in comment #0 for more information).