GNOME Bugzilla – Bug 576184
[Maildir] Full text search unreliable
Last modified: 2021-05-19 12:27:42 UTC
[Originally reported on the Evolution users list] --- problem --- For a while now I had this nagging feeling that full text no longer worked as well as it used to in older releases. I'm using Evolution 2.24.5, compiled from source, with maildir as storage format, and can now definitely say that full text search fails to find any mail with the word "sandbox" in a folder which has several mails which contain that word. I did a "grep -r -w" and found 13 mails. Doing the same search via Evolution found only one. I then stopped Evolution, removed all *.ibex.index* and *.cmeta files plus the folders.db, restarted Evolution. "Send/Receive" ran a while and produced shell output ("camel-WARNING **: Could not find key entry for word '0x00000000': No such file or directory") which seemed to come from text indexing. Once done, it doesn't find *any* email at all with the word "sandbox". --- how to reproduce --- I can reproduce it reliably, but I cannot share the mails where this occurs. --- random observations --- The folder where I have the problem has a subfolder. I created a new folder in the same account at the top level and copied all emails, without the subfolder. In that copy the search worked. I copied the folder and its subfolder, using Evolution's "copy folder" context menu item. The copied subfolder was empty although the original subfolder had emails. In that copy the search does not work: it finds 4 instead of just 2 mails, but that's still not correct. This result is reproducible, leading to the same 4 mails. After *deleting* the copied subfolder the search suddenly *works*!
Deleting the subfolder under the original folder does *not* fix the problem.
Srini, this is the issue that I demonstrated live in Gran Canaria. You offered to help if the problem still occurs in recent Evolution releases. I've finally moved to a new laptop and in the process tried out different Evolution versions. This problem still occurs in 2.26.3 (Debian unstable) and the latest sources as of today. Let me know what kind of information you need. I'll also hang around on IRC, just in case that you want to do an interactive debug session.
Thanks for a bug report. I see that runtime critical warning sometimes too, it's usually when moving/copying "too many" messages (last time I tried it was about 3000). If I do the same in chunks, then no runtime warning is shown. I guess you still see this on actual master (~2.30.0), do you? Is it an issue with the maildir only, or with On This Computer it's fine? What is your full-text search, something like an Advanced search with "body contains"? You know, if the word is not part of the body text, then evolution doesn't "see" it for this kind of search. To be honest, I didn't get the connection to the subfolder, it seems to me like a coincidence, but I can be wrong.
(In reply to comment #3) > I guess you still see this on actual master (~2.30.0), do you? I'm still on 2.28.x. I can try 2.30.0 if there is any reason to believe that it has been fixed. > Is it an issue > with the maildir only, or with On This Computer it's fine? I don't know. The folder it occurs in is large. I tried copying it around, with inconclusive results. > What is your > full-text search, something like an Advanced search with "body contains"? It is "body contains" with a single word ("sandbox"). > You know, if the word is not part of the body text, then evolution doesn't > "see" it for this kind of search. It is part of the message body. > To be honest, I didn't get the connection to the subfolder, it seems to me like > a coincidence, but I can be wrong. Yes, that seems to be pretty random.
Still occurs with 2.30 in the original folder. It does *not* occur with the same test folder after copying it into a local MBOX. But it also does *not* occur when copying into another maildir folder elsewhere in the hierarchy. In both cases I get results. Haven't done a check whether the results are complete. So it doesn't seem to be the format of the mail store which causes it, but rather the layout of the folder hierarchy, or the "freshness" of the folder (but note that I tried removing .cmeta and .ibex* files, without success), or whatever.
also reported at: https://bugs.launchpad.net/ubuntu/+source/evolution/+bug/582316
I suspect the problem is not just with full text search, but with search in general. I use a search folders to reproduce (combine) the contents of a couple of imap inboxes. There are no text search criteria, just "Match All". One of the search folders draws on a single imap inbox - couldn't be simpler. Sometimes newly received emails don't show up in the search folder(s) - refresh doesn't help, expunge doesn't help, switching focus to another folder and coming back doesn't help. What does help is going into the search folder's properties, changing one of the search criteria, accepting the changes, then returning to the original criteria and accepting those changes. That 'switching' rebuilds the index and the new emails appear ..usually. Sometimes I also have to visit the imap inboxes directly first and only after the email is 'seen' there does it appear in the search folder. This behaviour is intermittent, so not likely to ever meet the bugzilla triage criteria for 'reproducible'. So perhaps not likely to ever get fixed? The nearest I have noticed to reproducible is that indexing can break down badly if I ever start evolution quickly after login to the desktop. Sometimes only a couple of emails show up in the search folders instead of the hundreds or thousands that should be there. Logging out and back in - then starting evolution after a pause - sometimes fixes it temporarily. Long pauses in use with the screen-saver lock activated can also bring on the problem behaviour. I have experienced this problem for years using evolution in various versions on both fedora and ubuntu, sometimes missing important emails. At times everything seems to work fine and you think it is fixed only to be disappointed once more. Currently using evo 3.2.2 on F16 variously with lxde / gnome 3 desktops. Issues have been experienced on both desktop and laptop. My impression is the problem occurs because the timing of indexing / adding to indexes and re-indexing activities run to a schedule or priority that is not sufficiently tied into user actions, so the index might be out of date and evolution carries on working quite happily leaving the user in the dark. A work around would be not to use search folders!
Using 3.22.3 on Ubuntu 16.10 AMD64; I have all messages sync'd locally. I'm searching for the term "optic", with scope "Message contains". The search is finding this word in some emails, but not in others. Here is my best guess at what is happening. Evolution appears to be searching for partial text on the subject. It is finding subjects like this: "ThinOPTICS Always With You Lifetime Guarantee Activated". This email has no instance of "optic" in the body, but it does in the recipient email address. It is NOT finding several emails with subject "New reply: Unreliable Search" and with this phrase in the body: "the search should be text:optics." However, if I change the search term to "optics", then these emails ARE included in the search results, and when I select one of them, the text "optics" is highlighted. I'm concluding from this, perhaps erroneously, that searching for partial text is not being done on the body, but only full word search.
The original report claims an issue for a Maildir account, which is currently (3.22.x) the format used under On This Computer folders (which had been switched from mbox to maildir several years ago). I'd prefer to keep this bug report dedicated to it. (In reply to Jacek from comment #7) With respect of search folders, that's true, they had been unreliable since the move to the DB summary. There had been made extensive changes in the search folders for 3.6 and the current stable (3.22.x), but also some previous stable versions have the search folders reliable again, as far as the underlying folders report their changes properly. (In reply to Guy Rouillier from comment #8) I suppose it is an IMAPx account conecting to the Gmail and you are online, right? The "Message contains" searches some headers and the message body. While most of those picked headers can be searched locally, the body search is done on the server and the IMAPx fully relies on the server response. I just tried with my Gmail account and it works just as you described. Partial word matches are not considered by the server. Doing the same with a Dovecot server works properly. The RFC 3501 [1] defines the BODY search as "contains" the specified string, thus I'd say the error is on the server (in this case Gmail) side. I understand that you have messages stored for offline use, but these are ignored when doing the body search. I opened bug #777431 for this Gmail issue. [1] https://tools.ietf.org/html/rfc3501.html#page-51
I tend to close this in favour of bug #671470, where the Maildir indexing had been disabled. It makes body searches slower, but reliable.
Milan, you are correct, I am using Gmail. I apologize for not mentioning that. I will follow bug #771431.
*** Bug 790892 has been marked as a duplicate of this bug. ***
GNOME is going to shut down bugzilla.gnome.org in favor of gitlab.gnome.org. As part of that, we are mass-closing older open tickets in bugzilla.gnome.org (resources are unfortunately quite limited so not every ticket can get handled). If you can still reproduce the situation described in this ticket in a recent and supported software version, then please follow https://wiki.gnome.org/Community/GettingInTouch/BugReportingGuidelines and create a new bug report ticket at https://gitlab.gnome.org/GNOME/evolution/-/issues/ Thank you for your understanding and your help.