GNOME Bugzilla – Bug 714802
Messages not loaded chronologically
Last modified: 2021-07-05 13:26:40 UTC
---- Reported by rschroll@gmail.com 2013-03-23 10:22:00 -0700 ---- Original Redmine bug id: 6651 Original URL: http://redmine.yorba.org/issues/6651 Searchable id: yorba-bug-6651 Original author: Robert Schroll Original description: I have a label in my Gmail account for messages that I copied over from a previous account. When I select that label, the latest messages are not displayed. Then when I scroll down, messages are loaded from apparently random times. I suspect the problem is that I didn't copy the emails over chronologically, so Gmail's order of the messages doesn't resemble at all the chronological order. I assume that Geary is asking Gmail for the last n messages, and Gmail is giving the last n in its internal order. Unless there's a way to ask the IMAP server for messages in chronological order, I fear the only way to solve this would be to keep our record of the message order. Presumably some of this information is stored in Geary's database, but I don't know if it's enough for this. Related issues: related to geary - Feature #3767: Offline mode (Open) ---- Additional Comments From geary-maint@gnome.bugs 2013-09-04 17:51:00 -0700 ---- ### History #### #1 Updated by Jim Nelson 8 months ago * **Category** set to _engine_ * **Target version** set to _0.4.0_ "I assume that Geary is asking Gmail for the last n messages, and Gmail is giving the last n in its internal order." That's exactly what's going on. IMAP deals with messages as a vector from `1 to n`. Geary, in order to avoid loading your entire email account into the local database, only loads the top `m` messages, then allows you to scroll down and load more on demand. This generally works, since messages in IMAP are (as far as I can tell on the systems I've seen) listed from oldest to newest received. This doesn't correspond 100% with the `Date:` field ordering, but it's close enough for our purposes. It sounds like you have a folder than blows this assumption out of the water. What we really need to do here is load the entire folder, there's no way around it I can see, unfortunately. We could code a heuristic which detects this situation and falls back to fetching all messages. #### #2 Updated by Robert Schroll 8 months ago Jim Nelson wrote: > It sounds like you have a folder than blows this assumption out of the water. Yeah. I made this folder by copying emails over from another account. I didn't get much warning that it was being closed, so I just dragged-and-dropped them from one account to the other in Thunderbird, instead of finding out the right way to do it. The copying would fail randomly, so the copies ended up in a rather random order. I'm sure this is a rather rare occurrence, so feel free to reduce the priority as you wish. > What we really need to do here is load the entire folder, there's no way around it I can see, unfortunately. We could code a heuristic which detects this situation and falls back to fetching all messages. I would be fine with the loading happening "incorrectly" at first. But I'm wondering if once Geary has loaded the folder once and gotten whatever it gets into its database, Geary could compare the top m messages it gets from the server with the top m it expects from its local database. If it's expecting messages from a later time than it gets, it could specifically ask for those others. Of course, I'm making assumptions here about the info Geary stores that may be completely off base. This may relate to #3767. #### #3 Updated by Jim Nelson 8 months ago That's not quite how Geary works. The basic strategy with IMAP is to mirror the stack or vector of emails on the server locally. IMAP even uses positional addressing when referring to emails, and so it's vital to have the same ordering in our database. That's why I think we need this heuristic -- when the email order is so shuffled it starts to resemble a deck of cards, Geary needs all the cards in order to make sense of them. Of course, if this happens with All Mail, then we're essentially downloading all the email from the account. So that needs to be considerexd too. #### #4 Updated by Jim Nelson 3 months ago * **Target version** changed from _0.4.0_ to _0.5.0_ --- Bug imported by chaz@yorba.org 2013-11-21 23:11 UTC --- This bug was previously known as _bug_ 6651 at http://redmine.yorba.org/show_bug.cgi?id=6651 Unknown milestone "unknown in product geary. Setting to default milestone for this product, "---". Setting qa contact to the default for this product. This bug either had no qa contact or an invalid one. Resolution set on an open status. Dropping resolution
*** Bug 722713 has been marked as a duplicate of this bug. ***
*** Bug 742930 has been marked as a duplicate of this bug. ***
*** Bug 742804 has been marked as a duplicate of this bug. ***
Robert, I've created a potential candidate for this problem. It's not perfect but I believe it will solve this problem. It turns out this issue is more fundamental than it looks at first blush; if a user is actively moving/copying messages from folder to folder, it's trivial to create a vector that is completely out of order, which screws up ConversationMonitor and AccountSynchronizer. I've done some testing on a small out-of-order folder, but am curious if it fixes your problem. If it doesn't, a debug log would be helpful.
Created attachment 295505 [details] debug log This is better, but it's not completely working. Whereas I would only load a single message before, now I get a good number of conversations loaded, enough to be able to scroll back and trigger more. Unfortunately, these conversations don't start with the latest -- they begin around the time of that one message and go back a bit further. As I trigger more messages to be loaded, the resultant conversations aren't contiguous. I see obvious gaps of several weeks between groups of conversations. Additional loads fill in the gaps and provide newer conversations than any already loaded. There are two confounding factors I can think of. First, I already had a single message loaded, around which the initial conversations were loaded. If you think that's important, I can try again with an empty database and see if it gets the newest ones. Second, all of these messages are at least three years old, and I'm only set up to load messages from the past year by default. So getting any messages relies on an explicit trigger. I've attached a log. I opened Geary, waited a few seconds for things to settle, opened the problematic folder (UMass), and allowed for several cycles of loading additional messages.
So, what this patch does is tweak the AccountSynchronizer to find the first email in the vector (1 to n) that is older than the cut-off in the Accounts dialog. For example, if you have 10,000 messages in random order with a 2-week cutoff and the first email older than 2 weeks is message #5, Geary will pull mail #6 to #10,000. However, currently Geary will only do this when it detects something has changed in the folder. Looking at your log, Geary treated the folder as unaltered (no changes since the last time it visited it), so it didn't go through this process. So, yeah, it would be worth trying with a fresh database. To *really* fix the problem, we would need to detect when the folder is flat-out out of order and pull the entire folder in. There's a lot of edge cases and issues with that, though. The point of this branch is you will see all conversations from your cut-off point forward, although it's still possible (as in your case) to see conversations from before that w/ holes. But in the least you should see the chronologically newest conversations first, which doesn't sound like that's happening. I think we need to have some way to force the AccountSynchronizer to operate. But let me see first if the problem is solved with a fresh database.
GNOME is going to shut down bugzilla.gnome.org in favor of gitlab.gnome.org. As part of that, we are mass-closing older open tickets in bugzilla.gnome.org which have not seen updates for a longer time (resources are unfortunately quite limited so not every ticket can get handled). If you can still reproduce the situation described in this ticket in a recent and supported software version, then please follow https://wiki.gnome.org/GettingInTouch/BugReportingGuidelines and create a new ticket at https://gitlab.gnome.org/GNOME/geary/-/issues/ Thank you for your understanding and your help.