GNOME Bugzilla – Bug 775730
Very Long Delay before display of reply / forwarded message in conversation
Last modified: 2019-08-11 12:47:52 UTC
After replying, it takes on the order of 2 minutes to show the reply in the conversation thread.
Thanks! Okay, can you try the following for me so I can debug it: 1. Launch Geary from a terminal using the following command: geary -d | tee geary.out 2. When it has opened, loaded the account and started the initial sync in the background, send a reply to a message and wait for the reply to show up in the conversation (don't change mailbox during this time), then quit Geary. 3. Open up geary.out in a text editor, redact any personal info you don't want made public, then attach it here.
Other than folder names (and my email address), I don't see anything I need to redact. Is there any specific piece of info that folks fail to redact when posting debug, typically? If no, I'm ready to upload the log...
For the default debug log, mostly just the things you mention. Some people are sensitive about server names, but that's less common. The main thing is that you're happy with it.
Created attachment 341524 [details] Debug Log
Thanks for that. Looking at the log, there seems to be a bunch of things going on. Firstly while Geary first attempts to send the message at 15:37:23: > [deb] 15:37:23 0.003349 smtp-outbox-folder.vala:224: Outbox postman: Sending "RE: Following up" (ID:[SmtpOutboxEmailIdentifer:1])... But it's not until 15:37:41 (18s later) that the message is actually successfully sent and Geary saves it to the Sent mailbox. This problem looks like the server is using a self-signed TLS certificate, or at least one from an unrecognised CA. Geary realises the cert has been pinned, but not in time to accept the connection anyway. Then, although Geary queues a background sync of the Sent mailbox, it seems to complete before the message is copied there: > [deb] 15:37:41 0.000082 imap-engine-account-synchronizer.vala:372: Background ... > [deb] 15:37:41 0.000052 imap-engine-account-synchronizer.vala:447: Done background sync'ing Other:ken@hero.com:Sent (open_count=2 remote_opened=true) ... > [deb] 15:37:41 0.001968 imap-folder.vala:232: Sent RECENT 1 Finally, although the server notifies Geary of the message arriving in the Sent folder, it isn't updating the folder at that point: > [deb] 15:37:41 0.001488 imap-db-folder.vala:1208: Unable to detect duplicates for [-1/3965] ( available) And there's some other general weirdness going on as well, causing at least one connection to error out: > [0002/mail.hero.com/default:993 GEARY_IMAP_CLIENT_SESSION_STATE_SELECTED] Receive failed: No response to command(s) after 30 seconds So in the end it's not until another background sync is requeued, the connection re-established, and Sent is sync'ed that the new message picked up and loaded: > [deb] 15:39:58 0.188851 app-conversation-monitor.vala:662: 1 out of folder message(s) appended to Other:ken@hero.com:Sent (open_count=1 remote_opened=true), fetching to add to conversations... I'll try to dig into some of these issues, and if I can get some patches together for testing, are you able to compile Geary and try them out? Also, do you know which IMAP server is being used for this service?
Yes, this is a dovecot-2.2 server. I'm not sure why I had to pin the cert in the first place. The chain appears valid: ``` ken@ken: ~$ openssl s_client -connect mail.hero.com:993 CONNECTED(00000003) depth=3 C = SE, O = AddTrust AB, OU = AddTrust External TTP Network, CN = AddTrust External CA Root verify return:1 depth=2 C = GB, ST = Greater Manchester, L = Salford, O = COMODO CA Limited, CN = COMODO RSA Certification Authority verify return:1 depth=1 C = GB, ST = Greater Manchester, L = Salford, O = COMODO CA Limited, CN = COMODO RSA Domain Validation Secure Server CA verify return:1 depth=0 OU = Domain Control Validated, OU = PositiveSSL, CN = mail.hero.com verify return:1 --- Certificate chain 0 s:/OU=Domain Control Validated/OU=PositiveSSL/CN=mail.hero.com i:/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO RSA Domain Validation Secure Server CA 1 s:/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO RSA Domain Validation Secure Server CA i:/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO RSA Certification Authority 2 s:/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO RSA Certification Authority i:/C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root 3 s:/C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root i:/C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root --- Server certificate -----BEGIN CERTIFICATE----- MIIFSzCCBDOgAwIBAgIQVtuEAsQdibc5xcS5rp0q8TANBgkqhkiG9w0BAQsFADCB kDELMAkGA1UEBhMCR0IxGzAZBgNVBAgTEkdyZWF0ZXIgTWFuY2hlc3RlcjEQMA4G A1UEBxMHU2FsZm9yZDEaMBgGA1UEChMRQ09NT0RPIENBIExpbWl0ZWQxNjA0BgNV BAMTLUNPTU9ETyBSU0EgRG9tYWluIFZhbGlkYXRpb24gU2VjdXJlIFNlcnZlciBD QTAeFw0xNTA0MDcwMDAwMDBaFw0xODA0MDgyMzU5NTlaMFExITAfBgNVBAsTGERv bWFpbiBDb250cm9sIFZhbGlkYXRlZDEUMBIGA1UECxMLUG9zaXRpdmVTU0wxFjAU BgNVBAMTDW1haWwuaGVyby5jb20wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEK AoIBAQDCls2Yi5qm2lOODcaQChK3iLUNisgxU3lU/n9OQpnPiUMtRu1k6b/4sTR0 PTSNJgcWJj0QLW7xX/eOviEscspro8oPBZqrI80wakewmp95d2Ik5Hxat0HtV0cT zpNhaTgbniSTf0mYX04ndxHC8ubeDPp0Mw7gmxKU2GuSuxqQfIcqXG42j7pHow+I NI8VjNXlOVPNzfAGxl4KdMVtFvLciKSFDtnJJYOeyUP5I4j2+fLFNXnFsoJXbfhL MlBGh32SB5l/1NKhctRmhTOuSl5r+6Fwy90tBcF9WbZW2GOBFcoPOkUCjIc9mXjQ jAO4oUR3AFDqL+KikxhzVyiO8IKJAgMBAAGjggHdMIIB2TAfBgNVHSMEGDAWgBSQ r2o6lFoL2JDqElZz30O0Oija5zAdBgNVHQ4EFgQU2hxlVndTSukSJ3mt9jJlWV0x MtswDgYDVR0PAQH/BAQDAgWgMAwGA1UdEwEB/wQCMAAwHQYDVR0lBBYwFAYIKwYB BQUHAwEGCCsGAQUFBwMCME8GA1UdIARIMEYwOgYLKwYBBAGyMQECAgcwKzApBggr BgEFBQcCARYdaHR0cHM6Ly9zZWN1cmUuY29tb2RvLmNvbS9DUFMwCAYGZ4EMAQIB MFQGA1UdHwRNMEswSaBHoEWGQ2h0dHA6Ly9jcmwuY29tb2RvY2EuY29tL0NPTU9E T1JTQURvbWFpblZhbGlkYXRpb25TZWN1cmVTZXJ2ZXJDQS5jcmwwgYUGCCsGAQUF BwEBBHkwdzBPBggrBgEFBQcwAoZDaHR0cDovL2NydC5jb21vZG9jYS5jb20vQ09N T0RPUlNBRG9tYWluVmFsaWRhdGlvblNlY3VyZVNlcnZlckNBLmNydDAkBggrBgEF BQcwAYYYaHR0cDovL29jc3AuY29tb2RvY2EuY29tMCsGA1UdEQQkMCKCDW1haWwu aGVyby5jb22CEXd3dy5tYWlsLmhlcm8uY29tMA0GCSqGSIb3DQEBCwUAA4IBAQAQ WxWOZU40dCDMX2tV0pIU0yGCD62eKLvummD2yMN/HJSIVf2aNSQD1sWGHfsP30Uo WD1ji+qMU9K9MfRyotq2Iir4nO91/J3sjTU6wl3jsJjApcZ60KQw0qX/vVxWK6ZH JJ3ruGClLztmR3U6Z0qgqB8CEW4SZR/LaR+lgbSInf++hbdTNTWX0naRFjCy5FKp PMemCKjwzWP7f/YZT/zbn/XIrZi7EgG4yty/Be0/Vw/JPkIm4lJJGbAgQgP1Tq3o erbNsOFqcTVRfOjcK/JUjMOdwqBdHU/L5KyV1LjvK/fktDsxF3KFltBStId8j/tk 7xtbbc17KY4vcV/2BZJF -----END CERTIFICATE----- subject=/OU=Domain Control Validated/OU=PositiveSSL/CN=mail.hero.com issuer=/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO RSA Domain Validation Secure Server CA --- No client certificate CA names sent Peer signing digest: SHA512 Server Temp Key: ECDH, P-384, 384 bits --- SSL handshake has read 6089 bytes and written 463 bytes --- New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384 Server public key is 2048 bit Secure Renegotiation IS supported Compression: NONE Expansion: NONE No ALPN negotiated SSL-Session: Protocol : TLSv1.2 Cipher : ECDHE-RSA-AES256-GCM-SHA384 Session-ID: 9AA74E9DC32254433F5F502A479DC051EDC7CE24167BDE3BD95E495DAA5D28CF Session-ID-ctx: Master-Key: C8E1A395C524BFBFB424C0A3467B4DCD51CF931220402C446A600EE8C9BACD7BEE1A03A15A2FE2F0EF7B5AB01A2EABB6 Key-Arg : None PSK identity: None PSK identity hint: None SRP username: None TLS session ticket lifetime hint: 300 (seconds) TLS session ticket: 0000 - c7 f9 63 4c 98 74 96 8b-df 2d 2c a0 91 37 19 35 ..cL.t...-,..7.5 0010 - 15 a0 01 e2 54 72 29 0f-60 c7 24 54 72 d6 03 a4 ....Tr).`.$Tr... 0020 - d4 e0 38 bb bd 22 d6 43-55 9f 67 75 02 a4 e3 e9 ..8..".CU.gu.... 0030 - f3 37 ff 18 fb 0d 19 8e-eb 78 c5 a4 28 fe 52 34 .7.......x..(.R4 0040 - 14 d2 98 e6 a2 0e a2 99-9c 43 0b 72 12 2f 9b 15 .........C.r./.. 0050 - 6e 67 7d 2a 1d d2 02 d4-c1 ee 85 1e b7 bd fd 05 ng}*............ 0060 - 7a 67 a1 b4 7e 09 7f 3f-87 20 ca 42 1a a1 04 96 zg..~..?. .B.... 0070 - ee c6 ca 54 a9 07 f4 48-80 87 81 59 28 b0 80 7b ...T...H...Y(..{ 0080 - e3 33 21 bc 95 2e 7f 5f-00 82 fb d2 5d ba 9d 20 .3!...._....].. 0090 - ae 88 b4 df 07 e3 d1 6e-33 5b c8 fa 41 c3 5d 03 .......n3[..A.]. Start Time: 1481157727 Timeout : 300 (sec) Verify return code: 0 (ok) --- * OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE AUTH=PLAIN] Dovecot ready. ```
Hmm, so what do you get if you run Geary with `-d --revoke-certs` and send a message?
Created attachment 341591 [details] Unhappy Certificate
So it's just SMTP on port 587 that's the problem, and not IMAP? What does openssl say about for that port? I.e. > openssl s_client -quiet -starttls smtp -connect mail.hero.com:587
True. Let me take a look, I've obviously screwed up the cert chain. Let me fix it up and see if that's the issue. Seems plausible.
Oh, can you hold off doing so for a bit? I'd like to see if I can get a patch together to fix the initial connection failure when using a pinned cert - it would be useful if you could test it out. :)
Yes. No problem.
Created attachment 341593 [details] [review] Fix long delay sending mail when using pinned SMTP server cert Ken, can you please apply this patch (or checkout the wip/775730-long-delay-sent-mail branch in the repo) and rebuild geary, then try the following for me: 1. Start Geary logging with -d and try sending a message 2. You should be prompted about the SMTP cert - if so then click "Always Trust This Server" 3. Quit Geary, restart it, and send another message. You should not be prompted about the cert again, but it should finish sending almost straight away, and you shouldn't get the "SMTP login error: Unacceptable TLS certificate" error in the log 4. Open Seahorse, see if the certificate appears in there somewhere (you may need to select View > Show Any and search for you smtp server address Thanks!
Haven't built the project before. What's the target for building a .deb package?
Created attachment 341642 [details] Dockerfile
There's an Ubuntu-specific deb config in the source tree, so if you have the dpkg development tools and Geary's prereqs installed (see the INSTALL file), you should be able to just run dpkg-buildpackage from the source root to compile a deb. If you just want to test this out without installing it, you should be able to just do a "./configure && make" to build it, and you can then run it from the source tree using "./geary" Also, note since you are running 0.11, if you apply the patch to master you won't be able to go back to 0.11 without restoring a backup of ~/.local/share/geary, since it will upgrade any account databases it finds.
I'm running on a different machine than where I'm compiling. running `dpkg-buildpackage -d` cleans the build and then wants a tarball of it. I don't get a `.deb` to install, I get this error: ``` dpkg-source: error: can't build with source format '3.0 (quilt)': no upstream tarball found at ../geary_0.11.0.orig.tar.{bz2,gz,lzma,xz} ```
Ah, I think you'll need to specify a binary-only build: `dpkg-buildpackage -b`
That works. Attached a Dockerfile that builds the `.deb` However, pinning doesn't work. 1) Always asks to confirm the cert: ``` [deb] 18:05:40 0.000009 smtp-outbox-folder.vala:259: TLS connection warnings connecting to mail.hero.com/default:587, user must confirm connection to continue ``` 2) Nothing shows up in seahorse.
Created attachment 341653 [details] Dockerfile
(In reply to Ken Berland from comment #19) > That works. Attached a Dockerfile that builds the `.deb` > > However, pinning doesn't work. 1) Always asks to confirm the cert: > > ``` > [deb] 18:05:40 0.000009 smtp-outbox-folder.vala:259: TLS connection > warnings connecting to mail.hero.com/default:587, user must confirm > connection to continue > ``` > > 2) Nothing shows up in seahorse. Okay, can you run gcr-viewer and see if the cert is present there? Also, I assume you have gvfs installed?
`gcr-viewer` needs an argument, right? Yes: ``` ken@ken: ~$ dpkg -l | grep gvfs ii gvfs:amd64 1.28.1-1ubuntu1 amd64 userspace virtual filesystem - GIO module ii gvfs-backends 1.28.1-1ubuntu1 amd64 userspace virtual filesystem - backends ii gvfs-bin 1.28.1-1ubuntu1 amd64 userspace virtual filesystem - binaries ii gvfs-common 1.28.1-1ubuntu1 all userspace virtual filesystem - common data files ii gvfs-daemons 1.28.1-1ubuntu1 amd64 userspace virtual filesystem - servers ii gvfs-fuse 1.28.1-1ubuntu1 amd64 userspace virtual filesystem - fuse server ii gvfs-libs:amd64 1.28.1-1ubuntu1 amd64 userspace virtual filesystem - private libraries ```
Hey Ken, apologies disappearing for a while. I assume this is still a problem, are you still able to help out with testing possible fixes?
I'd like to help with this bug, as I have the same problem. Unfortunately at the moment I cannot test it, because every single message sent is sent but not saved in the sent folder. I get the message: "Message not saved. This message was sent, but has not been saved to your account". This was bug 727679 (fixed). Somehow it's come back... If I close Geary and start it again, the message is then copied from Outbox to Sent folder. This is what Ken experienced (see comment 16 of bug 727679). However, after the message has been saved correctly to Sent folder, I can wait hours but it will never appear in the conversation in the inbox. I have to close Geary and start it again to see it.
Created attachment 366134 [details] cannot save to sent folder
Created attachment 366135 [details] cannot save to sent folder (second attempt)
The WIP branch going up for Bug 778276 might actually help the non-TLS/SMTP aspects of this, since it reworks how Geary notifies and discovers new messages.
I must say that sometimes (very rarely, I think) my reply appears immediately in the conversation in inbox. It just happened with an email sent to five recipients (not to a mailing list, where you might get false positives if you have set the mailing list to send your own emails and you do not check carefully).
Federico, does the WIP branch I mentioned above help? Looking at those two different logs, in the first your Sent folder was already already open, hence Geary didn't need to establish a connection to it. In the second, the Sent folder was closed, and hence a connection was established to be able to save the message. Was the first one successful, but the second one successful? It's weird that no error was reported in either case, maybe I should add some more debugging in. In any case, please do use the WIP branch for testing this out further, I'm going to land it on master soon in any case and it substantially changes how Geary reports and looks for new messages showing up in folders.
Oh one other thing, lets look at the problem saving the sent mail over in Bug 727679, and this for it not showing up when it is successfully saved.
I cannot build that branch. $ ./configure [...] -- Vala fatal warnings: ON -- Checking for modules 'gthread-2.0;glib-2.0>=2.42.0;gio-2.0>=2.42.0;gtk+-3.0>=3.14.0;libsoup-2.4>=2.48;gee-0.8>=0.8.5;libnotify>=0.7.5;libcanberra>=0.28;sqlite3>=3.7.4;gmime-2.6>=2.6.17;libsecret-1>=0.11;libxml-2.0>=2.7.8;gcr-3>=3.10.1;gobject-introspection-1.0;webkitgtk-3.0>=2.4.0;enchant>=1.6' -- Package 'webkitgtk-3.0', required by 'virtual:world', not found CMake Error at /usr/share/cmake/Modules/FindPkgConfig.cmake:415 (message): A required package was not found Call Stack (most recent call first): /usr/share/cmake/Modules/FindPkgConfig.cmake:593 (_pkg_check_modules_internal) src/CMakeLists.txt:487 (pkg_check_modules) -- Configuring incomplete, errors occurred! See also "/home/fede/src/geary/build/CMakeFiles/CMakeOutput.log". Unable to prepare build directory. [geary (wip/775730-long-delay-sent-mail=)]$ The sent mail bug happened some times, not always. And today on that account (my primary account) I'm having lot of errors. I've already tried `geary --revoke-certs`. Here are the details provided by Geary error window: Geary version: master~g802c6b90 GTK version: 3.22.26 Desktop: GNOME Problem type: GEARY_PROBLEM_TYPE_SERVER_ERROR Account type: OTHER Error type: GearyImapError 2 Message: Not connected to mail.autistici.org/default:993 Back trace: - geary_account_problem_report_construct - geary_account_real_notify_account_problem - geary_imap_engine_generic_account_enumerate_folders_async_co - g_simple_proxy_resolver_set_uri_proxy - g_task_attach_source - geary_imap_engine_generic_account_enumerate_remote_folders_async_co - g_simple_proxy_resolver_set_uri_proxy - g_task_attach_source - geary_imap_account_fetch_child_folders_async_co - g_simple_proxy_resolver_set_uri_proxy - g_task_attach_source - geary_imap_account_send_list_async_co - g_simple_proxy_resolver_set_uri_proxy - g_task_attach_source - geary_imap_account_send_command_async_co - g_simple_proxy_resolver_set_uri_proxy - g_task_attach_source - geary_imap_account_send_multiple_async_co - g_simple_proxy_resolver_set_uri_proxy - g_task_attach_source - geary_imap_client_session_send_multiple_commands_async_co - g_simple_proxy_resolver_set_uri_proxy - g_task_attach_source - geary_nonblocking_batch_execute_all_async_co - g_simple_proxy_resolver_set_uri_proxy - g_task_attach_source - geary_nonblocking_abstract_semaphore_real_wait_async_co - _geary_scheduler_scheduled_instance_on_callback_gsource_func - g_list_sort_with_data - g_main_context_dispatch - g_main_context_dispatch - g_main_context_iteration - g_application_run - _vala_main - __libc_start_main - _start
(In reply to Federico Bruni from comment #31) > I cannot build that branch. Okay. Well I just merged it to master, so can you build master? ;) In particular, do you have libunwind-dev installed? > The sent mail bug happened some times, not always. > And today on that account (my primary account) I'm having lot of errors. > I've already tried `geary --revoke-certs`. > > Here are the details provided by Geary error window: > > Geary version: master~g802c6b90 > GTK version: 3.22.26 > Desktop: GNOME > Problem type: GEARY_PROBLEM_TYPE_SERVER_ERROR > Account type: OTHER > Error type: GearyImapError 2 The last commit to the WIP branch above, now on master, should resolve this error. So let me know if that resolves the issue saving to Sent. Again though, if there's a problem saving to Sent, lets keep digging into that over in Bug 727679.
Hi Ken, I'd be interested to know if this issue is now fixed for you on master. If possble, can you do a build from Git and see if that is the case? If you need a hand building from Git, let us know.
I was just testing it by sending an email to myself and it seems there's still some issues to get the Sent mail showing up immediately. With current master, I'm seeing the following, when the Inbox is selected: 1. Sent mail gets saved & background sync launched for Sent 2. Background sync opens Sent, finds new message, downloads it 3. Inbox is notified of the message, adds it to the conversation list, but doesn't find the message in Sent The ether of the two last steps is maybe the problem here. Will keep digging.
Created attachment 366701 [details] another try, still does not work Yes, I have libunwind-dev installed. I've just tried on master. The email is correctly saved to Sent, but it's never displayed in Inbox until I enter the Saved folder. See attached log. BTW, I cannot see what's the difference between this ticket and bug 714450 comment 11. Also, another related issue might be that the conversation which I've just replied to does not jump up the conversation list, as apparently the time of sent messages is not considered for sorting. I remember there was an open ticket for this, but I cannot find it.
Hi all, Since this bug now spans across at least three different issues (long delay of sent mail, TLS cert issues, and messages not being saved), and that some of those have been addressed 3.32, and in both GLib and GCR releases over the course of the year, or are being addressed 3.34, it has become unworkable and I'm going to close it. If any above issues are still a problem in 3.32 and/or 3.34 when that comes out, please open separate issues for each individual problem, or mention in existing bugs covering each. * In particular, the TLS pinning issue was worked around Geary 0.13 and fixed by the last GCR release. * Messages not being saved could be caused by a number of things, at least a few of which could have been GNetworkMonitor issues (fixed in GLib 2.60.4), and some issues with the ReplayQueue which I'm about to land on mainline. * Finally, sent messages not showing up is covered by https://gitlab.gnome.org/GNOME/geary/issues/131, which is on track to be fixed in 3.34