I'm in the middle of analyzing this but cannot finish it right now. Progress so far:


Original From that does shows up garbled:
=?utf-8?Q?WestLicht.=20Schauplatz=20f=C3=BCr=20Fotografie?= <info@westlicht.com>

This can be decoded by g_mime_utils_header_decode_text(), so I guess it's valid syntax. I tested it by adding the string to gmime-2.6.19/tests/test-mime.c and running "make check".

I tracked the problem down to rfc822-message.vala line 307:

from = new RFC822.MailboxAddresses.from_rfc822_string(message_sender);

message_sender at this point is

"=?utf-8?Q?WestLicht.=20Schauplatz=20f=C3=BCr=20Fotografie?=" <info@westlicht.com>

Note that quotes have been added! This string can not be decoded by g_mime_utils_header_decode_text().

Probably the string should have been decoded *before* adding the quotes?

I did a little research into this problem last week.  I believe the problem is this:

* When Geary adds the email address to the database, it uses GMime.utils_quote_string().
* When it's parsed from the database back into memory, the quoted string is passed directly to GMime.utils_header_decode_text().

This works for most strings, even when quoting is applied.  (utils_quote_string only modifies the string if necessary.)  My first thought was to use utils_unquote_string before utils_header_decode_text, but that's not quite accurate -- the quoted addresses are concatenated when stored in the database, much like they're listed in an RFC822 header.  Unquoting the entire string will fail when multiple addresses are stored, hence the reason for calling utils_header_decode_text to break them apart.

Looking at the gmime source code (https://git.gnome.org/browse/gmime/tree/gmime/gmime-utils.c#n1167 and specifically https://git.gnome.org/browse/gmime/tree/gmime/gmime-utils.c#n1133), it appears it's the period in the quoted-printable that's triggering the addition of quotes.  This is not what we want, and so we may need to do our own verification that quoting is desirable before running it through GMime.

For reference, this is how it looks like in the database:

sqlite> SELECT from_field,sender from MessageTable WHERE from_field LIKE '%?Q?West%';
"=?utf-8?Q?WestLicht.=20Schauplatz=20f=C3=BCr=20Fotografie?=" <info@westlicht.com>|"=?utf-8?Q?WestLicht.=20Schauplatz=20f=C3=BCr=20Fotografie?=" <info=westlicht.com@mail128.atl61.mcsv.net>

Created attachment 269592 [details]
Screenshot showing the sender OK in the list and garbled in the conversation

I dug a bit deeper this time, created a new empty Gmail account to debug this cleanly.

In the SQLite database, from_field looks good actually (probably dumped the wrong email last time?):

sqlite> SELECT from_field, sender from MessageTable;
"WestLicht. Schauplatz für Fotografie" <info@westlicht.com>|"<info=westlicht.com@mail128.atl61.mcsv.net>" <=?ISO-8859-1?Q?"WestLicht._Schauplatz_f=FCr_Fotografie"?=@>

This explains why the sender looks OK in the list (see screenshot).

However, if you look at the "header" column, you notice that everything has been rewrapped!
I will attach a screenshot of the meld diff in a second.

Original:
From: =?utf-8?Q?WestLicht.=20Schauplatz=20f=C3=BCr=20Fotografie?= <info@westlicht.com>

Geary DB:
From: =?utf-8?Q?WestLicht.=20Schauplatz=20f=C3=BCr=20Fotografie?=
 <info@westlicht.com>

Jim, I'm not sure how this fits in with your theory, but gmime cannot decode that rewrapped "From:". Is the header rewrapping intentional? Disabling it may solve other corruption problems, too. I have not found out how to do that, though. Hints are welcome :)

Created attachment 269600 [details]
Meld diff comparing header in the Geary DB and Thunderbird source view

Geary rewraps the header lines

(In reply to comment #5)
> Jim, I'm not sure how this fits in with your theory, but gmime cannot decode
> that rewrapped "From:". Is the header rewrapping intentional? Disabling it may
> solve other corruption problems, too. I have not found out how to do that,
> though. Hints are welcome :)

This might be the issue, but it's not really Geary's fault.

I suspect Thunderbird pulls down the entire RFC822 message at once.  With IMAP, that's done with a FETCH RFC822 command (there are other variants).

Geary was designed to build a sparse database of your email.  That is, it can fetch fragments of the email (headers, body, metadata, etc.).  This design was to quickly pull down the "envelope" and then later fetch the full email.  Because of this, Geary fetches the body and header separately using IMAP's FETCH RFC822.HEADER and FETCH RFC822.TEXT commands.

Whether or not this is a good idea, it's valid IMAP.  But, for whatever reason, Gmail (not Geary) re-wraps the headers when they're pulled with RFC822.HEADER:

$ openssl s_client -connect imap.gmail.com:993 -crlf
a login <email> <pw>
a select Inbox
a fetch * RFC822
<normal headers>
<body>
a fetch * RFC822.HEADER
<re-wrapped headers>

I haven't seen this with other servers, only Gmail.

In prior investigation of the problem, the re-wrapping seemed innocuous (i.e. it followed RFC822 wrapping rules and so was not technically causing any data loss).  It's entirely possible you've found a case where it is an issue, but we need to investigate more closely.

If you want to keep looking into this, we're welcome to hear more of what you find.

Created attachment 269841 [details] [review]
[PATCH] Initialize GMime with ENABLE_RFC2047_WORKAROUNDS

It turns out the wrapping was indeed innocuous and the problem is in fact the dot - as you suspected, Jim! 

The dot does seems to be technically illegal. The good thing is that GMime tolerates it once you enable ENABLE_RFC2047_WORKAROUNDS. The attached patch does that and I have not noticed regressions so far.

Great work!  Let me look into this further before committing.

(In reply to comment #8)
> It turns out the wrapping was indeed innocuous and the problem is in fact the
> dot - as you suspected, Jim! 

Charles Lindsay pointed out the dot problem to me, so he deserves the congratulations for that.

This patch also fixes bug #714339.

As mentioned, this closes a separate bug and potentially fixes other problems, so I'm sold.  Pushed to master, commit 1cbd39b.  Thanks for sticking with this!

Incidentally, the ticket describing our investigation into Gmail's header re-wrapping is bug #713326.

Jim, your responsiveness has created the best experience I have seen in a bug tracker so far.