GNOME Bugzilla – Bug 216738
gpg sigs getting munged when read in evo, causing verification failure
Last modified: 2013-09-10 14:02:48 UTC
Description of Problem: certain gpg-signed messages, when read by evo will have the sig boundary is munged. a message, sent to a mailman mailing list, signed by me, resent to me by the list (I'm on the list) showed as a bad sig. I knew the sig was good. I save-as'd the message from evo to a text file. Then I save-as'd the message from my sent-mail to a text file. I compared the boundaries and they were different. Since I'm using imap I went to my inbox on the server and looked at the boundary in my inbox, it matched the boundary from the sent-mail. So the problem was occuring somewhere in between. So I went into pine and checked what it reported as the boundary. It was correct and pine said the sig was good. so the sig only didn't match when read in evo. I put the message up on a webpage, asked luis to go try it out by shoving the message, from my inbox, into his maildir, he did so and the message screws up in the same way for him. I am running evo 1.0 from the gnomehide build. I am running on red hat linux 7.2. Steps to reproduce the problem: 1. luis has the message that can be used to show the problem. I'd rather it not be posted to bugzilla b/c it is not for general consumption 2. view the message, select view-email source 3. notice how the mime boundary header before the signature-attachment changes. that invalidates the sig Actual Results: the sig is marked as bad b/c the boundary is changing Expected Results: it should be good and data shouldn't be modified one the fly by evo How often does this happen? every time. Additional Information: notice how the boundary changes each time you click off the message and the click back on to it. This should be consider a major security problem b/c it means sigs will return as bad when they are really not - if people rely on sigs to know who is sending them a message (and there are people who do) then this will screw them up, badly.
Jeff: I've got the email and can confirm the behavior; let me know what you want me to do with it.
actually the problem has nothing to do with the boundary at all: Looks like the message as was sent by Evolution was QP encoded but the message he received is not, so the problem seems to be that one of the MTAs that the message went through decoded the message to raw 8bit format and thus the signature no longer validates. This is one of the problems with the PGP/MIME specification, there is nothing we can do :( Jeff
What is QP encoded? I've sent hundreds of message through this same system. Only one or two of them have this problem - why the lack of consistency?
ok, ignore my last comment - QP == quoted printable I just wasn't thinking. but here is the problem. the message is in my mailbox as quoted printable but when I view the message from evo (view email source) it claims that it is 8bit. This message was sent through mailman so I checked in the mailman inbox archive and it is also in there as quoted printable. it is only when the message is viewed in evo that it gets converted to 8bit. go to: http://www.phy.duke.edu/~skvidal/misc/gpg/ the message is in there - please disregard the body of the message. there are 4 messages there they are named fairly obviously. so from what I can see my MTA is not touching the mail in anyway - evo is.
okay, so when you use Evolution's Save As... menu item, it always converts to 8bit before saving so that the user can read the email without having to QP or base64 decode the text in his head. so disreguard the messages saved via that interface in Evolution, it has nothing to do with signature verification inside evo. The one interesting thing I did notice was that the received message that you saved with evo had a CHARSET=US-ASCII param in the Content-Type. Are you by any chance using IMAP? I found out the hard way that some IMAP servers add a charset param to the Content-Type params when the message parts are fetched individually but not when fetched as a complete message. If you are using IMAP, I suspect that is where the problem is. (evolution did not add a CHARSET=US-ASCII param by itself)
I am using imap for all my messages, Why would this one be special? I will test this from a local mail spool to verify, though. And if this is the case, does that mean that y'alls imap server does the same thing if content-type is set to text/plain? b/c luis encountered the same problem. We're doing a lot of tests w/this. Our environment is: all red hat linux 7.[12] MTA postfix 20011125-1SASL IMAP: uw-imap 2000c
ok I've verified - it works correctly from a local mailbox but its fubarred coming from the imap server. so I did a manual session from the server to see if it was the server a nd its not. I'm going to attach the session to this bug
Created attachment 40849 [details] manual connection to server and selecting the problem message
fejj: we're corrupting data. This is not FUTURE.
we are NOT corrupting data, we are merely failing to verify the signature. There is a difference. This is NOT fixable in the 1.0.x time-frame, hell it's not fixable without rewriting all of Camel. So this is future. At best this is 1.4.
Every single time I open the message, it is a different message. That's about the most obvious form of corruption I can imagine.
what are you talking about? show me.
seth: can you do a FETCH BODYSTRUCTURE command? also, since the message is larger than 5k, Evolution will download each part individually so a FETCH BODY[TEXT] doesn't prove anything. You'll need to grab each part individually.
Attaching new log of imap session with: fetch bodystructure fetch body[0] fetch body[1] fetch body[2] I think this is what you wanted.
Created attachment 40855 [details] log of session separately selecting the body segments from the message
* 1 FETCH (BODYSTRUCTURE (("TEXT" "PLAIN" ("CHARSET" "US-ASCII") told ya Camel wasn't pulling that out of it's butt ;-) that says that the first part is: Content-Type: TEXT/PLAIN; CHARSET=US-ASCII the IMAP server should *not* be pulling parameters out of it's butt and giving them to us as if they were really there. This is why the signature is failing to verify. this is clearly a NOTXIMIAN
Then why does it do it on other imap server too? how many imap servers "pull things out of their butt"?
I've just tested the sig from mutt, using the same imap server. and the sig is good. its got to be something on the client's interpretation.
the reason mutt probably verifies it is because it gets the message in 1 lump sum whereas evolution does not (when a message is > 5k, evo fetches each mime part individually as it is requested rather than fetching the entire message) mutt is very simplisting in it's imap implementation - it always fetches the entire message as far as I know. It doesn't fetch individual parts, and if it did - it would also have the same problem evo is having.
Then, I ask again, why did luis get the same problem with the message from a courier-imap server, in fact, YOUR courier imap server at ximian?
This is clearly evo's fault. Here is the sniffing done using ethereal as evo was making the connection and retrieving the message in question: ...[after the login]... a002 SELECT "mail/seth-test" * 1 EXISTS...[smallskip]... a003 FETCH 1 UID * 1 FETCH (UID 1) a003 OK FETCH completed a004 UID FETCH 1 BODY * 1 FETCH (UID 1 BODY (("TEXT" "PLAIN" ("CHARSET" "US-ASCII") NIL NIL "QUOTED-PRINTABLE" 3555 80)("APPLICATION" "PGP-SIGNATURE" NIL NIL NIL "7BIT" 240) "SIGNED")) a004 OK UID FETCH completed a005 UID FETCH 1 BODY.PEEK[1.MIME] * 1 FETCH (UID 1 BODY[1.MIME] {73} Content-Type: text/plain Content-Transfer-Encoding: quoted-printable ) a005 OK UID FETCH completed a006 UID FETCH 1 BODY.PEEK[2.MIME] * 1 FETCH (UID 1 BODY[2.MIME] {43} Content-Type: application/pgp-signature ) a006 OK UID FETCH completed a007 UID FETCH 1 BODY.PEEK[1] * 1 FETCH (UID 1 BODY[1] {3555} encryption-server@phy.duke.edu a couple of commments on the design: Karsten,=20 ...[etc-etc-etc]... The response to a005 is what we need to be looking at, as it clearly DOESN'T say "charset=us-ascii". This is something that evo adds all on its own, looking at the response for a004. Let me re-iterate: a005 UID FETCH 1 BODY.PEEK[1.MIME] * 1 FETCH (UID 1 BODY[1.MIME] {73} Content-Type: text/plain Content-Transfer-Encoding: quoted-printable ) a005 OK UID FETCH completed I don't see a "charset=us-ascii" here, yet it magically appears in the final evo representation of the message, making the signature fail. I think it's clear that this error is, in fact, a bug in evo. It shouldn't be adding anything to the mime headers of the parts it downloads off the IMAP server, since this causes pgp-validation failures.
uh, that is not a log from evolution - evolution for 1 does not use lowercase tags. I'm telling you, the CHARSET=US-ASCII in the BODYSTRUCTURE is the problem.
Look, it is the EXACT follow-up of the steps evo takes to retrieve the message. I just re-did it in a telnet session to minimize the garbage, while looking at the ethereal report. These are in fact THE COMMANDS issued by evo to retrieve the message. I don't see why you keep marking it as NOTXIMIAN, when it clearly is. Seth, please re-open it when you have a chance. This is really silly.
show me where evo magically puts CHARSET=US-ASCII in the content-type header and I'll believe you. Until then, I'm telling you it doesn't do it on its own. I even just grepped the sources and there is only one place that could possibly set charset=us-ascii and that function is only ever called in the smtp code.
just dumped the message on my imap server and evo isn't adding a CHARSET=US-ASCII param at all.
Here are the commands issued by evo to retrieve that message: B00021 SELECT "mail/seth-test" B00022 UID FETCH 1:1 (FLAGS) B00023 UID FETCH 1:* (FLAGS RFC822.SIZE) B00024 UID FETCH 1 BODY.PEEK[HEADER] B00025 UID FETCH 1 BODY B00026 UID FETCH 1 BODY.PEEK[1.MIME] B00027 UID FETCH 1 BODY.PEEK[2.MIME] B00028 UID FETCH 1 BODY.PEEK[1] B00029 UID FETCH 1 BODY.PEEK[2] Here are the mime headers of the first attachment the way it is in the mailspool: --=-EpEuX4sDt54eLT2YIONv Content-Type: text/plain Content-Transfer-Encoding: quoted-printable Here is how they look when I look at them in View->Message Display->Message Source: --=-Ng+pxah7Emcq5xtkgl+k Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; CHARSET=US-ASCII This is how they look when I do File->Save as: --=-qrTbugyav6YWItbWqgXo Content-Type: text/plain; CHARSET=US-ASCII Content-Transfer-Encoding: 8bit This is how it looks when I FORWARD this message to someone else: --=-cu2q1GPSzxIGlkVlrIw2 Content-Type: text/plain; CHARSET=US-ASCII Content-Transfer-Encoding: quoted-printable Looks like it's different everywhere, and CHARSET=US-ASCII is present at every point. This behavior invalidates the gpg signature and thus should be handled as data corruption.
yea fine, but why isn't a CHARSET=US-ASCII appearing on *my* end? it's because CAMEL ISN'T CORRUPTING ANY DATA As I stated earlier, Save-As mangles content-* headers to convert to [7,8]bit so that the user can actually *read* the message in a text editor - so this is not something you should even be looking at. Unlike you, I do *not* get a CHARSET=US-ASCII param appended onto the Content-Type header. I just get: Content-Type: text/plain Content-Transfer-Encoding: quoted-printable
are you even actually positive that evo is passing CHARSET=US-ASCII to gpg? Maybe you should write a gpg wrapper and find out... - better yet, diff it between IMAP vs local and attach the files here (and diff).
Well, I believe this is final: icon@cho:[~]$ diff wrapper.imap wrapper.local 0a1 > Content-Type: text/plain 2d2 < Content-Type: text/plain; CHARSET=US-ASCII The mime headers: wrapper.imap: Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; CHARSET=US-ASCII wrapper.local: Content-Type: text/plain Content-Transfer-Encoding: quoted-printable Not only did it add the "CHARSET=US-ASCII", it also moved the strings around. Ick.
Here is my wrapper: icon@cho:[~]$ cat /usr/bin/gpg #!/usr/bin/perl $blah = ""; while ($what=<STDIN>){ $blah .= $what; } open (BLAH, ">/home/icon/wrapper"); print BLAH "$blah"; close (BLAH);
okay, I'm getting the header swapping too but I am not getting any CHARSET params. So this is at least partially an imap server bug.
anyways, leaving this marked as closed because there's another bug report about the content-* header swapping "bug".
It does seem that the IMAP server is adding "CHARSET" "US-ASCII" if it finds that these parameters are missing. However, this is something it returns in the "FETCH BODY" response, and IIRC the IMAP RFC correctly, it is supposed to handle all text/plain messages without a "charset" parameter as of a "us-ascii" charset. You shouldn't rely on this command already because it completely disregards the upper/lower case (US-ASCII vs us-ascii), and any changes in letter case will cause gpg verification to fail. It seems to me that you should really stick to what the BODY.PEEK[1.MIME] returns and leave these headers unmodified. I should also remind you that this is uw-imap 2000c. While I'd like to rid the world of UW code as well, this stuff probably runs on 50% of all IMAP servers out there.
I disagree with closing this bug. This is something that will be a recurring motif and should be fixed now as opposed to later. The messages ARE, after all, being corrupted, and the IMAP server does return the correct information in the BODY.PEEK[1.MIME] command.
well, it can't be fixed without rewriting all the imap code, so this isn't going to happen for a while and there is already a bug report about rewriting the imap code so reopenning won't be at all helpful.
*** bug 217564 has been marked as a duplicate of this bug. ***
We continue to see reports of this munging; I'll look for and find at least one more in the morning. I don't think this should be closed until then. If it takes a rewrite of large portions of code to fix then we need to maybe think about that.
Just to add my comments into this bug since mine was marked as a duplicate of this bug.... Most PGP systems do not include the following (example) multi-part bountaries for e-mails: --=-DsZCrRxgkOC0EErqjVso Content-Transfer-Encoding: 7bit Content-Type: application/pgp-encrypted Version: 1 --=-DsZCrRxgkOC0EErqjVso Content-Type: application/octet-stream; name=encrypted.asc Content-Description: This is an encrypted message part Content-Transfer-Encoding: My bug is having problems going the other way from what it seems that the rest of the people here are discussing (i.e. IMAP is translating). Mail sent FROM evolution to other clients, most notably Outlook and Outlook Express, do not know how to implement action to handle these headers. I've been using Outlook + PGP for years now and never has the encrypting of an e-mail added any sort of multi-part header. You can see these headers added to outgoing mail in the ~/evolution/local/Sent/mbox file. I can encode the same message in both Evolution + GPG and Outlook + GPG, sent them via SMTP and pull the message out of the queue both by hand and also via POP. In all tests, the Evolution-created message is unreadable in Outlook, Outlook Express and Eudora becuase the e-mail is blank and has randomly-named attachments that are the message. These attachments must be manually saved to disk and hand decrypted, which defeats the purpose of PGP/GPG. I don't beleive it's an IMAP corruption becuase in the process of testing, e-mails have been locally delivered by three different mail servers using POP (1 MDaemon, 1 QPOP and 1 Courier POP).
yea, they aren't the same bugs at all. anyways, the real problem here is that somehow the way the imap code uses the mime parser, the Content-Type header gets moved to the end of the Content headers... I haven't been able to figure out why either. As louie did note to me earlier, it would also be nice if we re-used the multipart boundary from the original message if possible. We do this for locally delivered mail but apparently not for IMAP mail. I've found this particular problem but I see why danw called camel_multipart_set_boundary (mp, NULL); rather than calling it with header_content_type_param (ci->type, "boundary"). I'm guessing that 'ci' and 'ci->type' are constructed from summary info and thus the 'ci->type' struct doesn't have a boundary param nor does it have the protocol or micalg params that are usually found on a multipart/signed Content-Type header. Anyways, we need to have some way of getting these...maybe request the encapsulating multipart part?
i think the set_boundary(NULL) is historical cruft and can probably be replaced as you suggested. actually, CamelMultipart should override CamelDataWrapper:set_mime_type_field to parse out the boundary if it's there.
I at first tried modifying it to use header_content_type_param but that didn't work (I think it's because the params are not saved in the summary??) Of course, it's completely possible that my install was borked at the time (I had been swapping back and forth with 1.0.x and 1.1.x) but I'm pretty sure my install was ok. I'll give it another go tho... I think you're right about the CamelMultipart thing.
On the radar for 1.2.
fixed in CVS