can't do this anymore for 2.0 so closing

Reopening.  Milan Crha is able to reproduce this bug in Evolution 2.10.

I could reproduce this by simply typing new mail with text:"
From line1
From line2
>From line3 already marked
From line4
". And when I save this mail as draft on local folder, then I could see after opening this message this text:"
>From line1
From line2
>From line3 already marked
>From line4
", also in message source in text/plain part is this instead:"
>From line1
>From line2
>From line3 already marked
>From line4
".

Created attachment 88647 [details] [review]
proposed patch 1/2

for evolution;

Created attachment 88649 [details] [review]
proposed patch 2/2

for evolution-data-server;

I added support of encode/decode direction for CamelMimeFilterFrom and improved mbox support, so it works with "From " in text like described in http://qmail.org/qmail-manual-html/man5/mbox.html

The qmail man page is wrong.


To quote JWZ:


      > You should assume that "From " lines are quoted, not mangled.
      > On systems that mangle, mail gets corrupted no matter what you do.
      > If you assume quoting is used, you might save mail sometimes.

I completely disagree.

I was describing what the historical truth of this file format is: the facts about what is out there in the world today. If you do quoting, not mangling, you are following different rules than have been followed for decades.

If, when I send a message, I type ``>From '' at the beginning of a line, and you treat it as quoting instead of mangling, you will alter the message. And you will alter it in a different way than all the other software out there which is following the ancient de-facto standard.

Whether you treat it as quoting or mangling, either method will cause certain messages to be changed: neither is foolproof, as both make the assumption that all software is following the same rules, whichever they may be.

I say, since neither method is foolproof, use the older and more prevalent rules, since that's what will get you maximal interoperability.

You may think that turning ``>From '' into ``From '' is somehow more harmless than turning ``From '' into ``>From '', but I don't see why.

For example, consider the case of a message with a clear-signed cryptographic signature (multipart/signed, say.) Any change to the body will cause the signature verification to fail. Now, suppose that the software doing the signing, being aware of the prevalence of From-mangling, were to do that mangling before signing. This would interoperate perfectly with software that did mangling in the traditional way (like, say... sendmail, or /bin/Mail).

But software that tried to ``improve'' matters by changing the rules -- by mangling ``>From '' to ``>>From '', or by mangling ``>From '', to ``From '' -- would cause the signature to no longer match.

It is in nobody's best interest for random pieces of software to try and ``improve'' on the BSD mbox format. It's a crummy format, but it is what it is. If you don't want to use it, don't -- use something else. But please, use something else that can be self-identified, that is programatically distinguishable from the historical format, so that software can know what it's parsing.

Don't make us need to guess even more than we do now. 

fwiw, the logic in your patch is also broken.

there are a couple possible ways to solve this problem:


1. make evolution use maildir instead of mbox (see the original suggestion, it was suggested because trying to make Evolution use From-quoting instead of the widely used and accepted From-munging is a broken idea)

2. in the composer (or mailer after the composer is done with it), scan for lines starting with "From " - if any exist, use Quoted-Printable encoding and transform "From " into "=46rom " (I believe the CamelMimeFilterCanon filter does this) before it gets appended to an mbox (Outbox or Drafts)

*** Bug 440634 has been marked as a duplicate of this bug. ***

(In reply to comment #8)
Thanks for the detailed explanation, Jeff.  It sounds like your second suggestion about using CamelMimeFilterCanon would be a good temporary solution to this, while your first suggestion about using maildir instead of mbox is really the long-term goal to shoot for.

yea, switching to maildir would be a bit of a pain at this point :(

I kinda regret pushing for mbox back when we made the decision, but oh well.

(In reply to comment #7)
> fwiw, the logic in your patch is also broken.
> 

Why do you think that?

My logic is very simple:
- make changes only when accessing data from/to mbox (when reading/writing on disk) and only after found proper message beginning
- when writing, add quote
- when reading, remove quote

This is broken only for already stored messages, and only there, where user already typed his own (one or more) quote(s) before From, in this particular case it could break a signature, but nowhere else.

another possibility is to have the mbox folder code record the locations of
the mangling ">"s in the summary, so that when it fetches the message out of
the mbox later, it knows which ">"s it should remove to recreate the original
message text

yea, but that would be overly complicated and would make the summary less regeneratable (ok, that's not a word...)

*** Bug 441865 has been marked as a duplicate of this bug. ***

Jeff, what about add new option to mbox file/dir, called "qmail compliant store" (or any better text, which I know exists), which will turn on those replacements.
I know it's nothing extra, but it may be better than nothing. User will have an option to get rid of these unnecessary ">From "s in the text, when he/she will know it's safe. This option could be turned on on Evolution's Draft,Sent,Inbox,... local folders as default.

What do you think, could this work?

not if the mbox already has mail in it, no.

All this seems to be a WontFix from the discussion above. There is a bug for a mialdir, I'm pretty sure, thus I tend to close this. Any objections? (Except of functionality, but that one cannot change with a broken design/rfc anyway.)

*** Bug 475359 has been marked as a duplicate of this bug. ***

*** Bug 401665 has been marked as a duplicate of this bug. ***

*** Bug 428728 has been marked as a duplicate of this bug. ***

*** Bug 586412 has been marked as a duplicate of this bug. ***

Please, please, please start using maildir as a storage format for Evolution, instead of the broken mbox.

Created attachment 142705 [details] [review]
proposed evo patch ][

for evolution;

This is a change on a composer side, using quoted-printable whenever possible and escaping "\nFrom " properly, thus the message is not broke by the underlying store. Of course, moving to maildir is a long term fix, with other couple benefits, but that will take much longer (and other bug is for that too).

disclaimer: i don't use evo any more, but since I originally filed this bug, I'm going to comment anyway. :)

it would be extra-awesome if you could make the Outbox->CamelTransport codepath undo the qp encoding. IIRC you should be able to just always undo it, because the transport code will re-encode it if needed (eg, if it's non-ASCII and the SMTP server doesn't support 8BITMIME).

Also:

+text_requeires_quoted_printable (const gchar *text, gsize len)

typo. ("requires".) Also, I think there definitely used to be a method that did that somewhere in Camel, though maybe it's not there any more, or maybe it's not available in a form that useful from the composer.

Created commit 3766e19 in evo master (2.29.1+)

Thanks for your comments. The typo is fixed (that was easy) :) Otherwise I kept the patch as it is, because I'm not sure with that transport part, and the function from camel, I saw there couple filters using something similar, but didn't find any particular function.