*** Bug 719611 has been marked as a duplicate of this bug. ***

*** Bug 724950 has been marked as a duplicate of this bug. ***

Hi guys, 

I don't know Vala at all so I cannot really help in this bug. Just wanna say geary is awesome as a lightweigth email client and this feature is really missing for my use. This is why I can't use geary for now.

Please implement it ! :)

I second Kharec's comment.

As a first step, it would be nice to have the option to sign messages and verify signatures of recieved messages.

As for the issue of searching through encrypted messages, this would require Geary to encrypt sent messages two ways: outgoing messages with the public key of the recipients, and their stored copies (in the sent mail folder) with our own public key so that we can actually decrypt them. This is what some mutt users do already (and probably others too).

Just wanted to express that there is more of us that find these features a must-have.

On the mailing list, we had a conversation about GPG support (see https://mail.gnome.org/archives/geary-list/2015-January/msg00007.html and following messages).  A request was made specifically for encrypting email in the PGP/Mime format.  In the discussion, we found that receiving encrypted mail is generally a precondition for sending it, as you will also encrypt it with your public key for your own sent mail folder.

Unrelated, I happened to stumble upon End-to-End (https://github.com/google/end-to-end), which includes a javascript implementation of OpenPGP signing and encryption.  If we ever enable userscripts (bug 732434), this could be another way to go.  I'm not necessarily recommending it; just throwing it out there.

Just second Kharec.

Geary is a really awesome client. But without support of encryption, it is usable only to housewives, no offense.

Just started to add inline PGP support. See this branch: https://github.com/emersion/geary/tree/pgp

It's great to see progress on this.  I didn't know that there was a vapi for gpgme.  That'll save some work.

This is obviously a work-in-progress, so I'll refrain from specific notes for right now.  But there's a general architectural issue we should consider before you go too far with this: Should the decrypted message be stored in our local database?  This would let us avoid shelling out to GPG everytime a message is loaded, and it would also allow encrypted messages to show up in full-text search results.   However, it would open another attack surface (though I'm not convinced it's a very important one).  It would also be more complicated to implement, since we can't count on decryption at reception.  It may require testing for decryption at several points in time.

I'd be interested to hear your thoughts in this issue.

As a Geary user, I think it would be more convenient to store the message unencrypted. That would require to first store it encrypted, and update it when decrypted (when opening it?).

But as a developer, I think that it would be better to let the user choose. There is no big difference, we just need to store the decrypted message or not. What's your advice?

(In reply to Emersion from comment #10)
> As a Geary user, I think it would be more convenient to store the message
> unencrypted. That would require to first store it encrypted, and update it
> when decrypted (when opening it?).
> 
> But as a developer, I think that it would be better to let the user choose.
> There is no big difference, we just need to store the decrypted message or
> not. What's your advice?

We should store the messages in their original form *encrypted* !!! But for searching purposes there could be an option to also store/index them in an un-encrypted form, which should be off by default!

Cheers!

(In reply to Emersion from comment #10)
> As a Geary user, I think it would be more convenient to store the message
> unencrypted. That would require to first store it encrypted, and update it
> when decrypted (when opening it?).

This is a good point.  We have to be able to store it encrypted and decode it later, since we can't count on it being decryptable on reception.  So let's keep with your approach for now and add caching of decrypted content later.

Let us know when you'd like a review.  I don't want to burden you with suggestions on code that's going to change anyway.

Okay, so I'll first start to implement decrypting when opening a message, we'll see later for storing decrypted messages. I'll keep the new gpg/ folder in engine/ instead of client/ for that future work.

I just modified my patch to add a new Geary.PGPEmail class: https://github.com/emersion/geary/blob/pgp/src/engine/pgp/pgp-email.vala

It's a decorator for Geary.Email, and it implements inline PGP and PGP/MIME in a much better way than my previous patch. Even though it's still wip, what do you think of it?

I'll soon add signature verification and encrypting when sending a message. I'll need some piece of advice for the UI. I need to:

* Show that a message is encrypted and show that the signature is correct
* Let the user enable/disable encryption and signing when sending a message

I don't think key management must be included in Geary (it's not the software purpose).

I think moving the decryption away from the conversation viewer is a step in the right direction.  But I think it could be removed completely from the conversation viewer and occur entirely within RFC822.Message.

A bit of background: Every email client has its own interpretation of how MIME messages should work, so figuring out what content should be displayed from a message is much more complicated than it has any right to be.  Since we need to extract the content from a message in several different places, we've put that logic into the get_body() method of RFC822.Message.  Thus, consumers in the client can not worry about the details and get a sensible body content from any message.

My feeling is that decryption should work the same way.  The client should just be able to call get_body() and get a decrypted body, if necessary and possible.  I think this should be possible by extending the existing code.  It's a bit convoluted (because email is convoluted), but the basic idea is that we walk through the mime tree and record content of the requested type as we find it.  What would need to be added is (1) code to handle "application/pgp-encrypted" mime parts and (2) code to decrypt inline encrypted text in plain text parts.  You've written that code, so it's just a matter of getting it hooked up in the right spots.

Two things worry me about this approach, though:
1) How performant will it be?  As I understand it, gpgme works by starting a separate process running gpg, feeding in the data, and reading its output.  This can't be that efficient.  What happens when you load a conversation with 20 encrypted messages?  Does the whole UI freeze for a second or two?
2) How do we indicate that a certain bit of text has was decrypted successfully?  Obviously, we could add some HTML around those bits to style them specially, but those client details shouldn't live in the engine.

We might be able to solve both of these with the trick we use with inline mime parts.  Some mail clients (*cough* Apple) like to stick pictures and other content in the middle of messages by adding them as inline mime parts.  The engine should include them in the result of get_body(), but the details of how they are included are issues for the client.  We deal with this by having the client pass an "InlinePartReplacer" function to get_body, which gets called whenever we run across such a part.  I wonder if we should have an PGPPartReplacer that does the same here.  This could do the decryption and style the result appropriately, or it could just include the cyphertext in a tag, so that that client can come along and decrypt it asynchronously later.

I realize that this expands the scope of work significantly, but I think it's worth it.  By hooking this into this section of code, you'll automatically get decryption when replying to messages, decryption in attached messages, and decryption of messages that don't bother to put the cyphertext as the first attachment.  If it would help, I can try to create a branch with stubs for you to fill in with the actual decryption code.

A few other brief points:
- While reviewing RFC822.Message, I was reminded that we get and store the searchable text separate from the message itself.  So regardless of what else we do, saving searchable text from an encrypted message is its own problem.  Let's not worry about it now.
- Verifying signatures is another problem.  Let's get this one squared away before we attack that.  (Unless you think that'd be easier to do first, in which case let's do them in the other order.  One at a time, is my point.)
- Similarly, composing encrypted/signed messages will be another huge issue.  Let's not worry about that until we've got this under control.

I modified the way I handle inline PGP: I created a Geary.PGPFilter which works like others Geary.RFC822.Filter*. It works pretty well, and follows your idea of replacer. As you said, since it's applied in RFC822.Message, the text is decrypted when replying/forwarding/... which is a good thing.

Here is the source code: https://github.com/emersion/geary/blob/pgp/src/engine/pgp/pgp-gmime-filter.vala

I quickly benchmarked it: it takes ~ 3 seconds to decrypt 20 messages. It's does not takes ages, but it's still slow for long conversations and the UI freezes. Moreover, a message is decrypted several times when opened (get_body() results are not cached, and returns plain text as well as HTML body).

I would like to add some properties to Geary.RFC822.Message like is_encrypted() and get_decryption_result() to add a way for the client to know encryption details. What do you think of that?

I'll try to add a PGPPartReplacer to do the same thing for PGP/MIME.

(In reply to Emersion from comment #15)
> I modified the way I handle inline PGP: I created a Geary.PGPFilter which
> works like others Geary.RFC822.Filter*.

Great!  I considered recommending this, but I held back because RFC822.Filters are a bit confusing.  I'm glad you managed to work out how to do it!

> I quickly benchmarked it: it takes ~ 3 seconds to decrypt 20 messages. It's
> does not takes ages, but it's still slow for long conversations and the UI
> freezes. Moreover, a message is decrypted several times when opened
> (get_body() results are not cached, and returns plain text as well as HTML
> body).

So, a tenth of a second per message, roughly.  Noticeable, but not a show-stopper.  Let's keep this in mind but not worry about it now.
> 
> I would like to add some properties to Geary.RFC822.Message like
> is_encrypted() and get_decryption_result() to add a way for the client to
> know encryption details. What do you think of that?

This would seem to assume that there's only one encrypted bit per message.  This is the sort of completely reasonable assumption that some mail client somewhere feels the need to prove wrong.  This is why I'm trying to figure out how we can annotate bits of the message as we find them, so we can handle complicated mime constructions.

Also, I'd rather not add API that we don't have an immediate need for.  If we find we need something like this, then I'd definitely reconsider.
> 
> I'll try to add a PGPPartReplacer to do the same thing for PGP/MIME.

Great!  For now, go ahead and hardcode something in.  We can worry about letting the client pass in a callback later.

General request for everyone: Please send me encrypted emails from as many clients as and in as many formats as possible, so I can start testing this.  Send them both encoded with my key and with someone else's key, so I can test decryption failures as well.  Thanks!

rschroll@gmail.com

-----BEGIN PGP PUBLIC KEY BLOCK-----
Version: GnuPG v1

mQENBFLUU00BCACoSMr4MgDtpBY0EXg1mqJp8yWt6tQR99YgG/D848kJa6IxG6+V
+hOj2CODUzVKU3VD3e3AG2DFfcMJCsC5CyPj1l3RIIEjTAJlo6yvRFjAXPRGbddq
rk8Fz2Be2iiqkf8vDaOud6P99dJ/iebwc0YsN+qVxZNcR/bzRUFkJ6oM1IdO5hVv
VtJXKcKvQSIhaMd+u+uuOtxAkSsB/FgoB0LQGQs2eZiR82NJWw7aZ+7uw+2yva9b
bL6nhD3abgkPad8AbxGjrEI8CzoN8HQkISffU98F/w27fAQ3g4/P/I77k3hA2uyM
6qGEGxjC7iaHouAJQebRnRBGd4cFXuyWF49rABEBAAG0I1JvYmVydCBTY2hyb2xs
IDxyc2Nocm9sbEBnbWFpbC5jb20+iQE4BBMBAgAiBQJS1FNNAhsDBgsJCAcDAgYV
CAIJCgsEFgIDAQIeAQIXgAAKCRDyztNro2ctARcoB/49olmuvg/grMn61p/HxpFY
zAp74dT8jRc0pzN81JLfS9MYSZSWD7/N/+MuRikQxgYx+93b+b9FKdo6fNpNthyD
8ZEQRdN37hXnbqzdB8ppI8PV/ARNID8/xQXeBMPiYqTCenI4fxcS/FM/pbHAT2IB
xFLRhc8sC0AEyqOCK8LlSyg/um20jIaxf0xe41sucP35KGqCF0/L/M+dBFQbXPlT
CfmI5BFSi9lZbHhWMFngfwh0Xb2PwqzEgVtYrIJqKW2gBr7ALfZgFd5s8JWMQVmj
9tIn0Urvl65b/47HaC6lQQdy/Bfgey92s3Wbjn/bncjJJwkszKK5+2BwksVCiJZa
uQENBFLUU00BCAC+5B1NXxu6KzFpnw09TWCtYK5u+9BvNZJIYApL2DQrWxd/5eJw
bNHDAPqTvfbqynuKjBa0wnsf9KZbutKZze2ZTmYB0LCCfl1In+lu+D/LTdgS3FJi
JgGmMNGjCv/X5EQR6Jjgrd4xgXmAGfCLXQUTQmxkRcoz0lxVOe/nQbxQSS9N9LAd
d3xrAXPggOkPoIkYvsnm3N7y/pSUgLNUAO1foLi15oS3j9PS1ZcAM/cf4qp8QI2o
2ZzhEs0DUlMo3kuTpCjfgb+0m6ztKxjOflr1ocU+hToQLPK/lkiGToNd6ollL3Xb
59ru9LVlC/PKQBgQFmBL1N/B8tqc5jA38c/xABEBAAGJAR8EGAECAAkFAlLUU00C
GwwACgkQ8s7Ta6NnLQEyrQf9G4CCkTZ+gng+YQJu1w8u7m514NQ2n/7nsT9vFDk2
NQE39CdhOIaPl8F8I0McNyNqpkuHBiIggiG4V38kz2jlARkRapQFyGP5ctaqxm+v
QnRZ7J39xya+xLVJsD0r0S7RveLP1YgPcd+b5qAylg/xyYlqWH1+/dlTuleViPWk
NIouPkXlJgZAN7O6YlPC6USOu+rNlOvyEZpbgBFtpRWNeA4FzknBPqq2joNNrzGd
RQlPPp+52r2c7k/+2sAoNHMNjFd8mjI/Sp5VHLWY4akr1AbCo4xjxGeIxZgAsp2C
AYpxJV1vzguyHPml5JK1jHGrhYsnKMVJB3U7fAAw7XNcqg==
=1v87
-----END PGP PUBLIC KEY BLOCK-----

> Also, I'd rather not add API that we don't have an immediate need for.

Another solution would be to wrap the decrypted text inside a <div data-encrypted>[decrypted message]</div> (we could add other attributes such as data-signed, data-signed-by and so on). The problem with this approach is that the client will have to search for these <div>s and prepend HTML nodes to be able to display/style this information. This could be done by another filter in client/ (hard), by the Webkit API (better) or by a JS code embedded in the Webkit view (if we can avoid that, I'll be happy :-) ).

> Let's keep this in mind but not worry about it now.

Moreover, if we store decrypted emails in the DB, this won't cause any delay when loading long fully encrypted conversations.

GPGME currently creates a new process each time a new operation is performed, so that's not very efficient. They promised they will keep one process per context in version 2.0, but I don't know when it will be released.

(In reply to Emersion from comment #17)
> Another solution would be to wrap the decrypted text inside a <div
> data-encrypted>[decrypted message]</div> (we could add other attributes such
> as data-signed, data-signed-by and so on). The problem with this approach is
> that the client will have to search for these <div>s and prepend HTML nodes
> to be able to display/style this information. This could be done by another
> filter in client/ (hard), by the Webkit API (better) or by a JS code
> embedded in the Webkit view (if we can avoid that, I'll be happy :-) ).

This is why I'm thinking about letting the client pass in a callback to handle the encrypted data.  It can set up whatever HTML it wants to indicate the state of that data.  The conversation viewer probably has different requirements than the composer does.  But, as I said, let's get basic decryption working before we worry about getting fancy. :)

Have you looked at GMimeMultipartEncrypted (https://developer.gnome.org/gmime/stable/GMimeMultipartEncrypted.html)?  It might be a help in parsing PGP/Mime messages.

Hum, seems that we don't need GPGME anymore, Gmime can do all the job. I'll have a look and see how to use Gmime crypto library. It seems a little more complicated than GPGME, but the docs are okay. Maybe that will solve our performance problems. And it will be easy to implement other encryption standards.

I just happened to notice it when I was reviewing GMime to refresh myself on our mime message handling.  I don't know if it'll work or not.  It doesn't help with inline PGP, I don't think, so we'd need something else for those.  I'm curious how it interacts with keys -- can we use it without managing keys in Geary?  (If not, I think it'd be okay to require handle the user's private key in Geary, but I wouldn't want to manage public keys for signatures.)

Sorry to keep changing direction on you.  This is new territory for me.

(In reply to Robert Schroll from comment #16)
> General request for everyone: Please send me encrypted emails from as many
> clients as and in as many formats as possible, so I can start testing this. 
> Send them both encoded with my key and with someone else's key, so I can
> test decryption failures as well.  Thanks!

Robert, I've just sent you 15 e-mails, all having "Test e-mail" on the subject.  Some may have ended on your spam box, I suppose :).

I finally added a "GMime.Object? pgp_part_replacer(GMime.Object node)" function, which is called in "construct_body_from_mime_parts()" and replaces an encrypted node by a decrypted one. It works well for the message body, but I didn't managed to get it working for attachments. I tried to add it to the "get_attachments_recursively()" method, but it doesn't seem to be called to get attachments. Can you say me which method is used to list attachments? (Is it in Geary.RFC822.Message?)

You're right, GMime doesn't support inline PGP. I tried to use it but I wasn't able to make it work. Maybe I'll try again if our current approach fails.

I found that Geary.RFC822.Message is not used to get attachments from the ImapDB: https://github.com/GNOME/geary/blob/3694140bb762f0ba1009b8e76b18b0c00a1963c9/src/engine/imap-db/imap-db-folder.vala#L1509

Attachments a passed to Geary.Email and Geary.RFC822.Message doesn't even know that there are attachments for the message. I wonder if we need to re-move PGP/MIME handling in Geary.Email. What do you think?

Created attachment 303094 [details] [review]
Decrypt encrypted parts as soon as loaded into RFC822.Message

I hadn't considered attachments at all, and that's a place where 
just-in-time decryption clashes with Geary's model.  As soon as it gets 
an email, Geary parses it and extracts attachments.  References to those 
attachments are put in the database; the attachments themselves are 
saved in Geary's config directory.  I'm not sure why this architecture 
was chosen, but it does make accessing attachments very easy.

As long as they're not encrypted.  If they're part of an encrypted 
message, we'd need a whole parallel system of finding attachments when 
we decrypt the message.  This is the sort of thing that makes me wonder 
if we should instead decrypt the message as soon as possible and store 
that decrypted version in the database.

I accidentally did that in the attached patch.  I had been intending 
just to move the decryption earlier in the process, so the various get_* 
methods of RFC822.Message didn't have to worry about it.  But it turns 
out that the first thing we do with a new message coming over the wire 
is send it through RFC822.Message, so this ended up decrypting the 
message before it was stored in the database.  This has benefits: The 
body and attachments are ready to go right away.  The email is 
searchable.  The preview string is relevant.  Each email is only 
decrypted once.

The situation isn't so nice if we can't decrypt the message when it's 
received.  These message bodies are still displayed decrypted, but 
they're getting each time we load the message from the database.  But 
the real attachments are not present, and the encrypted part is still 
shown as an attachment.

I known that I said before that we needed to handle messages that we 
can't decrypt on reception, but is that really true?  We need to be able 
to verify signatures each time we look at a message, but the user 
wanting to use encryption will always have their private key around, 
right?

And then of course, there's inline PGP, where the just-in-time approach 
seems much more natural.

So ... I'm confused.  Comments welcome, especially in regard to saving 
decrypted messages and being unable to decrypt messages at reception.

Hello,

I noticed you're discussing implementing inline PGP, so here's a thing I once read that should probably be kept in mind: https://dkg.fifthhorseman.net/notes/inline-pgp-harmful/. While inline PGP IS still used, its use seems to be declining from some time now.

Also, _IMO_ making the body of an encrypted message searchable could be implemented at a later stage, if at all. Storing encrypted messages in the clear/indexing them in a non-encrypted index should not be enabled by default.

P.S.
Keep up the good work! I'm looking forward to encryption support in Geary :)

Well, K9 is still stuck on inline PGP.  I don't know any better FLOSS alternatives for Android which support PGP/MIME, sadly.

https://github.com/k9mail/k-9/milestones/PGP/MIME

Just browsed through the k9 issues. Does Geary also use SQlite's full text search index? ( http://www.sqlite.org/fts3.html ) That seems like something one would like to use in a mail app. Of course messages need to be accessible for the database in order to usable. Thus, the database should be encrypted. There are several on-the-fly cryptography hooks. Here is a nice overview:

http://stackoverflow.com/questions/5669905/sqlite-with-encryption-password-protection

SQLCipher seems to be widely used, even on android:
https://www.zetetic.net/sqlcipher/design/

I think this could make users with a need for cryptography happy, as there is never any plain text stored -- even for unencrypted mails.

Geary does indeed use SQLite.  Encrypting the database is intriguing, but I'm very hesitant to go that direction.  Essentially, it's asking Geary to be a cryptographic vault.  Geary was not built to be cryptographically secure, and I'm sure it would end up leaking information all over the place.  We can't just add some cryptography on the existing architecture and hope it's secure.  It would need to be re-designed with this in mind.  (As an example of the problems with trying to add cryptography onto an existing system, see PGP email!)

Off the top of my head I know we'd have problems with attachments.  Currently, they're stored in your home directory.  We'd have to either encrypt on disk or save them in the (encrypted) database.  Assuming we did that correctly (a big assumption), we still need to offer the decrypted file so that you can open it with another program.  So we decrypt it in a temp directory when the user tries to open it.  When do we delete it?  When the child process exits?  (But maybe the application was already open.)   After n seconds?  (But how are we sure it's enough?)  What happens when Geary exits before it can delete it?  Of course simply unlinking a file doesn't clear the bytes from the disk....

It looks to me that the best we can do is "Stores your messages encrypted (but your attachments will sometimes end up on your home directory unencrypted.  So watch out for that)".  Frankly, I think it'd be better to advertise "Stores your messages unencrypted (use an encrypted home directory to protect against local attackers)".  We should let a program that was actually designed to protect your bits against local attackers do that so that Geary can do what it was designed to do: handle email.

I'm happy to be proven wrong.  If there's an attack that defeats an encrypted home directory but not an encrypted email store, I'd love to hear about it.  Likewise for any best-practices documents about dealing with encrypted email.

I agree with you: Geary's mission is not to keep local emails safe, an encrypted home directory is a better solution.

The only issue I find with your patch is that encrypted emails already in the db will stay encrypted, and won't be decrypted. A solution would be to set attachments to the Message object from Email. Any objections? :-P

I'd like to know why we're caching the attachments upon reception, rather than generating them each time, before we go changing that bit.  Jim, can you provide some context here?

Thinking along other lines: If we're going to decrypt messages upon reception, perhaps we should remove the decrypt step from the general RFC822.Message initialization and trigger it instead only at reception.  This would avoid needing to step through the Mime structure each time we make an RFC822.Message.  It's a small optimization, to be sure, but we do create RFC822.Messages often.

This would mean existing messages do not get decrypted automatically.  Three ideas, which aren't mutually exclusive:
1) Do a database upgrade to search for and decrypt messages along with this patch.  This solves the existing encrypted messages problem, but doesn't help future messages where you're not set up with GPG.
2) Add a command line option to Geary to trigger the above.  The idea is that you'd need to run this once you've gotten GPG set up, but never again.
3) Detect encrypted messages and display a button to trigger decryption.  This would do the decryption, replace the encrypted message with the decrypted on in the database, and then reload the conversation with the decrypted message.  This would be the nicest from a UI perspective, but it would be a fair amount of work to solve a problem most people won't encounter.

Jim, the current approach (decrypt then store) means that we're saving different bits than the server sent us.  Is this going to cause problems?

> I'd like to know why we're caching the attachments upon reception, rather
> than generating them each time, before we go changing that bit.  Jim, can
> you provide some context here?

Yeah, a couple of reasons.

First, there was a big desire to allow for dragging attachments out of the client (which, as it turns out, was never implemented -- I think WebKit was an issue there).  This was a big bugaboo in Shotwell and is still a problem -- w/ DnD, you *must* have the file on-disk when then drag operation begins, otherwise you're in trouble.  For Geary, that means we can't show an attachment icon and, when the user tries to drag it to a Nautilus folder, extract the file from the db and put it on disk -- too late.

Second, we considered a "just-in-time" attachment operation where attachments were unpacked when the user clicked on a conversation, but ran into broader questions, such as what happens if the user clicks the same conversation several times in a session, where to store the files, when/if to delete them, and so on.

Third, a "just-in-time" system would mean viewing image-intensive HTML messages very slow.  Because all the inline images are pre-decoded and on-disk, Geary can serve them up instantly.

We finally punted on the issue and said we'll unpack all attachments as they come in, promising to come up with a better solution later.  Which, of course, never happened.  But the system we have today does address the above problems, which are very real.

One point I've not seen addressed (but perhaps I missed it): what happens if you receive a message before your key is installed on your keyring?  An odd situation, but imagine a user w/ a fresh system running Geary for the first time, downloading all his messages, and later adding his keys to his keyring.  It's a corner case, but from some of the approaches I've seen above, will he be hosed trying to read messages he downloaded before updating his keyring?  Or, since GPG failed, are they irrevocably unreadable?

> Jim, the current approach (decrypt then store) means that we're saving
> different bits than the server sent us.  Is this going to cause problems?

No.  There's a "fields" column in the MessageTable, a bitfield of what's been downloaded and stored in that db row.  If that's set, then Geary won't re-download the columns rep'd by those bits (i.e. Subject, To, From, Body, Headers, etc.)  That's a two-edged sword, of course, because it's difficult to coerce Geary to re-download that info if it was stored wrong...some of our database updates have to work around this by clearing the appropriate bit(s).  (Note that email flags are mutable and will be re-downloaded if requested -- there's exception-case code in various paths to handle that.)

Remember, IMAP makes specific guarantees that all elements of an email (except its flags) are immutable.  Geary takes advantage of that.  We can use that to our advantage here, too.

One possibility I'll throw out:

Treat decrypted messages like attachments, that is, write the decrypted body to a file on disk and mark in the database that "this email's body is decrypted in a file stored in a particular location".  When that mark is detected, load the file into the Geary.Email rather than from the MessageTable, where it's stored encrypted.  (Or, do the same in a separate SQLite table.)

Also, when it's decrypted, add the text to the FTS table for search and write its attachments out.

Both of the above allow for decryption "on-demand" or automatically when it's first downloaded from the server.  It doesn't require re-decrypting a message each time it's accessed -- once is enough.  The GC code (in imap-engine-gc) should clean all this up when the email is removed from the database.

I'm not saying this is easy, and perhaps something like this is the current coder's approach.  I'm also not saying this is bullet-proof -- just a grab-bag of ideas in my head.  I don't know from the past messages if GPGme is still on the table, I realize that slows down the on-demand process considerably.

I know some people want the encrypted messages to remain encrypted, only decrypted in-memory and never written to disk.  I appreciate the desire for that, but as has been said elsewhere, an encrypted filesystem (or home directory) is the better solution here.  (Do other email clients really not write a single plaintext byte to disk?  It would be worth looking at exemplars here.)

We could consider a mechanism where the client can inform Geary that a group of emails are "released" (i.e. not being viewed), and Geary can use that call to clean up all the decrypted bits.

(In reply to Jim Nelson from comment #32)
> > I'd like to know why we're caching the attachments upon reception, rather
> > than generating them each time, before we go changing that bit.  Jim, can
> > you provide some context here?
> 
> Yeah, a couple of reasons.

My first instinct is to leave this bit of code alone (which means requiring decoding upon reception).  But to play devil's advocate.
> 
> First, there was a big desire to allow for dragging attachments out of the
> client (which, as it turns out, was never implemented -- I think WebKit was
> an issue there).

Will we implement this?   I can't say I've ever missed this capability.  We shouldn't let current architectural decisions be decided by things that might happen in the future maybe.

> Second, we considered a "just-in-time" attachment operation where
> attachments were unpacked when the user clicked on a conversation, but ran
> into broader questions, such as what happens if the user clicks the same
> conversation several times in a session, where to store the files, when/if
> to delete them, and so on.

If we don't have the files around for a potential drag, then this is a non-issue.  We just unpack the files into /tmp when the user tries to open it.  (This would also solve another bug which asks us to make the attachments appear to come from /tmp, so you don't save changes accidentally.)
> 
> Third, a "just-in-time" system would mean viewing image-intensive HTML
> messages very slow.  Because all the inline images are pre-decoded and
> on-disk, Geary can serve them up instantly.

Do we have any evidence that this is actually a choke point?  IIRC, we put those images in as base64 data URLs, not as file URLs.  If we just read from the MIME part, we'll already have the data base64 encoded!

> One point I've not seen addressed (but perhaps I missed it): what happens if
> you receive a message before your key is installed on your keyring?  An odd
> situation, but imagine a user w/ a fresh system running Geary for the first
> time, downloading all his messages, and later adding his keys to his
> keyring.  It's a corner case, but from some of the approaches I've seen
> above, will he be hosed trying to read messages he downloaded before
> updating his keyring?  Or, since GPG failed, are they irrevocably unreadable?

This is the big problem with the current patch.  Right now, the body does get decrypted, since the decryption RFC822.Message will occur as we read it out of the database.  However, the various encryption attachments will show up and real attachments will not.  Additionally, a reasonable optimization would be to skip this second decryption step (since it's usually unnecessary), which would further break this.

I list three potential solutions in my previous comment.  The easiest to implement would be to write a little routine that searches the database for encrypted messages, decrypts them, and replaces the cypher text with the plain text.  This could run as part of dummy database upgrade now and/or be triggered by a command line option.  This wouldn't be user friendly, but you'd only have to run it at most once per machine.

> Treat decrypted messages like attachments, that is, write the decrypted body
> to a file on disk and mark in the database that "this email's body is
> decrypted in a file stored in a particular location".  When that mark is
> detected, load the file into the Geary.Email rather than from the
> MessageTable, where it's stored encrypted.  (Or, do the same in a separate
> SQLite table.)

Is this really any easier than just replacing the encrypted email with a decrypted version, as we do with the current patch?  Also, remember that there's no "canonical body" -- the client passes a callback to the get_body() method to handle inline images.

> I don't know from the past messages if GPGme
> is still on the table, I realize that slows down the on-demand process
> considerably.

Emersion has done other nice work that we need to get into a reviewable state that uses GPGme to decode inline encryption.  (Right now, this happens in a just-in-time fashion.)  But I don't know that there's any advantage to using that over GMime for the PGP/Mime messages.  (I think GMime is actually doing something similar to GPGme -- shelling out to a gpg command line.  We're just largely hidden from the details.)

> I know some people want the encrypted messages to remain encrypted, only
> decrypted in-memory and never written to disk.  I appreciate the desire for
> that, but as has been said elsewhere, an encrypted filesystem (or home
> directory) is the better solution here.  (Do other email clients really not
> write a single plaintext byte to disk?  It would be worth looking at
> exemplars here.)

I'd be interested in hearing how other clients work, but I'm still wary of trying to store the data encrypted.  That seems too much like a promise to protect that data, and I don't think we'll actually live up to that promise.

It's probably past the time where we should split this into sub-bugs to cover all the details.  Shall I do that?  Should we continue the PGP/Mime decryption discussion here, or create a new bug for it?

Just wanted to chime in and say that I use Geary and GPG at work and I'd *REALLY* like the two to be integrated.

Beyond that I believe it's generally vital to add encryption features to email clients.

Has there been any additional work done on signing / encrypting in the last year?  Has this been split into multiple bugs, and if so, what are their bug numbers?

I'd really like to get signing and encryption support into Geary. After re-reading the comments and the mailing list thread above, this seems to be a summary of the requirements specific for GPG support:

 - Sign new messages using PGP/MIME (i.e. multipart/signed + application/pgp-signature)
 - Encrypt new message bodies and attachments using PGP/MIME (i.e. multipart/encrypted)
 - Verify messages signed using PGP/MIME or inline PGP
 - Decrypt message bodies encrypted as PGP/MIME or inline PGP
 - Decrypt message attachments encrypted as PGP/MIME

Also, since this is an email client, we want to retain things like full-text search, the ability to easily handle attachments and so on. I.e we don't want to cause a regression in any of Geary's existing features. Also, the processes of sending an receiving signed/encrypted email should require as minimal intervention from the user as possible, and manual configuration should be kept as minimal as possible - if you have a GPG key, it should Just Work™.

Specifically, a non-feature is requiring Geary to encrypt it's message store or similar, since it is not itself an encryption tool - such users should use encrypted home directories or partitions instead.

I'll create a bug for each of the requirements above and make them a dependency of this one.

Emersion: Are you still keen to work on this? I'm going to be busy with the WebKit2 port for a while but will very happily review patches and provide advice where possible.

Since it is possible to set up Geary on a machine without the GPG key configured, it's possible that emails will come in encrypted and cannot be decrypted before being displayed.  If the GPG key is later configured, Geary would have to be able to decrypt on display and not just on reception.

If Geary must have the decrypt on display ability anyway, wouldn't it be simpler to store the emails as they came in and *always* decrypt on display.  It would mean only one code path for decryption.

It would also mean that when a user moves or copies an encrypted email from folder to folder, possibly from one account to another, Geary couldn't possibly upload a decrypted copy of an originally encrypted email.  Writing a decrypted copy of the email or its attachments into /tmp is a minor security sin.  Uploading a decrypted copy of an email the user expects to remain encrypted is a serious security sin.

I would think that anyone dealing with encrypted emails is happy to take an extra second or two to decrypt a long, graphics-heavy email each time they view it.  I know I am.  In fact, I feel better for it because the performance hit is a subtle bit of feedback that the email takes more effort to show and is therefore safe.

For simplicity and safety, I think stored-as-sent, decrypt-on-display is the best option.

(In reply to John Franklin from comment #37)
> 
> For simplicity and safety, I think stored-as-sent, decrypt-on-display is the
> best option.

John, I tend to mostly agree. It is important to ensure that copies etc on the server remain encrypted, but it's also important to make decrypted text and attachments available for things like full text search and opening in external apps. So I'd advocate stored-locally-as-both, decrypt-on-display. When doing remote operations care must then be taken to never ship decrypted data over the wire, but that and decrypt-on-display are the costs to be borne.

NB all: Let's take general discussion about the right way to do PGP to IRC or the mailing list. There's too many people CC'ed on this bug to have a consensus-building discussion.

Further comments here should be about specific patches, or what they contain.

Ta!

A switch whether to store the messages in unencrypted form would be welcome though.

Why not just simply integrate pEp[0], "pretty Easy privacy"? It offers you a GNU licensed Engine[1] through a couple of Adapters[2][3][4] and is compatible to OpenPGP, S/MIME and OTR.

In the usage and the interface pEp follows the same aim: reducing the user experience to elementary and easy to use handling. To take the user away the hassle of "damn near unusable" crypto.

Best, Lix

[0] https:/pep-project.org/
[1] https://cacert.pep.foundation/dev/repos/pEpEngine/
[2] https://cacert.pep.foundation/dev/repos/pEpPythonAdapter/
[3] https://cacert.pep.foundation/dev/repos/pEpJSONServerAdapter/
[4] https://cacert.pep.foundation/dev/repos/pEpQtAdapter/

Why not just simply integrate pEp[0], "pretty Easy privacy"? It offers you a GNU licensed Engine[1] through a couple of Adapters[2][3][4] and is compatible to OpenPGP, S/MIME and OTR.

In the usage and the interface pEp follows the same aim: reducing the user experience to elementary and easy to use handling. To take the user away the hassle of "damn near unusable" crypto.

Best, Lix

[0] https://pep-project.org/
[1] https://cacert.pep.foundation/dev/repos/pEpEngine/
[2] https://cacert.pep.foundation/dev/repos/pEpPythonAdapter/
[3] https://cacert.pep.foundation/dev/repos/pEpJSONServerAdapter/
[4] https://cacert.pep.foundation/dev/repos/pEpQtAdapter/

Hi Lix,

(In reply to Lix from comment #42)
> Why not just simply integrate pEp[0], "pretty Easy privacy"? It offers you a
> GNU licensed Engine[1] through a couple of Adapters[2][3][4] and is
> compatible to OpenPGP, S/MIME and OTR.

This bug is about implementing PGP directly - please open a new bug to cover implementing a different encryption method. To have a discussion about the merits of implementing PEP vs PGP vs PGP via PEP, the Geary mailing list would be the best place instead of here in Bugzilla. Thanks!

Feel free to contact me if you have questions about GMime's GPG support.

As it seems some comments have noted, GMime does not support inline-pgp, but it does do everything you'd need it to do for PGP/MIME.

For PGP support at minimum we must be able to send as, 

PGP/MIME, aka RFC 3156


Receiving ideal would be 
PGP/MIME, and inline-pgp

Save as recived, decrypt on display, would be the most secure and direct implimentation.

Is the mailing list or IRC still happening for this now?  I just checked out Geary, and I'm stuck on thunderbird until GPG is working.

GMime 3.0 will have inline-PGP support (in both directions).

It will also have full S/MIME support.

This stuff is actually already implemented in git master, there's just a few more (unrelated) API's that I'd like to clean up a bit first before releasing 3.0.

I've increased the bounty on https://www.bountysource.com/issues/991629-transparent-encryption-and-signing-with-gpg for this issue.

However, currently the bountysource links to the old issue tracker at yorba. (http://redmine.yorba.org/issues/4919) Could one – who is able to do that – maybe change it?

I think that only the user who started the bounty can update the link in the bounty header.

However, the comments made here appear also on bountysource. So at least we can make the link visible in the comment section. Here it is:

https://bugzilla.gnome.org/show_bug.cgi?id=713403

No, someone official from Geary can also the claim the Geary "team" on BugBounty, so can edit all bounties or so and I think they can alo update the URL.

On https://www.bountysource.com/teams/geary/settings when you are logged in.

1d28ed33, thanks for the bounty contribution.

I just emailed support@bountysource.com and asked them to update the link.  I've found them to be pretty responsive in the past, so hopefully they can fix it soon.

I just pledged another $10. Do you know this is the ticket with the highest bounty?

So has there been any news? Any ETA or so, maybe? (Note when you are working on this, you can assign it to you at https://www.bountysource.com/issues/1353854-transparent-encryption-and-signing-with-gpg and also state the ETA there.)

Is Gmime being worked on here? https://github.com/jstedfast/gmime

Also pledged for Autocrypt support: https://www.bountysource.com/issues/53943682-support-autocrypt-requires-pgp https://bugzilla.gnome.org/show_bug.cgi?id=791778

I also wanted to ping this as I think it is the sole reason to use a local mail client over something like gmail. I'm currently looking into how I can help with this but it would be great of one of the regular developers of geary could comment here with thoughts or small tasks the community could do to get involved.

(In reply to Max Ehrlich from comment #52)
> I'm currently looking into how I can
> help with this but it would be great of one of the regular developers of
> geary could comment here with thoughts or small tasks the community could do
> to get involved.

The bugs listed as dependencies up the top are a high-level breakdown of the work needed to be done to fully implement GPG support. Admittedly the order is a bit confusing, so I'd suggest they would want to be done in the following order:

1. Bug 788275 - Port to GMime 3.0
2. Bug 766273 - Develop API for querying GPG state from Geary
3. Bugs 766276 and Bug 766277 - Implement verifying and decrypting
4. Bugs 766274 and Bug 766275 - Implement signing and encrypting

The first bug is some infrastructure work that will need to be done up front. The second involves some research and development about how best to integrate GPG and develop a simple API for querying it. The last two pairs of bugs need some UX design work still and each depend on (1) and (2) happening first.

So anyone who can help out with coding could really help by taking (1) and (2) on right now. Once those two get done and we have the UX sorted out, (3) and (4) would also be able to get knocked off as well.

(In reply to 1d28ed33 from comment #47)
> I've increased the bounty on
> https://www.bountysource.com/issues/991629-transparent-encryption-and-
> signing-with-gpg for this issue.
> 
> However, currently the bountysource links to the old issue tracker at yorba.
> (http://redmine.yorba.org/issues/4919) Could one – who is able to do that –
> maybe change it?

Bountysource decided to update their Terms of Service:

> 2.13 Bounty Time-Out.
> If no Solution is accepted within two years after a Bounty is posted, then the Bounty will be withdrawn and the amount posted for the Bounty will be retained by Bountysource. For Bounties posted before June 30, 2018, **the Backer may redeploy their Bounty to a new Issue by contacting support@bountysource.com before July 1, 2020**. If the Backer does not redeploy their Bounty by the deadline, the Bounty will be withdrawn and the amount posted for the Bounty will be retained by Bountysource.

This is covered by https://gitlab.gnome.org/GNOME/geary/issues/6, closing in favour of that.