Reassigning to E-D-S. This appears to be happening in camel-smime-context.c::smime_context_encrypt_sync() which just calls CERT_FindCertByNicknameOrEmailAddr(). And obviously fails.

It's not clear how E_CONTACT_X509_CERT was ever supposed to work, if the lookups are done *within* camel code. Should we be providing a PKCS#11 module to NSS which does relevant addressbook lookups to find a cert?

Is there a bug which says anything about inclusion of this years GSoC, which made the pkcs11 module for this purpose, thus this one can be closed in favour of it?

I've fixed up a bunch of stuff in EWS and EDS to get the module performing nicely — implementing cursors and DRA for EWS was the start of it, and then fixing up the indexing and optimising the SQL that EBookSqlite uses for the query, and getting the benchmark suites running again, and finding other DRA regressions in master and.... eventually the module itself was working fairly nicely.

However, Evolution's behaviour goes to pot as soon as it starts seeing thousands of certificates. The O(n²) behaviour starts in NSS and we just make it worse (partly by getting the same list three times — from the main thread!). This is bug 736808, which I think is the main blocker for merging the module and actually starting to use it automatically.

This may also fix bug #203251.

David, the bug #736808 is closed for some time now, what is the state of the module, please? Maybe it can be merged for 3.30 (3.29.1) once the sources are branched?

The state of the evolution-pkcs11 module?

On the ancient Ubuntu system I have connected to a work network with PKCS#11 certs in a GAL, it works.

On a more recent Fedora box it builds, but I have no PKCS#11 certs in address books to test it with.

I'm sure it'll need little bit of tidying up as it's merged, but I think it's a good idea to do so. Worst case we don't have to enable it by *default* but we should certainly be *intending* to do so.

(In reply to David Woodhouse from comment #5)
> The state of the evolution-pkcs11 module?

Yes, that thing. Let's wait for the gnome-3-28 branch, then it can be checked in the eds sources (I suppose it'll be part of eds). But I'd like to see it first too, to do ("proper") review of it, with which we can start earlier.

https://github.com/yuumasato/evolution-pkcs11 is the latest, I believe.

(In reply to David Woodhouse from comment #7)
> https://github.com/yuumasato/evolution-pkcs11 is the latest, I believe.

Okay, thanks. After a quick look through them:
a) coding style doesn't match the one used in evolution* code
b) function names related to the module itself should be prefixed with something
   unique, to avoid symbol clash in runtime
c) if I understand it properly, then the only required public function of the
   module is "C_GetFunctionList". Can the other functions be named differently
   and maybe be also static in the module?
d) how does the module work in reality? According to the tests it looks like
   the application is supposed to call dlopen(), get the list of functions
   and then initialize the module, thus NSS knows about it, right? First
   of all, I'd like to avoid direct usage of dlopen() and dlsym(), it's
   better to use glib wrappers for these (GModule). Then this initialization
   might go into camel_init(), right? But then it means the library
   knows about certain module, which breaks abstraction a bit. Not a real
   problem, I'm just saying it.
e) can we avoid inclusion of pkcs11f.h and pkcs11t.h? They look like something
   provided by NSS. The license bits mentioning RSA Security Inc. in there
   scare me somehow.
f) the sources might be re-licensed to match license used in evolution-data
   -server, like the one here:
   https://git.gnome.org/browse/evolution-data-server/tree/src/camel/camel.c
   with left Yuuma Sato as the copyright owner and author, of course.

I tried to compile it, and it failed with:
.libs/object.o: In function `get_or_create_object':
   evolution-pkcs11/src/object.c:51: undefined reference to `PORT_NewArena'
and several other. I link with:
   LDFLAGS='-Wl,--as-needed -Wl,-z -Wl,relro -Wl,-z -Wl,now -Wl,-z -Wl,defs'
Removing those LDFLAGS helps. I get only one compiler warning:
   object.c:124:12: warning: unused variable ‘_a’ [-Wunused-variable]
but I'm afraid it's more about not using all the compiler warnings which evolution-data-server uses, which can mean it'll print more of them once the sources are used in eds. For example the test-app.c declares:
   void *pkcs11_so;
in the middle of the code block, not at the beginning, which generates a compile-time warning in eds & others.

Created attachment 368305 [details]
startup work - compile in eds

This is just a startup work, to add the module into eds sources. It only compiles, but nothing else. I did mimic what the Makefile.am & co does, maybe except I installed the module into private eds lib directory, under name madule-nss-pcks11 (do not rely the name will have .so extension, be platform independent in the code, use G_MODULE_SUFFIX). Quite some work is needed to make it behave as expected. You may even consider renaming tests, just to match their binary counterpart names (the test.c is compiled into load-test?) and maybe instead of suffixing the names, then prefix them, like:

   addressbook-access-test ~> test-addressbook-access
   load-test ~> test-load

I'm also wondering whether nss-pkcs11 is a good name for the module. Yes, it's installed under eds' private library directory, which avoids confusion, but if you think the nss-pkcs11 is not a good name in the sources, like to add into the name something about "book lookup certs" or such, and not making it too long, then I'm happy to hear your suggestions, because I didn't have any good idea myself.

Very briefly (I'll pay more attention later). The module provides a standard interface through C_GetFunctionList(), yes. Everything else can be private.

This is the PKCS#11 standard for loadable crypto object stores. It isn't NSS-specific, and can be used by all decent crypto libraries. When we're done, we should put it in the p11-kit configuration as one of the modules which automatically gets made available to applications.

NSS is a little bit of a special case here because it's buggy and doesn't honour the system-wide configuration for which modules to load, but we can work around that by manually loading it for now.

All the dependencies are addressed now. I guess we can move forward here.
What do you think, David?

Duplicate:
https://gitlab.gnome.org/GNOME/evolution-ews/issues/3

While Yuuma's work is good for PKCS#11 and allows sharing the certificates between any applications which can use that module, it doesn't cover PGP certificates, thus I chose a different approach. It was possible, because the gpg2 has --recipient-file option, which can have stored the key to use for encryption, instead of a key which is part of the gpg2 storage.

There is a difference between S/MIME and PGP here. With S/MIME is asked for the contact certificates and then is searched in the contact's store and "the best" certificate is used among those provided. This is not possible with PGP, not that easily at least, and I hesitate to write some parsers of the (machine-readable) output of gpg2. Thus with PGP the keys found in address books are always used, despite a better certificate can be found in the user's stored keys.

The approach adds CamelSession::get_recipient_certificates_sync() virtual method, which is optional. The descendants can use it for these look ups. To make it easier for anyone I also added e_book_utils_get_recipient_certificates_sync() which can be used within the implementation of the CamelSession method (it's part of libebook).

Evolution has an option org.gnome.evolution.mail lookup-recipient-certificates, which can have three values:
 - “off” - completely disables certificate lookup
 - “autocompleted” - provides certificates only for auto-completed contacts
 - “books” - uses certificates from auto-completed contacts and searches
    in books marked for auto-completion
where the default is “books”. The reason is that the lookup in books can be slow, thus users can avoid it and use only certificates for auto-completed contacts in the composer. For those whom want to use the certificates/keys they have imported only the "off" option is the way to go. The certificates in auto-completed contacts have precedence, thus when present, the books are not searched for them again.

Also, the first book which provides requested certificate/key wins. And the contact should not have stored more than one certificate/key (it can have both S/MIME and PGP, but ideally only one for each type), otherwise also the first, in the order of the existence in the vCard, wins. I guess (and hope) this is expected behaviour.

Created commit ea9ed9224 in eds master (3.29.4+)
Created commit_3627d75e1 in evo master (3.29.4+) [1]

[1] https://gitlab.gnome.org/GNOME/evolution/commit/3627d75e1c

*** Bug 203251 has been marked as a duplicate of this bug. ***