GNOME Bugzilla – Bug 588557
Adding bodystructure stores to Camel's IMAP
Last modified: 2009-09-14 11:09:12 UTC
I would like Evolution to store IMAP's bodystructure as often as possible. The reason for this is storing RDF graphs. Let me explain. I wrote a EPlugin for Evolution that is compiled and distributed by the Tracker project. This plugin fetches as much metadata about E-mails as possible and pushes it into Tracker's RDF store. Tracker's RDF store is a Nepomuk-ontology based RDF store that offers SPARQL as query language, and SPARQL Update as storage language. We have enhanced the Nepomuk message ontology so that it can store the entire structure of an E-mail. This means the structure that you can request using the BODY or BODYSTRUCTURE FETCH requests. These return you the MIME structure of the message in a pre-parsed skiplist format. Using this info our plugin can reconstruct the message's skeleton, but then in RDF as a graph. I don't just talk. I wrote the patch to do this too. It's attached. I tried to keep the changes small, and I implemented the migration code so that the tables' schemas will automatically be converted. The patch needs a good review, though. And testing. This is the point of the patch (a stored bodystructure string): sqlite> select bodystructure from 'INBOX/100' LIMIT 2; ("TEXT" "PLAIN" ("CHARSET" "ISO-8859-1") NIL NIL "8BIT" 2304 53 NIL NIL NIL NIL) ("TEXT" "PLAIN" NIL NIL NIL "7BIT" 3829 80 NIL NIL NIL NIL) sqlite> Note about adding BODYSTRUCTURE to the IMAP FETCH query that IMAP servers usually cache the body-structures. It wont slowdown an IMAP server, one that is worth being called "an IMAP server", much. It will add some bandwidth. But apparently isn't Evolution trying to save bandwidth, otherwise camel-imap-folder.c would use ENVELOPE instead of BODY.PEEK[HEADER.FIELDS(-a long list-)] (right?). Here are some pointers about things I just wrote: The Nemomuk message ontology http://git.gnome.org/cgit/tracker/tree/data/ontologies/34-nmo.ontology How we want this stuff in Tracker, how it could be used http://live.gnome.org/Tracker/Documentation/EmailSparql The Evolution plugin: http://git.gnome.org/cgit/tracker/tree/src/plugins/evolution/tracker-evolution-plugin.c Let me know!
Created attachment 138403 [details] [review] Patch that implements this
Created attachment 138404 [details] [review] Patch that implements this
mbarnes, ping
Philip, I thought I told you this. Don't extend the table, instead create a new table and store it. I really want to make Evo's table smaller, as static info, vs the modifyable info (flags, etc). To avoid it, for Anjal, I store the text_preview conditionally inside the db, but as a table per folder. I would like you to do it that way, instead of creating a new version and extending the existing table. Of course you know that you can read multiple table's columns on a single query. [ you know sqlite better than me :-) ]
Ok, allow me a few days to rework the patch then.
Working on this today, expect a patch in a few minutes
Created attachment 140963 [details] [review] Reworked version of same patch Uses '%s_bodystructure' as table name, as requested earlier.
Ping, attached patch is reworked as requested. Can we have a time-estimate for when the patch will be reviewed?
Philip, I'm sorry. I would be busy for a couple more days :(. I would look at it after that.
Couple of things... @@ -1195,7 +1199,12 @@ camel_db_migrate_folder_prepare (CamelDB *cdb, const gchar *folder_name, gint ve /* Migration stage one: storing the old data */ - if (version < 1) { + if (version < 2) { + + /* Between version 1-2 the following things are changed + * ADDED: bodystructure: text + * */ + /* Between version 0-1 the following things are changed * ADDED: created: time * ADDED: modified: time This migration is not necessary. It will be needed only when you add things to the same table. Just updating the versions should be fine. You might have to write/modify code to evolution-exchange and evolution-mapi for the api changes *_new_from_message() thing. Otherwise it seems fine to me. Akhil, can you test the patch once it is done. Matt/Chen should approve this.
ok, such a trivial change can/will be done during commit time.
Sure. Chen/Matt your call.
Pushed: http://git.gnome.org/cgit/evolution-data-server/commit/?id=e41c2f1e84a15db63d8c292190148b57f1d37f2b
(ps. approval was over IRC)