After an evaluation, GNOME has moved from Bugzilla to GitLab. Learn more about GitLab.
No new issues can be reported in GNOME Bugzilla anymore.
To report an issue in a GNOME project, go to GNOME GitLab.
Do not go to GNOME Gitlab for: Bluefish, Doxygen, GnuCash, GStreamer, java-gnome, LDTP, NetworkManager, Tomboy.
Bug 588557 - Adding bodystructure stores to Camel's IMAP
Adding bodystructure stores to Camel's IMAP
Status: RESOLVED FIXED
Product: evolution-data-server
Classification: Platform
Component: Mailer
unspecified
Other Linux
: Normal normal
: ---
Assigned To: evolution-mail-maintainers
Evolution QA team
Depends on:
Blocks:
 
 
Reported: 2009-07-14 16:30 UTC by Philip Van Hoof
Modified: 2009-09-14 11:09 UTC
See Also:
GNOME target: ---
GNOME version: ---


Attachments
Patch that implements this (18.39 KB, patch)
2009-07-14 16:31 UTC, Philip Van Hoof
none Details | Review
Patch that implements this (18.39 KB, patch)
2009-07-14 16:33 UTC, Philip Van Hoof
needs-work Details | Review
Reworked version of same patch (18.72 KB, patch)
2009-08-17 13:54 UTC, Philip Van Hoof
reviewed Details | Review

Description Philip Van Hoof 2009-07-14 16:30:41 UTC
I would like Evolution to store IMAP's bodystructure as often as
possible. The reason for this is storing RDF graphs. Let me explain.

I wrote a EPlugin for Evolution that is compiled and distributed by the
Tracker project. This plugin fetches as much metadata about E-mails as
possible and pushes it into Tracker's RDF store.

Tracker's RDF store is a Nepomuk-ontology based RDF store that offers
SPARQL as query language, and SPARQL Update as storage language.

We have enhanced the Nepomuk message ontology so that it can store the
entire structure of an E-mail. This means the structure that you can
request using the BODY or BODYSTRUCTURE FETCH requests. These return you
the MIME structure of the message in a pre-parsed skiplist format.

Using this info our plugin can reconstruct the message's skeleton, but
then in RDF as a graph.

I don't just talk. I wrote the patch to do this too. It's attached.

I tried to keep the changes small, and I implemented the migration code
so that the tables' schemas will automatically be converted. The patch
needs a good review, though. And testing.

This is the point of the patch (a stored bodystructure string):

sqlite> select bodystructure from 'INBOX/100' LIMIT 2;
("TEXT" "PLAIN" ("CHARSET" "ISO-8859-1") NIL NIL "8BIT" 2304 53 NIL NIL NIL NIL)
("TEXT" "PLAIN" NIL NIL NIL "7BIT" 3829 80 NIL NIL NIL NIL)
sqlite> 

Note about adding BODYSTRUCTURE to the IMAP FETCH query that IMAP
servers usually cache the body-structures. It wont slowdown an IMAP
server, one that is worth being called "an IMAP server", much.

It will add some bandwidth. But apparently isn't Evolution trying to
save bandwidth, otherwise camel-imap-folder.c would use ENVELOPE instead
of BODY.PEEK[HEADER.FIELDS(-a long list-)] (right?).


Here are some pointers about things I just wrote:

The Nemomuk message ontology
http://git.gnome.org/cgit/tracker/tree/data/ontologies/34-nmo.ontology

How we want this stuff in Tracker, how it could be used
http://live.gnome.org/Tracker/Documentation/EmailSparql

The Evolution plugin:
http://git.gnome.org/cgit/tracker/tree/src/plugins/evolution/tracker-evolution-plugin.c


Let me know!
Comment 1 Philip Van Hoof 2009-07-14 16:31:16 UTC
Created attachment 138403 [details] [review]
Patch that implements this
Comment 2 Philip Van Hoof 2009-07-14 16:33:33 UTC
Created attachment 138404 [details] [review]
Patch that implements this
Comment 3 Chenthill P 2009-07-29 18:11:30 UTC
mbarnes, ping
Comment 4 Srinivasa Ragavan 2009-07-29 19:01:35 UTC
Philip, I thought I told you this. Don't extend the table, instead create a new table and store it. I really want to make Evo's table smaller, as static info, vs the modifyable info (flags, etc). To avoid it, for Anjal, I store the text_preview conditionally inside the db, but as a table per folder.

I would like you to do it that way, instead of creating a new version and extending the existing table.

Of course you know that you can read multiple table's columns on a single query. [ you know sqlite better than me :-) ]
Comment 5 Philip Van Hoof 2009-07-29 20:24:10 UTC
Ok, allow me a few days to rework the patch then.
Comment 6 Philip Van Hoof 2009-08-17 13:32:04 UTC
Working on this today, expect a patch in a few minutes
Comment 7 Philip Van Hoof 2009-08-17 13:54:56 UTC
Created attachment 140963 [details] [review]
Reworked version of same patch

Uses '%s_bodystructure' as table name, as requested earlier.
Comment 8 Philip Van Hoof 2009-09-01 08:34:23 UTC
Ping, attached patch is reworked as requested. Can we have a time-estimate for when the patch will be reviewed?
Comment 9 Srinivasa Ragavan 2009-09-01 11:28:07 UTC
Philip, I'm sorry. I would be busy for a couple more days :(. I would look at it after that.
Comment 10 Srinivasa Ragavan 2009-09-04 04:36:54 UTC
Couple of things...

@@ -1195,7 +1199,12 @@ camel_db_migrate_folder_prepare (CamelDB *cdb, const gchar *folder_name, gint ve
 
 	/* Migration stage one: storing the old data */
 
-	if (version < 1) {
+	if (version < 2) {
+
+		/* Between version 1-2 the following things are changed
+		 * ADDED: bodystructure: text
+		 * */
+
 		/* Between version 0-1 the following things are changed
 		 * ADDED: created: time
 		 * ADDED: modified: time


This migration is not necessary. It will be needed only when you add things to the same table. Just updating the versions should be fine. 

You might have to write/modify code to evolution-exchange and evolution-mapi for the api changes *_new_from_message() thing.

Otherwise it seems fine to me. Akhil, can you test the patch once it is done. 

Matt/Chen should approve this.
Comment 11 Philip Van Hoof 2009-09-07 11:16:33 UTC
ok, such a trivial change can/will be done during commit time.
Comment 12 Srinivasa Ragavan 2009-09-08 07:05:35 UTC
Sure. Chen/Matt your call.
Comment 14 Philip Van Hoof 2009-09-14 11:09:12 UTC
(ps. approval was over IRC)