After an evaluation, GNOME has moved from Bugzilla to GitLab. Learn more about GitLab.
No new issues can be reported in GNOME Bugzilla anymore.
To report an issue in a GNOME project, go to GNOME GitLab.
Do not go to GNOME Gitlab for: Bluefish, Doxygen, GnuCash, GStreamer, java-gnome, LDTP, NetworkManager, Tomboy.
Bug 158105 - Index maildirs
Index maildirs
Status: RESOLVED FIXED
Product: beagle
Classification: Other
Component: General
unspecified
Other Linux
: Normal enhancement
: Milestone 2
Assigned To: Joe Shaw
Joe Shaw
Depends on:
Blocks:
 
 
Reported: 2004-11-12 21:25 UTC by Nat Friedman
Modified: 2005-06-30 17:05 UTC
See Also:
GNOME target: ---
GNOME version: ---


Attachments
Beagle Patch to support MailDir (22.33 KB, text/plain)
2005-05-16 12:21 UTC, Noam Rathaus
  Details
Unified patch (3.20 KB, patch)
2005-05-16 12:24 UTC, Noam Rathaus
none Details | Review
Filter for Maildir files (4.78 KB, text/plain)
2005-05-16 15:27 UTC, Noam Rathaus
  Details
MailDir.tar.gz (6.01 KB, application/x-compressed-tar)
2005-05-26 06:53 UTC, Noam Rathaus
  Details

Description Nat Friedman 2004-11-12 21:25:58 UTC
Index mboxes and maildirs.
Comment 1 Joe Shaw 2004-12-01 22:52:27 UTC
Filter out "joe milestone spam" to ignore these.

Moving milestones from versions to target milestones, and adding in actual versions
Comment 2 Joe Shaw 2004-12-02 19:56:09 UTC
mbox is done.  we don't do maildirs.  how do you set up a maildir?
Comment 3 Nat Friedman 2004-12-02 20:08:09 UTC
JP, how do you make evo do maildir?
Comment 4 Joe Shaw 2004-12-02 20:12:11 UTC
No, I know how to make it do maildir, I need to know how to actually CREATE a
maildir.  Mine on imap.ximian.com is waaaay too big to download locally, and I
don't understand all the "cur" and "new" directories.
Comment 5 Joe Shaw 2004-12-07 20:27:18 UTC
maildirs are kept outside the .evolution universe, so this will take a good
amount of reworking things like crawling, and it's rare enough that I am punting
it until M2.
Comment 6 Noam Rathaus 2005-05-16 08:57:55 UTC
The Util/Mozilla.cs parser is Maildir compatible. I have created a MailDirDriver
that indexs the /MailDir folder I have on my computer using the Mozilla class
(MessageReader and Message object).

I am not sure about a few things found there, like Uri (Maildir doesn't have
Uri, I have used Message-Id instead)...

Should I post the files here?
Comment 7 Fredrik Hedberg 2005-05-16 12:21:34 UTC
Yes you should. 

Unfortunatly, the Mozilla.cs is probably the worst code ever produced. Maybe we
should get something that's allready done and tested into Beagle for this, and
so that we actually could get the Thunderbird backend working. Something like
SharpMimeTools perhaps? (http://anmar.eu.org/projects/sharpwebmail/)
Comment 8 Noam Rathaus 2005-05-16 12:21:55 UTC
Created attachment 46480 [details]
Beagle Patch to support MailDir
Comment 9 Noam Rathaus 2005-05-16 12:22:47 UTC
Hi,

After some work, and help from #dashboard people, got it working... should work
against CVS.

Note the FIXME things, the work isn't complete, but its a good starting point...
BTW, it will only index the /Mail/ directory not the DImap directory or Imap
directory...

Once this has been introduced to the tree, I will continue work...

I will be also happy to receive comments on the way the code has been writen
(beagle wise, has it been structured correctly).
Comment 10 Noam Rathaus 2005-05-16 12:24:59 UTC
Created attachment 46481 [details] [review]
Unified patch
Comment 11 Noam Rathaus 2005-05-16 15:27:57 UTC
Created attachment 46495 [details]
Filter for Maildir files

Hi,

If the following Filter is used, any text/x-mail or message/rfc822 will be
parsed according to the content of it and not as a regular file.
Comment 12 Noam Rathaus 2005-05-26 06:52:34 UTC
The following patch will make the CVS version index MailDir directories:
Index: Util/Makefile.am
===================================================================
RCS file: /cvs/gnome/beagle/Util/Makefile.am,v
retrieving revision 1.53
diff -r1.53 Makefile.am
66a67
<       $(srcdir)/Rfc822DateTime.cs                             \

Index: Makefile.am
===================================================================
RCS file: /cvs/gnome/beagle/beagled/Makefile.am,v
retrieving revision 1.108
diff -r1.108 Makefile.am
246a247,252
< maildirqueryable = $(srcdir)/MailDirDriver
< MAIL_DIR_QUERYABLE_CSFILES =                                  \
<       $(maildirqueryable)/MailDirDriver.cs            \
<       $(maildirqueryable)/MailDirIndexableGenerator.cs        \
<       $(maildirqueryable)/MailDirCrawler.cs
<
275a282
<       $(MAIL_DIR_QUERYABLE_CSFILES)                           \

See MailDir.tar.gz for additional files incessary
Comment 13 Noam Rathaus 2005-05-26 06:53:11 UTC
Created attachment 46892 [details]
MailDir.tar.gz
Comment 14 Joe Shaw 2005-06-03 17:36:44 UTC
Sorry it took me a while to look into this.

I think the filter approach you took originally is the right one.  The nice
thing about maildir is that it's one file per message, so we can easily
piggyback on the file system backend for doing things like actually crawling,
creating indexables, etc.  It would dramatically simplify things.  The only
catch is if you are trying to load maildirs from locations which aren't normally
crawled (like a dot directory).

The filter you posted is using the Mozilla parsing code, which Fredrik says is
pretty uniformly bad.  We're already requiring gmime to parse mboxes, so we
might as well use it for maildir files as well.
Comment 15 Noam Rathaus 2005-06-04 16:52:53 UTC
How would you solve the hidden directory problem of MailDir?

I have no problem converting my MailDir crawler (not the filter) to support
Gmime, but I am not going to do work which won't be used (if the Beagle project
is going to take a different approach to indexing MailDirs).

P.S I am currently using the crawler, and the filter simply doesn't cut it, as
it doesn't crawl into hidden directories.
Comment 16 Joe Shaw 2005-06-07 20:17:27 UTC
I'd solve the hidden directory problem probably by creating a very simple
backend that crawls the hidden directory.  Look at the gaim backend to get an
idea of how that might work.  It might be possible to integrate that into the
file system backend at some point in the future, but we can worry about that later.

The bulk of the work needs to be done in the filter, that's the only appropriate
place to parse the data.  (And you get the added side benefit of maildirs
outside the hidden directory)
Comment 17 Noam Rathaus 2005-06-09 14:47:19 UTC
Hi, have you looked into my Filter for Maildir files? (attached below)

Is it work in the right direction?
Comment 18 Joe Shaw 2005-06-09 16:55:49 UTC
The filter I'd like to see ported to GMime.
Comment 19 Noam Rathaus 2005-06-13 19:14:55 UTC
Hi,

I will start working on porting it to GMime, but please "promise" that you will
at least commit it, once it is ready to the CVS, as I hate doing work for nothing.
Comment 20 Joe Shaw 2005-06-30 17:05:57 UTC
I've checked in an implementation of this.