Can you test if this is still an issue in Evolution 3.11 or later?

3.11 uses GLib's TCP stream class, but is still wrappered for IMAP parsing.

That should help isolate where the error is.

Not easily. I did try getting the test Fedora 21 Evolution SRPM (from rawhide) working on my F20 PC. I could rebuild the package, but it wouldn't run due to library incompatibilities.

I'll see if I can build a virtual F21 system using development/rawhide from a mirror. Never done it before, so a bit of a learning curve there...

Rats... probably easier just to build Evo from source (not SRPM)!

If you have a rawhide yum repo set up (but disabled), you can do something like "yum install --enablerepo=rawhide evolution" and yum should pull any additional packages needed from rawhide.

That is, if you're willing to.  A virtual machine would be ideal, but more work to set up.

Created an F20 virtual machine, and as suggested installed evolution from the rawhide repo, then rebooted. Unfortunately when I try and run evolution from the command line it crashes just as the 'account assistant' (?) starts. I get a SIGTRAP. Using strace, the tail end of the output shows:

============================
access("/home/john/.config/evolution/mail/sortorder.ini", F_OK) = -1 ENOENT (No such file or directory)
access("/home/john/.config/evolution/mail/sortorder.ini", F_OK) = -1 ENOENT (No such file or directory)
write(12, "\1\0\0\0\0\0\0\0", 8)        = 8
futex(0x1db0f90, FUTEX_WAKE_PRIVATE, 1) = 1
write(2, "\n(evolution:1464): GLib-GIO-ERRO"..., 176
(evolution:1464): GLib-GIO-ERROR **: The schema default value for key 'junk-default-plugin' in schema 'org.gnome.evolution.mail' was rejected by the binding mapping function.
) = 176
--- SIGTRAP {si_signo=SIGTRAP, si_code=SI_KERNEL} ---
+++ killed by SIGTRAP (core dumped) +++
Trace/breakpoint trap (core dumped)
============================



John.

(In reply to comment #5)
> (evolution:1464): GLib-GIO-ERROR **: The schema default value for key
> 'junk-default-plugin' in schema 'org.gnome.evolution.mail' was rejected by the
> binding mapping function.

Can it be some new (anti)feature of rawhide's GLib, that it rejects binding even when a transform function is provided? The [1] looks correct, and I do not get the crash on my machine with glib2-2.38.2-2.fc20.x86_64.

[1] https://git.gnome.org/browse/evolution/tree/modules/settings/e-settings-mail-session.c#n90

I couldn't test 3.11.x, and perhaps debugging 3.11/rawhide is not targeting this bug exactly. The test environment to recreate the bug is very easy to setup: Just create a few hundred empty mailboxes on an imap server.

Just install evolution-bogofilter.  I already fixed it in git.

https://git.gnome.org/browse/evolution/commit/?id=5a89422182cc9e5e75af808beef19479bab6e9de

Okay, I installed the 'fedora-release-rawhide' package to get the rawhide repo. I then ran 'yum --enablerepo=rawhide update evolution' and it updated evolution, evolution-data-server and evolution-bogofilter to the 3.11.4 versions.

However, when running evolution I still get the same error (Error fetching folders: list: expecting...).
After that evolution did show 7 or 8 folders (I have about 80 or so), but clicking on Folder/Subscriptions gives the same error.

Running evolution (3.11.4) with strace I now get:

========================
(evolution:3822): evolution-mail-WARNING **: (mail-send-recv.c:1126):receive_update_got_folderinfo: runtime check failed: (info != NULL)

(evolution:3822): evolution-mail-WARNING **: receive_update_got_folderinfo: Error fetching folders: list: expecting '(' or NEWLINE
========================

Just to say that I have now installed evolution/evolution-data-server 3.9.5-2. It works fine (I had to create a couple of soft links for missing earlier libraries; and a couple of mail filters needed fixing). I got the RPMS from rawhide (the Koji system), and downloaded the FC20 versions. Anything higher than 3.9.5-2 gave the error (...expecting '(' or NEWLINE).

At least I can see my mail now :-)

If you need testing of a new version to fix the problem, then let me know.

Should all Fedora 20 users downgrade to 3.9.x? Should I open a new bug at bugzilla.redhat.com?

I configured a test IMAP account with 500 mailboxes over the weekend, but so far haven't reproduced this problem on Evolution 3.10.x or 3.11.x.

It may be a server-specific issue though.  Can you try 3.10 or 3.11 again, but capture some debugging output for me by running Evolution from a terminal?  I need to see exactly what your server is returning for its LIST result.

   CAMEL_DEBUG=imapx:io evolution

Created attachment 266709 [details]
CAMEL_DEBUG imapx output

(Whoops! forgot to tell yum not to update evolution...!)

Okay, attached is the output from the CAMEL_DEBUG command using FC20 evolution 3.10.3-1.

(In reply to comment #12)
> Should all Fedora 20 users downgrade to 3.9.x? Should I open a new bug at
> bugzilla.redhat.com?

Personal opinion - no to both. It is far from an ideal solution (which it isn't at all, it's just a way to get email to be readable) and a large backwards step really (there seemed to be quite a few intermediate releases of evolution between 3.9.5 and 3.10.3). In my case it required a bit of fiddling with libraries, and requires automatic updates for evolution being disabled in yum.conf (which in turn then requires re-enabling when the problem is fixed). For a user just using evolution, and with not too much Linux savvy, it would probably cause too many problems. I mentioned that 3.9.5 worked for me as a possible clue for the developers, not as a possible solution.

I'm still unable to reproduce this locally because, like you said, the key seems to be that the incoming data is split mid-response across multiple ethernet packets, as your debug log shows:

   [imapx:B] camel_imapx_read: buffer is '
   ...
   * LIST (\Subscribed) "." Sys'  <-- response cut off


Would you be willing to assist with a little debugging?

Unfortunately I didn't indicate which token _was_ received in the "expecting '(' or NEWLINE" error message, mainly because I never expected it to be hit.

Not sure if you're familiar with gdb.  If you are, can you try running Evolution 3.11.x under gdb and set a breakpoint like so:

   (gdb) break camel-imapx-list-response.c:409
   (gdb) continue

That should be exactly where the error message gets set.

Once you hit that breakpoint, I'd like to know the values of 'tok' and 'token':

   (gdb) print tok
   (gdb) print token

That would help me figure out what corner case I've missed.

Okay, well unfortunately gdb itself was giving me errors. It reached the breakpoint and then bombed out with an internal error itself! I tried pushing it on further, but it couldn't display the values you wanted. Sorry, I don't have the details anymore, they went off the top of the screen.

I decided it might just be easier to modify the code to print out the values as part of the error message, so that is what I did. The error now became:

===================
(evolution:23567): evolution-mail-WARNING **: receive_update_got_folderinfo: Error fetching folders: list: expecting '(' or NEWLINE: token: Linux.LUG.Devon    tok: 43
===================

This is actually very interesting! The 'Linux.LUG.Devon' folder should actually be 'Linux.LUG.Devon+Cornwall'. And the decimal tok value of 43 is the ASCII character '+'. So I would say that it seems that evolution is reading the '+' character as something other than part of the folder name.

When I back-installed to evolution 3.9.5 I mentioned before that a mail filtering rule seemed to have been corrupted. It was in fact a rule to the 'Devon+Cornwall' folder, and evo renamed the folder just as 'Devon'. Sorry, I should have spotted that.

That's helpful, thank you!

I was trying to understand the low-level IMAP parsing code which someone else wrote but didn't comment very well.  I think the problem is this 'while' loop here:

https://git.gnome.org/browse/evolution-data-server/tree/camel/providers/imapx/camel-imapx-stream.c#n817

It's reading bytes from the input stream and, based on whether the byte is a special character, deciding whether to stop reading and return a symbolic token or extend the input buffer if necessary and keep reading.  '+' is one of the special characters which returns a token, though I don't yet understand why.

If I'm right, then the previous LIST parsing code in 3.9.5 and older just happened to get past this because it had poor error checking.

And I do have a lot of C++ related mailboxes ...

So is this a coding error from the IMAP server implementation? I notice that both John and myself are using dovecot, I believe.

(Yes, I am using dovecot).

A very quick look at RFC 3501 (the IMAP protocol), and I see nothing special about the '+' character in mailbox names. The formal ABNF specification of a 'mailbox' simply shows it to be a character string with some excepted characters. None of them is the '+' character though.

However. Section 2.2 about client/server responses does mention that '+' may be used:

   The protocol receiver of an IMAP4rev1 client reads a response line
   from the server.  It then takes action on the response based upon the
   first token of the response, which can be a tag, a "*", or a "+".

This may be a red-herring, or it may be that, as said, evolution is seeing the '+' character and misinterpreting it.

A test, I guess, would be to remove the mailboxes with '+' in their name, and then see if evolution works as expected.

I *think* what the logic is doing is trying to chop a continuous input stream into tokens, rather than buffering a full response line and then parsing that.

Frankly I'm a little terrified to touch this part of the code because it's so obfuscated and I'm trying to port this stuff over to GLib-based streams anyway.

Since you obviously know how to build E-D-S, I'll see if I can throw a test patch together for you.  I need to review the RFC a bit first.

Before I embark on a more complicated patch, can you try just removing the '+' character from the "notid_specials" constant in camel-imapx-utils.c?

If that's sufficient to solve the problem for you, then I'll just leave it at that until I can get the thing ported to GLib-based streams.  Because this parsing code is pretty gross, with all kinds of undocumented subtleties.

Yes, removing the '+' character from 'notid_specials' has worked. I built E-D-S using the 3.11.4 code.

Starting evolution I get no error; the folders all appear; and my 'Devon+Cornwall' folder/mailbox appears correctly. If I click on the 'Folder->Subscriptions' link, then all the folders appear (which is correct).
I left the program running for a short while, and duly received a message which was filtered correctly into the 'Devon+Cornwall' folder.
So it all seems to be working correctly.

(Sorry, I noticed that I tend to call these things folders - have done for years, whereas I suspect they should be called mailboxes really.)

Excellent, thanks for helping me with that.

Fixed for E-D-S 3.11.5 and 3.10.4:

https://git.gnome.org/browse/evolution-data-server/commit/?id=34f0f99f715bb3e308f1d838fac435b3d5fb6925

https://git.gnome.org/browse/evolution-data-server/commit/?h=gnome-3-10&id=a5350d6e8b1ef9e3b54839724ea68abb210fe10c

*** Bug 720757 has been marked as a duplicate of this bug. ***

*** Bug 720339 has been marked as a duplicate of this bug. ***

I haven't gone through what everyone else has said, but let me contribute my experience.

I upgraded to Fedora 20 on my spare machine as practice for doing so on my main machine.  I found that I couldn't get evolution to work.  I posted a question in the evolution user mailing list and was told about this bug.  I was also told to upgrade evolution, which I did, but it made no difference.

evolution 3.03 worked previously on the same machine under Fedora 17.  In setting up evolution under Fedora 20, I tried to use the same settings as work on my main machine under Fedora 117, but this was not possible because the choices are different.

I chose

Rec Mail: Server: imap.math.northwestern.edu port 993 SSL on a dedicated port
Authentication pwassword
(The server is running scientific linux, a version of redhat, and I think it is up to date.)

Send Mail: Server: mailhost.math.northwestern.edu port 587 STARTTLS after connecting Authentication Login

Sending email works, but whatever I do, I can't get Receiving email to work.
On the above settings when I click Send/Recieve, I get a Windows
saying
Public-Math(len@imap.math.northwestern.edu)
Scanning Folderds on 'IMAP server ima[.math.northwestern.edu'
but unlike what happens under Fedora 17, there is no progress shown.
It never gets anywhere.

The len@imap.math.northwestern.edu
looks suspicious to me.   Under Fedora 17, it says
Public-Math(imap)imap.math.northwestern.edu.

I hope someone can help me out because I can't continue much longer under Fedora 17 on my main machine, but I can't go to Fedora 20 without knowing that I can get email.

It seems that Thunderbird works fine both for sending and receiving mail, so the problem is with evolution.

I hope someone can fix it.  As I noted, the problems seems to be just for receiving mail

balsa also seems to work.

I am now running evolution 3.10.4-2 with evolution data-server 3.20.4-2.

Sending mail works but receiving mail still doesn't work.

As noted above thunderbird and balsa do seem to work fine.

I hope someone is still trying to fix this!

Can you run evolution from a terminal window like this:

    CAMEL_DEBUG=imapx:io evolution >& logfile

and post the logfile here?  The parser may be tripping over a different character in one of your folder names.

Also check the logfile for any personal information or exposed passwords before posting.  The logfile will contain the raw IMAP commands and responses so I can hopefully see where things are going wrong.

Created attachment 271368 [details]
logfile from running evolution > Send/Receive>Receive

I may be repeating myself, but I let me say what I did.

I ran evolution from the command line as suggested, ran tail -f on logfile2 and stopped evolution when the file stopped changing.

Okay I did as you suggested.  When evolution came up, I ran Receive Mail and did a tail -f on logfile2 untill it stopped changing.  Then I closed evolution.  I will attach the logfile

Created attachment 271451 [details]
logfile after trying Manage subscriptions

Some additional comments:

1.  I notice from the first logfile I uploaded that evolution was running through my many files when it stopped part way  through.  I wondered if the large number of files was an issue, but evolution 3.0.3 had no problem and neither did thunderbird nor balsa.

2.  I tried just manage subscritopsn and I got the usual '(' or NEWLINE erro.  I attached the logfille for that.

(In reply to comment #37)
> 2.  I tried just manage subscritopsn and I got the usual '(' or NEWLINE erro. 
> I attached the logfille for that.

Right, the thing needed is an imapx log, to see what was received from the server that it prevented evolution to work properly. As Matthew wrote at comment #32, please do:
   $ CAMEL_DEBUG=imapx:io evolution &>log.txt
and repeat the same procedure with manage subscription.

I thought I had already done that.  See the Attachments below.  However, I will do it again.  This time the results seem different so I will not attach the files but just include the results here:

loga.txt:  Result of running
CAMEL_DEBUG=imapx:io evolution &>loga.txt
and then trying Send/Receive"
------------------------------------------------------------------
java version "1.7.0_51"
OpenJDK Runtime Environment (fedora-2.4.5.1.fc20-x86_64 u51-b31)
OpenJDK 64-Bit Server VM (build 24.51-b03, mixed mode)

(evolution:5231): evolution-mail-WARNING **: (mail-send-recv.c:1145):receive_update_got_folderinfo: runtime check failed: (info != NULL)

(evolution:5231): evolution-mail-WARNING **: receive_update_got_folderinfo: Error fetching folders: list: expecting '(' or NEWLINE

** (evolution:5231): CRITICAL **: em_utils_folder_is_sent: assertion 'CAMEL_IS_FOLDER (folder)' failed
----------------------------------------------------------------------------------
CAMEL_DEBUG=imapx:io evolution &>logb.txt
and then choosing Manage Subscriptions:

-----------------------------------------------------------------------
java version "1.7.0_51"
OpenJDK Runtime Environment (fedora-2.4.5.1.fc20-x86_64 u51-b31)
OpenJDK 64-Bit Server VM (build 24.51-b03, mixed mode)

(evolution:5152): evolution-mail-WARNING **: (mail-send-recv.c:1145):receive_update_got_folderinfo: runtime check failed: (info != NULL)

(evolution:5152): evolution-mail-WARNING **: receive_update_got_folderinfo: Error fetching folders: list: expecting '(' or NEWLINE

** (evolution:5152): CRITICAL **: em_utils_folder_is_sent: assertion 'CAMEL_IS_FOLDER (folder)' failed
-----------------------------------------------------------------------

I hope that helps.

It would help me if I understood just what people are doing, if anything, to resolve this problem.   As I noted previously,  I upgraded to Fedora 20 on a subsidiary machine and encountered this problem.   I was putting off upgrading my main machine to Fedora 20 until I knew I could get evolution to work, but I can't wait much longer.

There is something going wrong for some reason, because the CAMEL_DEBUG=imapx:io is supposed to tunr on debugging of I/O operations of the IMAPx account, which works for me nicely, and means, when I enter Folder->Subscriptions, lines like these:

[imapx:F] I/O: 'F00038 LIST "" "*" RETURN (CHILDREN SUBSCRIBED STATUS (MESSAGES UNSEEN UIDVALIDITY UIDNEXT HIGHESTMODSEQ))
'
[imapx:F] I/O: '* LIST (\HasNoChildren) "/" "Chats"
* STATUS "Chats" (MESSAGES 0 UIDNEXT 520 UIDVALIDITY 1 UNSEEN 0 HIGHESTMODSEQ 16600)
* LIST (\HasNoChildren) "/" "Contacts"
* STATUS "Contacts" (MESSAGES 16 UIDNEXT 686116 UIDVALIDITY 1 UNSEEN 0 HIGHESTMODSEQ 1571836)
* LIST (\Subscribed \HasNoChildren) "/" "Drafts"

but they are not in your log.

(In reply to comment #40)
> It would help me if I understood just what people are doing, if anything, to
> resolve this problem.

There had been found previously that a '+' in a folder name causes the same error message as you encounter, which had been fixed (see above). It seems that your folder list contains another symbol which confuses the parser, and the task is to figure out what symbol it is.

Created attachment 272470 [details]
Using Send/Receive

Created attachment 272472 [details]
Using Manage Subscriptions

Thanks for the update. The parsing stops on folder:
   restored/538614080670880068-web-restore_2/from_schur/security]
where the ']' in the folder dame confuses the imapx parser, the same as it confused the '+' in folder names. I see that '[' also confuses the parser, the same as a '.' on a Dovecot server, though I thought the '.' is safe to have, when it is properly encoded

(I cannot create a folder containing the dot on a Dovecot server:
  The folder name "A.B" is invalid because it contains the character ".").

I'm reopening this.

I'm working on a fix for the problem I described in comment 17 and comment 22, but it involves reworking some extremely critical and delicate IMAP code, so I won't be backporting it to a stable release until it gets extensive testing in a development release or two.

I removed the offending folder security].   So now evolution works.

That means that I think I can safely upgrade my main machine to Fedora 20.

However, I think I should transfer the entire folder containing that folder to my local machine.  It contains a lot of old mail, which I still want access to, but it doesn't have to reside on the server.

Unfortunately,  I don't see how to do that.  I can right click on the folder in the server account in evolution and choose either copy or move and specify that I want to copy or move to the local account.  But when I do that, I get error messages:

Cannot move folder "old_schur_mail" to "maildir:/home/len/.local/share/evolution/mail/local".

Cannot open target "Invalid folder URI 'maildir:/home/len/.local/share/evolution/mail/local'".

I've posted something about this to the evolution users mailing list.

(In reply to comment #46)
> Cannot open target "Invalid folder URI
> 'maildir:/home/len/.local/share/evolution/mail/local'".

It sounds like a fallout similar to bug #649939, in yours 3.4.4 (I guess that's the evolution version you have installed right now).

Testing with 3.4.4, I can move folders without any issue when I choose a subfolder of On This Computer, but when I choose the On This Computer itself as a destination, then it fails with your error message. That means, create a folder under On This Computer first, like "Server Archive", and copy/move your server folders into it.

I followed Milan's advice, created a folder in On This Computer and started copying folders into.   It works fine, but there are a lot of folders that I need to copy.  I can't coy the top level folder that I want to copy, but I can copy its subfolders one at a time.

Oops!   The whole folder did get copied as a subfolder of the local folder.

But I still have to move its contents up one level one at a time.  Is there some way to select a whole bu ch of folders and ove them in one operation?

I managed to fix astring values reading for IMAPX. The fix will be available in:

Created commit f3b1cae in eds master (3.13.1+) [1]
Created commit 94e1402 in eds evolution-data-server-3-12 (3.12.1+)

[1] https://git.gnome.org/browse/evolution-data-server/commit/?id=f3b1cae

(In reply to comment #50)
> I managed to fix astring values reading for IMAPX.

Ehm, the fix has a regression, see bug #730979 for more information.