On Thu, 2017-03-09 at 12:49 +0000, evolution-ews wrote:
> …

That isn't related to your local time zone at all. That is the time zone in the Date: header of the message to which you're replying.

I think what you're reporting here is that Exchange is a steaming pile of crap and fails to correctly preserve the Date: header of incoming emails?

Thanks for a bug report. As David said, the value comes from the Date header of the message. I just tried it locally and I see that the sending message (which is passed to the server) contains Date header with correct value:

   Date: Fri, 10 Mar 2017 08:27:19 +0100

but when it's received by the evolution-ews and read from the server back, then it contains this Date header:

   Date: Fri, 10 Mar 2017 07:28:40 +0000

Thus it's the server which changes this header value. The times are equivalent, just the later is in UTC.

If you'd like to prefer local timezone over the timezone being used in the original message's Date header in the reply credits, then it's a question for evolution itself. Unfortunately, in this case, to workaround Exchange server oddity.

FWIW this varies from one Exchange setup to another. At Intel this would always happen — all incoming messages were mangled to the timezone of my server regardless of where they were actually sent from. Amazon's setup doesn't do that, and I get to see the original date stamps.

If I interpret comment 0 correctly, Roman was saying that even when Exchange is mangling the timezone on the headers reported through EWS, it *does* manage to recreate the original Date: header when we fetch the same message through IMAP? That implies that the information *is* there somewhere and we might have a chance of fetching additional properties of the message in order to fix it up for ourselves? That *would* be nice... :)

Yes, exactly, but the situation gets worse. I have several scenarios. One company server screws up both EWS and IMAP. The server from other company screws only EWS, but the same server, when accessed via IMAP works OK. So it indeed must be some exchange crap. But, why don't you set it to always recalculate the time to local timezone? If it ends with +0000 and the local timezone is not UTC, then recalculate. Or does such handling need to be done on composer level?

Regarding that additional info, I've looked into the source of mails and if the time is OK, all times in source are the same. Same when it is wrong. So I think that the only possible approach is recalculation...

Whose local timezone? The timezone of the recipient who's *replying* to
the message? We *used* to do that, but I actually fixed it years ago.

Because that's a lie. The fact that you're in Australia and you sent me an email at "01:30 +1000" might be *relevant* (and if not, why include it at all?).

It's the middle of your night; we might expect a brief and not-entirely-researched message that points in the right direction, but would be considered "unprofessional" if you were to send it during your working hours.

It also sets expectations about when people might expect a further contribution from you to the email thread, if they're looking at my citation and not your original message.

We should always use the original timestamp with the correct timezone. 

Yes, Exchange sucks, and we should try harder to *get* that original data if we can. But we certainly shouldn't override it ourselves just because some crappy "messaging" services are broken.

From that perspective indeed you are right. It should show the original timezone in which the sender sent the mail. But then the problem is, where to get that timezone if exchange arbitrarily leaves that in UTC throughout the whole message. I know that Thunderbird and Outlook (I know it's crap, but it's used in enterprise) show the local timezone of the replier. Maybe introducing an option for switching it would be a compromise for everyone? One option would be "show the reply header time in original timezone" and the other would be "show the reply header time in local timezone". But I guess this would more or less be a modification of Evolution itself, not evolution-ews... Should I reassign the bug to product Evolution?

Try Edit->Preferences->Mail Preferences->Headers tab->[ ] Show original header value, at the very bottom.

Hi, I've tried that already before reporting the bug. The only change it does is that when you see the Mail preview in the preview pane, the mail header for date/time that is displayed there either shows the original timezone or local timezone, and this works perfectly OK. But it doesn't influence the reply header date/time format in the composer.

Oh, right, I'm sorry, the reply "header", aka sender credits in the composed reply. My fault, I misplaced the two. I'll check whether there's a way to declare message date other than through MIME content, thus it'll be taken properly, but no promises, because these dates can be set by the SMTP (or basically any transport) service on the server side, thus out of hands of the evolution-ews completely. I'll let you know when I know more.

Oooookay, the good news is that the original Date header value is still there. The bad news is, well, it'll be ugly to do.

To explain: When I read all stored properties of one such message, then I see in the <t:MimeContent CharacterSet="UTF-8">...</t:MimeContent> this:
>   Date: Fri, 17 Mar 2017 07:59:36 +0000
which corresponds to:
>   <t:DateTimeSent>2017-03-17T07:59:36Z</t:DateTimeSent>
there are also other date/time related elements, namely:
>   <t:DateTimeReceived>2017-03-17T08:00:58Z</t:DateTimeReceived>
>   <t:DateTimeCreated>2017-03-17T08:00:57Z</t:DateTimeCreated>
>   <t:LastModifiedTime>2017-03-17T08:03:08Z</t:LastModifiedTime>
They all are in UTC, but there is also this element:
>   <t:InternetMessageHeader HeaderName="Date">Fri, 17 Mar 2017 03:59:36 -0400</t:InternetMessageHeader>
which contains the value as it should be, the same as PigTagInternetHeaders:
>   <t:ExtendedProperty>
>      <t:ExtendedFieldURI PropertyTag="0x7d" PropertyType="String"/>
>      <t:Value>...
>      Date: Fri, 17 Mar 2017 03:59:36 -0400&#xD;
>      ....</t:Value>

Thus it seems to me that the Exchange server (in this case version 2013), constructs the <MimeContent/> from its properties and overrides the Date header with the value in <t:DateTimeSent/>, instead of using the <t:InternetMessageHeader/> value.

Created attachment 348153 [details] [review]
ews patch

for evolution-ews;

When the Date header is available in EWS item properties, use it and overwrite the server-provided Date value in the local MIME message with it, thus it corresponds to the actual value being used by the sender, and not a UTC value being provided by the Exchange server.

Woow thank you very much for such a quick resolution. I'll be waiting impatiently for the new version to come from the repositories. Do you maybe happen to know what will be the version number of the evolution-ews that will contain this patch?

The sources are under hard code freeze till Monday, then, after 3.24.0 release it can be applied. I'll update this bug report (and close it) when it's done.

Created commit 5a62b9e in ews master (3.25.1+)
Created commit 3c98f1d in ews gnome-3-24 (3.24.1+)

Follow-up change in bug #780391.

Do we need to do the same for outbound messages? If you send a message while offline and it's stored to your Outbox, then later reconnect to the network and it gets sent, the recipients seem to see the later time.

Do we need to explicitly set the Date: header when we send, because we can't trust Exchange to use it from the MimeContent?

(In reply to David Woodhouse from comment #16)
> Do we need to do the same for outbound messages? If you send a message while
> offline and it's stored to your Outbox, then later reconnect to the network
> and it gets sent, the recipients seem to see the later time.

Later time, later time, do you mean the time when the message had been uploaded to the Exchange server, not the value of the Date header in the message, which is most likely the time when it had been added to the On This Computer/Outbox folder? I'm always a bit confused by these 'later time' term, I'm sorry.

To answer your question, I do not know. I do not know whether the server would even accept the Date value this way. It needs testing.

> Do we need to explicitly set the Date: header when we send, because we can't
> trust Exchange to use it from the MimeContent?

I've been always wondering whether it's correct to use the Date when the message "left composer", not the date when it had been passed for actual sending. I have set evolution to save message to Outbox by default and send them after 5 minutes, thus I have time to verify what will be sent, change thing sin it and so on. It's fine, though the receiving part things that there is an issue in the connection, because the message is always up to 5 minutes late (I sometimes flush the Outbox manually).

(In reply to Milan Crha from comment #17)
> Later time, later time, do you mean the time when the message had been
> uploaded to the Exchange server, not the value of the Date header in the
> message, which is most likely the time when it had been added to the On This
> Computer/Outbox folder? I'm always a bit confused by these 'later time'
> term, I'm sorry.

The one which is later, not earlier :)

So yes, recipients currently see a Date: header which represents the time the message was later submitted from Evolution to Exchange, not the time I hit 'Send'. That is explicitly forbidden by RFC5322 (see below).

> I've been always wondering whether it's correct to use the Date when the
> message "left composer", not the date when it had been passed for actual
> sending. 

It is absolutely correct (and required) to do so. See RFC5322 §3.6.1: which covers this explicitly:

3.6.1.  The Origination Date Field

   The origination date field consists of the field name "Date" followed
   by a date-time specification.

   orig-date       =   "Date:" date-time CRLF

   The origination date specifies the date and time at which the creator
   of the message indicated that the message was complete and ready to
   enter the mail delivery system.  For instance, this might be the time
   that a user pushes the "send" or "submit" button in an application
   program.  In any case, it is specifically not intended to convey the
   time that the message is actually transported, but rather the time at
   which the human or other creator of the message has put the message
   into its final form, ready for transport.  (For example, a portable
   computer user who is not connected to a network might queue a message
   for delivery.  The origination date is intended to contain the date
   and time that the user queued the message, not the time when the user
   connected to the network to send the message.)

I gave it a quick try, but an Exchange server (2013 in this case) doesn't let me change DateTimeSent, DateTimeCreated, neither the InternetMessageHeaders properties, each ends with (only the FieldURI differs):

        <m:CreateItemResponseMessage ResponseClass="Error">
          <m:MessageText>Set action is invalid for property.</m:MessageText>
          <m:ResponseCode>ErrorInvalidPropertySet</m:ResponseCode>
          <m:DescriptiveLinkKey>0</m:DescriptiveLinkKey>
          <m:MessageXml>
            <t:FieldURI FieldURI="item:DateTimeCreated"/>
          </m:MessageXml>
          <m:Items/>
        </m:CreateItemResponseMessage>

FWIW this succeeds (it sends the message) but doesn't work (the Date: header is not affected):

          <ExtendedProperty>
            <ExtendedFieldURI PropertyTag="57" PropertyType="SystemTime"/>
            <Value>2017-05-22T09:24:14Z</Value>
          </ExtendedProperty>
          <ExtendedProperty>
            <ExtendedFieldURI PropertyTag="3590" PropertyType="SystemTime"/>
            <Value>2017-05-22T09:24:14Z</Value>
          </ExtendedProperty>

This *must* be possible though, surely? What does Outlook do? I know Microsoft doesn't care about violating standards but there are real business problems here, caused by the ensuing miscommunication when messages cross in transit and are mis-dated.

Hi guys,

Just checking this bug again, since I've tried it with Evolution 3.26 and the reply header still has +0000 timezone instead of the original timezone. I see that this bug has been put to Fixed, but it still doesn't work. Any idea?

Thanks for checking it out.

Could it be that your server doesn't provide that value? It would not surprise me.  You might be also more specific and provide some data, just we know we talk about the same. I reread this bug comments and to be honest the term "the reply header in compositor" had been quite confusion to me. I guess it's for the credits line when replying in the quoted mode, thus something like this line:

   On Fri, 2018-02-23 at 05:11 -0600, User wrote:

right? This comes from my outlook.com address, to which I connect using evolution-ews, thus the change does work, as long as the server provides necessary data. Could you check what value the Date header has of the original message, please? Mine shows:

   Date: Fri, 23 Feb 2018 05:11:55 -0600

If yours not, then the server didn't provide the value or the Date header really had that UTC timezone. As a hint, my message had the Date header as the previous of the last header when I looked into its source (Ctrl+U), which can be a clue that the extra Date header had been received from the server, because otherwise it uses to be around To/From/Subject headers, if I recall correctly.

You can see what the server returned to evolution-ews when you run evolution from a terminal like this:

   $ EWS_DEBUG=2 evolution

but it shows the information for new messages only. You can remove
   ~/.cache/evolution/mail/<ews-account-uid>
which will cause redownload of all the iformation from the server the next evolution start, but it means really everything, thus if you've many messages, then it'll take its time and the log will be very long.

Also, do not reopen the bug, please. One can add comments to closed bugs too. It can be reopened (or a new bug filled) once we figure out the cause and whether it's doable at all. Thanks.

Hi,

Well I've analyzed the source code of the message that came from ews connector and indeed nowhere in the message is the date/time with correct timezone. The Date field contains +0000 time. However, I have the same account setup via IMAP in Thunderbird and there the source code of the message looks exactly the same EXCEPT the Date field is in the correct timezone (in my case +0100). That is very strange, and it means that this same exchange account via IMAP works OK, but via the ews connector doesn't. Any idea?

It's known that the EWS protocol as provided by Microsoft doesn't work the same as the IMAP protocol provided by Microsoft. There's really nothing one can do about it, regardless how much we would like to. Of course, there's always a chance that what works on my side doesn't necessary work on your, or some server update fixed these things, and so on, but without some co-operation with debugging on your side it's quite impossible to know for sure. I already gave you my ideas how to debug the things on your side (see comment #22), which I'm not going to repeat again.

OK, I've tried it and was lucky that I got some new e-mails :) But the output is huge - the question is what are you looking for? All Date fields that I saw are in +0000 timezone (GMT). Also for incoming messages there are XML fields like DateTimeReceived, DateTimeCreated, DateTimeSent - they are all in +0000. Some of the Received fields also have timestamps, some of them are in +0000 and some are in a completely different zone like -0600 and so on. I've seen NO mention of my own timezone ;-) Doesn't it maybe work in a way that Outlook manages timezones locally? So that it ignores what is in the mail and just puts in the reply header what's correct for the sender?

Looked at it again, all Date fields are in +0000 timezone for all incoming messages. XML fields DateTimeReceived, *Sent and *Created as well. This is an Office 365 account, but all senders use Outlook 2013 themselves. Moreover, when I send to this account a test mail from my Gmail account, the timezone is correct. I can see it also in the debug window, the Date field is +0100 and when I do a reply the header contains the correct timezone. So I guess the server works fine, but Outlook 2013 doesn't care about the timezones apparently and probably handles them locally. Would there be a possibility to implement an option to put a local timezone into the reply header?

Right, it can be that Outlook stores everything in UTC for some reason and converts this time to the user's/local time zone in the reply credits (word 'header' is ambiguous here), thus makes this odd.

> Would there be a possibility to implement an option to put a local timezone
> into the reply header?

I'm really unsure whether I want to workaround the misbehaviour of 4 years old Outlook. Maybe they already fixed it in the newer version. Maybe not.

The above change added a code which also received only the Date field, which means that your log can contain something like:
   <t:InternetMessageHeaders>
      <t:InternetMessageHeader HeaderName="Date">Fri, 13 May 2016 04:57:21 -0400</t:InternetMessageHeader>
   </t:InternetMessageHeaders>

It also reads the message as the MimeContent, which is not shown in the log (not the received base64-encoded content), and this had different Date header value here. It's all about the EWS protocol working differently and as you found also about the Outlook, which seems to convert the time into UTC.

Well, to be honest this was the problem for me when my colleagues were using Outlook 2010 as well. So whether it's 2010 or 2013, the problem is the same. I guess Outlook 2016 will have the same issue, since Microsoft probably decided to handle timezones locally. Wouldn't that make a good business case for implementing the option into Evolution, since it is mostly used in enterprise environments, where it mostly interacts with Outlook? :-) Back then I used an ews connector in Thunderbird, which had the same problem, but then in Thunderbird I solved it with a reply credits generating extension, which had an option to use local timezone. However, Thunderbird is not an option anymore for me...

I've read the comments to this bug again and what got my attention again was David's comment #5. When I think about it, the timezone in the reply credits should be of the *sender*, because it gives information at what time the *sender* sent the original mail and I think that is the important information. Maybe it would be interesting to add such an option in Evolution.

Finally one additional finding from me :-) When I use the IMAP access to the same Office 365 account, all Date fields are converted into *MY* timezone (+0100). So all mails that are for example sent to me from timezone -0600 also get converted into +0100. So for all mails that come to me from my own timezone, all is OK. But for mails that come to me fom other timezones, the date in the reply credit is also wrong (taking into account what David said in comment # and I in my previous comment), because it is not in the sender's original timezone. So IMAP access also has an issue, but this is not Evolution's problem, because I see the same behavior in Thunderbird as well. So, the information about the timezone is NOT in the mail itself - it's just converted during receiving.

(In reply to Roman Dinga from comment #29)
> When I think about it, the timezone in the reply credits should be of
> the *sender*, because it gives information at what time the
> *sender* sent the original mail and I think that is the important
> information. Maybe it would be interesting to add such an option in
> Evolution.

What option? That's exactly what Evolution does, it does use sender's time zone in the reply credits.

(In reply to Roman Dinga from comment #30)
> So, the information about the timezone is NOT in the mail itself -
> it's just converted during receiving.

Except it's the opposite. The time zone information is in the mail itself, in the Date header. It's not client's fault that the *server* converts the Date value to a different timezone than what the sender used. And there is no way back on the client side, I strongly disagree with any guessing on the sender's time zone. Just rely on the data provided by the *server*. If it modifies the Date header then it's the *server* fault, not client. Yes, some clients (read Outlook) can mangle the Date header when sending, to change the time zone to UTC or any other, but then there's even less chance to guess the Sender's time zone (and I'm still against any guessing).

What you suggested earlier, to use local time zone, instead of UTC, in reply credits does make sense. Maybe it can be done only with a hidden option, just check whether the sender's time zone is UTC and if the local time zone is not UTC, then convert the time to the local time zone, which may cover Outlook clients and Exchange servers, both IMAP and EWS. The option would be enabled by default and people could disable it when needed. If we agree on this, then I'd rather see a new bug report only for this new behaviour being opened, against Evolution, to avoid all the back discussion and noise.

Indeed, you are absolutely right, there is no way how to guess the sender's timezone if the date field in the incoming mail contains UTC timezone. And from that perspective I agree that guessing anything is not a correct approach. I will open a new bug report against Evolution, with a link to this discussion, so that a new option can be implemented in Evolution.

I've opened a new bug report against Evolution: https://bugzilla.gnome.org/show_bug.cgi?id=793915