(In reply to comment #0)
> I am building tracker git master via JHBuild on Fedora 16. The build fails with
> the log below. Evolution has removed libemail-utils/e-account-utils.h in this
> commit:

What a surprise, another commit breaks our build. *sigh*.

Sorry Craig, this is not your fault, only I am sick of all the changes they've made breaking Tracker's build and likely everyone else over the past months.

CCing Matthew here to get some discussion going about this. Matthew, can we please have one library to include to avoid all these errors. I have fixed our plugin's includes so many times this last 6 months it's just not funny any more :)

Let me just clarify here, I meant "single include" for the evo libraries. I don't mind if we have one include per directory (given they usually don't come and go so much), but including specific files is killing us here with bug reports and fixes all the time. This is why we, GTK+ and other system libraries have single include files.

See: http://mbarnes.livejournal.com/4631.html

I strongly advise against linking to Evolution libraries as there are absolutely no backward-compatibility guarantees there.  Stick with E-D-S libraries.

Hello Matthew,

I am unsure about which pkgconfig libraries you mean here. We currently link with:

TRACKER_MINER_EVOLUTION_3_2_REQUIRED="glib-2.0                  >= $GLIB_REQUIRED
                                      evolution-shell-3.0       >= 3.1
                                      evolution-plugin-3.0
                                      evolution-data-server-1.2 >= $EDS_REQUIRED
                                      camel-1.2                 >= $CAMEL_REQUIRED"


or more modern set ups require:

TRACKER_MINER_EVOLUTION_3_3_5_REQUIRED="glib-2.0                  >= $GLIB_REQUIRED
                                        evolution-shell-3.0       >= 3.1
                                        evolution-plugin-3.0
                                        libemail-utils
                                        libemail-engine
                                        evolution-data-server-1.2 >= $EDS_REQUIRED
                                        camel-1.2                 >= $CAMEL_REQUIRED"

Do you mean evolution-shell here?

--

All that aside, I was trying to do some testing here on Pangolin and I can't get the plugin to work any more, the camel_store_get_folder_info() call we make seems to never call us back and I am unsure if this is a blocking issue with the threading set up we have here (which I would like to rewrite and avoid if possible) or some new bug in that function.

Any clues?

I will probably end up writing a test case to figure this out more cleanly at some point. I found it quite hard to find any examples for how to traverse folders and emails easily using the APIs. Is there something you can point to perhaps?

Thanks,

I should add, EDS_REQUIRED = 3.2, CAMEL_REQUIRED = 3.2.

(In reply to comment #4)
> Do you mean evolution-shell here?

I meant all of Evolution's shared libraries.  They're not really proper shared libraries like the E-D-S libraries... they're not libtool versioned, there's no stability guarantees, and they're really only meant to be used by 3rd party groupware packages like evolution-exchange.  I'm working toward weaning even those packages off of Evolution libraries.

Since the Tracker plugin has to run in the Evolution process, I understand some amount of linking is necessary.  If I can get a better idea of what Evolution APIs you're using I can be a little more careful about changing them.  That said, it's better to lean on Camel and libedataserver as much as you can.

Also, the changes in 3.5.3 are dramatic enough that you might consider dumping backward compatibility and starting fresh, or at least splitting off a new Evo plugin for >= 3.5.3.  I'd be happy to help with that.  Otherwise I fear the code will become horrendous to maintain, if it's not already.  As of 3.5.3 we no longer use GConf, and account data has moved to plain text files with a whole new (and hopefully better) client API.


> All that aside, I was trying to do some testing here on Pangolin and I can't
> get the plugin to work any more, the camel_store_get_folder_info() call we make
> seems to never call us back and I am unsure if this is a blocking issue with
> the threading set up we have here (which I would like to rewrite and avoid if
> possible) or some new bug in that function.
> 
> Any clues?

camel_store_get_folder_info() is an async wrapper that just invokes camel_store_get_folder_info_sync() from the GIO thread pool using g_simple_async_result_run_in_thread().

If the GAsyncReadyCallback is never invoked, it could very well be a mutex deadlock in the synchronous function.  Deadlocks have been an ongoing problem in Camel for as long as I've worked on Evolution.  It's getting better but we're still not there yet.  Should be able to spot that easily in a backtrace, if that's indeed the problem.

(In reply to comment #6)
> (In reply to comment #4)
>
> Since the Tracker plugin has to run in the Evolution process, I understand some
> amount of linking is necessary.  If I can get a better idea of what Evolution
> APIs you're using I can be a little more careful about changing them.  That
> said, it's better to lean on Camel and libedataserver as much as you can.

If I just keep the libedataserver/* includes and the camel/camel.h include then it shows that we're using:

  e_mail_folder_uri_from_folder() - to build a message uri for an email (we combine the uid and folder)

  This is to get the CamelSession, if there is a better way, I am all ears :)

    shell = e_shell_get_default ();
    shell_backend = e_shell_get_backend_by_name (shell, "mail");
    s = e_mail_backend_get_session (E_MAIL_BACKEND (shell_backend));
    session = CAMEL_SESSION (s);

  Of course we need some includes for EPlugin, but that's expected.

  We also get the accounts list and iterate using:

    priv->accounts = g_object_ref (e_get_account_list ());

    for (it = e_list_get_iterator (E_LIST (priv->accounts));
         e_iterator_is_valid (it);
	 e_iterator_next (it)) {
      ...
    }

    Sometimes this too e_iterator_get (it);


  So type wise, we're using:

    EAccountList
    EAccount
    EMailSession
    EShell
    EShellBackend
    EIterator


  I would love to drop the use of all of these for just Camel (other than EPlugin which is necessary as I 
  see it).

> Also, the changes in 3.5.3 are dramatic enough that you might consider dumping
> backward compatibility and starting fresh, or at least splitting off a new Evo
> plugin for >= 3.5.3.  I'd be happy to help with that.  Otherwise I fear the
> code will become horrendous to maintain, if it's not already.  As of 3.5.3 we
> no longer use GConf, and account data has moved to plain text files with a
> whole new (and hopefully better) client API.

Are there any examples which could help with the migration?
 
> > All that aside, I was trying to do some testing here on Pangolin and I can't
> > get the plugin to work any more, the camel_store_get_folder_info() call we make
> > seems to never call us back and I am unsure if this is a blocking issue with
> > the threading set up we have here (which I would like to rewrite and avoid if
> > possible) or some new bug in that function.
> > 
> > Any clues?
> 
> camel_store_get_folder_info() is an async wrapper that just invokes
> camel_store_get_folder_info_sync() from the GIO thread pool using
> g_simple_async_result_run_in_thread().

I see. Yea, I would really like to drop all threading we use because I really don't think it's necessary, especially if there are decent async APIs available.

I think one of the reasons Philip used threads in the first place, was because the UI would stall and we had no way to know when Evolution was idle to begin indexing (which is when we would prefer to do it, not just on start up. Is there any mechanism to do that?
 
> If the GAsyncReadyCallback is never invoked, it could very well be a mutex
> deadlock in the synchronous function.  Deadlocks have been an ongoing problem
> in Camel for as long as I've worked on Evolution.  It's getting better but
> we're still not there yet.  Should be able to spot that easily in a backtrace,
> if that's indeed the problem.

Good to know, thanks.

(In reply to comment #7)
>   This is to get the CamelSession, if there is a better way, I am all ears :)
> 
>     shell = e_shell_get_default ();
>     shell_backend = e_shell_get_backend_by_name (shell, "mail");
>     s = e_mail_backend_get_session (E_MAIL_BACKEND (shell_backend));
>     session = CAMEL_SESSION (s);
> 
>   Of course we need some includes for EPlugin, but that's expected.

That's a common pattern seen in EPlugins... nothing wrong with it, but a note about EPlugin itself:

We've been slowly decommissioning EPlugin in favor of a nicer extension framework that lives in Evolution-Data-Server.  It works through GTypeModule and subclassing, and allows an extension to target a specific class such as EMailSession.  So you'd have full access to EMailSession's API.

If you think you might make a clean break for 3.5.3, this may be a good time to switch.  I'm happy to help or add whatever Evolution APIs you may need.

I wrote a simple "Hello World" example on our wiki:
https://live.gnome.org/Evolution/Extensions



>   We also get the accounts list and iterate using:
> 
>     priv->accounts = g_object_ref (e_get_account_list ());
> 
>     for (it = e_list_get_iterator (E_LIST (priv->accounts));
>          e_iterator_is_valid (it);
>      e_iterator_next (it)) {
>       ...
>     }
> 
>     Sometimes this too e_iterator_get (it);

This is the part that will be completely different now, as EAccount and EAccountList are gone.


> Are there any examples which could help with the migration?

I tried to document the API changes as well as I could:

Migration guide (FAQ-style, work-in-progress):
https://live.gnome.org/Evolution/ESourceMigrationGuide

The new account APIs are fully documented here:
http://mbarnes.fedorapeople.org/account-mgmt/docs/libedataserver/

And if you're curious, an explanation of the new account format:
https://live.gnome.org/Evolution/ESourceFileFormat


> I see. Yea, I would really like to drop all threading we use because I really
> don't think it's necessary, especially if there are decent async APIs
> available.

Not sure how the Tracker / Camel timeline pair up, but Camel's API used to be entirely synchronous, and it was never clear to me which functions blocked and which ones didn't.  That _might_ be why Philip wrote things the way he did.

The arrival of GIO was a game changer for us.  A massive Camel API rewrite ensued.


> I think one of the reasons Philip used threads in the first place, was because
> the UI would stall and we had no way to know when Evolution was idle to begin
> indexing (which is when we would prefer to do it, not just on start up. Is
> there any mechanism to do that?

I was actually thinking along those same lines recently, as I'd like to do some background operations like database garbage collection and downloading messages for offline viewing while the whole system is idle.

I was thinking of tapping into upower to try and determine when the system becomes idle, and then broadcast an application-wide signal through EShell or something.  And other signal when the system becomes active again.

Haven't written anything along these lines yet, and I'm open to better ideas.

(In reply to comment #8)
> (In reply to comment #7)
> >   This is to get the CamelSession, if there is a better way, I am all ears :)
> > 
> >     shell = e_shell_get_default ();
> >     shell_backend = e_shell_get_backend_by_name (shell, "mail");
> >     s = e_mail_backend_get_session (E_MAIL_BACKEND (shell_backend));
> >     session = CAMEL_SESSION (s);
> > 
> >   Of course we need some includes for EPlugin, but that's expected.
> 
> That's a common pattern seen in EPlugins... nothing wrong with it, but a note
> about EPlugin itself:
> 
> We've been slowly decommissioning EPlugin in favor of a nicer extension
> framework that lives in Evolution-Data-Server.  It works through GTypeModule
> and subclassing, and allows an extension to target a specific class such as
> EMailSession.  So you'd have full access to EMailSession's API.
> 
> If you think you might make a clean break for 3.5.3, this may be a good time to
> switch.  I'm happy to help or add whatever Evolution APIs you may need.

I might start something in master for 0.16.

> I wrote a simple "Hello World" example on our wiki:
> https://live.gnome.org/Evolution/Extensions

Is the dynamic type system available for 3.2? I tried to play with it tonight and it compiled, etc but didn't ever load my plugin for some reason. :/

> I tried to document the API changes as well as I could:
> 
> Migration guide (FAQ-style, work-in-progress):
> https://live.gnome.org/Evolution/ESourceMigrationGuide
> 
> The new account APIs are fully documented here:
> http://mbarnes.fedorapeople.org/account-mgmt/docs/libedataserver/
> 
> And if you're curious, an explanation of the new account format:
> https://live.gnome.org/Evolution/ESourceFileFormat

Ah right, I saw those thanks :)
 
> > I see. Yea, I would really like to drop all threading we use because I really
> > don't think it's necessary, especially if there are decent async APIs
> > available.
> 
> Not sure how the Tracker / Camel timeline pair up, but Camel's API used to be
> entirely synchronous, and it was never clear to me which functions blocked and
> which ones didn't.  That _might_ be why Philip wrote things the way he did.

Yea I think it is.
 
> The arrival of GIO was a game changer for us.  A massive Camel API rewrite
> ensued.

Nice.

> > I think one of the reasons Philip used threads in the first place, was because
> > the UI would stall and we had no way to know when Evolution was idle to begin
> > indexing (which is when we would prefer to do it, not just on start up. Is
> > there any mechanism to do that?
> 
> I was actually thinking along those same lines recently, as I'd like to do some
> background operations like database garbage collection and downloading messages
> for offline viewing while the whole system is idle.
> 
> I was thinking of tapping into upower to try and determine when the system
> becomes idle, and then broadcast an application-wide signal through EShell or
> something.  And other signal when the system becomes active again.
> 
> Haven't written anything along these lines yet, and I'm open to better ideas.

We were perhaps looking for some signal from the framework... Looks like we have a shared goal :)

Thank you for the support and replies here. Quite appreciative of the help! ;)

Following up on our IRC discussion, since we parted with a slight misunderstanding of the class layout.

You actually don't need to implement EExtensible.  That wouldn't do anything useful for you.

What I had in mind was:

1) EMailSession already implements EExtensible, which is really just a tag
   that means you can write extensions for EMailSession.

2) You would (re)write TrackerMinerEvolution, derived from TrackerMiner.
   And I would suggest having it take an EMailSession instance as a
   G_PARAM_CONSTRUCT_ONLY property and holding a weak reference on it.
   ("Weak" so that the extension doesn't outlive the thing it's extending.)

3) You would also write, say, EExtensionTracker, derived from EExtension.
   The extension will target E_TYPE_MAIL_SESSION the same way the "hello
   world" example targets E_TYPE_SHELL.

   i.e.  extension_class = E_EXTENSION_CLASS (class);
         extension_class->extensible_type = E_TYPE_MAIL_SESSION;

   This would be the "glue code" between Evolution and Tracker whose only
   job is to asynchronously create the TrackerMinerEvolution instance and
   feed it the EMailSession, and then of course finalize the miner object
   from its own dispose() or finalize() method.

Hope that's clear.  This stuff can get confusing quick!

Is this work on track for 3.6?

Should release-team do some workaround in the moduleset such as disabling the evolution miner?  (What would the consequences of that be?)

Er nevermind, it's already disabled in the moduleset, I just needed to re-autogen.

http://git.gnome.org/browse/jhbuild/commit/?id=3021c46d6d6c807f8bfdec86c16c239f4cbcaf48

(In reply to comment #12)
> Er nevermind, it's already disabled in the moduleset, I just needed to
> re-autogen.
> 
> http://git.gnome.org/browse/jhbuild/commit/?id=3021c46d6d6c807f8bfdec86c16c239f4cbcaf48

Colin, work to fix this issue is in a branch right now, so jhbuild shouldn't be affected with that work. But last time I got time to work on this branch there were issues with the GOA system which meant I couldn't progress. It's a bit of a mess for me to test with here.

I will try again soon.

Any news on this bug. It would be nice to enable the evolution miner again

Javier, I've not had a chance to check out the GOA stuff and to see if things work better here now. It's bottom of my priority list right now for Tracker.

I think I pushed a branch somewhere, I can check/sync/push if you're interested in helping out at all? :)

Maybe this plugin should finally be dropped :/ We are needing to disable it for a long time downstream due to it being incompatible with recent evolution versions. Also the same is happening with firefox and thunderbird plugins :|

GNOME is going to shut down bugzilla.gnome.org in favor of gitlab.gnome.org.
As part of that, we are mass-closing older open tickets in bugzilla.gnome.org
which have not seen updates for a longer time (resources are unfortunately
quite limited so not every ticket can get handled).

If you can still reproduce the situation described in this ticket in a recent
and supported software version, then please follow
  https://wiki.gnome.org/GettingInTouch/BugReportingGuidelines
and create a new enhancement request ticket at
  https://gitlab.gnome.org/GNOME/tracker/-/issues/

Thank you for your understanding and your help.