Created attachment 184298 [details]
bootchart (data)

Created attachment 184299 [details]
bootchart

We're debugging this right now, it's a long chain of problems.

* gnome-settings-daemon initializes libcanberra
* libcanberra creates a thread to talk to pulseaudio
* [thread] libcanberra auto-starts pulseaudio
* pulseaudio connects to the system bus
* pulseaudio tries to talk to bluez over the bus
* bluez gets auto-spawned (maybe - this may be where the problem is)

Then gnome-settings-daemon in the main thread, tries to talk to pulseaudio for volume controls, which ends up blocking on the above train.

Hmm, PA can be *very* slow at startup. Due to limitations of ALSA we have to open the audio device in a gazillion of different combinations to figure out what is available (basically, the ALSA device enumeration is too naive to be useful). 

In GNOME2 nothing that mattered was really blocking on PA. The volume applet and the welcome sound were the only thing. And if those started eventually this wasn't much of a problem.

In GNOME3 nothing should block on PA either. The broken enumeration in ALSA is unlikely to be fixed anytime soon (because the problem is incredibly hard and the manpower limited), so PA will be very unlikely to be quick at startup anytime soon either. (And no, we cannot reliably cache the probing data, this was discussed quite often already).

Hmm, the bootchart plot doesn't actually suggest a PA issue? PA is started very late, and consumes a bit of CPU there due to the probing, but it seems to be finished fairly quickly.

We discovered that this was an interaction between all of:

gnome-session,gnome-settings-daemon,libcanberra,pulseaudio,dbus,systemd,bluez

The core bug was that in systemd conversion, the Exec= lines in e.g. bluez.service were changed to Exec=/bin/false.  But there was some broken code in dbus which compared by Exec= key, thus effectively merging together all /bin/false activations.

See: https://bugs.freedesktop.org/35750

This should be closed, probably ?

I'd like to profile this again with the Fedora 16 alpha if possible... do you have particular recommendations except just installing bootchart and seeing how it turns out?

The reason I'm still a bit skeptical until I've seen it is:
- My up to date Fedora 15 machine still takes 20+ seconds to log in
- I've never seen gnome login faster than 20 seconds since 2006 (I kinda hoped that 3.0 would help with that).

With GNOME 3.1.91/up to date Fedora 16:
- cold login time = 30 seconds
- warm login time = 15 seconds

Created attachment 196438 [details]
bootchart in 3.1.91

Created attachment 196439 [details]
bootchart in 3.1.91 (data)

@Jean-Francois: keep in mind that the Fedora kernel is built with a lot of debugging overhead, which might cause some slowness that won't be present in the released Fedora 16.

You might want to try again with this kernel, it has debugging disabled:
    https://admin.fedoraproject.org/updates/kernel-3.1.0-0.rc6.git0.0.fc16
(I don't think it has been pushed out yet, at least not when you last tried)

I also find Fedora 16 login quite slow, but haven't had a chance to test this kernel yet. I was thinking about doing it tonight, but feel free to beat me to it. :)

Fedora 16 is to be released in the matter of days, it's still slow and I expect it to remain similarly slow given that this problem has been consistent for the last five years or so.

gnome2 was a different beast, but not that different it seems. I just thought that gnome3 would be a nice opportunity to do some serious profiling again, as it has been a long time since the initial SoC study from 2005.

It's kind of embarrassing that it does not log-in instantly with a machine with a solid state drive.

Tested again on a Thinkpad T43p (2.13 GHz Pentium M with 2 GB of RAM),
with the latest updates:
- 19 seconds cold login
- 15 seconds warm login

The amount of I/O might have changed for the cold login, but the amount of work/"blocking stuff" seems to have stayed the same for a warm login.

Created attachment 227025 [details]
bootchart in 3.6.1

Created attachment 227026 [details]
bootchart in 3.6.1 (data)

Created attachment 227027 [details]
bootchart in 3.6.1 using autologin

From what I've been told, using autologin provides cleaner bootcharts as gdm just hands over the session to gnome shell, so this one should be more accurate.

Created attachment 227028 [details]
bootchart in 3.6.1 using autologin (data)

Hi folks, did you miss me? ;)
Here are some updated bootcharts with a fully updated Fedora 18 and GNOME 3.6.1. Those were not done on my Thinkpad T43, but on an Acer Aspire 1410 netbook with a dualcore U4100 CPU, Intel GMA 4500HD GPU and a conventional SATA hard disk drive. The machine is thus a bit beefier this time, but login times are still unacceptably slow.

The autologin bootchart is quite interesting and seems more accurate. If I'm not mistaken, some of the easily spotted "top offenders" are:

- Evolution eating up I/O and CPU
- gconf still being present and eating up I/O
- colord-sane being started* and eating up I/O and CPU
- gnome-shell eating a crapton of CPU
- zeitgeist doing a little bit of I/O in three different processes
- Empathy and all telepathy accounts being started, even though the Empathy UI is not shown and GNOME Shell does not actually connect the accounts at login.
- Tracker.

*: Why should colord-sane ever be started at login!?
   I don't even have a scanner.

Not denying that we have big problems in GNOME but... what is firewall-cmd? /me makes a note to not ever install that on his laptop.

I have a similar issue on Fedora 17. HDD use is pathological. I strongly suspect tracker and zeitgeist, from what iotop says. Have you tried disabling them?

As I mentioned above, Zeitgest does a bit of I/O but it's not as big as the other offenders.

Tracker is, well... Tracker.

The problem is that disabling components like that is not going to make the problem go away for the intended default user experience, it's just hiding dirt under the carpet. I'm looking at the overall combined performance (or lack thereof).

I didn't suggest you disable them permanently, but you obviously have to check whether they are the main offenders, and that's the best way to find it out.

I think this is a proper place to suggest it: maybe a splash/login screen would be a good idea here?
I know that it actually *decreases* the login time, for the inevitable reasons, but it would inform the user that the session did not hang. Contrary to some beliefs, it might not always be obvious.
There was an option to enable the splash screen before in Gnome 2 in, now mostly obsolete, gnome-session-properties. Probably, there was some valid reason behind removing it, but honestly, I don't remember it.
The traditional splash screen may be inconsistent with the current Gnome 3 interface, but this is not a place to discuss it.

The only thing I'm asking here is if it's a good idea at all to consider such a solution when talking about the login process. It will most certainly not solve the problem, but will provide an input that the session is alive and is just struggling. The discussion about how should it look like would take place in Gnome Design, obviously.

No, we prefer to fix reasons for slow login instead of adding splash screens.

I have tried to measure the problem on my computer. It's a quad-core core2 at 2.8GHz, I bought the hard drive in 2012, 7200rpm 2Tb.

On a fedora18 computer which was upgraded from fedora17...

The measurements are approximate.

The time from "Loading initial ramdisk" to the fedora logo flashing is 17 seconds (measured that only once).

The time from the fedora logo flashing to a working gnome3 desktop is most of the time 38 seconds.

NOTE: I have autologin, and I have configured online accounts with my google account. With regards to that I have opened the bug 692826, which may or may not affect my startup time.

Here is what I measured:

-----------------
my configuration
-----------------

       | desktop background visible |  top menu visible |
---------------------------------------------------------
Boot#1 |        6-7 seconds         |       38 seconds  |
Boot#2 |           -                |       38 seconds  |
Boot#3 |          33 seconds        |       1:00 minute |
---------------------------------------------------------

-----------------
disabled the proprietary DROPBOX
-----------------

       | desktop background visible |  top menu visible |
---------------------------------------------------------
Boot#1 |           -                |       36 seconds  |
Boot#2 |           -                |       38 seconds  |
Boot#3 |           -                |       38 seconds  |
---------------------------------------------------------

-----------------
disabled the proprietary DROPBOX, disabled tracker [1]
-----------------

       | desktop background visible |  top menu visible |
---------------------------------------------------------
Boot#1 |           33 seconds       |       35 seconds  |
Boot#2 |           -                |       34 seconds  |
Boot#3 |           -                |     33.5 seconds  |
---------------------------------------------------------

[1] to disable tracker, I have renamed /usr/libexec/tracker-miner-fs and /usr/libexec/tracker-store by appending "1" at the end of their filenames.

So my conclusion for now would be that in my case, I could probably shake at best 2 seconds by dropping dropbox from startup and at best 3 extra seconds by removing tracker.

But it seems fully 33.5 seconds of startup time are the shell's responsability.

I do have the following shell extensions installed, but I doubt they affect startup time:
* No IM status
* Remove accessibility
* TopIcons
* Window fade in

I am available if you think extra tests could help fixing the problem, but I can't do I don't know what, I need this computer.

I think the startup time was much better and acceptable in gnome3.4/fedora 17 but I didn't measure anything.

Another problem I have with gnome3.6 is that on top of that boot time, once it's initialized, when I go to activities, there is another pause which feels like 1 second or so, could be only a couple of hundred milliseconds, but it really feels long. That problem I am absolutely certain that it was NOT present in fedora 17/gnome 3.4.

Hi Emmanuel, just clarifying some things:
- As you can see with all the analysis/benchmarking I've done above, this has affected versions 3.0 to 3.6. Generally speaking though, the slow login performance has been that way since 2006-2007.

- The particular regression you mention in 3.6 regarding gnome-shell's Activities button is bug #687364

*** Bug 683765 has been marked as a duplicate of this bug. ***

I want to add my observations as well. I have 3.6 on a Pentium 4 CPU, 3GB of DDRII and a quite old 2006 HDD (7200 RPM I think). I also use the Linux-ck kernel[1] on Arch Linux.

I have come to the conclusion that Contacts, Documents, Tracker and Zeitgeist are making my life miserable(for Contacts and Documents I'm referring to their providers not the actual apps. Kudos for fixing that in 3.8) so I removed them all.

I also disabled Dropbox and the like from the auto-start family and only left Mailnag. I have also disabled some gnome-settings-daemon plugins that I think were not useful (a11y-keyboard, a11y-settings, orientation, smartcard) .

Lastly, I use around 13 extensions.

A warm start takes ~11 seconds, the cold start is little more than that. Even with that number I still think the seconds the shell freezes before starting up are wasteful.




[1] https://wiki.archlinux.org/index.php/Linux-ck

*** Bug 678221 has been marked as a duplicate of this bug. ***

as a note: don't write down approximate times - use a proper boot charting tool, like Bootchart. wallclock times are completely useless: in order to debug the boot process you need to instrument the system and get what is taking up CPU, disk, or both. if you feel the urge to comment on this bug, do so by attaching a bootchart profile.

Created attachment 238829 [details]
bootchart data for GNOME 3.6.3 (Fedora 18)

My results at least seem to indicate the culprit is not pulseaudio.

The attached file documents what I see on my laptop, but on my desktop (a far, far more powerful machine) I can see more or less the same trend, as most of the issue is I/O bound, instead of CPU bound.

Can't we have tracker being a bit less aggressive at startup, please? It's trashing the disk and eating CPU like a werewolf.

I tried also to profile gnome-shell, it seems most of the issues can be traced back to the JS engine. I will try again, and see if I can attach also that profiling info.

Created attachment 238831 [details]
bootchart graph for GNOME 3.6.3 (Fedora 18)

I ran a new X session (xinit -- :1), and launched from the xterm:

  dbus-launch perf record -- gnome-session

You can get the archive from http://goo.gl/0mP5V. It's a bit big, because it includes all dependencies and DSOs needed for the instrumentation. It will stay there for the foreseeable future.

To use it, after downloading: 

  tar xvf bug645756.tar.bz2 -C ~/.debug

Unfortunately, I am not aware of a good method to profile an application for I/O. Advice is welcome, since I think most of the issues are not CPU-bound.

Maybe it would make sense to file separate bugs against Tracker and gnome-contacts, as they seem to eat much CPU and I/O that they could delay a little bit at least. systemd-udevd seems to be an important offender too.

All systemd-udevd does is wait for udev events, which means it's likely a hardware or kernel issue.

Did someone file bugs against tracker and gnome-contacts?  Improving start up time from gdm sounds like a worthy goal for the 3.10.

This has massively improved in gnome3.8/fedora19 for me.

Sri, Tracker's bug #689302 might be the one.
As for gnome-contacts, couldn't find an existing bug so I opened bug 721443.

I wanted to try generating bootcharts again with 3.10, however Fedora decided to not package bootchart anymore (since F19 I think), because "hey we have systemd-analyze, why do we still care about bootchart?"...

The problem is that the GNOME session is not handled by systemd and we thus lose the ability to chart its login time like we used to. Unless gnome session switches to using systemd behind the scenes, but I don't really expect that to happen soon.

We need an *easy* way for gnome testers to continue profiling/charting this.

AFAICS, there isn't even a way for users to disable tracker these days. It's not in the startup applications in gnome-tweak-tool, neither in gnome-session-properties (which is hidden and I'm not sure it's actually wired up to anything still), and you can't remove it because it would also remove nautilus, totem, boxes, etc.

FWIW, measuring with a stopwatch on a F20 laptop with regular hard drive,
GNOME 3.10 login still takes 20+ seconds. No perceptible improvement for me.

Bootchart is now part of systemd, just look at /usr/lib/systemd/systemd-bootchart binary and its manpage on how to use it...

(In reply to comment #39)
> Sri, Tracker's bug #689302 might be the one.
> As for gnome-contacts, couldn't find an existing bug so I opened bug 721443.

Looking at the latest bootchart graphs...
I have to say 13s and 14s for tracker-miner-fs and tracker-store respectively is quite a lot longer than I would expect, but then it really depends on what Tracker is doing.

If I am to go by the bug report you opened (above), you are indexing A LOT of content and possibly locations which are ill-advised to be indexed (like source directories with 10 versions of the Linux kernel checked out) :)

Now, we *can* deal with that, but I doubt people starting GNOME for the first time ever have these level of content for Tracker to deal with on first boot. If this content is accumulated, the issue isn't so prevalent as if you just dump hundreds of GB into $HOME and expect Tracker to index it on first boot without any noticeable resource usage. I say all this - but actually on my desktop, I don't notice anyway - but I would on my laptop.

> I wanted to try generating bootcharts again with 3.10, however Fedora decided
> to not package bootchart anymore (since F19 I think), because "hey we have
> systemd-analyze, why do we still care about bootchart?"...
> 
> The problem is that the GNOME session is not handled by systemd and we thus
> lose the ability to chart its login time like we used to. Unless gnome session
> switches to using systemd behind the scenes, but I don't really expect that to
> happen soon.
> 
> We need an *easy* way for gnome testers to continue profiling/charting this.
> 
> AFAICS, there isn't even a way for users to disable tracker these days. It's
> not in the startup applications in gnome-tweak-tool, neither in
> gnome-session-properties (which is hidden and I'm not sure it's actually wired
> up to anything still), and you can't remove it because it would also remove
> nautilus, totem, boxes, etc.

Yea, this is true to some extent.

There is tracker-preferences - which you can run and configure Tracker with. This includes options for:

- How aggressive Tracker is
- If it sets up file monitors at all
- How often it crawls and updates your indexed files
- When it indexes (i.e. with battery or low disk space conditions)
- WHERE it indexes (i.e. your content)

Changing these options can have a huge impact on your system. These are not available in the system settings dialog thingy in GNOME and I've thought a long time that they should be in some form. Recently there were indexing options added, but more around WHAT is indexed (so quite basic). So unless you know about the tracker-preferences GUI, you're really buggered. Perhaps we should have a link to tracker-preferences from the tweak tool?

It's important to note, that Tracker has options to alter the way it starts including:

1. $ gsettings set org.freedesktop.Tracker.Miner.Files crawling-interval -2

   This will disable indexing and you get to use the DB even if the data is old.
   See the gsettings schema details to know what the settings do here:
   https://git.gnome.org/browse/tracker/tree/data/gschemas/org.freedesktop.Tracker.Miner.Files.gschema.xml.in#n58

2. $ gsettings set org.freedesktop.Tracker.Miner.Files initial-sleep 15

   Where 15 is the number of seconds before Tracker does _ANYTHING_.

...
There are more.

One thing is clear to me, no 2 situations are the same when it comes to users asking why Tracker is taking up so many resources and it's often a combination of configuration and content being indexed.

> FWIW, measuring with a stopwatch on a F20 laptop with regular hard drive,
> GNOME 3.10 lgin still takes 20+ seconds. No perceptible improvement for me.

I would add, 14 seconds for tracker-store is quite suspicious. It sounds like it didn't finish indexing the LAST time you used your machine and it is continuing from where it left off. IF that's the case, you simply can't expect Tracker to sit still and do nothing with the current default options.

We're happy to improve Tracker where it makes sense, but I am concerned that in this case, Tracker is busy because of quite uncommon circumstances surrounding the configuration and content on the machine creating this bootcharts. I would love to find out what is actually happening :)

Frederic: I took a look at systemd-bootchart, but I can't try it out yet, as a Linux kernel feature it depends on is disabled in Fedora... you get "open /proc/schedstat: No such file or directory" (see https://bugzilla.redhat.com/show_bug.cgi?id=1013225 ). For the time being I can only rely on rough observations when it comes to GNOME 3.10, and it seems that tracker is indeed eating lots of resources during login.


Warning: somewhat long comment ahead.
         While I'm covering specifically the case of Tracker below, 
         the principles there can be applied to pretty much any software, 
         so I hope it will prove insightful regardless of the specifics.


Martyn, thanks for taking the time to comment on this.

Correct me if I am wrong in my perception, but as far as I can remember and presume, a big part of Tracker's reason to exist (besides not being written in C# back when Mono apps were the target of heavy bashing circa 2007) was that it would aim to be a lighter, more efficient alternative to Beagle.

As such I am looking at it from the viewpoint of a low-priority background service that should never impact foreground. Any amount of processing that eats CPU or I/O is therefore expected to happen when the system is completely idle, not on battery power, not in active use by the user, when the screen is locked, etc. For that reason I am a bit surprised to learn through your comment that tracker starting indexing during session login is not completely unexpected and unacceptable (on that note, please do not take anything I say below the wrong way, I'm voicing my surprise here :)



> There is tracker-preferences - which you can run and configure Tracker with

AFAIK, it is not shipped by default as part of the core GNOME desktop (at least not on Fedora). We cannot expect users to know about the possibility of an indexing daemon being the cause for their slow login times, to know about the existence of a tweaking tool for it, to install the additional software and start tweaking stuff. If we expect Joe Plumber to know (or care) about that, we have lost. We're sweeping problems under the rug by putting the onus on users.


> How aggressive Tracker is

It can be as aggressive as it wants when the computer has been idle for a while and is not on battery. Call me crazy, but I consider this decision to be "the computer's job" and expect it to be fully automated (or maybe configured by the distro or device maker if not targetting a desktop/laptop). I suspect others have beaten this horse to death before me, or I'm missing something here (but that's probably a story for a separate discussion).

You hint a bit at this throughout your points further in your comment, but I say again: I don't think it should have to be configured/tweaked at all. My parents/aunts/joe-plumber won't ever know how to make such performance-impacting decisions*, nor do they care.

  *: neither would they ever be able to make decisions on security prompts.
     Stef Walters is making some bold new decisions in that regard,
     his GUADEC 2013 talk on the matter is a must-see.


> 1. $ gsettings set org.freedesktop.Tracker.Miner.Files crawling-interval -2
> 2. $ gsettings set org.freedesktop.Tracker.Miner.Files initial-sleep 15

If the user launched tracker manually, fair enough, it can start grinding immediately, that's the result of a direct, conscious action.

Autostart, however, is a special case and should be treated as such. I think this is an important, somewhat separate point that makes all the "configurability" argument moot. It should never start touching anything for the whole duration of login + some time afterwards. Login is the wrong time to hammer the system, slowing down perceived UI/shell readiness or impeding the user who wants to launch various applications as soon as possible.

Obviously I'm not a Tracker developer so you know best how to achieve this, but from a naïve high-level perspective, I would imagine the following: wait for the login to be completed, add 20 seconds of waiting time "during which there has been no further IO/system load" before you initialize your db in "read-only" mode (crawling-interval -2). Then, wait 1 to 5 minutes of "fully idle time" before you actually start indexing (giving users enough time to do whatever they want to do immediately at login). In other words, continually defer/delay processing at startup until startup is over and the user is taking a break from launching stuff all over the place.

> I would add, 14 seconds for tracker-store is quite suspicious. It sounds like
> it didn't finish indexing the LAST time you used your machine and it is
> continuing from where it left off. IF that's the case, you simply can't expect
> Tracker to sit still and do nothing with the current default options.

Hell yes I can. Even if indexing was not complete last time (is it ever?), I do expect it to sit still until my desktop is not being actively used in a significant way.

On that note, on a system where tracker hadn't been run for a significant amount of time (or never run at all, where no .cache/.local tracker folders exist), I wouldn't be shocked if tracker takes a while to index stuff... just NOT at system startup. Only afterwards in a transparent manner, "ideally" when I'm actually not in front of the computer.

> I am concerned that in this case,
> Tracker is busy because of quite uncommon circumstances surrounding
> the configuration and content on the machine creating this bootcharts

I'm not the person who reported bug #689302, but even I have 40 000 files in my affected laptop's home directory - I consider that a fairly small and reasonable amount (my dev desktop machine is expected to have hundreds of thousands of files sitting around). I doubt this is considered too much for Tracker to handle in an unobtrusive way (otherwise, I'll be extremely blunt and say that we might as well be using "locate" behind the scenes ;)


I wish I could call my laptop "uncommon".

It's been sitting around idle for hours and days on end. It's been in use for months and years. I can't imagine that Tracker hasn't already indexed the whole thing a couple times. A few years back I sometimes wiped tracker's data so it could start from a "clean state" to avoid the possibility of "bad data" being the cause.

Now, I've been writing this comment for the past two hours - during that time the laptop has been sitting idle besides me and I haven't seen the HDD I/O LED flash once, somewhat of an indication that Tracker had nothing to index further. But if I reboot it now, I can guarantee that a cold GNOME login will still take 20+ seconds. For the sake of verification, I just did now: and launching gnome-terminal to run "iotop" as soon as I can during login, I see tracker-store and tracker-miner-fs doing all the HDD activity indeed (an eventual bootchart might find other things that occur before I'm able to actually launch gnome-terminal).


The catch? It's not a Core i7 with a solid state drive. It's a normal laptop from 2009 with a dualcore CPU, Intel GM45 graphics and a conventional SATA hard disk drive.

I don't know under which hardware environment you are testing, so I must ask: did you or anyone here try on older machines recently (say, a Pentium 4 with 1-2 GB of RAM and a 5200 rpm HDD)? Booting a distro on that kind of hardware?

To take things further and make it more representative, I suggest rsyncing your home into such a machine (minus the software dev/build/checkout folders) and using it as if it was your main computer for a few days. If you haven't tried that for a few years, it can be quite eye-opening :)

Sure, my overpowered quadcore desktop with a SSD boots in 5-10 seconds whereas this laptop will take 58 seconds (!) to get from LUKS to GDM, but that's exactly the kind of hardware that we can expect "Joe Plumber" to be using out there.


Sorry about the length of this comment; I wanted to provide a complete reasoning and argument. Hope it is helpful.

(In reply to comment #42)
> Correct me if I am wrong in my perception, but as far as I can remember and
> presume, a big part of Tracker's reason to exist (besides not being written in
> C# back when Mono apps were the target of heavy bashing circa 2007) was that it
> would aim to be a lighter, more efficient alternative to Beagle.

We're not really comparing ourselves to any other project like Beagle.

In terms of being lighter and more efficient, it really depends what you're measuring there. We're certainly trying to be that way based on our database choice (SQLite instead of MySQL or PostgreSQL, etc). We also have heavily optimised the project for embedded devices thanks largely due to Nokia. So in a way yes.
 
> As such I am looking at it from the viewpoint of a low-priority background
> service that should never impact foreground. Any amount of processing that eats
> CPU or I/O is therefore expected to happen when the system is completely idle,
> not on battery power, not in active use by the user, when the screen is locked,

OK.

- completely idle: config option + we use SCHED_IDLE for this (we could use other approaches)

- not on battery power: config option and NOT the default (could fix this)

- not in active use / screen locked: TODO. We could certainly make this happen if you don't care about finding documents that you just created. Again, it would have to be a config option and I doubt it makes GNOME better to use if you make this the default.

> etc. For that reason I am a bit surprised to learn through your comment that
> tracker starting indexing during session login is not completely unexpected and
> unacceptable (on that note, please do not take anything I say below the wrong
> way, I'm voicing my surprise here :)

;)

All things being equal, assuming that your situation is not subject to weird config or setup issues of course :) you expect Tracker to possibly NOT have your content indexed at the expense of your machine booting faster by 10-30 seconds?

We often battle with this trade off. People in the community (like Bastien) have said for a long time, that Tracker should be more passive, i.e. applications should tell us to index content (to avoid resource use and other issues), we've had to cater for many different expectations that users have about what Tracker should be doing and when it should do it.

One very clear expectation I see, is that Tracker shouldn't be slowing the system down or using its resources to detriment. I will add that, you can configure Tracker that way if you prefer.
 
> > There is tracker-preferences - which you can run and configure Tracker with
> 
> AFAIK, it is not shipped by default as part of the core GNOME desktop (at least
> not on Fedora). We cannot expect users to know about the possibility of an
> indexing daemon being the cause for their slow login times, to know about the
> existence of a tweaking tool for it, to install the additional software and
> start tweaking stuff. If we expect Joe Plumber to know (or care) about that, we
> have lost. We're sweeping problems under the rug by putting the onus on users.

You're right. I agree.
 
> > How aggressive Tracker is
> 
> It can be as aggressive as it wants when the computer has been idle for a while
> and is not on battery. Call me crazy, but I consider this decision to be "the
> computer's job" and expect it to be fully automated (or maybe configured by the
> distro or device maker if not targetting a desktop/laptop). I suspect others
> have beaten this horse to death before me, or I'm missing something here (but
> that's probably a story for a separate discussion).

The "on-battery" point (as mentioned above) is clearly a configuration default that if changed, would improve things for you.

Sadly, we've generally taken the standpoint that data should be available as fast as possible over Tracker being less resource hungry. I think this can be device specific (e.g. on a phone or in a car, you really need this when you insert some removable device for example).

In some ways, I would like to have these sort of decisions and policies in the wiki so we can point people to it when they want to know why Tracker is the way it is and also to allow for discussion about what is preferred more generally. Nokia even struggled with this over the years, at some point they wanted data available immediately, but then they didn't want the resource hog, and it goes on like that.

Maybe we need presets based on targets that we run on?
 
> You hint a bit at this throughout your points further in your comment, but I
> say again: I don't think it should have to be configured/tweaked at all. My
> parents/aunts/joe-plumber won't ever know how to make such
> performance-impacting decisions*, nor do they care.

My counter to that is, I don't expect Joe public to have the same data set. I concede that's quite a weak argument, but you can't deny that *how* people expect their data to be available (either immediately or after some period) is subjective.

Currently we have priority queues to allow applications to make sure their content is indexed ahead of the other stuff on demand. We also have had a way to pause the miners for some years now. It wouldn't take much to add a feature where we wait for inactivity and then do the processing (i.e. resume things). The pause API is even public so people? or apps can choose to do this themselves.

So we can fix this. The question is, should we make this the default?

While thinking about this some more, I wondered if we could find some middle ground. Perhaps we have some policy where we prioritise files JUST created over files that are days old (or older), that way newly created content is always available and older content becomes available given enough idle time?
 
>   *: neither would they ever be able to make decisions on security prompts.
>      Stef Walters is making some bold new decisions in that regard,
>      his GUADEC 2013 talk on the matter is a must-see.
> 
> 
> > 1. $ gsettings set org.freedesktop.Tracker.Miner.Files crawling-interval -2
> > 2. $ gsettings set org.freedesktop.Tracker.Miner.Files initial-sleep 15
> 
> If the user launched tracker manually, fair enough, it can start grinding
> immediately, that's the result of a direct, conscious action.
> 
> Autostart, however, is a special case and should be treated as such. I think
> this is an important, somewhat separate point that makes all the
> "configurability" argument moot. It should never start touching anything for
> the whole duration of login + some time afterwards. Login is the wrong time to
> hammer the system, slowing down perceived UI/shell readiness or impeding the
> user who wants to launch various applications as soon as possible.

So we want the tracker-miner-fs process to start ASAP because we don't want to miss new files created. We set up monitors (usually with inotify) and catch the events to keep the index up to date as much as possible and so that applications using Tracker (like GNOME Documents, GNOME shell, etc) can see them immediately.

We do have a crawl or sweep which we can do when starting the miner (depending on config settings) to catch those files that get through the net too. The default is to do this and a simple mtime check but ONLY when we don't cleanly shutdown, otherwise, we assume the index is up to date.

We could turn off auto-start, but it's entirely possible the applications you launch call the DBus API and tracker is started a tiny bit later anyway. So I think disabling auto-start alone won't accomplish as much as if you only index when idle. It does depend on the applications you use of course.
 
> Obviously I'm not a Tracker developer so you know best how to achieve this, but
> from a naïve high-level perspective, I would imagine the following: wait for
> the login to be completed, add 20 seconds of waiting time "during which there

How do we know when login is completed?

> has been no further IO/system load" before you initialize your db in

What load do we gauge as sufficient?

> "read-only" mode (crawling-interval -2). Then, wait 1 to 5 minutes of "fully
> idle time" before you actually start indexing (giving users enough time to do

I think we could certainly do this.

> whatever they want to do immediately at login). In other words, continually
> defer/delay processing at startup until startup is over and the user is taking
> a break from launching stuff all over the place.

I should add, Tracker is NEVER delaying my start up because all my content is already indexed. If it isn't, well, it can be a different story :)

So this is very much a solution for content yet to be indexed.
 
> > I would add, 14 seconds for tracker-store is quite suspicious. It sounds like
> > it didn't finish indexing the LAST time you used your machine and it is
> > continuing from where it left off. IF that's the case, you simply can't expect
> > Tracker to sit still and do nothing with the current default options.
> 
> Hell yes I can. Even if indexing was not complete last time (is it ever?), I do
> expect it to sit still until my desktop is not being actively used in a
> significant way.

I did say "with the current default options" :)

And the current default is to make sure you're data is available ASAP, not make sure the system is idle before indexing.
 
> On that note, on a system where tracker hadn't been run for a significant
> amount of time (or never run at all, where no .cache/.local tracker folders
> exist), I wouldn't be shocked if tracker takes a while to index stuff... just
> NOT at system startup. Only afterwards in a transparent manner, "ideally" when
> I'm actually not in front of the computer.
> 
> > I am concerned that in this case,
> > Tracker is busy because of quite uncommon circumstances surrounding
> > the configuration and content on the machine creating this bootcharts
> 
> I'm not the person who reported bug #689302, but even I have 40 000 files in my
> affected laptop's home directory - I consider that a fairly small and

Well, 40k on a laptop is not small IMO. But again it depends on the content.

I have ca. 60k files on my server (audio/video/documents/etc), but the Linux kernel alone is ca. 44k files. So if you have software projects on your laptop, well, it probably isn't a lot of files :)

> reasonable amount (my dev desktop machine is expected to have hundreds of
> thousands of files sitting around). I doubt this is considered too much for
> Tracker to handle in an unobtrusive way (otherwise, I'll be extremely blunt and
> say that we might as well be using "locate" behind the scenes ;)

It depends on the type of content too. For example, we index the content of source files (up to a configurable max length) so you can find your g_setenv() function calls or whatever you might want to look for. This (known as FTS) can be disabled too but isn't by default because clear text (not source) files usually have useful content.

We've discussed disabling FTS (full text search, i.e. content of files) for source files only, because often those that have problems with indexing periods being too long or database size being too big, etc. are those with 10+ copies of the linux kernel checked out or generally more extreme cases.

It's a feature we've just not put in place yet and would undoubtedly make indexing faster and the database size smaller *by default*. A simple option in the GNOME system settings for "[x] Index content of my source files" could be available then for those that want to continue down that road.
 
> I wish I could call my laptop "uncommon".
> 
> It's been sitting around idle for hours and days on end. It's been in use for
> months and years. I can't imagine that Tracker hasn't already indexed the whole
> thing a couple times. A few years back I sometimes wiped tracker's data so it
> could start from a "clean state" to avoid the possibility of "bad data" being
> the cause.

It sounds like we should investigate your case more closely.
 
> Now, I've been writing this comment for the past two hours - during that time
> the laptop has been sitting idle besides me and I haven't seen the HDD I/O LED
> flash once, somewhat of an indication that Tracker had nothing to index
> further. But if I reboot it now, I can guarantee that a cold GNOME login will
> still take 20+ seconds. For the sake of verification, I just did now: and
> launching gnome-terminal to run "iotop" as soon as I can during login, I see
> tracker-store and tracker-miner-fs doing all the HDD activity indeed (an
> eventual bootchart might find other things that occur before I'm able to
> actually launch gnome-terminal).
> 
> 
> The catch? It's not a Core i7 with a solid state drive. It's a normal laptop
> from 2009 with a dualcore CPU, Intel GM45 graphics and a conventional SATA hard
> disk drive.

Still, that doesn't seem right to me. If you cleanly shutdown, it should be doing nothing on start up. The store should just be updated with states of the volumes on your machine (to disable files for removable devices) and then wait for work from tracker-miner-fs. If it's writing to the database (which it sounds like it is if you see it in iotop), something else is happening.

Can we continue with your case in the other bug report?
 
> I don't know under which hardware environment you are testing, so I must ask:
> did you or anyone here try on older machines recently (say, a Pentium 4 with
> 1-2 GB of RAM and a 5200 rpm HDD)? Booting a distro on that kind of hardware?

I have 2 machines:

1. Desktop (self made): Intel® Core™ i7-2700K CPU @ 3.50GHz × 8, 8Gb Memory, Samsung SSD 840 Series 250 GB running (Ubuntu: 13.04)
2. Laptop (Lenovo x301): Intel® Core™2 Duo CPU U9400 @ 1.40GHz × 2, 2Gb Memory, Samsung SSD 128 GB running (Debian: Jessie)
 
> To take things further and make it more representative, I suggest rsyncing your
> home into such a machine (minus the software dev/build/checkout folders) and
> using it as if it was your main computer for a few days. If you haven't tried
> that for a few years, it can be quite eye-opening :)
> 
> Sure, my overpowered quadcore desktop with a SSD boots in 5-10 seconds whereas
> this laptop will take 58 seconds (!) to get from LUKS to GDM, but that's
> exactly the kind of hardware that we can expect "Joe Plumber" to be using out
> there.

I agree. My desktop isn't really comparable and I do use the laptop and occasionally notice Tracker - plus my data set is much smaller (22k files - I don't index source code).
 
> Sorry about the length of this comment; I wanted to provide a complete
> reasoning and argument. Hope it is helpful.

No problem, I think we're getting somewhere at least :)

To summarise, I think we could improve things by:

- default config to not index on battery
- default + add config option to index ONLY when completely idle (i.e. a stage past using SCHED_IDLE)
- default + add config option to not index content of source code based files (*.c, etc).
- prioritise new files (within the last 5 minutes for example) over older content
- possibly introduce presets for different hardware abilities?

> if you don't care about finding documents that you just created.

You can add a watch on the homedir using inotify or something, to index files as they are being created. I'm sure indexing one small file every once in a while even when not idle won't hurt performance that much. Don't index anything on miner startup, just add the watch. Once idle, index everything. Prioritize indexing small files instead of big ones. 


> How do we know when login is completed?

gnome-shell should announce its ready-ness in some way.

I have one question tho: I don't know how tracker works. What does tracker do when it encounter big ISO file, or a big archive, or a big video file? does tracker read the entire file or is it already optimized to only read metadata (file list of archive, metadata tags from the video file, etc etc)?

Another thing I should point out is that you really need good defaults. Most users aren't going to tweak the settings too much. You should assume by default that the computer is a weak laptop. If you really want presets, you should automatically pick the right one for the user based on the type of hardware.

(In reply to comment #44)
> > if you don't care about finding documents that you just created.
> 
> You can add a watch on the homedir using inotify or something, to index files
> as they are being created. I'm sure indexing one small file every once in a
> while even when not idle won't hurt performance that much. Don't index anything
> on miner startup, just add the watch. Once idle, index everything. Prioritize
> indexing small files instead of big ones. 

Hello Elad, sorry to say it, but the solution you present is the simplest use case that Tracker has been handling for years now and the scenario Jean-François describes is more complex than that.

Yes we can use inotify and yes we do that in Tracker right already. But when you have 50k+ files in the queue (for example) that didn't get indexed before you shutdown last and you create a new file, how does Tracker know it's more or less important than the others? In an ideal world or what I would think is more common, is your queue is empty because everything was indexed before you shutdown last - making this a non-issue.

We need something in place for that circumstance. The only way right now is using the Tracker APIs to promote the file in the priority queue. My suggestion was to add more logic to prioritise files better.
 
> > How do we know when login is completed?
> 
> gnome-shell should announce its ready-ness in some way.

Should or does?
 
> I have one question tho: I don't know how tracker works. What does tracker do
> when it encounter big ISO file, or a big archive, or a big video file? does
> tracker read the entire file or is it already optimized to only read metadata
> (file list of archive, metadata tags from the video file, etc etc)?

In ALL cases, we index the *file* only metadata (in tracker-miner-fs), that is, name, size, mtime, etc.

On top of that we index as much as the extractors can extract (in tracker-extract):

- ISOs: see the ISO extracter RedHat wrote:
  https://git.gnome.org/browse/tracker/tree/src/tracker-extract/tracker-extract-iso.c

- Archives: no extra metadata is indexed. we used to have one, but no more.

- Video: the same as audio in some ways, we have specific and generic extractors.

  A specific extractor would be the MP3 extractor
  https://git.gnome.org/browse/tracker/tree/src/tracker-extract/tracker-extract-mp3.c

  A generic extractor would be the GStreamer extractor.
  https://git.gnome.org/browse/tracker/tree/src/tracker-extract/tracker-extract-gstreamer.c

  All configurable at build time and found at run-time.
  3rd parties can even provide their own.

The size of the file makes very little difference. Depending on the implementation, we may mmap or read from the file using standard APIs. In some cases we know exactly where to read from in the file based on the spec for that format. We're rarely doing things like reading the actual content (e.g. video frames) to deduce details like frame rate or other information. Extractors must be fast!
 
> Another thing I should point out is that you really need good defaults. Most
> users aren't going to tweak the settings too much. You should assume by default
> that the computer is a weak laptop. If you really want presets, you should
> automatically pick the right one for the user based on the type of hardware.

You're right. But good defaults for who?
Tracker is used in quite some places:
- Desktops
- Servers
- Laptops
- Phones
- Cars
- Other embedded environments?

Plus the requirements of tracker in each of these target areas changes according to a number of details. You never get this completely right. We've been tweaking defaults for years on and off ;)

Jean-François, a recent bootchart from GNOME 3.10 would really help a lot. I redid a lot of our startup process in GNOME 3.10 to fix race conditions and it seemed to get faster.

Jasper, I'd be happy to provide one... problem being that in Fedora 20, "bootchart" does not exist as standalone package and the kernel prevents systemd-bootchart from working: https://bugzilla.redhat.com/show_bug.cgi?id=1013225

Closing this as incomplete until an updated bootchart comes along. When the Fedora bug gets fixed, please reopen and attach a new bootchart! Thanks.

(In reply to comment #3)
> We're debugging this right now, it's a long chain of problems.
> 
> * gnome-settings-daemon initializes libcanberra

The patch in https://bugzilla.gnome.org/show_bug.cgi?id=735777 might be enough to break that chain early.