I forgot to mention that I had asked a nvidia developer about this and he said it was due to the overhead of many xlib calls.

I can also confirm this issue, gnome-shell hits around 80% CPU usage on one core when running glxgears, It also causes stuttering in all games I have tried as well. Reverting the patch mentioned by Hussam brings the CPU usage back down to normal levels.

I have also encountered this issue on the Nvidia proprietary driver. Reverting the patch fixes the problem.

I make a screenshot of cpu usage before and after  applying  
Call-coglxlibrenderersetthreadedswapwaitenabled.patch
http://storage4.static.itmages.ru/i/17/0411/h_1491948046_9928347_1598f98b62.png


I also noticed that this patch have improved responsiveness when moving windows. But now, the animations of the Gnome Shell to lag even more. Now without the load on the CPU, the decrease in FPS became visible. This is all tested on a proprietary Nvidia + 650GTX.

(In reply to Maxim from comment #4)
> I make a screenshot of cpu usage before and after  applying  
> Call-coglxlibrenderersetthreadedswapwaitenabled.patch
> http://storage4.static.itmages.ru/i/17/0411/h_1491948046_9928347_1598f98b62.
> png
> 
> 
> I also noticed that this patch have improved responsiveness when moving
> windows. But now, the animations of the Gnome Shell to lag even more. Now
> without the load on the CPU, the decrease in FPS became visible. This is all
> tested on a proprietary Nvidia + 650GTX.

I have also noticed this behaviour

I am seriously wondering why no devs have noticed this yet. This bug seems to affect literally everyone who uses an NVIDIA gpu.

I've been experiencing sluggish animations in gnome-shell after updating to v3.24.1 on ArchLinux on my laptop with dedicated Nvidia GPU (NVS 5100M); no integrated GPU is present.

The previous Gnome version did not show this behaviour.

I reverted the commit:
https://git.gnome.org/browse/mutter/commit?id=383ba566bd7c2a76d0856015a66e47caedef06b6
as mentioned by somebody on the Arch Forums and this fixed the issue for me.

(In reply to Frank from comment #7)
> I've been experiencing sluggish animations in gnome-shell after updating to
> v3.24.1 on ArchLinux on my laptop with dedicated Nvidia GPU (NVS 5100M); no
> integrated GPU is present.
> 
> The previous Gnome version did not show this behaviour.
> 
> I reverted the commit:
> https://git.gnome.org/browse/mutter/
> commit?id=383ba566bd7c2a76d0856015a66e47caedef06b6
> as mentioned by somebody on the Arch Forums and this fixed the issue for me.

Sorry I forgot to mention that I use the closed source Nvidia driver 340.xx and run Gnome on xorg.

Same for me, this is a really annoying issue.

I am on ArchLinux and it seems that with the new NVIDIA release (381.22) and latest mutter release the high cpu usage is finally gone. Also the animations are back to normal as in having very smooth animations and window movements, whatever. I am not sure if it's the NVIDIA release or mutter which has included a fix but I would assume that NVIDIA has somehow fixed the locking issue.

(In reply to Peet from comment #10)
> I am on ArchLinux and it seems that with the new NVIDIA release (381.22) and
> latest mutter release the high cpu usage is finally gone. Also the
> animations are back to normal as in having very smooth animations and window
> movements, whatever. I am not sure if it's the NVIDIA release or mutter
> which has included a fix but I would assume that NVIDIA has somehow fixed
> the locking issue.

The update to mutter-3.24.2 (ArchLinux) has not resolved the issue for me.

(In reply to Frank from comment #11)
> (In reply to Peet from comment #10)
> > I am on ArchLinux and it seems that with the new NVIDIA release (381.22) and
> > latest mutter release the high cpu usage is finally gone. Also the
> > animations are back to normal as in having very smooth animations and window
> > movements, whatever. I am not sure if it's the NVIDIA release or mutter
> > which has included a fix but I would assume that NVIDIA has somehow fixed
> > the locking issue.
> 
> The update to mutter-3.24.2 (ArchLinux) has not resolved the issue for me.

Since this is NVIDIA related have you tried upgrading to the lastest NVIDIA release? I am currently using 381.22.

I use the nvidia-340.102 legacy drivers which is up-to-date.

(In reply to Peet from comment #10)
> I am on ArchLinux and it seems that with the new NVIDIA release (381.22) and
> latest mutter release the high cpu usage is finally gone. Also the
> animations are back to normal as in having very smooth animations and window
> movements, whatever. I am not sure if it's the NVIDIA release or mutter
> which has included a fix but I would assume that NVIDIA has somehow fixed
> the locking issue.

The Mutter 3.24.2 and Nvidia 381.22 updates have not fixed the issue for me

The nvidia changelog says:
"Disabled OpenGL threaded optimizations by default, initially enabled in 378.09, due to various reports of instability."
Could this be related?

(In reply to alecdc272 from comment #14)
> (In reply to Peet from comment #10)
> > I am on ArchLinux and it seems that with the new NVIDIA release (381.22) and
> > latest mutter release the high cpu usage is finally gone. Also the
> > animations are back to normal as in having very smooth animations and window
> > movements, whatever. I am not sure if it's the NVIDIA release or mutter
> > which has included a fix but I would assume that NVIDIA has somehow fixed
> > the locking issue.
> 
> The Mutter 3.24.2 and Nvidia 381.22 updates have not fixed the issue for me

Can confirm, same setup, did not fix issue for me either

This is was my recent upgrade:

[2017-05-12 00:46] [ALPM] upgraded dialog (1:1.3_20170131-1 -> 1:1.3_20170509-1)
[2017-05-12 00:46] [ALPM] upgraded nvidia-utils (378.13-6 -> 381.22-1)
[2017-05-12 00:46] [ALPM] upgraded libproxy (0.4.13-2 -> 0.4.15-1)
[2017-05-12 00:46] [ALPM] upgraded mutter (3.24.1+1+geb394f19d-1 -> 3.24.2-1)
[2017-05-12 00:46] [ALPM] upgraded gnome-shell (3.24.1+2+g45c2627d4-1 -> 3.24.2-1)
[2017-05-12 00:46] [ALPM] upgraded gnome-shell-extensions (3.24.1+1+gfbf3cf3-1 -> 3.24.2-1)
[2017-05-12 00:46] [ALPM] upgraded konsole (17.04.0-1 -> 17.04.1-1)
[2017-05-12 00:46] [ALPM] upgraded lib32-nvidia-utils (378.13-3 -> 381.22-1)
[2017-05-12 00:46] [ALPM] upgraded v4l-utils (1.12.3-1 -> 1.12.5-1)
[2017-05-12 00:46] [ALPM] upgraded lib32-v4l-utils (1.12.3-1 -> 1.12.5-1)
[2017-05-12 00:46] [ALPM] upgraded libcdio-paranoia (10.2+0.94+1-1 -> 10.2+0.94+1-2)
[2017-05-12 00:46] [ALPM] upgraded libkexiv2 (17.04.0-1 -> 17.04.1-1)
[2017-05-12 00:46] [ALPM] upgraded libkomparediff2 (17.04.0-1 -> 17.04.1-1)
[2017-05-12 00:46] [ALPM] upgraded libreoffice-fresh (5.3.2-3 -> 5.3.3-1)
[2017-05-12 00:46] [ALPM] upgraded nvidia-dkms (378.13-6 -> 381.22-1)
[2017-05-12 00:46] [ALPM] upgraded okteta (17.04.0-1 -> 17.04.1-1)
[2017-05-12 00:46] [ALPM] upgraded okular (17.04.0-1 -> 17.04.1-1)
[2017-05-12 00:46] [ALPM] upgraded openvpn (2.4.1-2 -> 2.4.2-1)
[2017-05-12 00:46] [ALPM] upgraded qt4 (4.8.7-19 -> 4.8.7-20)
[2017-05-12 00:46] [ALPM] upgraded sudo (1.8.19.p2-1 -> 1.8.20-1)
[2017-05-12 00:46] [ALPM] upgraded wildmidi (0.4.0-1 -> 0.4.1-1)

As I said before, only several packages seem relevant to me, for example:

[2017-05-12 00:46] [ALPM] upgraded nvidia-utils (378.13-6 -> 381.22-1)
[2017-05-12 00:46] [ALPM] upgraded mutter (3.24.1+1+geb394f19d-1 -> 3.24.2-1)
[2017-05-12 00:46] [ALPM] upgraded gnome-shell (3.24.1+2+g45c2627d4-1 -> 3.24.2-1)
[2017-05-12 00:46] [ALPM] upgraded lib32-nvidia-utils (378.13-3 -> 381.22-1)
[2017-05-12 00:46] [ALPM] upgraded nvidia-dkms (378.13-6 -> 381.22-1)

Not resolved for me too.

[2017-05-12 17:32] [ALPM] upgraded mutter (3.24.1+1+geb394f19d-1 -> 3.24.2-1)
[2017-05-12 17:32] [ALPM] upgraded gnome-shell (3.24.1+2+g45c2627d4-1 -> 3.24.2-1)
[2017-05-12 17:32] [ALPM] upgraded nvidia (378.13-6 -> 381.22-1)

I am not sure if this can make a difference but, @Léo could you try using nvidia-dkms instead of the nvidia package?

Hi, I'm a Manjaro user. I was redirected to here from Manjaro Forum and I think I have the issue explained here, and if it's true, I have something to add.

When I was using GNOME 3.22 the environment worked perfect, but since the update to GNOME 3.24 I have problems with the stability with the environment, and I think the Mutter has the blame.

I have two computers. One of them is my main computer, a desktop equipped with an ASUS P5K Motherboard, a NVIDIA GTX 1050 as GPU and an Intel Core 2 Quad Q8300 as CPU. I use the proprietary blob driver 375.66-1 version from Manjaro repos, Linux 4.9 and Intel Microcode.

The second computer I have is an old Toshiba laptop Satellite Pro P200, with an Intel Core 2 Duo T7300 and an ATI Mobility Radeon HD 2600 as GPU identified as AMD® Rv630 by the free drivers stack (Linux 4.9 LTS and Mesa 17.0.5). I use Intel Microcode here too.

Well, the problem I have is sometimes the environment crashes randomly. It recovers perfect after crashing, but when the bug occurs is very annoying.

I couldn't catch any log or message to know what is the exact error, the only thing I have clear is the bug runs always when I have an amount of windows spreaded through some virtual desktops.

Yes, the bug occurs on Mesa too if it's the same problem I'm responding.

I forgot to say that Wayland looks much more stable than Xorg, but the bug happens on both servers.

(In reply to Eduardo Medina from comment #20)
> Well, the problem I have is sometimes the environment crashes randomly. It
> recovers perfect after crashing, but when the bug occurs is very annoying.
> 
> I couldn't catch any log or message to know what is the exact error, the
> only thing I have clear is the bug runs always when I have an amount of
> windows spreaded through some virtual desktops.
> 
> Yes, the bug occurs on Mesa too if it's the same problem I'm responding.


Is your bug _only_ crashing? If so, it's quite possibly not the same or related. As in the title, this report is about high CPU usage, and none of the other reporters have mentioned any crashing following that. Do you log high CPU usage before these crashes occur?

Created attachment 352150 [details]
CPU log from Eduardo Medina's old Toshiba laptop

I ran this command:
$ while true; do ps -eo pcpu,pid,user,args | sort -k 1 -r | head -50 >> logfileToshiba.txt; echo "\n" >> logfileToshiba.txt; sleep 1; done

It seems the problem is systemd-coredump, it appears with a high CPU usage more or less in the same times than GNOME Shell crashed. You can see it in the final part of the big file inside the ZIP.

Well it seems I have to report this to Manjaro or systemd.

Sorry for the inconvenience.

That's definitively a different issue. This bug is about high CPU usage cause by mutter/gnome-shell on Nvidia GPUs.

A crash is of course a bug too, but it's not a matter of CPU usage (you could say that the CPU usage of a crashed process is too low, namely 0% ...)

I see, I'm collecting all data I can to fill another bug.

From coredumpctl and journalctl I see something related with libc.

(In reply to Eduardo Medina from comment #23)
> It seems the problem is systemd-coredump, it appears with a high CPU usage
> more or less in the same times than GNOME Shell crashed.

This doesn't mean systemd-coredump is the culprit - quite the opposite. It using significant CPU coincident with a crash is to be expected, because it needs to dump the core image of the process that crashed if needed for debugging purposes.

Well, I think I'm following now the correct bug: https://bugzilla.gnome.org/show_bug.cgi?id=781799

Maybe a9f139cab66b532e83fca31d35f01b1b5650ca24 to 383ba566bd7c2a76d0856015a66e47caedef06b6 can be reverted? They are supposed to help in NVIDIA's situation but ended up causing more issues. And it was verified that reverting helps.

Unless anyone has an objection, I am going to email NVIDIA with a link to this bug report (as there is obviously a bug in the driver causing this).

More notices: With 381.21, the high CPU issue is less as they turned off gl threading optimizations by default. But if you switch tty or hibernate/resume, gnome-shell uses high CPU again when constantly rendering till I alt-f2 > r and then it goes quiet again.

@Hussam Al-Tayeb Did you get any response from NVIDIA?

(In reply to Peet from comment #30)
> @Hussam Al-Tayeb Did you get any response from NVIDIA?

I emailed them on the 28th on linux-bugs@nvidia.com with a nvidia-bug-report.log.gz file and a full description of the bug and easy reproduction steps. They never replied.

They replied last year when I reported a 13 year old Linux computer game was not running well and they even quickly fixed the bug. I thought this was worth the shot but it seems they only care for computer games.

Well, yeah, my next GPU is definitely not going to be an NVIDIA card (GTX 1060 6GB). The linux drivers are buggy as hell. But yeah, you can't really compare the performance to Intel HD Graphics or the recent AMD gpus, so I guess I am stuck here.

Confirmed here too on two different computers with Gnome 3.24.3 on Fedora 26/Linux 4.11 + NVIDIA 381.22 (GTX 560 and GTX 760), i also tried with NVIDIA 375.66 in the same computers and see the same lag after updated to Gnome 3.24.

However, with integrated intel graphics and Wayland on a macbook everything works better than ever.

So if NVIDIA 381.22 and 375.66 works normal on Gnome 3.22, i think this bug is more on gnome 3.24 than in nvidia proprietary driver.

So, nobody in the GNOME community is able to do something for a such obvious performance issue?

Maybe everybody is using nouveau instead NVIDIA. In my case i downgrade every pc with Nvidia card to GNOME 3.22. Someday perhaps Nvidia will do a better job, instead force the community to implement eglstreams instead GBM, for Wayland scenario. Nevertheless this bug is related to Xorg use only.

*** Bug 785609 has been marked as a duplicate of this bug. ***

this also happens on intel hardware, tested witha a sandy and a baytrail series laptops

In theory, commit 383ba566bd7c2 is correct. It will definitely reduce CPU usage because it's stopping things from running that are meant to be running. The real question is what does gnome-shell need to be running in idlers that's using so much CPU? That's the heart of the problem that needs more work.

Scrolling up, it appears this is mostly with the NVIDIA driver. And I can confirm from previous experience that yes the NVIDIA driver is very CPU intensive compared to Intel. It doesn't just use your GPU but also uses as much CPU as it can get. 

But that's not an excuse for this bug... I know gnome-shell uses unreasonably high CPU [1] on Intel graphics too, and hope to find time to investigate it in detail during October. But even better would be if someone else could look sooner.

[1] https://bugs.launchpad.net/ubuntu/+source/gnome-shell/+bug/1696305

Confirming bug, but I have a bit strange behavior.
When gnome-shell is idle, I see fps drop in applications, i.e. Chromium, or games.
Symptoms - launch Chromium, scrolling page is smooth. Wait 15 secs, scrolling fps drop to 30 fps and don't go up until I do something in shell, i.e. start new app, or switch few windows. The same happens in games like Terraria, Prison Architect, etc.
This almost looks like CPU or GPU drops frequency on idle and doesn't restore it.
In fact, for over a month I thought this is exactly the case with NVIDIA blob.
But then I reverted that commit (383ba566bd7c2a76d0856015a66e47caedef06b6) and problem has gone.
The only downside of this reverting is that window moving around the screen become not so smooth, like 30 fps sometimes, but at least my programs aren't affected anymore.

Archlinux, i3 4130, gtx 960 with prop. nvidia driver.
P.S. I didn't have CPU usage problem, only what i wrote above.

Frequency scaling is definitely something that exists, and can hurt frame rates. But I still suspect that reverting commit 383ba566 is the wrong answer.

It's possible the clutter frame clock is the problem. If it was to try and measure frame intervals based on past frames rather than calculating it from the monitor's expected refresh rate then you could get some unfortunate feedback resulting in artificially low frame rates. Just a theory (haven't looked at that code yet), but worth checking.

Someone should also check that glXWaitVideoSync() is being used correctly (introduced via commit 383ba566) ...

https://www.khronos.org/registry/OpenGL/extensions/SGI/GLX_SGI_video_sync.txt

Very interesting observation, please test and confirm:

1) - Launch nvidia-settings, watch PowerMizer frequencies while doing something in parallel - i.e. scroll Chromium page. when you start doing that, nvidia righfully raises it's frequency to max, but Chromium performance is still low, as is overall shell performance.

2) - Set PowerMixer to Prefer Maximum Performance, then try to scroll in Chromium and to move some windows in shell, try different animations. Performance is absolutely great, every animation in gnome-shell is smooth also.

---
May it be that mutter somehow conflicts with nvidia's PowerMizer, I don't know, SETTING, not REAL frequency. Maybe you were right about clutter,s frame clock?

I experienced similar problems way back with Intel graphics actually. Rendering was only smooth while the CPU was being stressed. The solution in that case was to change the way Mir scheduled frames. This is what I was hinting at in comment #40.

My similar experience is documented here:

https://bugs.launchpad.net/mir/+bug/1388490

And it sounds like the problem with Nvidia seems to be related to the use of glXWaitVideoSync (introduced with commit 383ba566 ?):

https://www.khronos.org/registry/OpenGL/extensions/SGI/GLX_SGI_video_sync.txt

However, we are now off-topic. This bug is about CPU usage being too high.

Ok, I found it. Tested with and without 383ba566 commit:

With this commit - nvidia drops frequencies and Chromium drops fps to 30, shell starts to lag. When nvidia raises frequencies on load, it's again 60fps. You can trigger load by switching\opening new windows. You can monitor nvidia frequency in parallel.

Without commit - nvidia drops frequencies, but Chromium is still smooth. gnome-shell fps drops to 30, you can clearly see it when moving Chromium window around the screen.

This is tested with and without HW acceleration in chromium.

So, you are right.

>> However, we are now off-topic. This bug is about CPU usage being too high.

I see it now, thanks.

I created bagreport 
https://bugzilla.gnome.org/show_bug.cgi?id=789186

Hi, Did we get to the bottom of this? I hope the filing of another bug (NV only?) doesn't draw the attention away from this issue.

I am using an intel video card. I see high CPU usage of gnome-shell when hovering mouse over the desktop background, or if I ran glxgears. Both are close to 25%. 

gnome-shell uses a few percent to zero if I am hoving the mouse over gnome-terminal or firefox.

(In reply to rainwoodman from comment #47)
> I am using an intel video card.

This issue is about the fallback code for drivers that don't support the INTEL_swap_event extension. The intel driver is clearly not one of them, so if you are seeing unusual high CPU usage, then that's a different issue unrelated to this bug.

Adding 
__GL_YIELD="USLEEP"
__GL_THREADED_OPTIMIZATIONS=0
to /etc/environment and then restarting stopped the CPU spikes. Using usleep supposedly means a client bug according to nvidia forums but right now I have low cpu usage on gnome-shell no matter what I have open. Perhaps it can help someone else.
KDE's wiki also suggests using usleep.

commenting to follow the issue -- seeing `gnome-shell` taking `106+%` CPU pretty consistently

AMD graphics card

3.24 didn't have any issues like this

Upgrading from Ubuntu 17.04 to 17.10 (running under Virtualbox) I also get this issue.

gnome-shell was using max CPU and the Wayland session was unusable.  Switching to command-line console was functional, but any UI tasks were impossible.

I tried adding to /etc/environment:
__GL_YIELD="USLEEP"
__GL_THREADED_OPTIMIZATIONS=0

But this did not help.

Turning off the second monitor in the VM settings and enabling "3D acceleration" in the VM settings fixed it for me.

Note: Another 17.10 VM installed from scratch did not have this issue, despite having 3D acceleration disabled.

*** Bug 792643 has been marked as a duplicate of this bug. ***

*** Bug 789186 has been marked as a duplicate of this bug. ***

Is there seriously no way to fix it after a year? This is becoming a bit annoying. Or it is NVIDIA fault? The only real workaround is actually to revert the commit...

https://bugzilla.gnome.org/show_bug.cgi?id=789186

Please check and comment there if you have the same problem.
Thanks in advance.

I worked out that if I first log into my windows partition, then do a warm boot into my Linux partition, the problem goes away.

I'm guessing there's some kind of initialization of the graphics card that Windows does that the Linux OSS driver doesn't do.

PS: I'm using Fedora, but the same should apply to Ubuntu.

Since this is still an issue, I looked into this and found out that this is reinventing the wheel. The same approach has already been taken by Mozilla:
https://bugzilla.mozilla.org/show_bug.cgi?id=1197954
Most interesting to learn from that is:
>> what happens if we use the main thread X display?
>>If we use a separate X11 display, which physical monitor's vsync does this listen to?
>
>If we use the main X display, depending on the GLX implementation we can either:
>
>a) crash due to threading issues (mesa)
>b) block a lot (NVIDIA)
>
>Neither are preferable. An X11 display doesn't really correspond to a monitor, so nothing effectively changes with regards to what gets synced.
>
>In this case we synchronize with the default CRTC (at least on NVIDIA).

In the current implementation I can't see this using a separate display. Has this been taken into account?

As I understand this bug should be migrated to Gnome's GitLab otherwise nobody will fix it, is that correct?

All open bugs (e.g. this one) will eventually be migrated to gitlab.

I am now testing nvidia-390 and gnome-shell/mutter 3.30 and reverting commit 383ba566bd7c2a76d0856015a66e47caedef06b6 does not seem to make any difference at all. There is high CPU usage still, so that seems to be caused by something else.

Is anyone able to find that reverting the infamous commit actually makes a difference with the current mutter code and more recent Nvidia driver? If not then this bug should be closed.

I feel this bug has become a general discussion about Nvidia performance when the Description at the top intended for it to be about that specific commit.

I just checked and while the situation seems to have somewhat improved, this is still far away from being resolved.
- GK208M using PRIME output
- Nvidia driver 410
- gnome-shell-3.30.1
- mutter-3.30.1

Previously, just wiggling the mouse raised gnome-shell cpu usage to 30-40%, this effect has vanished, no noticeable difference, about 4% cpu usage with or without threaded-swap-wait.
Moving a window (gnome-terminal) slowly around still puts gnome-shell to >80% cpu usage opposed to being 30-40% without the threaded swap wait.
While people running a desktop might find this acceptable, this represents a critical regression for me since affecting battery life being on a notebook.

This bug should not be about general Nvidia performance when the original reporter was citing a specific commit.

I am analysing the performance of Nvidia right now, and certainly it is not good. But if we're no longer talking about commit 383ba566bd7c2a76d0856015a66e47caedef06b6 then this bug should probably be closed and replaced with new more current bug reports.

I don't understand what you're trying to tell me. This bug report is about high cpu usage when threaded swap wait is enabled, which is done by this specific commit.
And I just confirmed that this is still the case.

Or maybe you misunderstood me:
Yes, reverting that commit still makes a difference in cpu usage.

Yes, sorry. I misunderstood the statement about "no noticeable difference, about 4% cpu usage with or without threaded-swap-wait.", and then failed to notice your second mention of it with respect to window movement.

I did not include window movement in my testing because the original reporter only mentioned glxgears. I will now retest with window movement, which is also mentioned a major problem for Nvidia users in https://gitlab.gnome.org/GNOME/mutter/merge_requests/168.

Nope. Reverting commit 383ba566bd7c2a76d0856015a66e47caedef06b6 makes no difference to window-moving performance for me (nvidia-390 with mutter/gnome-shell 3.30.2 master branches). So I can't endorse removing it myself.

Someone who does benefit from the change should propose a merge request here:
  https://gitlab.gnome.org/GNOME/mutter
and include performance stats of before and after.

With mutter versions 3.26-3.28 opening windows preview (with Windows key) for the first time after logging in or after not using it for long time was really choppy on X11.
Reverting this commit or running Wayland made it smooth.

Somebody also said (I think it was on Reddit but cannot find it) after rebooting from Windows makes this issue disappear but it appears after full shutdown.

Daniel, what kind of system (desktop/optimus notebook) and gpu (Fermi/Kepler/Maxwell/Pascal/Volta/Turing) were you using for test? I would based on that do more tests to see if this is system specific.

While you're here, please allow me to ask a question about the specifics of this implementation. The current flow of gl commands on buffer swap currently is:

glFinish
start new thread --------------->glxWaitVideoSync
glXSwapBuffers
return
dostuffordont
glFinish

The impression I got from research about glXSwapInterval and glxSwapBuffers is that the nvidia implementation is behaving differently than other ones.
I learned (correct me if I'm wrong) that with nvidia glx, glXSwapBuffers is just put on the pipeline, only executed when glFinish is called. Since glFinish is blocking, I would expect that it blocks until the VSync is happening, which would mean that
glXSwapInterval (1) + glXSwapBuffers + glFinish = glxWaitVideoSync
 so that the swapping logic for nvidia could be changed to:

start new thread --------------->glXSwapBuffers + glFinish
return
dostuffordont

Would that be worth experimenting with?
The reason I'm asking is, if you're following the discussions about Mozilla using the same approach mentioned in comment #57, a Nvidia dev mentioned "glxWaitVideoSync? They shouldn't use that old extension."

Mateusz,

The windows preview performance is not an Nvidia problem, but one that affects all GPUs. The main fixes I know improved that are:

  https://gitlab.gnome.org/GNOME/mutter/merge_requests/105
  https://gitlab.gnome.org/GNOME/gnome-shell/merge_requests/73

But you need Gnome 3.30 to benefit from those.

Certainly some low-level rendering changes such as being described here will help, but the main fixes for the preview are nothing to do with any graphics driver. See the links above.

Similarly, the icon spring animation's poor performance has very little to do with the graphics driver. The fixes for that are general CPU fixes:

  https://gitlab.gnome.org/GNOME/gnome-shell/issues/349


---

Maik,

I am presently testing with a Quadro K620 (Maxwell?).

No, there should not be any calls to glFinish being used by default because that function stalls the pipeline and kills performance. I have checked a couple of times and I don't think mutter is using glFinish by default. You will find it in the source code but it's (hopefully) never called.

OpenGL in general, including the final glXSwapBuffers or eglSwapBuffers are already threaded by design. GL commands are only requests to the GPU to do the work /eventually/. Even after glXSwapBuffers returns that does not mean the GPU has finished rendering the last frame. So threading would be redundant. Also threading creates new synchronization problems that would take a lot of effort to solve, AND the nouveau driver is not thread safe and likely to crash (https://bugs.freedesktop.org/show_bug.cgi?id=92438).

So I think you're guessing there. The main lesson I have learned in most of a year working on mutter performance is that your initial guesses are usually wrong and the main causes of poor performance are much stranger than you imagined.

Daniel, I think you completely missed my point. This was about the current implementation of the threaded swap wait which is affecting the nvidia proprietary driver only. So nouveau has nothing to do with it anyway, it not being thread safe is already mentioned in comment #57. Nevermind, anyway.
glFinish is always used as _cogl_winsys_wait_for_gpu in cogl-winsys-glx.c but IMHO not always in a sensible way.
My guess was based on an article where someone investigated and explained the implementation differences of swapbuffers/vsync in the different drivers mesa/amd/nvidia, probably outdated. Still a guess though.

(In reply to Daniel van Vugt from comment #62)
> This bug should not be about general Nvidia performance when the original
> reporter was citing a specific commit.
> 
> I am analysing the performance of Nvidia right now, and certainly it is not
> good. But if we're no longer talking about commit
> 383ba566bd7c2a76d0856015a66e47caedef06b6 then this bug should probably be
> closed and replaced with new more current bug reports.

The issue with the commit is still relevant but for me, adding
__GL_YIELD="USLEEP"
_GL_THREADED_OPTIMIZATIONS=0
to /etc/environment and restarting Xorg worked around the issue.
So it's either those environment variables or reverting the commit.

Maik,

Re-reading your comment #68, I still have much the same concerns:

1. If the Nvidia driver is consistently different to other drivers, then why can't I reproduce the problem using the Nvidia proprietary driver?

2. Moving GL commands into a new thread is non-trivial (you need to manually set up the context so that the commands will work), and it may cause crashes for nouveau users (https://bugs.freedesktop.org/show_bug.cgi?id=92438). I know this bug is not about nouveau but your suggested solution may be unacceptable because we have to write code that is compatible with nouveau.

3. Any real solution should not involve glFinish at all anyway. glFinish really hurts performance so the sooner we can get rid of that the better.

Also, I proposed a new fix yesterday that might help some people here:

  https://gitlab.gnome.org/GNOME/mutter/merge_requests/277

Thank you, Daniel. This specific info destroys my idea:
> Moving GL commands into a new thread is non-trivial (you need to manually set
> up the context so that the commands will work)

The rest of your comments do not apply. The threaded swap wait is an nvidia proprietary driver only code path, emulating the intel extension glx_intel_swap_event. So changing only that does not influence the nouveau code path or any other driver's one. This specific path already contains a glFinish, so I was talking about moving that, not adding another one.

PS:
Daniel, in general I completely agree with you, curently mutter uses too many glFinish and glxWaitVideoSync which are both blocking. The prop. driver to my knowledge needs exactly one glFinish per frame for proper operation so my idea was about using that as efficiently as possible.
A web-search trying to re-find the mentioned article brought up that many gl devs were hitting the same behaviour of the prop. driver and used the same solution I was thinking of.
I didn't look into why mutter uses glxWaitVideoSync so often to wait a little here, wait a little there.

Just a reminder to all:

If you have frame rate problems then please see bug 789186.

This bug is about high CPU usage only.

For those of you who still find reverting commit 383ba566bd7c2a76d0856015a66e47caedef06b6 helps to reduce CPU, can you please note the Nvidia driver version you are using? I've tested some theories today and retested drivers 340 and 390 but can't find any problem. Or at least can't find any problem that's unique to Nvidia.

When testing for this bug please take care to not move the mouse at all. Just let rendering proceed untouched. High CPU from moving the mouse is a whole different family of issues (https://gitlab.gnome.org/GNOME/mutter/issues/283) so please don't move the mouse or discuss that here.

Please also note if the bug affects Xorg or Wayland sessions.

Correction: Please DON'T note if the bug affects Xorg or Wayland sessions.

It's rather obvious from 383ba566bd7c2a76d0856015a66e47caedef06b6 that this discussion should be about Xorg sessions only.

And remember, this bug is about high CPU only. Other issues should not be discussed here.

Those simply wanting smoother performance for Nvidia should use this instead:

  https://gitlab.gnome.org/GNOME/mutter/merge_requests/281

Ok, nvidia driver 418.43, gnome-shell 3.31.91 + mutter 3.31.91 freshly built without any additional patches.
Running only glxgears in a small window, not anything else open, not touching anything, gnome-shell cpu usage spikes to 85%.

Even just the small circle animation in the g-c-c bluetooth pane puts gnome-shell to constant 85% cpu usage.

https://www.phoronix.com/scan.php?page=news_item&px=NVIDIA-KDE-High-CPU-Fix
gnome-shell-3.32.0+mutter-3.32.0
Setting __GL_MaxFramesAllowed=1 returns cpu usage to normal when running glxgears or opening the bluetooth pane. Mystery solved?

Interesting, so GNOME is also affected by this glXSwapBuffers misimplementation?
Furthermore, setting this environment variable also fixes some big freezes that I had with GNOME Shell (or Mutter ?) 3.32.

The fix in comment 81 sounds the same as what this does:
https://gitlab.gnome.org/GNOME/mutter/merge_requests/281

Although not entirely the same.

Mutter was already trying to do things right (compared to KDE) so if __GL_MaxFramesAllowed=1 helps you then that suggests to me the issue is missing frame notifications from the backend/driver (COGL_FRAME_EVENT_SYNC/COGL_FRAME_EVENT_COMPLETE). If they are unsupported or absent then clutter_stage_cogl_schedule_update returns early and the clutter master clock would revert to a dumb fallback throttling method (which is the same as KDE's).

This also explains why reverting 383ba566b would seem to help some people. It's a workaround, but not a fix for the real problem.

I guess what we need now is a developer who can reproduce the problem (I can't) to find out why COGL_FRAME_EVENT_SYNC/COGL_FRAME_EVENT_COMPLETE events aren't arriving. Note: Arriving 150ms+ late is counted the same as not arriving, so that's possible too but would be extra strange.

Leo, I can confirm that setting __GL_MaxFramesAllowed=1 also mitigates gnome-shell freezes in startup animations. The icon zoom animation would sometimes freeze for some seconds when starting an application from the dash. Just as additional info.

Daniel, I came around doing some testing on a desktop system and there, the high cpu usage is indeed not noticeable. So maybe this is only affecting hybrid graphic setups using PRIME Sync?

Yes, it sounds like a good theory that COGL_FRAME_EVENT_SYNC/COGL_FRAME_EVENT_COMPLETE are missing/invalid in hybrid/PRIME setups. Someone should test that. Maybe me when I get time :)

A quicker fix for people might just be to use:

  https://gitlab.gnome.org/GNOME/mutter/merge_requests/363.patch
  https://gitlab.gnome.org/GNOME/mutter/merge_requests/363

That should avoid the offending spinning and high CPU. And it should work better than reverting 383ba566bd7c2a76d0856015a66e47caedef06b6. Because doing the revert has bad consequences for the responsiveness and throughput of mutter/gnome-shell's main loop.

If 363.patch works for you, that's great. But even that's not the optimal long term fix I would like to figure out still.

As a PRIME (NVIDIA+Intel) user, I would be happy to help you test but I don't know how to trace the events.

Doing some more testing on the desktop system, this now gets weird and weirder. My desktop is also effected by this, just not when running glxgears, I bought a bluetooth dongle to have the circle animation.
gnome-shell cpu usage:
Threaded swap wait enabled - Desktop:
glxgears - normal, no cpu spikes
g-c-c bluetooth animation - 85% cpu
dragging windows - 85%cpu

Threaded swap wait enabled - notebook/PRIME:
glxgears - normal, spiking to 85% cpu
g-c-c bluetooth animation - 85% cpu
dragging windows - 85% cpu

Now setting __GL_MaxFramesAllowed=1 on both systems:
desktop: no change! cpu usage still high (?)
glxgears - normal, no cpu spikes
g-c-c bluetooth animation - 85% cpu
dragging windows - 85%cpu

notebook:
glxgears - normal, 10% cpu, no spikes
g-c-c bluetooth animation - normal, 10% cpu
dragging windows - okayish, 35% cpu

Next I'll check if the mentioned patch changes anything.

@Maik I think you should also try with or without PRIME Synchronisation, just in case.

With mr363 and mr281 (minus the first commit of 281), moving the mouse spikes the CPU up to 3% and then settles down. dragging windows around spikes CPU usage up to 10% but then immediately goes back to 0% once the dragging stops.

With glxgears, I get spikes up to 3% without moving the mouse and 4% while moving the mouse. This is a massive improvement (mr281 + mr363). My desktop feels very fast!

PRIME Sync on/off changes:
__GL_MaxFramesAllowed=1 unset:
g-c-c bluetooth animation - 85% -> 30%
dragging windows - 85% -> 40%

__GL_MaxFramesAllowed=1 set:
g-c-c bluetooth animation - 10% -> 20%
dragging windows - 35% -> 45%

Entertaining but not really helping to shed light on this.

Now applied mr363+mr281, results are mixed:
Desktop:
g-c-c bluetooth animation - 85% -> 9% Good!
dragging windows - 85% -> 75% slightly better

Notebook, PRIME Sync
__GL_MaxFramesAllowed=1 unset:
g-c-c bluetooth animation - 85% -> 85% no change 
dragging windows - 85% -> 75% slightly better

__GL_MaxFramesAllowed=1 set:
g-c-c bluetooth animation - 10% -> 13%
dragging windows - 35% -> 30%

Overall, desktop is snappy with those patches, PRIME sync notebook still sluggish. With patches + prime sync disabled, mutter will now render unthrottled (animation speeds up) so high cpu usage in general.

On the desktop system with single nvidia gpu, I can only second Hussam: it's a blast when using mutter-3.32 plus both patches. Snappy, fast, great user experience.
Great work Daniel, thank you.
In the prime sync case, mutter obviously somehow derails taking different code paths.

Maik,

Great to hear. But please don't use dragging windows as a CPU test. Even without this bug (e.g. Intel GPUs) dragging windows is very heavy on the CPU. That's a separate bug (not yet reported upstream?). The fixes required for high CPU usage when dragging windows are:

  * Closure of https://gitlab.gnome.org/GNOME/mutter/issues/283 and
  * Something like https://gitlab.gnome.org/GNOME/mutter/merge_requests/270 but better.

And I don't think we need to worry about __GL_MaxFramesAllowed=1. That's what mr281 does.

---

Hussam,

I'm very glad you find those patches work and I would like mutter to get both. But for this bug I think it's work pointing out you only need mr363. Once you have that this bug is fixed, and adding mr281 only reduces output latency (which is nice, but not relevant here).

I almost forgot, dragging windows is extra expensive with the Nvidia driver:

  https://launchpad.net/bugs/1799679

So everyone please don't use window dragging as a CPU test for this bug. It has its own causes not related to this bug.

Random thought: This might be the real root cause:

  https://gitlab.gnome.org/GNOME/mutter/merge_requests/216
  https://gitlab.gnome.org/GNOME/mutter/merge_requests/216.patch

Can anyone experiencing this bug please try that fix alone?

Just to put it straight:
- mr363 fixes this bug on single GPU systems
- mr363 does not have any effect on hybrid GPU systems using PRIME sync
- __GL_MaxFramesAllowed=1 is a workaround on PRIME systems currently.

So all testing is now done on PRIME sync with __GL_MaxFramesAllowed unset.
I'm sorry, 
applying mr216 does not have any effect on my PRIME sync system.
g-c-c bluetooth animation - 85% -> 85% no change.

Of course, this is now mutter-3.32.0+mr216+mr363+mr281.

Daniel, re-reading your comment, I think I might have misunderstood you. So I now restested the following setup:
Single GPU, mutter-3.32+mr216 only (mr363+mr281 left out)
I can confirm that this setup also fixes this specific bug ( High cpu while constant rendering) but only this. (mr363 also fixes high cpu on mouse movement.)

Thanks, that is very encouraging because mr216 would explain the root cause here, and why the bug randomly affects some Nvidia systems sometimes and not others.

But I would also be curious to hear what Hussam thinks about testing just

  https://gitlab.gnome.org/GNOME/mutter/merge_requests/216
  https://gitlab.gnome.org/GNOME/mutter/merge_requests/216.patch

alone.

Maybe I need to provide a backport for older mutter releases?

My mistake. If you ever do experience the problem fixed by:

  https://gitlab.gnome.org/GNOME/mutter/merge_requests/216

then the symptom for that is a frozen screen. Not this bug.

So the real fix for this bug is still:

  https://gitlab.gnome.org/GNOME/mutter/merge_requests/363.patch
  https://gitlab.gnome.org/GNOME/mutter/merge_requests/363

And an extra, final nail in the coffin for this bug:

https://gitlab.gnome.org/GNOME/mutter/merge_requests/602

Although you only need !363 to solve this bug. !602 clarifies things further by showing after !363 it is now safe to revert commit 383ba566bd7c2a76d0856015a66e47caedef06b6 and more.

Thanks again, Daniel.
Unfortunately, I'm unable to apply MR602, tried mutter 3.32.2+MR363, mutter 3.33.2+(rebased)MR363 and mutter-HEAD+(rebased)MR363
Any prerequisite I'm missing?

Nevermind, didn't get that MR602 includes MR363.

I now tested on my Optimus system using PRIME sync with __GL_MaxFramesAllowed unset using mutter/gnome-shell-3.33.2+MR602.
The good news is, while previously MR363 had no effect on CPU usage while constantly rendering on this system, MR602 now fixes it. CPU usage on bluetooth-pane circle is ~12%.
The bad news, unrelated to this bug is that without setting __GL_MaxFramesAllowed=1 on PRIME sync, mutter is stuttering and intermittently freezing. Let alone still being laggy with or without that setting. 

I did not test this so far on the single gpu desktop system since Gnome 3.33 is quite a mess right now with lots of api breakage and increased dependencies. Is it possible to rebase MR602 to Gnome-3.32.x ?

Tested mutter 3.32.2 +MR281+520+576(-1Hunk)+602 on the single gpu system, runs fine. No noticeable issues until now. Due to the removal of threaded wait moving windows is also back to normal/expected cpu usage now.

For the remaining issues with PRIME sync where these patches don't have any effect, I updated the existing issue at:
https://gitlab.gnome.org/GNOME/gnome-shell/issues/1202

Closed via https://gitlab.gnome.org/GNOME/mutter/merge_requests/363