After an evaluation, GNOME has moved from Bugzilla to GitLab. Learn more about GitLab.
No new issues can be reported in GNOME Bugzilla anymore.
To report an issue in a GNOME project, go to GNOME GitLab.
Do not go to GNOME Gitlab for: Bluefish, Doxygen, GnuCash, GStreamer, java-gnome, LDTP, NetworkManager, Tomboy.
Bug 741915 - Builder is slow and deadlocks hard when working over SSH mounted partition
Builder is slow and deadlocks hard when working over SSH mounted partition
Status: RESOLVED FIXED
Product: gnome-builder
Classification: Other
Component: general
unspecified
Other Linux
: Normal normal
: ---
Assigned To: GNOME Builder Maintainers
GNOME Builder Maintainers
Depends on:
Blocks:
 
 
Reported: 2014-12-23 16:08 UTC by John (J5) Palmieri
Modified: 2015-01-05 19:26 UTC
See Also:
GNOME target: ---
GNOME version: ---


Attachments
make diff renderer use async git operations (14.03 KB, patch)
2014-12-24 02:23 UTC, Christian Hergert
committed Details | Review
make snippet context creation lazy (was calling git config user.email) (5.33 KB, patch)
2014-12-24 02:38 UTC, Christian Hergert
committed Details | Review
avoid hitting disk when generating search results (4.21 KB, patch)
2014-12-24 06:13 UTC, Christian Hergert
committed Details | Review
workbench: load git repository asynchronously (3.42 KB, patch)
2015-01-05 19:24 UTC, Christian Hergert
committed Details | Review

Description John (J5) Palmieri 2014-12-23 16:08:23 UTC
At work we have to keep code on our workstations so when working off my laptop I usually ssh mount my workstation and use gedit to edit my code.  I tried to use gnome-builder but it does not like working over an ssh mounted file system.

First and more critical is the deadlocks:

* When searching for a file with the global search bar the application SOMETIMES becomes completely unresponsive and has to be killed on the command line

* When selecting a file to open on an ssh mounted partition the application ALWAYS becomes completely unresponsive and has to be killed on the command line

I'm guessing either the latency is either causing a race condition or the app is trying to perform file operation that is not supported by the ssh file system. I did not see any output in the terminal.

Now for the slowness:

* When searching for a file the app becomes laggy and unresponsive for short periods of time

This is a common thread with IDE's which do intelligent searching.  gEdit doesn't have issues here because it only hits disk to load a directory or load/save files. Some suggestions are to put file lookups in a thread, cache the git index in memory or on the local file system (i.e. the tempdir which is almost guaranteed to be local) and don't perform any unnecessary reads or writes inside the project directory proper. If you need to read from files in the project directory cache the hell out of those reads. Chances are files are not going to change while I'm searching them.


To reproduce:

1) mount a filesystem with a git repo on it
2) Go to that directory (is mounted under nautilus it will be put in /run/user/250565/gvfs/sftp:host=<host>/<path_to_git_dir_on_host>)
3) run gnome-builder
4) search for file and try to open it
Comment 1 John (J5) Palmieri 2014-12-23 16:32:37 UTC


  • #0 access
    from /lib/x86_64-linux-gnu/libc.so.6
  • #1 git_path_exists
    at /home/palmieri/jhbuild/checkout/libgit2-0.21.2/src/path.c line 463
  • #2 locate_object
    at /home/palmieri/jhbuild/checkout/libgit2-0.21.2/src/odb_loose.c line 472
  • #3 loose_backend__read
    at /home/palmieri/jhbuild/checkout/libgit2-0.21.2/src/odb_loose.c line 626
  • #4 git_odb_read
    at /home/palmieri/jhbuild/checkout/libgit2-0.21.2/src/odb.c line 776
  • #5 git_object_lookup_prefix
    at /home/palmieri/jhbuild/checkout/libgit2-0.21.2/src/object.c line 166
  • #6 git_object_lookup
    at /home/palmieri/jhbuild/checkout/libgit2-0.21.2/src/object.c line 201
  • #7 git_tree_lookup
    at /home/palmieri/jhbuild/checkout/libgit2-0.21.2/src/object_api.c line 51
  • #8 git_tree_entry_bypath
    at /home/palmieri/jhbuild/checkout/libgit2-0.21.2/src/tree.c line 864
  • #9 git_tree_entry_bypath
    at /home/palmieri/jhbuild/checkout/libgit2-0.21.2/src/tree.c line 867
  • #10 ggit_tree_get_by_path
    at ggit-tree.c line 189
  • #11 gb_source_change_monitor_load_blob
    at src/editor/gb-source-change-monitor.c line 387
  • #12 gb_source_change_monitor_reload
    at src/editor/gb-source-change-monitor.c line 536
  • #13 gb_source_change_monitor_set_file
    at src/editor/gb-source-change-monitor.c line 566
  • #14 gb_editor_document_notify_file_location

Comment 2 John (J5) Palmieri 2014-12-23 16:33:40 UTC
Stacktrace from GDB.  Looks like it is still running but running a bunch of git commands that may be extremely slow over a connection with high latency.
Comment 3 John (J5) Palmieri 2014-12-23 17:27:46 UTC
So I let this run and went to lunch.  When I came back the file was loaded.  This seems to me an issue with ggit_tree_get_by_path being really slow on latent file systems.
Comment 4 Christian Hergert 2014-12-23 23:50:46 UTC
Makes sense. I did this synchronously because I wasn't sure of the thread safety of libgit2-glib (although libgit2 underneath should be thread safe).

We do the blog comparision in a thread at least (to determine the diff), just the repository discover and blog loading that is done synchronously.

I'll see if I can come up with something quick, otherwise I'll look next week.
Comment 5 Christian Hergert 2014-12-23 23:51:07 UTC
s/blog/blob
Comment 6 Christian Hergert 2014-12-24 02:22:27 UTC
I even had a comment in that code that mentioned that this would occur :)

I've pushed a fix for some of this to master.

In particular, we load git data in 2 places. The diff renderer in the source editor, and the search index.

The global search index builds a fulltext index of all the filenames. I think that was done all asynchonrously. I need to double check.

The diff renderer was not, it was loading repository info and gitblob information synchronously. That is now performed in a thread using GTask.

I'm still not happy with the slowness loading a new file (it seems like its blocking when using ctrl+shift+o and i need to track down why).

Anyway, just pushed a patch that should make some of this better.
Comment 7 Christian Hergert 2014-12-24 02:23:13 UTC
Created attachment 293312 [details] [review]
make diff renderer use async git operations
Comment 8 Christian Hergert 2014-12-24 02:38:24 UTC
Created attachment 293314 [details] [review]
make snippet context creation lazy (was calling git config user.email)

This patch should speed up startup when running from an sshfs mounted directory. We were creating the snippet context on snippet creation which would hit the filesystem when running 'git config user.email'. Pretty braindead. Startup is much faster for me when running from a sshfs directory now.
Comment 9 Christian Hergert 2014-12-24 06:13:32 UTC
Created attachment 293319 [details] [review]
avoid hitting disk when generating search results

We were hitting the disk once to read refs and master file of the git repository. Not necessary since we can cache that when loading the GgitRepository. This makes searching when using sshfs as fast as local for me.
Comment 10 Christian Hergert 2014-12-24 06:27:15 UTC
John, I'm still not happy with the state of the open dialog (it looks like it hangs a bit for me). But I'm curious if things are better for you now otherwise.
Comment 11 John (J5) Palmieri 2015-01-05 18:04:46 UTC
Much better. It still lags while loading the index but doesn't hard freeze and once I select a file the latency is what I would expect for loading over a network.

A note - keeping latency issues in check with good caching and only writing to local disks for tmp files and internal indexes would be a killer feature as most modern IDE's don't handle this well causing many people to revert to vim/emacs when doing any remote editing.

Sorry for my own latency. I've been on vacation.

Fixed for me for now. I will open up a new bug if it becomes too slow again. Thanks.
Comment 12 Christian Hergert 2015-01-05 18:21:48 UTC
Sounds good. I wonder what we could do in terms of replicating with a traffic shaping script using `tc'. For now, I've just been testing with a hosted VM.
Comment 13 Christian Hergert 2015-01-05 19:24:59 UTC
Created attachment 293869 [details] [review]
workbench: load git repository asynchronously

This slipped through the cracks when fixing loading in sshfs directory.
Comment 14 Christian Hergert 2015-01-05 19:26:03 UTC
Comment on attachment 293869 [details] [review]
workbench: load git repository asynchronously

Attachment 293869 [details] pushed as e66c6fa - workbench: load git repository asynchronously

Found another one that was slowing down application startup on sshfs.
Looks like loading the git index was falling through the cracks when
loading the search bar.