GNOME Bugzilla – Bug 358479
Bounty: Rewrite symbol-browser plugin
Last modified: 2009-01-10 13:20:27 UTC
A rewrite of symbol-browser plugin is in place. The rewrite is about view/model/store of the symbols database. It will use sqllite for the database for primary storage and implement a GtkTreeModel interface exclusively for using it in Views. A store class will be created that proxies the sqllite database and provide GtkTreeModel interface. The design for this work is still under sketch. Following the given thread for more details: http://sourceforge.net/mailarchive/message.php?msg_id=36842046
Massimo, please assign the bug to yourself.
Created attachment 74103 [details] Entity-Relationship Diagram for the upcoming sqlite database. This is a beta version This schema describes a possible database to use with the new symbol browser plugin. It's a "beta" because we still have to define the parent/child relationship against scope and inheritance. Naba gives some ideas: "We should also give a good thought on how parent child relationship would work out. It might be a good idea to come up with an (associative) entity that defines such relationships. In fact, we might even consider defining multiple parent-child relationship entities, each based on different criteria (for example parent/child relationship in view is different from in inheritance based parent/child relationships and scope based parent/child relationships). This is just a random thought though. The good thing with this would be that we would not end up figuring out the parent/child relationships in the code everytime."
Created attachment 81522 [details] test program for sqlite database population from ctags tags
Created attachment 81523 [details] Dia schema and SQL tables for Anjuta's database. It will be populated by 'sqlite_pop_from_ctags' test program
Use of 'sqlite_pop_from_ctags' program. First of all you should check to have installed the following libraries: libgda-1.99, glib-2.x, sqlite3-3.3.11, ctags 5.6 Secondly, untar the program somewhere, run ./configure and make. If everything goes good you're in luck :) Thirdly, set up libgda. This isn't done automatically by the test program, because I scheduled to do it when integration with Anjuta will be complete. Well, launch gda-config. at prompt "gda>" give "config". Following the instructions and helps of the interactive shell, you should be able to add a section to manage the database. Mine appears like this: Section 2: /apps/libgda/Datasources/sqlite_pop_from_ctags name: DSN type: string value: DB_DIR=/com/c/sqlite_pop_from_ctags/src/;DB_NAME=db_anjuta name: Provider type: string value: SQLite name: Password type: string value: name: Username type: string value: name: Description type: string value: This is just a little test All ok? good. Come back into 'sqlite_pop_from_ctags' src directory [mine is /com/c/sqlite_pop_from_ctags/src], copy the contents of the other attachment you'll find here "Dia schema and SQL tables...". Launch sqlite3 with "sqlite3 db_anjuta.db". When the prompt appears " sqlite>" type in ".read tables.sql". Ok now you should have tables created, check with command ".tables" to see them. The work has almost been done. Just other two steps are remaining. Go on the root of your favourite project [Anjuta is too big for now, try something little]. Type in "ctags -R -o tags --fields=afmiKlnsStz --languages=+c,+c++,-Make *" a 'tags' files is created on the current directory. Ok, just copy this on "sqlite_pop_from_ctags/src" directory and set the #DEFINE FILE_TO_OPEN on main.c pointing at this file. great, make && ./sqlite_pop_from_ctags and you should see the population of the db. Once finished you can issue all the queries you want. Please note this: in this test program there's no support for "project" and "workspace" tables. I'll do that when integration with Anjuta will start. "Heritage" is still unimplemented, but scope works. Have fun and tell me what you think
Created attachment 82257 [details] test program [version 0.2] for sqlite database population from ctags tags second test case. Now heritage and scope are working good, at least on my test-case projects. Typedefs and others scope structures have been added to parsing. Changes on tables.sql: just rename 'field_typedef' into 'field_typeref' on table '__tmp_heritage_scope'.
Could you please fix your testcase? It is broken im many way: ./configure has wrong input files and does not check for glib, libgda, etc You seem to use a weird version of libgda (1.99). Please either use 3.0 (from svn, preferable as it has a better API) or 1.2.
Created attachment 83243 [details] test program [version 0.3] for sqlite database population from ctags tags Updating of symbols is working now. Some bugfix are included in this release of test program. On libgda versions there's a big confusion IMHO. The site http://www.gnome-db.org/ is very weird and without any info/docs up-to-date. My dropline-gnome uses libgda.1.99 [aka libgda 2.0 beta]. I'll use 3.0/svn when I'll convert the program to be Anjuta plugin, hoping the gnome-db guys will have something stable soon. Anjuta plugin is already started.
Hi Massimo. Nice to hear that you are making progress on this. I went through your main.c file and would like to point out some important things. Coding style: ============= Please follow anjuta coding style. You can find it in anjuta sources, but if you want to know it quickly, you can find a short example code in HACKING file that demonstrates it. Also please try to trim function calls to 80 chars width. Some lines can not be trimmed properly (such as long running database statement), which is fine and leave them. The HACKING file also mentions some other important things that you need to keep in mind for anjuta codes. It is better to follow the style right from the beginning then fix later (by which time it could become boring). Object oriented: ================ Begin writing the class structures first and methods later, not the other way round. This will save you lot of time later trying to 'fit' those already written functions in your new class structure. For example, you can create a new class called AnSymbolEngine and figure out the necessary methods that you see would be required in rest of your implementation. Come up with signals and properties that you see fit for the class etc. And then port your functions to this class (appropriately renamed, of course). I would suggest doing that right away to avoid unnecessary porting work later. Then ensure you start with the class not methods for other implementations. Code: ===== Consider looking into incremental updates using g_main loop now (the outputs from AnjutaLauncher is a source loop). This will require some changes from what is being currently done in main.c, such as context maintenance during the execution. Please note that your primary focus should not be on database operations (inserts and selects), but on doing the implementation (that uses these operations) in proper and optimal way. While you got the db operation part right, you left out the more important aspect the implementation, namely code structure and quality. I couldn't quite go through the code in details and so can't comment much on the implementation right now. I plan to do that as soon as you have properly formated and structured code. However, I did notice a couple things. You are using g_strdup_printf() to formate all db statement strings. I am pretty sure the db library would provide a function to create statements from formated strings (taking care of any db specific character escapings). Please consider using that instead. The other thing is that I *didn't* you using prepared statements anywhere. I can speculate that there would be many repetitive inserts considering the things it does, and doing it all with statement_execute is very expensive. Please consider using prepared statements wherever appropriate, such as in a loop of inserts. I hope you will find my comments helpful. Looking forward to more updates.
Hi Naba. I'll synthesize here our converstation on IRC just to keep others up to date. The test program attached on this bug report aimed to have a quick and debuggable support to Anjuta's db. It would have required some time trying to do that using an Anjuta plugin. So contexts, coding styles, code structures used there are just indicative. The new plugin, based on SymbolDBEngine class, can now create a new database and populate it with the tables.sql. I'll implement now the tags input side. At a third step I'll merge the sql-populating-code of the test program into the plugin. I'll use prepared statements which give more performances. This task should be quick. Last step will be the GtkTreeModel stuff. I still have to look inside it, but I suppose that, using signal connections with SymbolDBEngine, the thing is doable. Ctags actions. I thought a lot about this. There are several ways to pipe in/out the symbols/text buffers. I suggest to try with this implementation: * New Project: - buffer updating on the fly: take the buffer, save it into a temporary file [e.g. /tmp/fileXYZ] and then call ctags using Anjuta Launcher. Parse then the 'tags' file with ctags-library and update the database. - On saving file: call ctags on the saved buffer and then parse the 'tags' file with the library to update the db. * Existing project/Importing project: - if Anjuta's db exists, load it. If the user clicks refresh then update files newer than scan time. - if Anjuta's db doesn't exist then perform a full scan of project and populating of db. ctags --filter case: I won't use this feature coz it's too complex. It works as this: it listens for file names on stdin, and then outputs the parsed symbols. This would require the same the saving of a temporary file, and then a reparsing with ctags-library. More: to have everything on the fly it would require an hack on ctags-code, which isn't so simple and error-prone.
Created attachment 87172 [details] first compilable draft of symbol-db plugin Here it is a first working copy of new symbol-db plugin. It's compilable against libgda-svn revision 2888. Probably it's not compilable against libgda-3.0.0, but libgda 3.0 isn't compilable itself... so.. As you can see from symbol-engine.h, right now it can add_new_files, update_project_symbols, remove_file, update_files_symbols also create a new db, add a project etc. To test it you can follow these steps: 1. create a new simple anjuta project. 2. copy some .c files with some symbols inside [you can copy them from other projects inside the new test project, it's not important where they come from now] 3. add one file to project and see the db file created and populated. 4. you can sqlite inside it with sqlite3 .anjuta_sym_db.db, and have 'select * from symbol' etc. There are still some functions missing, like the buffer updating, but it should be a matter of few hours to be completed. I think we're ready to study the gtk-tree-view approach, and together with that, implement the find_scope_members(), get_symbol_parents () etc functions. I decided to post this first version because I encounter a problem with anjuta-launcher. As you can see from plugin.c code, symbol_db_engine_add_new_files () will fork a ctags process, and the function will return *before* the ctags process end the scanning. I don't know how to handle this thing in a event-driven approach. If it was multi-threaded it would just be a matter of GCond or a semaphore.. but how to do with Anjuta? Is there a way to 'wait' for the complete scanning/symbol-parsing? I'm not speaking about signal catching, I'm already doing that, but I would like to hide to the caller what's happening inside that forking_function. A polling method should also be done, but I dislike that approach too. Any ideas? thanks and regards, Massimo
Great Massimo! I am trying it out soon
Hi Massimo, (In reply to comment #11) > As you can see from plugin.c code, symbol_db_engine_add_new_files () will fork > a ctags process, and the function will return *before* the ctags process end > the scanning. I don't know how to handle this thing in a event-driven approach. > If it was multi-threaded it would just be a matter of GCond or a semaphore.. > but how to do with Anjuta? Is there a way to 'wait' for the complete > scanning/symbol-parsing? I'm not speaking about signal catching, I'm already > doing that, but I would like to hide to the caller what's happening inside that > forking_function. > A polling method should also be done, but I dislike that approach too. > Any ideas? > > I think the way you do is run ctags in server mode and keep on feeding the file paths to its STDIN. Then separately, keep listening to STDOUT of the process which will come via callback from AnjutaLauncher. In that callback, you maintain a buffer that grows as more output arrives and at some point where you deem appropriate (such as a when a full file has been scanned by ctags -- I presume it can send a 'marker' at the end of the file's output) you just take the buffer and added them all in database. Instead of waiting adding all the file's tags at once, you may also consider flushing the buffer after N tags have arrived. Check out message-view plugin where something similar is done with build outputs. It flushes the buffer at the end of every line, but you can easily do so for N tags in your case. > thanks and regards, > > Massimo >
Hi Naba, my design doesn't use ctags in server mode, but executes a new instance every time for a group of files [see a GPtrArray of files-paths]. This avoids to save the scanned symbols into a new temp file by hand. Infact The temp file with ctags-symbols is necessary to use readtags.[c|h] library to parse them. Where the problems are: the process of taking the GPtrArray with file paths, scan them and import into db must be atomic. If we call symbol_db_engine_add_new_files () more than once before it has finished the first job then we're in trouble because of concurrency problems. If we want to add two or more files with 'Add Source File' dialog, unfortunately on_project_element_added () gives out a separate file per time, which causes the concurrency problems said above. I cannot find a good solution to this problem. I last thought to implement a sort of queue on symbol-db-engine which could serialize the scanning, but also here there can be mutexes problems. Note: these problems are present into different forms also with ctags in server mode. I keep thinking about a solution. We should find it now because this behaviour will be more complex going on with plugin developing. ps: please note that the message link at the top of this bug report brings to a spam page. thanks and regards, Massimo
(In reply to comment #14) > Hi Naba, > > my design doesn't use ctags in server mode, but executes a new instance every > time for a group of files [see a GPtrArray of files-paths]. This avoids to save > the scanned symbols into a new temp file by hand. Infact The temp file with > ctags-symbols is necessary to use readtags.[c|h] library to parse them. > I understand your point. The problem in using a separate temp file is inefficiency. Perhaps we can come up with memory based file (e.g FIFO)? readtags(anjuta) can then start reading the file while ctags is running. This also means you don't have to wait for ctags to exit before starting the scan (and the i/o buffer is always maintained in memory). > Where the problems are: the process of taking the GPtrArray with file paths, > scan them and import into db must be atomic. If we call > symbol_db_engine_add_new_files () more than once before it has finished the > first job then we're in trouble because of concurrency problems. > If we want to add two or more files with 'Add Source File' dialog, > unfortunately on_project_element_added () gives out a separate file per time, > which causes the concurrency problems said above. > I cannot find a good solution to this problem. I last thought to implement a > sort of queue on symbol-db-engine which could serialize the scanning, but also > here there can be mutexes problems. Note: these problems are present into > different forms also with ctags in server mode. > AnjutaLauncher is only useful when you are intending to use the output from the process. If you are going to run ctags only, the better approach is just fork it. Something like: 1) fork ctags tmp_file 2) start scanning tmp_file 3) waitpid() The reason I hate this approach is because we start ctags everytime and with lots of files, it will be quite an overhead. Another approach, which I would prefer, is to run in server mode. You start ctags once in the beginning (or the first time it is required). Initialization: =============== 1) Maintain a queue of files. 2) Maintain a tags output *shared memory* buffer (see shm_open). Give a unique file name, such as /anjuta-tmp-PID-COUNT. Invocation: =========== 3) Each call to symbol_db_add_new_files(): 3.1) a) If ctags hasn't been running, setup launcher and start ctags in server mode. b) push the files both to the queue and to ctags STDIN (using AnjutaLauncher API). Processing: =========== 4) When output arrives via AnjutaLauncher, separate it out into lines. 4.1) For each line, check if it is the filter-terminator (indicating the end of the file scan). 4.1.1) If yes, a) Pop out the topmost file from the files queue (use this filename to populate the DB). b) Invoke readtags with the share memory filename (/anjuta-tmp-PID-COUNT) c) Reset or truncate the shared-memory to 0 size. 4.1.2) Else, a) Append the line to the share memory (use the handle returned by shm_open). 5) When ctags process dies (you get the notification, we want to recover where we left: a) Reset shared memory to 0 size. b) Restart ctags and push current files in the queue to it (they are the one unfinished). Deinitialization: ================= 5) a) Disconnect all signals from the launcher. b) If launcher is there, destroy it (will also destroy ctags if running). c) If shared memory is there, close it. d) Other necessary cleanup. The point of using shared memory is to allow opening the buffer as file for readtags. If you don't like this, you can perhaps hack readtags to work with memory buffer, but I suppose that is not worth doing now and the performance gain is very slight. And the point of maintaining a files queue is to get the filename at the end of each scan so that you can use it to update the database. You also know exactly whose output the shared memory currently holds. I hope this approach takes care of the synchronization you pointed out.
Created attachment 88084 [details] symbol-db plugin v.0.2 Here it is a new version that takes care of ctags in server mode. Many bug fixes are included here. It compiles and works with libgda 3.0.1
Created attachment 88843 [details] Symbol-db v 0.3 This version supports symbol-db-engine-iterator within its interface. When querying for a symbol the caller will be able to use some defaults infos like the symbol-name, the file-position, the signature [if it's a function], and also can query for 'extra-infos'. These can be implementation kind, access kind, the file path, the language of the file and so on. By separating these infos from the default one we can achieve a better performance. In fact every extra info needs a SQL JOIN with other tables. Currently implemented are symbol_db_engine_get_current_scope (), symbol_db_engine_get_symbol_parents (), symbol_db_engine_get_scope_members (). I'll probably add something about getting info on the single symbol itself. The parameters to the function will be the scope_path, the symbol name and kind. It would be very useful for language-support-plugins, coz they'll be able to parse simple and complex expressions while we can take here things clean and simple. With symbol_db_engine_get_scope_members () I'm ready to populate a GtkTreeView and probably also to release this plugin. regards, Massimo
Created attachment 88967 [details] Symbol tree view layout Hi all, I thought about symbol tree view layout. The following on attached file should suite good for the 'global' tab. There will also be a tab for the local files symbols and one for search. Let me know your ideas. regards, Massimo
Hi, reporting an email with LibGda maintainer about LibGda DataModel signaling: ---- snip ---- On 6/8/07, Massimo Corà <maxcvs@email.it> wrote: > > Hi guys, > > I'm gonna ask here this question because I didn't find the answer elsewhere. > > Well suppose I have a GdaDataModel, which has been obtained from a > gda_connection_execute_select_command () on a sqlite database. > I connected the data_model to 'row_inserted', 'row_updated' etc. > signals, then I created another command which inserted a row into that > sqlite database. I was using the same GdaConnection too. > But unfortunately no signal was raised, and so no callback called. > > Is GdaDataModel a rowset which detects changes into database? Or is it > like a CachedRowSet in Java [an offline rowset with no notifications of > changes in db data]? The data model retuned when executing a query does not detect changes made in the database (in fact I don't know if any DBMS offers an API to do so, if you have any example, please tell me). I think it's like a CachedRowSet in Java. A GdaDataModel emits the signals you mention only when its contents changes, but the GdaDataModel returned when you execute a query generally (this depends on the DBMS provider and of course on the actual SELECT query) cannot be modified (the data it contains is read-only). I any case the data model returned is not impacted by any other modification you make on the database (even if it's on the same connection). Now there is the GdaDataModelQuery object which lets you run a SELECT statement and any modification you make to the data model is translated into the correcponding UPDATE, INSERT or DELETE queries, so the database is modified in the same way as the data model is modified. > > what about if I insert/update a row from an external sqlite console? Then you'll see the changes you made if you execute the query again. Regards, Vivien ---- snip ---- said this I think we should find another approach to accomplish the signaling to the gtktreeview structures. I thought that connecting to the symbol-engine and raising a signal when a symbol is inserted/updated can be a solution. The problem comes when we remove a symbol: it's a triggered action, and so we should fill a temporary table with the removed syms, then select it and sending the signals. On the gtktreestore I'll store the also symbol_id, that will be signaled by the engine. In this manner we can achieve some performance over a string saving. But how to search for the symbol_id(s) in a quick way with gtktreestore? I'm not familiar with that interface, probably you have some more ideas Naba? Said this the state of the tree can be updated in realtime. Probably removing a parent node will affect also the children? And what about updating its name? Will the children remain its children? thanks and regards, Massimo
Created attachment 90499 [details] Symbol-db v 0.4 Here it is a new version of symbol-db plugin. Here we have a 'working' locals gtk-tree view tab. To try it just create a new project, e.g. /tmp/foo-bar. Copy some *.c into /tmp/foobar/src directory, load symbol-db plugin, then select project tab, right click and 'Add file'. Select one of the just copied files. Open the file into documents and select the new 'symbols' tab [the one with the fish]. You should be able to see the local symbols of the file. This is still a apha version, it has loading/unloading problems within plugin.c|.h files. Many 'visible' things have to be improved, but we're there! I'll post a screenshot. regards, Massimo
Created attachment 90500 [details] Symbol-db v 0.4 screenshot A screenshot for symbol-db 0.4 version. As you can see the pixbufs reflects the current tagmanager, but the backend it totally different!
Created attachment 92417 [details] Symbol-db v 0.5 The main changes in this version are: - many stability bug-fixes - real time updating of symbols [with visual feedback] - inserting/updating/removing signals are now emitted by the engine. Loading/unloading the plugin now should be crash free. It's nice the real time updating of symbols on the gtktreeview, at least for 'Locals' tab. I have still to think about the 'Global' one. Refreshing the local symbols every 10 seconds probably isn't the best solution. I thought about doing a refresh after 2-3 seconds after the user has typed something, but is it good enought? In debug mode this 'refresh' can freeze the editor for 0.5 - 1 seconds, which is not the better thing in this world. Is there a way to raise a sort of thread to vanish this behaviour?
Created attachment 93310 [details] Symbol-db v 0.6 beta1 Ok here we are. I think this is a good point to mark this release a beta. The main changes in this version are: - added 'global' tab - added 'search' tab - all is managed by real time updating of symbols. I've added a feature that once user has finished enter text, after 5 seconds an update of symbols'll take place. I've tested this stuff with test-class.h file of 'old' symbol-browser plugin. Try to use that to see namespace handling and global tab functionality. Under the global tab it's still missing the 'click on row' and 'go to symbol in file#line' feature, but it's just a matter of connecting some signals. Globals symbols are structured so that if you add some method under a class, or some class under a namespace, it will be promptly updated under the right parent node into the gtktreeview. Try that feature with test-class.h, or build your own file. Note: there is a problem, I think it's quite quick to fix anyway, if you do a new->new gobject class and add that to the project. Simply the symbol plugin receive a signal that something has been added to the project, but the file added isn't already present on disk, causing obviously some problems if you're gonna parsing a file that still doesn't exist... There's a need to switch some calls in project manager perhaps, but I didn't investigated that. What else? It's still missing a preference page (but what should be put there?) and the implementation of isymbol interface [probably some changes will be required here, as now we have much more freedom of queries and informations about stored symbols]. Last but not least, the only dependencies are libgda 3.0.1 and sqlite3 3.4.0. I left much debugging code in the plugin, but testing it is necessary to squash bugs that may come out. regards, Massimo
Hi! > Note: there is a problem, I think it's quite quick to fix anyway, if you do a > new->new gobject class and add that to the project. Simply the symbol plugin > receive a signal that something has been added to the project, but the file > added isn't already present on disk, causing obviously some problems if you're > gonna parsing a file that still doesn't exist... > There's a need to switch some calls in project manager perhaps, but I didn't > investigated that. It might be easier in the moment to just check if the file exists and skip it otherwise. Of course we could find better solutions later. > > What else? It's still missing a preference page (but what should be put there?) > and the implementation of isymbol interface [probably some changes will be > required here, as now we have much more freedom of queries and informations > about stored symbols]. You can change the implementation of isymbol as you like they editor completion code has to be rewritten and moved to the language-support plugins anyway. The current implementation of the interface is broken anyway. If we need no preferences - even better - but what about global tags handling?
While trying to test the plugin I only got lots of messages like that: ** (AnjutaGda:18996): WARNING **: sync problems between db and ctags filenames entries. File was /src/main.c (base_path: (null), fake_file: /src/main.c, tag_file: /dev/shm/anjuta-18996-1188464216-main.c) but the trees remained empty. Another thing: Massimo, Could you please make sure that the symbol-db plugin really replaces the symbol-browser plugin and that anjuta does really work without symbol-browser. You will need to patch some files outside the plugin and it maybe won't work for the scintilla editor as it uses a lot of tagmanager stuff internally. Anyway, for the sourceview editor you only need to implement the interfaces that are need by the language-support-plugin (of comment those for now). There are also some ActionGroup problems for the popup-menu of the old symbol-browser but it should not be that difficult to scope them. This way we could create a "new-symbol-db" branch and start to get rid of the old symbol-browser and the tagmanager directory which would make development easier for all of us. Thanks for the great work.
Hi Johannes, > It might be easier in the moment to just check if the file exists and skip it > otherwise. Of course we could find better solutions later. I'm doing that infact: if the file doesn't exist, and by now it's always the case, the file won't be added to the database. > > You can change the implementation of isymbol as you like they editor completion > code has to be rewritten and moved to the language-support plugins anyway. The > current implementation of the interface is broken anyway. If we need no > preferences - even better - but what about global tags handling? > well in my opinion the symbols' interface should be something incremental. We would know what we really need only when we'll write completion code on languages-support plugins. By now I can just implement the necessary functions for sourceview to work. About the global-tags handling me and Naba thought about creating another database on ~/.anjuta/anjuta-db.db [or similar place] where we'll store the infos collected by the installed plugins. It's easy infact to parse a high number of files and add them to the sqlite db throught the symbol-engine. We would then manage two dbs on the symbol-db plugin: one for the current opened project, the other one for the global-system stuff. Queries should be really quick in my opinion.
(In reply to comment #25) > While trying to test the plugin I only got lots of messages like that: > > ** (AnjutaGda:18996): WARNING **: sync problems between db and ctags filenames > entries. File was /src/main.c (base_path: (null), fake_file: /src/main.c, > tag_file: /dev/shm/anjuta-18996-1188464216-main.c) > > but the trees remained empty. > Hmm pretty strange. It works quite good here. How do you test it? Do you create a fresh new foobar-cpp project, say on /tmp/foobar-cpp, then copy some *.c files taken somewhere else into /tmp/foobar-cpp/src/ and then add them with right-click 'add files to project' ? > Another thing: Massimo, Could you please make sure that the symbol-db plugin > really replaces the symbol-browser plugin and that anjuta does really work > without symbol-browser. You will need to patch some files outside the plugin > and it maybe won't work for the scintilla editor as it uses a lot of tagmanager > stuff internally. Anyway, for the sourceview editor you only need to implement > the interfaces that are need by the language-support-plugin (of comment those > for now). sure. I suppose I have to edit the *.plugin.in files and add 'Interfaces=IAnjutaSymbolManager' to let Anjuta select the right plugin to load? I wouldn't know where to look for other stuff to edit. Any ideas? > There are also some ActionGroup problems for the popup-menu of the > old symbol-browser but it should not be that difficult to scope them. yeah, they're just easily-fixable details. > > This way we could create a "new-symbol-db" branch and start to get rid of the > old symbol-browser and the tagmanager directory which would make development > easier for all of us. > yeah, I really need some tests to see how much stable is the plugin. thanks and regards, Massimo
To sum up some things we discussed on IRC and others that came into my mind to make your life easier: - get rid of the browser toolbar because it scales bad and uses deprecated egg-combo-toolbar - don't care about the scintilla editor in the moment because it is hard-coded to tagmanager - you can remove the old symbol-browser from your local copy (and in the patch). Minimum requirements for creating a branch: - Builds cleanly - Does not crash (too often) - Loading projects works What can be done later: - IAnjutaSymbol interface - editor autocompletion
Created attachment 95222 [details] Symbol-db v 0.7 beta 2 Hi all, I think I've reached a commitable/branchable version. * It builds cleanly with the following dependencies: libgda 3.0.1, ctags 5.6, sqlite 3.40, Anjuta svn HEAD. * It does not crash (too often), at least in tests I've done here with my linux box. * Loading projects works. I must say a note here: little-middle sized projects import good, in a little amount of time. The 'little' project is composed by 10 sources [c/c++/java] by now, 'middle' grows till 80 files. Anjuta [775] still takes ages to import.. and we must think a solution here. Unfortunately sqlite decreases its speed in insertion when number of rows increases. We go from a 50 insertions/second on first seconds to < 10 insertions/second after 60+ seconds. I had this behaviour also in the first stages of this plugins, with just a command-line test program. Trying to import a middle project in a tmpfs filesystem does not grow the performances. Probably my solution would be to split the 'big' import into as many parts as the 'targets' number. But also in that case there will be a little freeze of the gui. With a thread everything would be better IMHO.. back to the patch: it's has done against head, and when applied it will completely remove symbol-browser/ directory and will change some templates on project-wizard/. It would be nice to create a branch to have some 'serious' test on all this stuff. I don't know how to branch with svn, please help :) thanks and regards, Massimo
(In reply to comment #29) > Created an attachment (id=95222) [edit] > I think I've reached a commitable/branchable version. Cool! > Unfortunately sqlite > decreases its speed in insertion when number of rows increases. We go from a 50 > insertions/second on first seconds to < 10 insertions/second after 60+ seconds. > I had this behaviour also in the first stages of this plugins, with just a > command-line test program. > This happens because of the indexing on-the-fly. The usual way to come around it is to disable/remove indexing while the project is being imported and re-enabled/added later. But seriously, 50 insersions/sec is still very slow from standard DB operations point of view. Is it only db insersions or it includes usual processing of the tags? Trying to import a middle project in a tmpfs > back to the patch: it's has done against head, and when applied it will > completely remove symbol-browser/ directory and will change some templates on > project-wizard/. It would be nice to create a branch to have some 'serious' > test on all this stuff. I don't know how to branch with svn, please help :) > I think it would be better if we keep it parrallel to existing SB. Anjuta can easily choose the right one (and remember) on first run. This way we can allow a fallback while it is being developed.
Some UI suggestions apart from the speed-problem: Locals: IMHO a tree view would be superior here. Of course expanders should be open by default and maybe expanders should even be hidden using "show-expanders" property of GtkTreeView Global: To sort things, create a parent nodes "Global Macros", "Global Functions", Global variables", "Global structs/classes". For C++/Java there could also be "Namespace/Macros", etc. Search: Maybe include an option to also search in system-wide tags if performance allows it. Otherwise it looks very impressive and apart from loading everything is pretty fast!
It doesn't work for me :(. Not even with a newly created sample project. All I get is empty rows in global tags with bunch of assertion failures. The suspected error I get is: ** (AnjutaGda:29999): CRITICAL **: symbol_db_view_get_pixbuf: assertion `node_type != NULL' failed Any idea?
(In reply to comment #30) > This happens because of the indexing on-the-fly. The usual way to come around > it is to disable/remove indexing while the project is being imported and > re-enabled/added later. But seriously, 50 insersions/sec is still very slow > from standard DB operations point of view. Is it only db insersions or it > includes usual processing of the tags? You made me curious and I tried with a tables.sql without any index* not 'unique' keywords... but nothing. I'm suspecting there's some 'magic' keyword to pass to sqlite schema to make it quick. The timing is considered with all the process of parsing and inserting. Right now I'm testing it in this way: I launch anjuta, open a project that does not have a .anjuta_sym_db.db file inside, then when messages like 'sdb_engine_populate_db_by_tags ()' start to be printed I give a `sqlite3 .anjuta_sym_db.db'. Roughly every second I give a `select count(*) from symbol` and look at the number returned. In the first stages I see a exponential growth which decreases at a rate o 5-10 when the inserted symbols are > 500-600. This is why I'm convinced that there's something with sqlite [or with my sql tables declaration] that defects its performances; the process of symbol inserting does not change from a file and the other, and the scopes/inheritance are calculated at the end of all the group of files. On the other side I never saw an application running with sqlite and so I cannot have comparisons. > I think it would be better if we keep it parrallel to existing SB. Anjuta can > easily choose the right one (and remember) on first run. This way we can allow > a fallback while it is being developed. Well, ok then. Where is the setting to let Anjuta choose one or the other? Just writing on plugin.in file 'Interfaces=IAnjutaSymbolManager' ? thanks, Massimo
(In reply to comment #32) > It doesn't work for me :(. Not even with a newly created sample project. All I > get is empty rows in global tags with bunch of assertion failures. > > > The suspected error I get is: ** (AnjutaGda:29999): CRITICAL **: > symbol_db_view_get_pixbuf: assertion `node_type != NULL' failed > > Any idea? > hmm that's pretty strange. Can you please attach an output of the messages?
(In reply to comment #34) > (In reply to comment #32) > > It doesn't work for me :(. Not even with a newly created sample project. All I > > get is empty rows in global tags with bunch of assertion failures. > > > > > > The suspected error I get is: ** (AnjutaGda:29999): CRITICAL **: > > symbol_db_view_get_pixbuf: assertion `node_type != NULL' failed > > > > Any idea? > > > > hmm that's pretty strange. Can you please attach an output of the messages? > This was of course fixed after I installed libgda from svn trunk.
I have committed plugins/symbol-db directory in svn trunk. You can start using it for further developments. Couple of things I did not commit: 1) Changes in configure.in, plugins/Makefile.am: libgda needs to be properly detected in configure.in and symbol-db conditionally built only when right version of libgda is found. 2) *.anjuta files: Currently we keep using the old symbol-browser until we fix default.profile to load IAnjutaSymbolBrowser interface instead of unique plugin. This would allow anjuta to make choice of the two implementations on startup. BTW, there is double definition of PACKAGE_DATA_DIR in symbol-db/Makefile.am, perhaps an error.
(In reply to comment #36) > 2) *.anjuta files: Currently we keep using the old symbol-browser until we fix > default.profile to load IAnjutaSymbolBrowser interface instead of unique > plugin. This would allow anjuta to make choice of the two implementations on > startup. > sorry, my bad. s/default.profile/*.anjuta/ files.
(In reply to comment #33) > Well, ok then. Where is the setting to let Anjuta choose one or the other? Just > writing on plugin.in file 'Interfaces=IAnjutaSymbolManager' ? > All *.anjuta files need to be fixed to load the interface instead of particular plugin.
(In reply to comment #38) > (In reply to comment #33) > > Well, ok then. Where is the setting to let Anjuta choose one or the other? Just > > writing on plugin.in file 'Interfaces=IAnjutaSymbolManager' ? > > > All *.anjuta files need to be fixed to load the interface instead of particular > plugin. > <plugin name="Symbol Browser" url="http://anjuta.org/plugins/" mandatory="yes"> <require group="Anjuta Plugin" attribute="Interfaces" value="IAnjutaSymbolManager"/> </plugin>
Created attachment 96141 [details] Symbol-db v 0.8 beta 3 Hi all. In this patch you can find: implementation of IAnjutaIterable, IAnjutaSymbol, IAnjutaSymbolManager [only isymbol_manager_search ()]. I rewrote symbol-db-engine-iterator.[c|h] and added symbol-db-engine-iterator-node.[c|h], which take the places of an_symbol_iter and an_symbol classes respectively. At load time now Anjuta detects the two Symbol Managers and let you choose your favourite. * changes in IAnjutaSymbol interface: what will still work and what won't: + everything connected with symbol-browser internals will still functioning [i.e. local tags, globals, search]. + completion tips with scintilla and gtksourceview will *both* work, due to the working ianjuta_symbol_manager_search (). [hey you did a pretty cool work there! ;)] - class-inheritance/ plugin won't be able to use the interfaces because they have been changed. I think it's not so important if we temporary let down that plugin. Ok now the questions/proposals: we have to define some types on IAnjutaSymbol interface so that we can unify 'implementation' and 'access' for all the languages. We cannot rely on ctags ones. 'inheritance' and 'scope' means nothing to me right now. These and other informations should be given once we know how to use these symbols infos. I'm thinking about the use of bison/yacc to build a tree to parse complex expressions. I know Naba knows more than me here. The same is for IAnjutaSymbolManager: we can have all the infos we want, obviously based on the schema/relations of the database; at the same time I think we cannot know them all know. So adding them incrementally would be great I think. Hoping that libgda grows its sql-parsers to permit more complex prepared-statements we can achieve good performances on queries, probably by proxy them etc. Last but not the least I suggest that queries to db would be done with brain in gear. I mean, it's absurd to query for ALL_INFO when you just need only the name and the pixbufs. The more infos you want the more time it'll take. regards, Massimo
Created attachment 96163 [details] [review] Some updates from jhs Hi Massimo! I tried to review the patch but ran into some problem. Seems like the symbol_db_engine_iterator_node_get_extra_string method is completely broken and thus brakes all views (Locals, etc.). I tried to fix that but did not succeed. Anyway, attached is a new diff which fixes some of the compile warnings, memory curruptions and missing return types. Please make sure to fix all compiler warnings because it makes it damn difficult to find really important ones if there is so much noise. And yes, I won't apply it until the views do not work. Also consider to use a hash_table for get_extra_strings() and probably use gda_parameter_get_value_str(). Thanks!
Created attachment 97551 [details] Symbol-db v 0.8 beta 4 Hi all, this is the changelog for this patch. 2007-10-21 Massimo Cora' <maxcvs@email.it> * plugins/symbol-db/plugin.c, plugins/symbol-db/symbol-db-view.c, plugins/symbol-db/symbol-db-engine.h, plugins/symbol-db/symbol-db-engine-iterator-node.c, plugins/symbol-db/symbol-db-view-locals.c, plugins/symbol-db/symbol-db-view-locals.h, plugins/symbol-db/symbol-db-engine.c: Fixed functions declarations with 'const gchar*' instead of 'gchar*'. Added a new algorithm for dynamic population of the local symbols' tab. Now the default view is tree-like. Some fixes on engine correct some population issues. regards, Massimo
To summarize some things we discussed on IRC: - The locals tab looks really great know and the global tab should work about the same way. - For C++ source files (.cc, .cxx) it still has some issues because the member methods are not part of the class. This might be a ctags issue we have to investigate that - For other languages (e.g python) it looks OK, as they do not have the stupid header/source file separation. - We might want to add a popup menu (goto declaration/definition, mark all occurences, etc) I also added the LIMIT bug in libgda as dependencies as this is one of the performance bottlenecks.
Created attachment 97668 [details] Symbol-db v 0.8 beta 5 Hi, here it is the patch that should fix the different file declaration of a symbol. Simply if a file.cpp doesn't find the declaration of MyFooClass [which is in file.hh] then we build a MyFooClass node and append it on the view of file.cpp. regards, Massimo
Hmm, I couldn't get it working with C++ classes, even after regenerating the symbol-db. I also noticed some other things: - It gets confused by namespaces and doesn't populate a tree then - When the symbol-db plugin is first activated, it generates the symbol_db file but the local tab is not working before anjuta is restarted
Hi Johannes, can you please me give me more info about the C++ classes which are wrongly parsed? Project etc. It would be really useful to have a test case. about the first point: it may be a conseguence of the same error, probably it's on the engine. about the second point you mention: did you try just closing and reopening the file in the editor [without closing Anjuta]? I never had such behaviour that forced me to close and reopen the IDE to see symbols appearing. thanks and regards, Massimo
Hi Massimo! Well, I tried with several files (from anjuta, from glom and from a custom project and I couldn't get it working for one. Unfortunately most C++ projects I have are far two big to import them now and thus I cannot test those. Closing and reopening the editor did not help in my case. I use libgda from svn, just in case it could matter. Regards, Johannes
Created attachment 98032 [details] Symbol-db v 0.8 beta 6 Hi, Changelog for this patch. 2007-10-28 Massimo Cora' <maxcvs@email.it> * plugins/symbol-db/plugin.c, plugins/symbol-db/symbol-db-engine.h, plugins/symbol-db/symbol-db-view.c, plugins/symbol-db/symbol-db-view.c, plugins/symbol-db/symbol-db-view-locals.c, plugins/symbol-db/symbol-db-view-locals.h, plugins/symbol-db/symbol-db-engine.c: Better end-of-file-scan detection, permitting an improved scope/inheritance parsing. Locals tab gtktree now displays correct namespace->class->children tree, even if in a C++ file there isn't class declaration. TODO: global tab [next week], improve file loading. I would like that on project loading the locals tab wouldn't be populated if not at the last session-restored file. This would speed up the opening of a project. I had a look at anjuta_session_get_string_list (session, "File Loader", "Files"), but the signal 'load-session' is emitted after 'project loaded', making the files to be parsed and the local-tab-tree to be populated the same. Is there any solution on this? btw [@Johannes]: I tried to load Glom project with the 'old' symbol-browser plugin, and it's really slow too to parse and load all the symbols. thanks, regards, Massimo
Created attachment 98512 [details] Symbol-db v 0.8 Hi Johannes, thanks to your test-project I was able to discover and fix many things on population. Here it is the changelog. Despite it not so important, I call this patch 0.8 release, because for me it's quite stable. I think I'll reach the feature-complete on 1.0. thanks and regards, Massimo 2007-11-04 Massimo Cora' <maxcvs@email.it> * plugins/symbol-db/plugin.c: add all files at once on project_import. No need to split them into more languages: ctags and the engine will take care of that. * plugins/symbol-db/symbol-db-engine.h, plugins/symbol-db/symbol-db-engine.c, plugins/symbol-db/tables.sql, plugins/symbol-db/test/Makefile.am, plugins/symbol-db/test/main.c, plugins/symbol-db/symbol-db-view-locals.c: some memory leaks fixed. Ported the thing to libgda 3.1.2 [or better svn HEAD]. Thanks to a fresh new algorithm to detect parent scope we're able to display correcly a local gtktree, including classes not directly defined inside that file. More performance improvements to come.
Works really good, but as usual I have some points: - I get a lot of Implementation missing: parsed_create_global_query_field() in gda-query-parsing.c line 1212 warnings. Don't know if the matter in any way - It seems to ignore pure virtual methods in C++. If you have a look at libentwickler/edit.h you will notice that the pure virtual functions are not shown. That's possibly a ctags issue though (in this case we should report a bug...)
Hi, (In reply to comment #50) > - I get a lot of > Implementation missing: parsed_create_global_query_field() in > gda-query-parsing.c line 1212 > warnings. Don't know if the matter in any way well, this is libgda related. It's just a function un-implemented in some libgda's provider. > > - It seems to ignore pure virtual methods in C++. If you have a look at > libentwickler/edit.h you will notice that the pure virtual functions are not > shown. That's possibly a ctags issue though (in this case we should report a > bug...) > Yes, that's ctags that doesn't list the functions. If you give a `ctags edit.h` and then `less tags` you'll see what ctags has parsed. Please note that I added a bug for a libgda crash here http://bugzilla.gnome.org/show_bug.cgi?id=493360. thanks and regards, Massimo
Hi guys, I downloaded Anjuta 2.3.0 to see the status of symbol-db plugin. I noticed something strange: there's no tables.sql nor symbol-db.png files inside plugin/symbol-db directory. I didn't try to compile here at work but I suppose it won't, or at least won't work once installed. Probably I'm missing something on Makefile.am? thanks and regards, Massimo
(In reply to comment #52) > I downloaded Anjuta 2.3.0 to see the status of symbol-db plugin. I noticed > something strange: there's no tables.sql nor symbol-db.png files inside > plugin/symbol-db directory. > I didn't try to compile here at work but I suppose it won't, or at least won't > work once installed. > Probably I'm missing something on Makefile.am? > Unfortunately, if we disable a plugin during 'tarball creation', the files of the plugin will not be part of the tarball. This is known problem (see indent plugin for similar effect). I haven't figure out what is the best way to add disabled plugin files in distribution, but only solution seems to put them in EXTRA_DIST in the 'else' conditional (and at the same time avoid re-listing the files, perhaps using separate variables). So yes, the disabled plugins can not be built from the tarball, unfortunately.
> Yes, that's ctags that doesn't list the functions. If you give a `ctags edit.h` > and then `less tags` you'll see what ctags has parsed. I don't really know what Elliot wants to tell me here, but take a look at my ctags but report: https://sourceforge.net/tracker/?func=detail&atid=106556&aid=1825873&group_id=6556
Hi Johannes, > I don't really know what Elliot wants to tell me here, but take a look at my > ctags but report: > > https://sourceforge.net/tracker/?func=detail&atid=106556&aid=1825873&group_id=6556 > hmm seems like there's an option to let ctags parsing prototype functions... I've been able to see them on gtktreeview after adding "--c++-kinds=+p " on Anjuta launcher command line. What I'm thinking now is: should we add a --<LANG>-kinds for every language parsed by ctags? Probably yes.. Oh, another thing: I'm missing some icons for prototype functions, where did you get the /plugins/symbol-browser/images? It should be good to add different icons for declaration and definition, seen that ctags can differentiate them. thanks and regards, Massimo
Created attachment 99288 [details] Symbol-db v 0.9 beta1 Hi all, Still in working progress: combo-box to let user select global symbols language. Performance improvements in loading/caching GtkTreeStore(s). regards, Massimo ChangeLog entry: 2007-11-18 Massimo Cora' <maxcvs@email.it> * plugins/symbol-db/test/main.c, * plugins/symbol-db/test/Makefile.am: fixed some compilation issues on test program. * plugins/symbol-db/plugin.c, plugins/symbol-db/symbol-db-view.c, plugins/symbol-db/symbol-db-engine.h, plugins/symbol-db/symbol-db-view.h, plugins/symbol-db/symbol-db-engine-iterator-node.c, plugins/symbol-db/symbol-db-view-locals.c, plugins/symbol-db/tables.sql, plugins/symbol-db/symbol-db-engine.c: Rewrote symbol-db-view global tab. Now every expandable node is a query. Started using LIMIT keyword to speed up things. Deprecated GdaCommand in favour of GdaQuery into engine. This should make providers use prepared statements. * configure.in: requred gda version to 3.1.3. [svn trunk]. Without this Anjuta can crash due to bug #493360. Still present anyway bug #495843
Committed the last patch! Some comments (and things we already discussed on IRC): - progress indication when the symbols are created - reorganization of global view to cope better with namespaces and to keep things clean, more emphasis on the icons for types - performance needs to be improved a lot, esspecially tab-switching Thanks!
Hi Naba, I've done more tests with population in importing a project to avoid gui freezing. I try to use g_idle_add () everywhere but nothing. I tried to use g_threads but some weird bug make libgda crashing happily at some random point. http://www.sqlite.org/cvstrac/wiki?p=MultiThreading can confirm this, especially at the second bullet "Do not use the same database connection at the same time in more than one thread". The point where to call the g_idle or the g_thread was on the callback from anjuta_launcher process. I'm start thinking that sqlite is too much 'lite'. Insertion speed is a problem and probably http://www.mail-archive.com/sqlite-users@sqlite.org/msg27802.html confirm this. Despite that we cannot stay without indexes ot TEXT fields. GUI freeze can happen everywhere when scanner is active, because it's too much hungry of cpu. We (me and Johannes) thought to have a look at dbus API(s) to try to solve the problem and have a parallel process that scans in background. The big picture of the new implementation would be to put a glue on symbol-db-engine and let it communicate throught dbus with Anjuta's main process. Anjuta-helper will stay there waiting for scanning requests. Of course db access will be mutexed. What do you think about this? Have you ever had experiences with dbus? I'm afraid it's a slow architecture but Johannes says it could work. Don't know, probably coming up with some test-programs should be the right choice. thanks and regards, Massimo
Hi Massimo, (In reply to comment #58) > http://www.sqlite.org/cvstrac/wiki?p=MultiThreading can confirm this, > especially at the second bullet "Do not use the same database connection at the > same time in more than one thread". This is quite normal if sqlite is not prepared for multi-threading. What it means is that you should not access the same connection (object) *simultaneously* from different threads. This can clearly be solved in Anjuta my using a mutex for accessing any libgda/sqlite resources. Also the article claims that sqlite connection (object) can be used from different thread than the one where it was created. So it also helps the mutex solution above. However, one thing which would make it not work seems to be that you can not use the connection when a prepared statement is active on one thread. This is the idea I got from the article, but may require confirmation. If this is true, it may not solve our problem of non-blocking UI, since the insert thread (with active prepared statement) will be running for quite a bit of time. The mutex won't solve anything for this. > The big picture of the new implementation would be to put a glue on > symbol-db-engine and let it communicate throught dbus with Anjuta's main > process. Anjuta-helper will stay there waiting for scanning requests. Of course > db access will be mutexed. > What do you think about this? Have you ever had experiences with dbus? That seems like a good approach. DBus is a very nice and convenient IPC method as long as you are not transporting tons of data across it. If you use it mostly for signaling it should be fine.
Hi, I've posted a bug report to ctags tracker. It took a little bit to discover it, anyway this is stopping symbol-db parsing at file 147/780: yeah now I can see how many files have been parsed out of a total. http://sourceforge.net/tracker/index.php?func=detail&aid=1840798&group_id=6556&atid=106556 if they solve it I can claim that I have a working g_thread_background architecture with non-freeze gui. I thought infact that having a DBus architecture was not solving our problem, but just moving it from Anjuta's main process to a helper one. regards, Massimo
Created attachment 99980 [details] Symbol-db v 0.9 beta2 Hi, here it is a patch that implements the g_threads architecture. GUI usability should be improved now. Please test it. It fixes also some problems with previous version about symbols population and its triggers. I'm working on editor-switch caching and better global-tab symbols display. Anyway it seems to me that threads are quite stable so here it is the changelog: 2007-12-01 Massimo Cora' <maxcvs@email.it> * plugins/symbol-db/plugin.c: (on_editor_destroy), (on_editor_update_ui), (on_single_file_scan_end), (on_importing_project_end), (project_root_added), (on_session_load), (symbol_db_activate), (symbol_db_instance_init): * plugins/symbol-db/plugin.h: show on status bar files being scanned. * plugins/symbol-db/symbol-db-engine.c: (sdb_engine_get_query_by_id), (sdb_engine_populate_db_by_tags), (sdb_engine_ctags_output_thread), (sdb_engine_timeout_trigger_signals), (sdb_engine_thread_monitor), (sdb_engine_ctags_output_callback_1), (sdb_engine_scan_files_1), (sdb_engine_init), (sdb_engine_finalize), (sdb_engine_class_init), (sdb_engine_connect_to_db), (symbol_db_engine_db_exists), (sdb_engine_get_table_id_by_unique_name2), (symbol_db_engine_open_project), (sdb_engine_prepare_executing_commands), (symbol_db_engine_add_new_files), (sdb_engine_add_new_sym_type), (sdb_engine_add_new_scope_definition), (sdb_engine_add_new_symbol), (sdb_engine_detects_removed_ids), (symbol_db_engine_get_class_parents), (symbol_db_engine_get_global_members), (symbol_db_engine_get_scope_members_by_symbol_id), (symbol_db_engine_get_scope_members), (symbol_db_engine_get_current_scope), (symbol_db_engine_get_file_symbols), (symbol_db_engine_get_symbol_info_by_id), (symbol_db_engine_get_full_local_path), (symbol_db_engine_find_symbol_by_name_pattern), (symbol_db_engine_get_parent_scope_id_by_symbol_id): * plugins/symbol-db/symbol-db-engine.h: * plugins/symbol-db/symbol-db-view-locals.c: (sdb_view_locals_init), (traverse_free_waiting_for), (on_scan_end), (symbol_db_view_locals_recv_signals_from_engine), (symbol_db_view_locals_update_list): * plugins/symbol-db/symbol-db-view-locals.h: * plugins/symbol-db/symbol-db-view.c: (traverse_free_waiting_for), (on_scan_end), (trigger_on_symbol_inserted), (add_new_waiting_for), (prepare_for_adding), (on_symbol_inserted), (sdb_view_init), (symbol_db_view_recv_signals_from_engine), (symbol_db_view_open): * plugins/symbol-db/symbol-db-view.h: Added g_thread architecture to scan in background. With this you can use Anjuta's GUI without freezing. Improved insertion speed by using a paradigm like 'insert' and 'check' replacing a 'check' and 'insert' one. This has been done for tables like symbol, scope, sym_type. Fixed two crashers. * plugins/symbol-db/tables.sql: * plugins/symbol-db/test/main.c: (get_global_members), (main), (thread), (print_message), (bastard_thread), (idle_signals): fixed a typo on tables that broke sql triggers.
OK, commited the last version. In general this is a huge improvement over previous version in the sense of usability and feel faster. Some points: * It crashes on deactivation * It hangs when you try to import the whole anjuta project * The global tab needs some love, we discussed that already. * It would be better if the symbol loading would not take place in the splash screen but when the UI is already fully loaded because it is in the background anyway. * When are the symbols updated? It seems that they are not updated when there is a old version of .anjuta_sym_db lying around. Maybe we need some timestamp to compare with. * Later: We need some global-tag database... Thanks for your nice work!
Created attachment 100368 [details] Symbol-db v 0.9 beta3 Hi. More, more and more speed with this patch! I added caching while tab-switching and IMHO this is a killer feature. Passing from an editor to another is instantaneous now and it improves the usability. About the comment #62 it fixes the crash on deactivation. I need more infos about the second point "It hangs when you try to import the whole anjuta project". It can be caused by the ctags error which I already discussed. About the fourth point always on comment #62: maybe while importing Anjuta you feel some freeze on process, I'll explain you what's happening there: when a project is opened and doesn't find the .anjuta_symbol.db file it starts importing. The importing process needs that all files of the project are sent to stdin via anjuta_launcher. While we're sending them [on Anjuta pj there are about 780 files] ctags starts to parse them *and* on anjuta_launcher output we're starting threads for population. This step is heavily cpu hungry, and despite I used a pool of at max 10 running threads to speed up the things, the send_stdin process takes a little bit. When it finishes Anjuta continues loading the other GUI components, plugins etc. So I think it's ok to let the user wait a few seconds while the 'stdin process' is running. Next patch will take care of global-tab and other things. But I think I squashed most of the crashers with this patch, so here it is. ChangeLog: 2007-12-05 Massimo Cora' <maxcvs@email.it> * plugins/symbol-db/plugin.c: (register_stock_icons), (value_added_current_editor), (value_removed_current_editor), (on_importing_project_end), (project_root_removed): added controls to clean/populate GtkTreeView(s) on projects unload/load. * plugins/symbol-db/symbol-db-engine.c: (sdb_engine_finalize), (symbol_db_engine_get_file_db_path): fixed a useless warning. * plugins/symbol-db/symbol-db-view-locals.c: (traverse_free_waiting_for), (file_view_status_destroy), (sdb_view_locals_create_new_store), (traverse_files_view_status), (symbol_db_view_locals_clear_cache), (sdb_view_locals_init), (sdb_view_locals_finalize), (on_symbol_inserted), (symbol_db_view_locals_recv_signals_from_engine), (symbol_db_view_locals_update_list): more speed on editors switching. GtkTreeStore caching method implemented. Fixed compiler warnings and a little memory leak. * plugins/symbol-db/symbol-db-view-locals.h: * plugins/symbol-db/symbol-db-view.c: (gtree_compare_func), (traverse_free_waiting_for), (symbol_db_view_clear_cache), (on_scan_end), (on_symbol_inserted), (sdb_view_locals_create_new_store), (sdb_view_init), (sdb_view_finalize), (symbol_db_view_new), (symbol_db_view_open): * plugins/symbol-db/symbol-db-view.h: fixed some crashers. Now project loading/populating should be quite quick.
Hi, (In reply to comment #62) > * When are the symbols updated? It seems that they are not updated when there > is a old version of .anjuta_sym_db lying around. Maybe we need some timestamp > to compare with. > I have a working project-importing method. On project opening the plugin will test if there are files newer than the timestamps on the database: if it's true it will update the symbols. I was thinking to add a menu somewhere to let the user rebuild completely the database, or force updating some files etc. Where can I add them? Is it ok to add them on 'global-tab'? thanks and regards, Massimo
OK, I tested your last patch a bit. Apart what I already mentioned on IRC (hangs on start-up) I also get random crashes while using anjuta most probably related to database updating.
Hi Johannes, (In reply to comment #65) > OK, I tested your last patch a bit. Apart what I already mentioned on IRC > (hangs on start-up) I think I squashed the hangs-bug. In the next few days I'll send another patch about global-tab-display rewriting. For speed reasons I'll maintain a node 'Global' always open even if there are no namespaces in the project. It would be a mess to put all global symbols on GtkTreeView root and regroup them under 'Global' once a namespace is added in the code. > I also get random crashes while using anjuta most probably > related to database updating. May it be linked to the fact that also symbol-browser is loaded? I use to delete the installed files of libanjuta-symbol-browser* to prevent it to be loaded for example in a new-created project. I know I can modify the project-wizard files to load "IAnjutaSymbolManager", but I'm not sure if it's ok for you to include these in a patch. Oh, one last thing. It would be great, if it's not much effort, if you could commit my last patch so that I can maintain a list of code changes: I usually go to ViewVC interface to check the progress of files. thanks and regards, Massimo
(In reply to comment #66) > > I also get random crashes while using anjuta most probably > > related to database updating. > > May it be linked to the fact that also symbol-browser is loaded? I use to > delete the installed files of libanjuta-symbol-browser* to prevent it to be > loaded for example in a new-created project. I know I can modify the > project-wizard files to load "IAnjutaSymbolManager", but I'm not sure if it's > ok for you to include these in a patch. Feel free to make this change in the project-wizard. Hopefully I will have more time to test your next iteration. > Oh, one last thing. It would be great, if it's not much effort, if you could > commit my last patch so that I can maintain a list of code changes: I usually > go to ViewVC interface to check the progress of files. No problem, I will commit it!
Created attachment 101644 [details] Symbol-db v 0.9 Hi guys, got symbol-db v 0.9! See ChangeLog for details. What is missing [and will be available in the final 1.0 version]: * preference page. * global-tags management. [use a separate db] * parsing of non-project files (?) [use a separate db] * probably some right-click options on GtkTreeView(s). Happy testing! regards, Massimo 2007-12-26 Massimo Cora' <maxcvs@email.it> * plugins/project-wizard/templates/java/project.anjuta: * plugins/project-wizard/templates/mkfile/project.anjuta: * plugins/project-wizard/templates/python/project.anjuta: * plugins/project-wizard/templates/terminal/project.anjuta: * plugins/project-wizard/templates/minimal/project.anjuta: Added attribute="Interfaces" and value="IAnjutaSymbolManager" to let new projects use new SymbolDB plugin. * plugins/symbol-browser/images/Makefile.am: added some images taken from MonoDevelop project. * plugins/symbol-db/plugin.c: (value_added_current_editor), (goto_file_line), (on_importing_project_end), (project_root_added), (on_session_load), (symbol_db_deactivate), (symbol_db_finalize), (isymbol_manager_search): Added project-updating feature when opening a project with some files modified externally [e.g. a svn up]. * plugins/symbol-db/symbol-db-engine-iterator-node.c: (sdb_engine_iterator_node_instance_init), (sdb_engine_iterator_node_finalize), (symbol_db_engine_iterator_node_set_conversion_hash): * plugins/symbol-db/symbol-db-engine-iterator-node.h: * plugins/symbol-db/symbol-db-engine-iterator.c: (symbol_db_engine_iterator_new): * plugins/symbol-db/symbol-db-engine-iterator.h: Moved Hash table initialization into the engine. This proxies and speeds up the process of creation and iteration of a GdaDataModel. * plugins/symbol-db/symbol-db-engine.c: (sdb_engine_ctags_output_thread), (sdb_engine_timeout_trigger_signals), (sdb_engine_thread_monitor), (sdb_engine_scan_files_1), (sdb_engine_init), (sdb_engine_unlink_shared_files), (sdb_engine_finalize), (symbol_db_engine_add_new_workspace), (symbol_db_engine_add_new_project), (sdb_engine_add_new_file), (sdb_engine_update_file), (on_scan_update_files_symbols_end), (symbol_db_engine_get_sym_type_conversion_hash), (symbol_db_engine_update_files_symbols), (symbol_db_engine_update_project_symbols), (symbol_db_engine_update_buffer_symbols), (symbol_db_engine_get_class_parents), (symbol_db_engine_get_global_members_filtered), (symbol_db_engine_get_scope_members_by_symbol_id_filtered), (symbol_db_engine_get_scope_members_by_symbol_id), (symbol_db_engine_get_scope_members), (symbol_db_engine_get_current_scope), (symbol_db_engine_get_file_symbols), (symbol_db_engine_get_symbol_info_by_id), (symbol_db_engine_find_symbol_by_name_pattern), (symbol_db_engine_get_parent_scope_id_by_symbol_id): * plugins/symbol-db/symbol-db-engine.h: * plugins/symbol-db/symbol-db-view-locals.c: (sdb_view_locals_get_iter_from_row_ref), (symbol_db_view_locals_clear_cache), (do_add_child_symbol_to_view), (traverse_on_scan_end), (on_scan_end), (on_symbol_removed), (on_symbol_inserted), (symbol_db_view_locals_update_list): Changes on some queries fuctions and some fixes. Added *_filtered functions. * plugins/symbol-db/symbol-db-view.c: (do_add_child_symbol_to_view), (add_new_waiting_for), (prepare_for_adding), (on_symbol_inserted), (do_recurse_subtree_and_remove), (on_symbol_removed), (sdb_view_do_add_hidden_dummy_child), (sdb_view_namespace_row_expanded), (sdb_view_global_row_expanded), (sdb_view_vars_row_expanded), (symbol_db_view_row_expanded), (sdb_view_locals_create_new_store), (sdb_view_init), (sdb_view_finalize), (sdb_view_class_init), (symbol_db_view_get_type), (sdb_view_load_symbol_pixbufs), (symbol_db_view_get_pixbuf), (sdb_view_build_and_display_base_tree), (symbol_db_view_open): New display for global tags. This is the definitive version, bugs apart. * plugins/symbol-db/tables.sql: fixed typo.
Hi, (In reply to comment #68) Forgot to mention that in the patch are included also some new images to put into symbol-browser/images directory. Another thing about global tags: to parse the .pc files I suggest to throw away the create_global_tags.sh&co which seem something hackish. As a replacement I think it's ok to use the sources of pkg-config itself. They're in C with GLib and I already have a test program working. On Slackware I have pkg-config 0.21. They are 4 files at most without any other dependence. If it's ok I'll implement this feature: * another SymbolDBEngine will be created, with it's own AnjutaLauncher. Once I know the paths of the library you want to scan I'll recursively scan all the *.c *.h inside there and populate a new database. A new database-project will be created for every package, with its files/symbols under it. In this way it'll become easy to add/remove new packages. thanks and regards, Massimo
This version is really about 90% of what we want. What remains is the ctags bug mentioned before which will hopefully be fixed soon. TODO: - Remove all local stuff from global tabs (no big deal) - Rethink interfaces with the editor to allow things like object->member and local variable completion. Implement this for symbol-db and add stubs to symbol-browser (return a empty list or simply NULL) to keep the other code working. Also needs some work in the language-support-c-cpp-java plugin. Commited the last patch! Thanks!
Created attachment 101766 [details] Symbol-db v 0.9.1 Hi, this patch implements the first TODO of comment #70. Now it filters out the static functions from the global-tab. Inside the patch there's a txt ChangeLog too. Reporting it here for completeness. regards, Massimo 2007-12-28 Massimo Cora' <maxcvs@email.it> * plugins/symbol-db/plugin.c: (on_importing_project_end), (symbol_db_activate), (symbol_db_deactivate), (symbol_db_instance_init), (isymbol_manager_get_parents): * plugins/symbol-db/plugin.h: * plugins/symbol-db/symbol-db-engine-iterator-node.c: * plugins/symbol-db/symbol-db-engine.c: (symbol_db_engine_get_global_members_filtered): * plugins/symbol-db/symbol-db-view.c: (prepare_for_adding), (on_symbol_inserted): Filtered out static functions on global-tab. E.g. when flag is_file_scope = 1 we won't add entries there. Fixed a little bug on 'Global'->'Vars/Others' node too.
Thanks for the last patch - it's committed! Some more comments: * Local tab is really looking like I expected now - great! * Global tab: - Opening the nodes is too slow to do this in a sync way. We definitly need to avoid UI freezes here and display a "Loading..." information. Maybe we can split it up to a call that waits for the database (thread?) and do the UI update afterwards in Gtk+. See the file-manager for an example how that could be handled. - The classes/functions need to be sorted alphabetically, can be done by a GtkTreeModelSort (good test case for many classes is importing glom) * Autocompletion: - Seems like the matching is not correctly because it shows lots of functions for "if" and "for" which is of course totally useless.
Found another small problem: When importing a project and exiting anjuta before symbol db is completely populated it does not continue population on next anjuta start. As population can take some time, it should keep track of the files already imported and start at the next file when not all files are in the database yet.
Created attachment 102274 [details] Symbol-db v 0.9.2 Hi, this patch fixes the freezing expanding nodes of the global tab. The method used is g_idle_add with G_PRIORITY_LOW. It's used on namespaces, Global, Vars/Others. Structs/Classes etc are instead pre-computed and then expanded: usually they have less entries and so are quicker. regards, Massimo
Applied this patch! Looks good, thanks! And thanks for the ChangeLog file! Would be nice if the generation of the file tree (when a new files is opened) could also be done async because it sometimes takes 3-4 seconds. I will rather report idividual bugs for now as it makes things easier to track.
(In reply to comment #75) > Would be nice if the generation of the file tree (when a new files is opened) > could also be done async because it sometimes takes 3-4 seconds. Sure. I already planned to do that and it's in my todo things. Fixing this should also fix bug #506831. > > I will rather report idividual bugs for now as it makes things easier to track. > Ok that's fine. Should then I post the patches here or on the opened bugs? I think that before 1.0 version it's good to maintain the fixes here, and reference these on the bugs. regards, Massimo
Hi, does anyone use isymbol_manager_get_completions_at_position () anywhere? Or related? I've marked it as deprecated. regards, Massimo
No, it sucked and crashed. We disabled it long time ago. Feel free to remove it. Instead we need something like, get_completion_at_scope(). Maybe best is to get the variable type from the symbol name in scope and than check with the type which -> or . completions are possible. But feel free to create an interface that fits good with symbol-db. symbol-browser will always return NULL (or equivalent) for those.
(In reply to comment #78) > Instead we need something like, get_completion_at_scope(). Maybe best is to get > the variable type from the symbol name in scope and than check with the type > which -> or . completions are possible. well, the logic of get_completions () and similar shouldn't be done on symbol_manager interface. Functions like those should be implemented in language-support-* plugins, or we risk to commit the same error as in symbol-browser where we give support only for c/c++/java. Instead get_scope_members (), get_scope_parent () etc. can be part of the interface, as they are language-independent functions. thanks and regards, Massimo
Created attachment 103284 [details] Symbol-db 1.0 beta1 Hi, ok here it the patch that implements the new interface IAnjutaSymbolManager. Of course more functions can be added. If fixes also bug #507072. thanks and regards, Massimo here it is the ChangeLog (there's an included file too): 2008-01-20 Massimo Cora' <maxcvs@email.it> * libanjuta/interfaces/libanjuta.idl: * plugins/class-inheritance/class-inherit.c: (class_inheritance_show_dynamic_class_popup_menu), (cls_inherit_add_node), (cls_inherit_draw_expanded_node), (class_inheritance_update_graph): * plugins/language-support-cpp-java/cpp-java-assist.c: (create_completion), (cpp_java_assist_create_scope_completion_cache), (cpp_java_assist_create_word_completion_cache), (cpp_java_assist_show_calltip): * plugins/profiler/gprof-view.c: (gprof_view_show_symbol_in_editor): * plugins/symbol-browser/an_symbol.c: (anjuta_symbol_get_name), (isymbol_get_name), (isymbol_get_sym_type), (isymbol_get_args), (isymbol_get_extra_info_string), (isymbol_get_line), (isymbol_get_icon), (isymbol_iface_init): * plugins/symbol-browser/an_symbol.h: * plugins/symbol-browser/an_symbol_view.c: (anjuta_symbol_view_get_file_symbol_model): * plugins/symbol-browser/plugin.c: (isymbol_manager_search), (isymbol_manager_get_members), (isymbol_manager_get_class_parents), (isymbol_manager_get_completions_at_position), (isymbol_manager_iface_init): reimplemented IAnjutaSymbolManager new functions in plugins that used those. Class-inheritance is still under test, keeping an open eye on it for bugs. * plugins/symbol-db/plugin.c: (symbol_db_deactivate), (isymbol_manager_search), (isymbol_manager_get_members), (isymbol_manager_get_class_parents), (isymbol_manager_iface_init): * plugins/symbol-db/symbol-db-engine-iterator-node.c: (symbol_db_engine_iterator_node_new), (sdb_engine_iterator_node_instance_init), (sdb_engine_iterator_node_finalize), (symbol_db_engine_iterator_node_get_symbol_extra_string), (isymbol_get_name), (isymbol_get_args), (isymbol_get_extra_info_string), (isymbol_get_line), (isymbol_get_icon), (isymbol_get_sym_type), (isymbol_iface_init): * plugins/symbol-db/symbol-db-engine.c: (sdb_engine_execute_select_sql), (sdb_engine_timeout_trigger_signals), (sdb_engine_thread_monitor), (sdb_engine_scan_files_1), (sdb_engine_init), (sdb_engine_set_defaults_db_parameters), (symbol_db_engine_open_db), (symbol_db_engine_add_new_files), (symbol_db_engine_fill_type_array), (sdb_engine_prepare_symbol_info_sql), (symbol_db_engine_get_class_parents_by_symbol_id), (symbol_db_engine_get_class_parents), (symbol_db_engine_get_global_members_filtered), (symbol_db_engine_get_scope_members_by_symbol_id_filtered), (symbol_db_engine_get_scope_members_by_symbol_id), (symbol_db_engine_get_scope_members), (symbol_db_engine_get_file_symbols), (symbol_db_engine_get_symbol_info_by_id), (symbol_db_engine_find_symbol_by_name_pattern), (symbol_db_engine_get_parent_scope_id_by_symbol_id), (symbol_db_engine_find_symbol_by_name_pattern_filtered): * plugins/symbol-db/symbol-db-engine.h: * plugins/symbol-db/symbol-db-view.c: (sdb_view_row_expanded_idle_destroy), (sdb_view_namespace_row_expanded), (sdb_view_vars_row_expanded): Cleaned code, implemented new interface. Added anjuta_launcher_disable_password_check () to avoid strange effects on output parsing. * plugins/symbol-db/tables.sql: removed dead code. Initialize parameters on symbol-db-engine at connection establishment.
Created attachment 103915 [details] Symbol-db v 1.0 beta2 Hi all, this patch fixes bug #506831. It' a cumulative patch: it includes symbol-db-beta1 changes too. Changes for the new part are described below. We can achieve a faster loading of files at Anjuta launch time too, but we must have an interface that specifies how many files were saved on previous session. thanks and regards, Massimo 2008-01-28 Massimo Cora' <maxcvs@email.it> * plugins/symbol-db/symbol-db-view-locals.c: (sdb_view_locals_get_iter_from_row_ref), (sdb_view_locals_init), (do_add_root_symbol_to_view), (consume_symbols_inserted_queue_idle_destroy), (consume_symbols_inserted_queue_idle), (on_scan_end), (do_recurse_subtree_and_remove), (on_symbol_removed), (on_symbol_inserted), (symbol_db_view_locals_recv_signals_from_engine), (symbol_db_view_locals_update_list): * plugins/symbol-db/symbol-db-view.c: (sdb_view_get_iter_from_row_ref), (prepare_for_adding), (on_symbol_removed), (sdb_view_row_expanded_idle_destroy), (sdb_view_namespace_row_expanded), (sdb_view_global_row_expanded), (sdb_view_vars_row_expanded): added non-freezing updating to local tab symbols. Adding of symbols to the GtkTreeView is now done with an g_idle_add () function and G_PRIO RITY_LOW.
Created attachment 104745 [details] Symbol-db v 1.0 beta3 Hi, remarkable changes in this patch: - added a preferences page. - continue population after abort. - start tests for global tags using pkg-config sources. thanks and regards, Massimo 2008-02-08 Massimo Cora' <maxcvs@email.it> * plugins/symbol-browser/plugin.c: (isymbol_manager_get_completions_at_position): deprecated. * plugins/symbol-db/Makefile.am: * plugins/symbol-db/anjuta-symbol-db.glade: * plugins/symbol-db/anjuta-symbol-db.schemas: * plugins/symbol-db/plugin.c: (do_import_sources_after_abort), (do_import_sources), (project_root_added), (symbol_db_activate), (on_prefs_executable_changed), (on_gconf_notify_prefs), (ipreferences_merge), (ipreferences_unmerge), (ipreferences_iface_init): * plugins/symbol-db/plugin.h: added a preferences page. * plugins/symbol-db/symbol-db-engine.c: (sdb_engine_ctags_output_thread), (sdb_engine_scan_files_1), (symbol_db_engine_add_new_files), (symbol_db_engine_update_project_symbols), (symbol_db_engine_get_files_with_zero_symbols), (symbol_db_engine_find_symbol_by_name_pattern_filtered): * plugins/symbol-db/symbol-db-engine.h: added a method to continue population of a project after an abort [e.g. closing Anjuta before symbol-db has finished its tasks]. Fixed also a little bug in output parsing. * plugins/symbol-db/symbol-db-view.c: (prepare_for_adding), (sdb_view_row_expanded_idle), (sdb_view_build_and_display_base_tree): removed unuseful debug messages. * plugins/symbol-db/symbol-db.ui: removed unused file. * plugins/symbol-db/test/Makefile.am: * plugins/symbol-db/test/main.c: (get_parents), (main): * plugins/symbol-db/test/parse.c: (read_one_line), (trim_string), (trim_and_sub), (parse_name), (parse_version), (parse_description), (split_module_list), (parse_module_list), (parse_requires), (parse_requires_private), (parse_conflicts), (_do_parse_libs), (parse_libs), (parse_libs_private), (parse_cflags), (parse_url), (parse_line), (parse_package_file), (backticks), (try_command), (get_compat_package): * plugins/symbol-db/test/parse.h: * plugins/symbol-db/test/pkg.c: (debug_spew), (verbose_error), (add_search_dir), (add_search_dirs), (ends_in_dotpc), (name_ends_in_uninstalled), (scan_dir), (add_virtual_pkgconfig_package), (package_init), (file_readable), (internal_get_package), (get_package), (get_package_quiet), (string_list_strip_duplicates), (string_list_strip_duplicates_from_back), (string_list_to_string), (get_l_libs), (get_L_libs), (get_other_libs), (get_I_cflags), (get_other_cflags), (get_conflicts), (get_requires), (get_requires_private), (pathposcmp), (spew_package_list), (spew_string_list), (packages_sort_by_path_position), (fill_one_level), (recursive_fill_list), (fill_list_single_package), (fill_list), (compare_req_version_names), (compare_package_keys), (add_env_variable_to_list), (verify_package), (get_merged), (get_merged_from_back), (get_multi_merged), (get_multi_merged_from_back), (package_get_l_libs), (packages_get_l_libs), (package_get_L_libs), (packages_get_L_libs), (package_get_other_libs), (packages_get_other_libs), (packages_get_all_libs), (package_get_I_cflags), (packages_get_I_cflags), (package_get_other_cflags), (packages_get_other_cflags), (package_get_cflags), (packages_get_all_cflags), (define_global_variable), (package_get_var), (packages_get_var), (rpmvercmp), (compare_versions), (version_test), (comparison_to_str), (max_len_foreach), (packages_foreach), (print_package_list), (enable_private_libs), (disable_private_libs): * plugins/symbol-db/test/pkg.h: Added some files from pkg-config tools [slackware sources, but should be general]. Started tests for last part of the plugin: the global tags.
Hi all, I'm parsing the *.pc files. I'm thinking that most of them are broken, and so must be filtered out. let's see some examples (output from a sample program I wrote): ** Message: key : xcb-damage, value : /usr/lib/pkgconfig/xcb-damage.pc ** Message: I flags: -I/usr/include this is broken. To include the whole /usr/include is nonsense. ** Message: key : evolution-data-server-1.2, value : /usr/lib/pkgconfig/evolution-data-server-1.2.pc ** Message: I flags: this is broken too. It has no I flags. ** Message: key : gtkhtml-sharp-2.0, value : /usr/lib/pkgconfig/gtkhtml-sharp-2.0.pc ** Message: I flags: -I:/usr/lib/pkgconfig/../../share/gapi-2.0/gtkhtml-api.xml this one too. What's that .xml? It's not a standard include directory. These instead are ok. They have their own include directory with their .h files. ** Message: key : valgrind, value : /usr/lib/pkgconfig/valgrind.pc ** Message: I flags: -I/usr/include/valgrind ** Message: key : libstartup-notification-1.0, value : /usr/lib/pkgconfig/libstartup-notification-1.0.pc ** Message: I flags: -I/usr/include/startup-notification-1.0 My plan of a filter is this: accept only strictly standards like -I/usr/include/startup-notification-1.0, check for duplicates includes and in case let them out. A regex would be enought. On database side there will be a new 'project' for every package. In this last case the project's name will be "/usr/include/startup-notification-1.0". Every file inside that directory will be scanned by ctags (symbol-db-engine) and will be part of the project. All the available 'projects' (=packages) will be displayed in a preferences page and will be checkable in a way like symbol-browser does now. thanks and regards, Massimo
Hi Massimo! I think you shouldn't read the .pc files directly but use pkg-config command instead. So basically you need to parse the output of "pkg-config --cflags package". Parsing the .pc can be dangerous as the format could change and maintaining a second parser is generally a bad idea. The gnome-build library also uses the output of pkg-config. (It lists all packages using pkg-config --list-all)
Hi Johannes, I'm not directly reading .pc files, I'm using the functions inside pkg-config to read them. You just have to give a PKG_CONFIG_PATH and it'll parse them, filling a structure 'Package'. Personally I dislike the idea of using AnjutaLauncher and the pkg-config command as I saw that often it bails out with some 'non-standard' error messages like 'Failed to open '/usr/lib/pkgconfig/mozilla-nspr.pc': No such file or directory', 'Variable 'datarootdir' not defined in '/usr/lib/pkgconfig/gnome-mag-1.0.pc'' etc. Instead using the sources inside the pkc-config command you would just have a NULL Package, meaning that something has gone wrong. thanks and regards, Massimo
Do you mean you copied code from pkg-config? That's not a very good solution either. Should pkg-config report errors to stderr and the information to stdout? And, it should return an error code (!= 0) if it fails.
Sorry, that I don't have much time for testing in the moment :( Anyway, some thoughts about this /usr/include stuff. I think we should add some other package named "system" which includes those things like stdio.h. Same applies for C++ standard headers (<iostream>, etc.). And second, later it should also be possible to scan packages for other languages, such as python. I think they also use pkg-config but can't say for sure. Thanks for all your work!
(In reply to comment #86) > Do you mean you copied code from pkg-config? That's not a very good solution > either. > I took sources from pkg-config as they come from release 0.21. > Should pkg-config report errors to stderr and the information to stdout? And, > it should return an error code (!= 0) if it fails. > Yes errors are reported on stderr, so they can be filtered out. Anyway scanning stdin for pkg-config output [like we do with ctags] cannot be performant, even if in this case the data should be relatively little. Pkg-config has also a 'strange' output format, I hope they'll maintain compatibility in future versions or the scanning would become a mess. I'll implement anyway this output scanning. thanks and regards, Massimo
(In reply to comment #87) > Anyway, some thoughts about this /usr/include stuff. I think we should add some > other package named "system" which includes those things like stdio.h. Same > applies for C++ standard headers (<iostream>, etc.). Well, at least on my linux box, /usr/include is a total mess. It has from stdlib.h to mpeg4ip.h to tiff.h, which are totally different packages. It would be a really hard task to separate system packages I think. > > And second, later it should also be possible to scan packages for other > languages, such as python. I think they also use pkg-config but can't say for > sure. > I think they should, but I really don't know too. thanks and regards, Massimo
Created attachment 107810 [details] Symbol-db v 1.0 beta4 Hi, finally I had a working port to libgda v 4.x. We need now to test the timings. I added a preferences page too, with the possibility of add tags for installed libraries. It's not fully working anyway, because of the difficulty of determining what's really installed on a system. Many libraries have infact more than a dependence in common, and those should be let out or they'll be scanned more than once. @Johannes: may you please compare the 3.x against the 4.x? We really need something more quick for population. An installed-library to scan may have 500+ files, with dependences, and it'll take ages. If you still have the old patch for 3.x it would be the right moment to see if something may be improved in 4.x. thanks and regards, Massimo
Committed your last patch. Not much clue about the perfomance. The callgrind output seems to have improved, at least nothing that makes me worry anymore. But things are still to slow. Thansk!
OK, I will try to merge this back into trunk now (still disabled by default...). Some notes: * All symbol tabs (Local/Global/Search) should be grayed out while the initial file scanning happens. * Clicking the first time on the "Global" node freezes anjuta for a while * When you close anjuta and reopen it while scanning files, the scanning speeds up again - maybe a bug but could you please investiage this? Maybe this is a key to better performance! * The pixbufs for the global tab still do not work * Global symbols are broken and freeze if you activate anything When adding things to a GtkTreeView in an idle handler with PRIORITY_LOW I feel you should not only add one symbol (or one package for pkg-config stuff) per idle handler but maybe 10 or 20 to reduce the overhead caused by calling the idle handlers.
Created attachment 108947 [details] Symbol-db v 1.0 beta5 Hi all, with this patch GdaSet* is set globally so that it's not created/freed everytime a query needs it. regards, Massimo
Thanks a lot for this patch - it has 30% better performance than before and now libsqlite is the worst offender in callgrind which is obviously correct! I had to run down the thread timeout to 10 ms because population is so fast now. There is only one point left that Vivien mentioned. Using gda_holder_take_value instead of gda_holder_set_value/g_value_free. I doubt that it has a big impact but it would be nice to change this. Other than that, we should concentrate on usuability now: * Fix global tags - feature parity with symbol-browser * When database is in "second_pass" mode and ianjuta_symbol* is used, anjuta locks up because the mutex is locked. This has to be fixed, either don't allow any ianjuta_symbol* calls (return NULL) or unlock the mutex in between. (tried the latter but it causes sync problems. * Typing feels a bit slow, maybe we should add a timeout like in the symbol-db plugin to only show completion when the user types slow. And maybe we can speed up the symbol-look-up a bit. * The types problem which is bug 495843 is still open of course.
Created attachment 109926 [details] Symbol-db v 1.0 beta6 Hi, new dynamic query architecture. See changelog for details. It's necessary to recreate the .anjuta_sym_db.db because I changed a table on tables.sql. With this patch the lookups and queries on db would be faster once populated because of prepared-query usage. thanks and regards, Massimo 2008-04-25 Massimo Cora' <maxcvs@email.it> * plugins/symbol-db/symbol-db-engine-iterator-node.c (symbol_db_engine_iterator_node_get_symbol_extra_string): * plugins/symbol-db/symbol-db-engine.c (sdb_engine_get_statement_by_query_id), (gtree_compare_func), (sdb_engine_get_dyn_query_node_by_id), (sdb_engine_dyn_child_query_node_destroy), (sdb_engine_insert_dyn_query_node_by_id), (sdb_engine_free_cached_queries), (sdb_engine_free_cached_dynamic_queries), (sdb_engine_get_tuple_id_by_unique_name), (sdb_engine_get_tuple_id_by_unique_name2), (sdb_engine_get_tuple_id_by_unique_name3), (sdb_engine_populate_db_by_tags), (sdb_engine_timeout_trigger_signals), (sdb_engine_thread_monitor), (sdb_engine_finalize), (symbol_db_engine_add_new_workspace), (symbol_db_engine_add_new_project), (sdb_engine_add_new_language), (sdb_engine_add_new_file), (sdb_engine_add_new_sym_type), (sdb_engine_add_new_sym_kind), (sdb_engine_add_new_sym_access), (sdb_engine_add_new_sym_implementation), (sdb_engine_add_new_heritage), (sdb_engine_add_new_scope_definition), (sdb_engine_add_new_tmp_heritage_scope), (sdb_engine_second_pass_update_scope_1), (sdb_engine_second_pass_update_heritage), (sdb_engine_second_pass_do), (sdb_engine_add_new_symbol), (sdb_engine_detects_removed_ids), (sdb_engine_update_file), (on_scan_update_files_symbols_end), (symbol_db_engine_update_project_symbols), (symbol_db_engine_get_full_local_path), (symbol_db_engine_get_file_db_path), (sdb_engine_walk_down_scope_path), (symbol_db_engine_get_files_with_zero_symbols), (sdb_engine_prepare_symbol_info_sql), (symbol_db_engine_get_class_parents_by_symbol_id), (symbol_db_engine_get_class_parents), (symbol_db_engine_get_global_members_filtered), (symbol_db_engine_get_scope_members_by_symbol_id_filtered), (symbol_db_engine_get_scope_members_by_symbol_id), (symbol_db_engine_get_scope_members), (symbol_db_engine_get_current_scope), (symbol_db_engine_get_file_symbols), (symbol_db_engine_get_symbol_info_by_id), (symbol_db_engine_find_symbol_by_name_pattern), (symbol_db_engine_get_parent_scope_id_by_symbol_id), (symbol_db_engine_find_symbol_by_name_pattern_filtered): * plugins/symbol-db/symbol-db-engine.h: new dynamic prepared queries architecture. Now every query used in the engine has its own compiled query in libgda-sqlite provider. This for speed improvements, code cleaning and auto-escaping of string parameters. Lookup of dynamic queries takes care of sym_info parameters and of parameters passed to functions. The compiled GdaStatements are stored in an array of GTree (of GTree(s)). Some parameters, as the filter_kinds, are bounded at n = 5, to avoid a third level of indirection on the store-trees. * plugins/symbol-db/tables.sql: * plugins/symbol-db/test/Makefile.am: * plugins/symbol-db/test/main.c (get_parents), (get_current_scope), (main): re-enabled test.
Thanks! There are still some problems: * The second_pass() takes very very long for a big project and there is no project indication. This is even worse, as every lookup in the database (for example typing three characters) freezes the whole UI waiting for the mutex lock. * I had to restart anjuta to get the symbols displayed correctly. I seems the end of the second_pass() is not detected correctly. * The progress bar is not disappearing as it should when everything is finished * Autocompletion is broken and all shown tooltips are empty (e.g "some_func()")
Created attachment 110202 [details] Symbol-db v 1.0 beta7 Hi guys, ok we're having N betas but I think this is more an atomic patch than a beta++. I tried to fix the usability bug when it's doing second_pass_do (). Now there are more mutex_lock () entries, which may facilitate the main_thread to take the control on db. This function is called by the last thread in the pool, and that's a good thing. Passing the execution to the main_thread with G_PRIORITY_LOW should suffer of gui freezing because of high cpu requested to complete that task (I tried this some time ago). Better then to let the OS the decision to swap the cpu between the threads. Anyway here they are the changes: 2008-05-01 Massimo Cora' <maxcvs@email.it> * plugins/symbol-db/plugin.c (on_single_file_scan_end): set message 'Generating inheritances...' when doing sdb_engine_second_pass_do () * plugins/symbol-db/symbol-db-engine.c (sdb_engine_cache_lookup), (sdb_engine_init_caches), (sdb_engine_ctags_output_thread), (sdb_engine_thread_monitor), (sdb_engine_init), (sdb_engine_add_new_sym_kind), (sdb_engine_add_new_sym_access), (sdb_engine_add_new_sym_implementation), (sdb_engine_second_pass_update_scope_1), (sdb_engine_second_pass_update_scope), (sdb_engine_second_pass_update_heritage), (symbol_db_engine_update_project_symbols), (symbol_db_engine_get_files_with_zero_symbols), (symbol_db_engine_get_file_symbols): fixed a population bug introduced with using of caches. It broke the kind of symbol in some cases (e.g. returning of table_id = -1). Avoid lazy initialization and inlined the lookup functions to speed up the thing. Performances improvement in sdb_engine_second_pass_update_heritage () using a prepared query (libgda parser now rulez!). Some fixes to the lock ()/unlock () logic in this part should avoid gui freezing, at least here it does not freeze. * plugins/symbol-db/symbol-db-view-locals.c (symbol_db_view_locals_recv_signals_from_engine): * plugins/symbol-db/symbol-db-view.c (symbol_db_view_recv_signals_from_engine): greyed out trees while population is in place.
Hi Massimo, good job with symbol-db. As we agreed, I transfered 1000 USD to your paypal account as advance payment for the bounty to appreciate your work so far. The rest will be paid at task's completion. Please confirm the transfer by a comment here. Thanks a lot and keep it up.
Hi Naba, payment received, thanks. regards, Massimo (In reply to comment #98) > Hi Massimo, good job with symbol-db. As we agreed, I transfered 1000 USD to > your paypal account as advance payment for the bounty to appreciate your work > so far. The rest will be paid at task's completion. Please confirm the transfer > by a comment here. Thanks a lot and keep it up. >
Hi guys, I had a look at the above comments with 'bugs', and I think symbol-db as is now is free from them. I need to focus on what have to be done to finish the plugin. Surely global-tags must be improved. We can discuss the nodes which need to be implemented. When preferences page is opened I need to parse all the packages to kill the ones that haven't the cflags or the ones which are broken in some ways. IIRC Johannes told me about the possibilty of implement a threaded architecture here too, but probably the output-parsing is good enought. Let me know, thanks and regards, Massimo
Hi Massimo! Your are quite right that there are no major bugs left in the current code though one thing still bothers me: - Autocompletion is really slow and I think it's because the query for searching is not limited. If we have more than 50 results, we can stop searching because we will never show more than 50 completions (for the simple word-autocompletion). This is also the maximum possible in the preferences. Other than that, we need to concentrate on global tags now. I hope simply output parsing will be sufficient for the preferences. You can also have a look at the code I recently added to the symbol-browser to auto-load the project systems. Though, for symbol-db I would prefer if the currently loaded symbols were stored in the session instead of the preferences, because this is clearly a session information. And there should be a check button "Use project settings" that greys out the whole tree of possible symbols. When global symbols are working, we should consider to improve C++ support (using our old and/or the CodeLite code). Thanks, Johannes
Created attachment 110538 [details] Symbol-db v 1.0 beta8 Hi Johannes, here it is a patch that introduces limit and offset keywords to filtered_search. Let me know if it speeds up, or I'll add an index to the table, even if I think it should already be at the max speed. regards, Massimo 2008-05-07 Massimo Cora' <maxcvs@email.it> * libanjuta/interfaces/libanjuta.idl: * plugins/class-inheritance/class-inherit.c (class_inheritance_show_dynamic_class_popup_menu): * plugins/language-support-cpp-java/cpp-java-assist.c (cpp_java_assist_create_word_completion_cache), (cpp_java_assist_show_calltip): * plugins/profiler/gprof-view.c (gprof_view_show_symbol_in_editor): * plugins/symbol-db/plugin.c (isymbol_manager_search): * plugins/symbol-db/symbol-db-engine.c (on_scan_update_files_symbols_end), (symbol_db_engine_find_symbol_by_name_pattern_filtered): * plugins/symbol-db/symbol-db-engine.h: add limit/offset keywords to search query. Adjusted APIs. Fixed a minor bug with caches in the engine.
Hi Johannes, (In reply to comment #101) > Other than that, we need to concentrate on global tags now. I hope simply > output parsing will be sufficient for the preferences. You can also have a look > at the code I recently added to the symbol-browser to auto-load the project > systems. Though, for symbol-db I would prefer if the currently loaded symbols > were stored in the session instead of the preferences, because this is clearly > a session information. ok I'll have a look. Just a question: see for example the isymbol_manager_search () api. After having searched for the symbols in the current opened project, should I also search for the global tags, concatenate the two iterators and return? I think I have to add another parameter to the api, specifying whether to return also the global tags or not. [e.g. the gboolean global_search parameter searches only for extern/public symbols]. > > When global symbols are working, we should consider to improve C++ support > (using our old and/or the CodeLite code). sure, but I think this is out of the scope of symbol-db plugin. I'll help and support the implementation, because I'm really interested in the undestanding of the completion mechanisms, but I also think that after the global stuff has been fixed this bug may be considered closed, bugs apart of course. thanks and regards, Massimo
Hi! > here it is a patch that introduces limit and offset keywords to > filtered_search. > Let me know if it speeds up, or I'll add an index to the table, even if I think > it should already be at the max speed. > Thanks! Actually it does not give us much more speed. Seems like it especially slow when it finds nothing while it fast when there are results. Do you have a clue there? Johannes
Hi, (In reply to comment #104) > Hi! > > > here it is a patch that introduces limit and offset keywords to > > filtered_search. > > Let me know if it speeds up, or I'll add an index to the table, even if I think > > it should already be at the max speed. > > > > Thanks! Actually it does not give us much more speed. Seems like it especially > slow when it finds nothing while it fast when there are results. Do you have a > clue there? > it seems like that when it reaches the limit of the results it returns the data model before reaching the end of the table, otherwise it does a full scan of the symbols. In particular when queries are done with the 'LIKE' keyword [i.e. you look for a "foo_sym*"] the sqlite engine may ignore indexes such that the results are slower. If it's so much slow I'll try to see if it's possible to tune the 'LIKE' queries. regards, Massimo
Created attachment 111509 [details] Symbol-db v 1.0 beta9 Hi, now search will be done also on global tags. Next step will be to improve global-tags population. regards, Massimo Cora' 2008-05-25 Massimo Cora' <maxcvs@email.it> * libanjuta/interfaces/libanjuta.idl: * plugins/class-inheritance/class-inherit.c (class_inheritance_show_dynamic_class_popup_menu), (cls_inherit_add_node), (cls_inherit_draw_expanded_node), (class_inheritance_update_graph): * plugins/language-support-cpp-java/cpp-java-assist.c (create_completion), (cpp_java_assist_create_word_completion_cache), (cpp_java_assist_show_calltip): * plugins/profiler/gprof-view.c (gprof_view_show_symbol_in_editor): * plugins/symbol-browser/plugin.c (isymbol_manager_search): * plugins/symbol-db/plugin.c (isymbol_manager_search): * plugins/symbol-db/plugin.h: * plugins/symbol-db/symbol-db-engine.c (symbol_db_engine_find_symbol_by_name_pattern_filtered): * plugins/symbol-db/symbol-db-engine.h: * plugins/symbol-db/symbol-db-view.c (prepare_for_adding): now search for symbols can be done also in global tags. Added a new parameter to search function and adjusted dependencies on different plugins.
Hi Johannes, (In reply to comment #101) > Other than that, we need to concentrate on global tags now. I hope simply > output parsing will be sufficient for the preferences. You can also have a look > at the code I recently added to the symbol-browser to auto-load the project > systems. Though, for symbol-db I would prefer if the currently loaded symbols > were stored in the session instead of the preferences, because this is clearly > a session information. > > And there should be a check button "Use project settings" that greys out the > whole tree of possible symbols. I saw that on gnome-build you return a list of all the packages on system. While this may work in most cases, it's not always true with symbol-db and cflags options. Infact many symbols haven't a cflags parameter set, so that you cannot know the ones that can be populated in the global-db. Hence a scan in a way as I'm doing on symbol-db-prefs must be done: user will see only packages which give an effective population/completion. From what I see ianjuta_project_manager_get_packages () returns packages required by the current opened project: I can compare the ones returned here with the ones in the prefs, and check/uncheck grey/ungrey the right ones, also checking if those packages have already been scanned [populated]. Another option would be this: 1. add an interface method to retrieve all packages in the system [as already done in project->properties->packages->add package]. 2. propose to user all the packages, with or without cflags [eventually notify him that there'll be no symbols on that package]. 3. grey out the project-required ones. 4. on project loading check if we had already done a population of the required packages, if it's not the case populate them. 5. lazily add/remove new packages when user toggles the checkbox, maintaining always the scanned packages on db avoiding conflicts with different projects adn redundant repopulation. pro and cons of this method: 1. (pro) standardized way to access packages, avoiding to reinvent the wheel over and over. 2. (cons) add methods to retrieve infos from packages on gnome-build e.g. package_get_cflags beign warned that they must respect some standards [see on_cflags_output () function on symbol-db-prefs, where I'm filtering the correct ones]. 3. (cons) add an interface to project-manager. what do you say? thanks and regards, Massimo
Hi, Naba pointed out that while populating project 'anjuta' itself, the process takes up to 191 MB of memory. I had a run with valgrind (I'll have a deeper analysis anyway) and it didn't find any relevant memleak on the plugin. It's possible that I missed some little leak in the code, but not in the huge form of 191 MB for a population I hope! I already tried disabling the PRAGMA = memory given to sqlite, but the situation doesn't change. Probably it's libgda allocating some (too much) memory?. Just a supposition anyway. regards, Massimo
> I saw that on gnome-build you return a list of all the packages on system. > While this may work in most cases, it's not always true with symbol-db and > cflags options. Infact many symbols haven't a cflags parameter set, so that you > cannot know the ones that can be populated in the global-db. Hence a scan in a > way as I'm doing on symbol-db-prefs must be done: user will see only packages > which give an effective population/completion. > From what I see ianjuta_project_manager_get_packages () returns packages > required by the current opened project: I can compare the ones returned here > with the ones in the prefs, and check/uncheck grey/ungrey the right ones, also > checking if those packages have already been scanned [populated]. Think this option would be ok though it should be clear to the user that those have been activated automatically somehow. I don't like the idea to add unrelated code to gnome-build.
(In reply to comment #108) > Hi, > > Naba pointed out that while populating project 'anjuta' itself, the process > takes up to 191 MB of memory. > I had a run with valgrind (I'll have a deeper analysis anyway) and it didn't > find any relevant memleak on the plugin. It's possible that I missed some > little leak in the code, but not in the huge form of 191 MB for a population I > hope! > I already tried disabling the PRAGMA = memory given to sqlite, but the > situation doesn't change. Probably it's libgda allocating some (too much) > memory?. > Just a supposition anyway. libgda-4.0 has quite some memory leaks. If you watch the gnome-db mailing list you will see some patches on this topic. I guess this will get better.
Hi, I'm encountering a strange behaviour when closing a project [with symbol-browser plugin loaded]: when symbol-browser plugin is finalized, automatically symbol-db gets loaded producing some [obvious] critical-warnings. See the one below. I don't know from where the 'plugin->priv->activated' is coming, problably from document-manager but I'm not sure. Do you have any clue? thanks, Massimo ** Message: Profile descoped: project ** Message: Removing widget from hash ** Message: Widget removed from container ** Message: GTodoPlugin: Dectivating Tasks manager plugin ... ** Message: Removing widget from hash ** Message: Widget removed from container ** Message: Removing widget from hash ** Message: Widget removed from container ** Message: Finalizing symbolview widget ** Message: Destroying symbolsearch ** Message: Destroying symbolsearch ** Message: Finalizing symbolsearch widget ** Message: SymbolDB: ipreferences_iface_init ** Message: SymbolDBPlugin: Activating SymbolDBPlugin plugin ... ** Message: symbol_db_engine_open_db (): opening/connecting to database... [New Thread -1246196848 (LWP 5948)] [Thread -1246196848 (LWP 5948) exited] [New Thread -1254585456 (LWP 5949)] [Thread -1254585456 (LWP 5949) exited] ** Message: connected to database DB_DIR=/home/pescio/.anjuta;DB_NAME=.anjuta_sym_db ** Message: sdb_view_locals_init () ** Message: Profile scoped: user ** Message: User profile loaded; Restoring user session ** Message: session_loading started ** Message: Setting geometry: 1276x953+0+24 ** Message: session_loading finished libanjuta-CRITICAL **: anjuta_plugin_deactivate: assertion `plugin->priv->activated == TRUE' failed aborting... Program received signal SIGABRT, Aborted.
+ Trace 199974
Thread NaN (LWP 3450)
Sorry, I am currently a bit short in time to have a look at it myself but some tips: When a project is closed, all project plugins are unloaded and the default profile is loaded. This means that some plugins might get deactivated/reactivated and probably in a strange order. About the report: I guess on_close_project_idle() is called after the project-manager plugin was deactivated so a possible fix would be to save the id of the close_project_idle() call and force it when the plugin becomes deactivated because there is no reason to call this any later.
Hi, (In reply to comment #112) > When a project is closed, all project plugins are unloaded and the default > profile is loaded. This means that some plugins might get > deactivated/reactivated and probably in a strange order. > > About the report: I guess on_close_project_idle() is called after the > project-manager plugin was deactivated so a possible fix would be to save the > id of the close_project_idle() call and force it when the plugin becomes > deactivated because there is no reason to call this any later. > I think that the wrong thing is /* Self destruct */ anjuta_plugin_deactivate (ANJUTA_PLUGIN (plugin)); inside on_close_project_idle (). After your patch I get the following backtrace: Program received signal SIGABRT, Aborted.
+ Trace 200311
Thread NaN (LWP 26759)
but removing the call to anjuta_plugin_deactivate () doesn't give a critical warning anymore. Probably this forced deactivation is not necessary.
Created attachment 112668 [details] [review] proposed project-manager fix for critical-warning. this a proposed patch to fix the critical warning.
(In reply to comment #114) > Created an attachment (id=112668) [edit] > proposed project-manager fix for critical-warning. > > this a proposed patch to fix the critical warning. > That's really not right. The point of self destruct is not to leave project-manager plugin around after the project is closed. Perhaps there is a different way to fix it?
> That's really not right. The point of self destruct is not to leave > project-manager plugin around after the project is closed. Perhaps there is a > different way to fix it? > Sorry, I commited this on 2008-06-13! Feel free to revert/change this.
Created attachment 114025 [details] Symbol-db v 1.0 beta10 Hi, I added a new class, SymbolDBSystem, that'll manage system packages population. As written in the changelog, it has some problems with libgda: it crashes randomly, probably due to a wrong management of threads/data in the library. I'm sending this so that libgda maintainer can test himself the bug and hopefully provide a patch! thanks and regards, Massimo 2008-07-05 Massimo Cora' <maxcvs@email.it> * libanjuta/anjuta-utils.c (anjuta_util_parse_args_from_string): fixed a little mem-leak. * plugins/symbol-db/Makefile.am: * plugins/symbol-db/anjuta-symbol-db.glade: * plugins/symbol-db/plugin.c (on_editor_update_ui), (on_char_added), (on_project_element_added), (on_project_element_removed), (on_system_scan_package_start), (on_system_scan_package_end), (on_system_single_file_scan_end), (on_project_single_file_scan_end), (on_importing_project_end), (do_import_sources_after_abort), (do_import_sources), (on_project_root_added), (on_project_root_removed), (symbol_db_activate), (symbol_db_deactivate), (symbol_db_instance_init), (isymbol_manager_search): * plugins/symbol-db/plugin.h: * plugins/symbol-db/symbol-db-engine.c (sdb_engine_disconnect_from_db), (sdb_engine_populate_db_by_tags), (sdb_engine_ctags_output_thread), (sdb_engine_scan_files_1), (symbol_db_engine_new), (sdb_engine_create_db_tables), (symbol_db_engine_db_exists), (symbol_db_engine_file_exists), (symbol_db_engine_project_exists), (symbol_db_engine_add_new_project), (sdb_engine_add_new_file), (symbol_db_engine_add_new_files), (sdb_engine_add_new_sym_type), (sdb_engine_add_new_sym_kind), (sdb_engine_add_new_sym_access), (sdb_engine_add_new_sym_implementation), (sdb_engine_add_new_scope_definition), (sdb_engine_add_new_tmp_heritage_scope), (sdb_engine_add_new_symbol), (symbol_db_engine_update_project_symbols), (on_scan_update_buffer_end), (symbol_db_engine_update_buffer_symbols), (symbol_db_engine_get_full_local_path), (symbol_db_engine_get_file_db_path), (symbol_db_engine_get_files_with_zero_symbols), (symbol_db_engine_get_file_symbols), (symbol_db_engine_find_symbol_by_name_pattern_filtered): * plugins/symbol-db/symbol-db-engine.h: * plugins/symbol-db/symbol-db-prefs.c (destroy_parseable_data), (on_listall_output), (on_listall_exit), (on_tag_load_toggled_parseable_cb), (on_tag_load_toggled), (symbol_db_prefs_init), (symbol_db_prefs_finalize): * plugins/symbol-db/symbol-db-prefs.h: * plugins/symbol-db/symbol-db-system.c (destroy_single_scan_data), (destroy_engine_scan_data), (sdb_system_init), (sdb_system_finalize), (sdb_system_class_init), (sdb_system_get_normalized_cflags), (on_engine_package_single_file_scan_end), (symbol_db_system_new), (symbol_db_system_is_package_parsed), (on_pkg_config_output), (sdb_system_files_visit_dir), (prepare_files_to_be_scanned), (on_engine_package_scan_end), (sdb_system_do_scan_package_1), (sdb_system_do_scan_next_package), (sdb_system_do_scan_new_package), (on_pkg_config_exit), (symbol_db_system_scan_package), (symbol_db_system_is_package_parseable): * plugins/symbol-db/symbol-db-system.h: * plugins/symbol-db/symbol-db-view.h: * plugins/symbol-db/test/Makefile.am: * plugins/symbol-db/test/benchmark.c (on_scan_end), (main): * plugins/symbol-db/test/main.c (add_new_files): brand-new system tags population system. It's still not completed. It crashes with libgda svn 3174: probably there's some thread bug on this library and must be fixed. The crash happens when system tags and project tags are scanned concurrently. It's a random crash, so it's not possible to find a point. Preferences page now support check-box toggle population. Anyway it's still missing a save-on-session method... to be implemented soon. * plugins/valgrind/preferences.c (build_general_prefs): be sure to set the correct executable path.
Created attachment 114086 [details] Symbol-db v 1.0 beta11 Hi, here it is the patch that fixes the concurrent population. @jhs: on svn head I don't find symbol-db-system.[c|h]. thanks and regards, Massimo 2008-07-07 Massimo Cora' <maxcvs@email.it> * plugins/language-support-cpp-java/cpp-java-assist.c (cpp_java_assist_create_word_completion_cache): added some debugging info. * plugins/symbol-db/plugin.c (on_project_root_added), (isymbol_manager_search): * plugins/symbol-db/symbol-db-engine.c (sdb_engine_get_statement_by_query_id), (sdb_engine_get_dyn_query_node_by_id), (sdb_engine_insert_dyn_query_node_by_id), (sdb_engine_get_query_parameters_list), (sdb_engine_free_cached_queries), (sdb_engine_free_cached_dynamic_queries), (sdb_engine_get_tuple_id_by_unique_name), (sdb_engine_get_tuple_id_by_unique_name2), (sdb_engine_get_tuple_id_by_unique_name3), (sdb_engine_init), (sdb_engine_finalize), (symbol_db_engine_new), (symbol_db_engine_find_symbol_by_name_pattern_filtered): * plugins/symbol-db/symbol-db-engine.h: * plugins/symbol-db/symbol-db-prefs.c (symbol_db_prefs_init): * plugins/symbol-db/symbol-db-system.c (destroy_single_scan_data), (destroy_engine_scan_data), (sdb_system_init), (sdb_system_finalize), (sdb_system_class_init), (sdb_system_get_normalized_cflags), (on_engine_package_single_file_scan_end), (symbol_db_system_new), (symbol_db_system_is_package_parsed), (on_pkg_config_output), (sdb_system_files_visit_dir), (prepare_files_to_be_scanned), (on_engine_package_scan_end), (sdb_system_do_scan_package_1), (sdb_system_do_scan_next_package), (sdb_system_do_scan_new_package), (on_pkg_config_exit), (symbol_db_system_scan_package), (symbol_db_system_is_package_parseable): * plugins/symbol-db/symbol-db-system.h: fixed threaded libgda stuff. It was the static prepared statement that broke the thing.
Could you attach a diff against trunk again. This patch fails horribly here so I don't know the exact cause (maybe because I added the two files in the meantime). Thanks a lot!
Created attachment 114191 [details] [review] Symbol-db v 1.0 beta12 Sure, here it is.
Thanks!
OK, as usual some random comments! * The database seems to work stable and reasonable fast know - congrats to that! Some problems I encountered: * When you should down anjuta before global symbol population has finished it seems to break things. * I fixed the prefs kind of but we have to do a better error checking when ctags is not installed (or the path is incorrect) and give the user advice how he can fix it. * I don't really get why there are over 500 files scanned for gtksourcview-2.0 when it only constist of 20 header files. Thanks and regards, Johannes
(In reply to comment #122) > > Some problems I encountered: > * When you should down anjuta before global symbol population has finished it > seems to break things. you're right. That happens probably because there's not an 'resume_from_abort' option for globals tags.. I had to implement that. > * I fixed the prefs kind of but we have to do a better error checking when > ctags is not installed (or the path is incorrect) and give the user advice how > he can fix it. I had a quick look at the code but I don't like the idea of passing an AnjutaPlugin instance to the engine just to retrieve the prefs string. Probably using a set () method would be the best choice. For instance an Anjuta-external benchmark program would not run anymore without an AnjutaPlugin. Anyway thanks for the idea, I'll see later how to improve it. > * I don't really get why there are over 500 files scanned for gtksourcview-2.0 > when it only constist of 20 header files. well... because of this: $ pkg-config --cflags gtksourceview-2.0 -I/usr/include/gtksourceview-2.0 -I/usr/include/gtk-2.0 -I/usr/include/libxml2 -I/usr/lib/gtk-2.0/include -I/usr/include/atk-1.0 -I/usr/include/cairo -I/usr/include/pango-1.0 -I/usr/include/glib-2.0 -I/usr/lib/glib-2.0/include -I/usr/include/freetype2 -I/usr/include/libpng12 -I/usr/include/pixman-1 I had to scan for all the dependencies. Of course when I pass the files to ctags I have to check that they don't already exist on db. thanks and regards, Massimo
> you're right. That happens probably because there's not an 'resume_from_abort' > option for globals tags.. I had to implement that. Good! It seems to break the database a bit otherwise! > I had a quick look at the code but I don't like the idea of passing an > AnjutaPlugin instance to the engine just to retrieve the prefs string. Probably > using a set () method would be the best choice. For instance an Anjuta-external > benchmark program would not run anymore without an AnjutaPlugin. > Anyway thanks for the idea, I'll see later how to improve it. well, you can also just do a g_object_set_data (sdb_engine, "__prefs", prefs) if you want. > > * I don't really get why there are over 500 files scanned for gtksourcview-2.0 > > when it only constist of 20 header files. > > well... because of this: > > $ pkg-config --cflags gtksourceview-2.0 > -I/usr/include/gtksourceview-2.0 -I/usr/include/gtk-2.0 -I/usr/include/libxml2 > -I/usr/lib/gtk-2.0/include -I/usr/include/atk-1.0 -I/usr/include/cairo > -I/usr/include/pango-1.0 -I/usr/include/glib-2.0 -I/usr/lib/glib-2.0/include > -I/usr/include/freetype2 -I/usr/include/libpng12 -I/usr/include/pixman-1 > > I had to scan for all the dependencies. Of course when I pass the files to > ctags I have to check that they don't already exist on db. > Do you check this already? Because gtk+-2.0 was already scanned for example. Thanks!
(In reply to comment #124) > > you're right. That happens probably because there's not an 'resume_from_abort' > > option for globals tags.. I had to implement that. > > Good! It seems to break the database a bit otherwise! > yes you're right. > > well, you can also just do a g_object_set_data (sdb_engine, "__prefs", prefs) > if you want. > I had a ready patch which is able to switch the ctags executable on the fly. I didn't send it because I found a crasher. I'll send it soon anyway. > > > Do you check this already? Because gtk+-2.0 was already scanned for example. > yes, but probably the count of the files to scan is wrong: in this number are included also the files already scanned, even if they won't be parsed. thansk and regards, Massimo
> > > > I had a ready patch which is able to switch the ctags executable on the fly. I > didn't send it because I found a crasher. I'll send it soon anyway. Cool! Anyway, my problem was more that the schema was not installed correctly but that's now fixed. (glade2schema.pl script...) > yes, but probably the count of the files to scan is wrong: in this number are > included also the files already scanned, even if they won't be parsed. I see! I looked at the member completion code (moved out of symbol-browser into language-support-cpp-java) and we definitly need a scope() interface for IAnjutaSymbol when I understood the code correctly. Greetings from Istanbul, Johannes
Created attachment 114340 [details] Symbol-db v 1.0 beta13 Hi, a patch for ctags checking. I hope this is working for you too. Let me know. 2008-07-10 Massimo Cora' <maxcvs@email.it> * libanjuta/anjuta-utils.c (anjuta_util_prog_is_installed): * libanjuta/anjuta-utils.h: fixed definition const *gchar. * plugins/symbol-db/anjuta-symbol-db.glade: * plugins/symbol-db/plugin.c (symbol_db_activate), (symbol_db_deactivate): * plugins/symbol-db/plugin.h: * plugins/symbol-db/symbol-db-engine.c (sdb_engine_get_dyn_query_node_by_id), (sdb_engine_insert_dyn_query_node_by_id), (sdb_engined_ctags_launcher_create), (sdb_engine_scan_files_1), (sdb_engine_init), (symbol_db_engine_set_ctags_path), (symbol_db_engine_new), (sdb_engine_prepare_symbol_info_sql): * plugins/symbol-db/symbol-db-engine.h: * plugins/symbol-db/symbol-db-prefs.c (on_prefs_executable_changed), (symbol_db_prefs_init): * plugins/symbol-db/symbol-db-view.c (sdb_view_namespace_row_expanded), (sdb_view_global_row_expanded): The engine will now check for a working (existing) ctags executable. If not found a message will be displayed. It's now possible to switch ctags executable on the fly. Fixed a crasher with dynamic queries (missing initialization).
Hi Johannes, (In reply to comment #126) > > I looked at the member completion code (moved out of symbol-browser into > language-support-cpp-java) and we definitly need a scope() interface for > IAnjutaSymbol when I understood the code correctly. > Probably what you are looking for is symbol_db_engine_get_current_scope (), symbol_db_engine_get_scope_members (), or symbol_db_engine_get_parent_scope_id_by_symbol_id () functions. Oh well, you meant 'interface'. Yes, we definely need that. Just try to write a summary of the problems you encounter with your implementations, then we'll be able to write down the interface. > Greetings from Istanbul, > wh0a! How lucky you are! Greetings from Italy :) Massimo
I have found a rather series problem occuring after the last patches. When the population is running anjuta renders more or less unusable because of the high load (other applications still run ok). Is it possible that you do some more stuff in process that was done in a thread before? Or any other idea? Thanks, Johannes
With 'population' do you mean global-tags population? If global-tags is run concurrently with project's population then we have one more thread than a normal situation, and obviously the load grows up. When global-tags population is running the first step is passing the files to scan to ctags, and probably this is what you are seeing: some seconds of freeze because of the high cpu required by ctags. A package may be composed by thousand of files, e.g. I saw something like 1100-1500 files. That means doubling the size of an 'Anjuta' project [~800 files]. I already put an 'implementation-fix' for this global-population high load: we'll scan a package per time, not all the required at once. I mean, suppose we have package x y z to scan, these are the steps: 1. get the cflags for x, y, z. 2. loop 3. visit the directories looking for interested files of x 4. send them to ctags 5. populate. 6. when, and only when step 5 is concluded loop to step 2 and process y, then z etc.. This is why I asked you some time ago to 'pre-prepare' .db packages for some common libraries, e.g. glib, gtk+ etc. So that the population could have been quickier. Anyway I didn't touch the engine process of population, but just the preferences. thanks, Massimo
(In reply to comment #130) > With 'population' do you mean global-tags population? No, it happens with every populatin. > because of the high cpu required by ctags. A package may be composed by > thousand of files, e.g. I saw something like 1100-1500 files. That means > doubling the size of an 'Anjuta' project [~800 files]. The rest of GNOME is still responsive so it can't be the ctags cpu load. > I already put an 'implementation-fix' for this global-population high load: > we'll scan a package per time, not all the required at once. > I mean, suppose we have package x y z to scan, these are the steps: > 1. get the cflags for x, y, z. > 2. loop > 3. visit the directories looking for interested files of x > 4. send them to ctags > 5. populate. > 6. when, and only when step 5 is concluded loop to step 2 and process y, then Sounds good! > This is why I asked you some time ago to 'pre-prepare' .db packages for some > common libraries, e.g. glib, gtk+ etc. So that the population could have been > quickier. I don't think it's a good idea. We will have to update those all the time and people should always have symbols that fit with their installed libraries. > Anyway I didn't touch the engine process of population, but just the > preferences. I didn't talk about your last patch (which crashed X for me when I changed the prefs last time) but maybe the change with the static queries could have caused the slowdown. Thanks! > > > thanks, > Massimo >
(In reply to comment #131) > > > Anyway I didn't touch the engine process of population, but just the > > preferences. > > I didn't talk about your last patch (which crashed X for me when I changed the > prefs last time) but maybe the change with the static queries could have caused > the slowdown. crashed X?! I cannot understand how X can crash with an user-level program... Can you reproduce it? Probably with gdb running? I'm not noticing any slowdowns here, nor X crashes. The change with the static query shouldn't have affected anything, as I just ported those initialization into an allocated structure which is now attached to a SymbolDBEngine... thanks, Massimo
Created attachment 114543 [details] Symbol-db v 1.0 rc1 Hi, ok guys. I think I've got it. A feature complete release. It's obvioulsy not bug-free but we can test it hard now. Now system packages are saved within the session. Plus a rewrite of SymbolDBPrefs object has been done. In the next patch I'll do some slight modifications to symbol-db-engine to manage the search in only some packages [i.e. now even if we toggle packages the global search is always done db-wide]; plus I'll add some interfaces. Last patch wasn't committed, but this one contains also those modifications. The changelog instead is only for this one. thanks and regards, Massimo 2008-07-14 Massimo Cora' <maxcvs@email.it> * plugins/symbol-db/plugin.c (on_editor_destroy), (on_session_save), (on_session_load), (on_project_element_removed), (do_import_system_src_after_abort), (do_import_project_src_after_abort), (do_import_sources), (on_project_root_added), (symbol_db_activate), (symbol_db_deactivate), (isymbol_manager_search), (g_list_compare), (on_prefs_package_add), (on_prefs_package_remove), (ipreferences_merge), (ipreferences_unmerge): * plugins/symbol-db/plugin.h: * plugins/symbol-db/symbol-db-engine.c (sdb_engine_get_dyn_query_node_by_id), (sdb_engine_insert_dyn_query_node_by_id), (sdb_engined_ctags_launcher_create), (sdb_engine_scan_files_1), (sdb_engine_init), (sdb_engine_finalize), (symbol_db_engine_set_ctags_path), (symbol_db_engine_new), (symbol_db_engine_project_exists), (symbol_db_engine_add_new_files), (symbol_db_engine_get_files_with_zero_symbols), (sdb_engine_prepare_symbol_info_sql): * plugins/symbol-db/symbol-db-engine.h: * plugins/symbol-db/symbol-db-prefs.c (on_prefs_executable_changed), (on_listall_output), (on_listall_exit), (on_tag_load_toggled_parseable_cb), (on_tag_load_toggled), (sdb_prefs_init1), (sdb_prefs_init), (sdb_prefs_finalize), (sdb_prefs_class_init), (symbol_db_prefs_new): * plugins/symbol-db/symbol-db-prefs.h: Rewritten symbol-db-prefs.[c|h]. SymbolDBPrefs is now an object. It's more usable/maintainable. Session packages are now saved and reloaded at session-start time. Code cleaning. * plugins/symbol-db/symbol-db-system.c (destroy_engine_scan_data), (sdb_system_do_engine_scan), (on_engine_package_scan_end), (sdb_system_do_scan_package_1), (on_pkg_config_exit), (symbol_db_system_scan_package), (symbol_db_parse_aborted_package): * plugins/symbol-db/symbol-db-system.h: * plugins/symbol-db/symbol-db-view.c (sdb_view_namespace_row_expanded), (sdb_view_global_row_expanded): Added 'continue global tags scan after abort' feature. Code cleaning.
(In reply to comment #129) > I have found a rather series problem occuring after the last patches. When the > population is running anjuta renders more or less unusable because of the high > load (other applications still run ok). Is it possible that you do some more > stuff in process that was done in a thread before? Or any other idea? > probably it's sdb_engine_second_pass_update_scope () function, which is quite slow. In my global db for example I have 34k symbols to process, and this function does a for loop. For every loop I do some necessary operations. I hope to find an hack to improve this speed.
Created attachment 114807 [details] Symbol-db v 1.0 rc2 Hi, some new interfaces implemented. This patch should make not crash Anjuta on bug #543637. 2008-07-19 Massimo Cora' <maxcvs@email.it> * libanjuta/interfaces/libanjuta.idl: * plugins/symbol-browser/plugin.c (project_root_added), (on_editor_saved), (update_editor_symbol_model), (value_removed_current_editor), (isymbol_manager_get_scope), (isymbol_manager_get_parent_scope), (isymbol_manager_get_symbol_more_info), (isymbol_manager_iface_init): * plugins/symbol-db/plugin.c (do_import_system_src_after_abort), (do_import_sources), (isymbol_manager_search), (isymbol_manager_get_scope), (isymbol_manager_get_parent_scope), (isymbol_manager_get_symbol_more_info), (isymbol_manager_iface_init): * plugins/symbol-db/symbol-db-engine-iterator-node.c (symbol_db_engine_iterator_node_get_symbol_extra_string): * plugins/symbol-db/symbol-db-engine.c (sdb_engine_init), (symbol_db_engine_add_new_files), (sdb_engine_second_pass_update_scope_1), (sdb_engine_second_pass_update_scope), (sdb_engine_second_pass_update_heritage), (sdb_engine_second_pass_do), (sdb_engine_prepare_symbol_info_sql), (symbol_db_engine_get_class_parents_by_symbol_id), (symbol_db_engine_get_current_scope), (symbol_db_engine_find_symbol_by_name_pattern_filtered): * plugins/symbol-db/symbol-db-engine.h: * plugins/symbol-db/symbol-db-system.c (symbol_db_system_parse_aborted_package): * plugins/symbol-db/symbol-db-system.h: * plugins/symbol-db/symbol-db-view-locals.c: * plugins/symbol-db/symbol-db-view.c: * plugins/symbol-db/symbol-db-view.h: * plugins/symbol-db/test/benchmark.c (main): temporary disabled do_import_system_src_after_abort (). It needs more performance tests. Implemented some new methods for IAnjutaSymbolManager. More performances on population and displaying.
Created attachment 115732 [details] Symbol-db v 1.0 rc3 Hi, this is a bugfix patch. It solves some critical warnings, a couple of crashers, some displaying bugs. I probably found the bug that led to a huge memory leak. I reported it here bug #545977 and marked as a dependence of this bug #358479. I temporary disabled the check of mutex in the engine. Probably we have reached an unpredictable point with last libgda and sqlite releases: they support access to db from multiple threads! It seemed to me that disabling symbol-db-engine mutexes brought to a improved speed in population. Some fixes on dynamic queries in the engine permit now to have no more problems with the limit of symbol-kinds or session packages to filter. Now the search on global-tags is done following the enabled packages in the preferences. Try it yourself: enable glib-2.0 for instance, then swich to the editor and see the completion appearing. Disable glib-2.0 and retry: no more completion appears. What remains to do now? test, test and again test. We need to tune this plugin for a stable release. regards, Massimo 2008-08-02 Massimo Cora' <maxcvs@email.it> * plugins/symbol-db/plugin.c (on_system_scan_package_start), (do_import_system_src_after_abort), (on_project_root_added), (symbol_db_activate), (symbol_db_deactivate), (isymbol_manager_search): disconnected signals and re-enabled do_import_system_src_after_abort (). * plugins/symbol-db/symbol-db-engine.c (sdb_engine_get_dyn_query_node_by_id), (sdb_engine_insert_dyn_query_node_by_id), (sdb_engine_free_cached_queries), (sdb_engine_free_cached_dynamic_queries), (sdb_engine_disconnect_from_db), (sdb_engine_init), (symbol_db_engine_new), (symbol_db_engine_open_db), (symbol_db_engine_add_new_files), (symbol_db_engine_get_global_members_filtered), (symbol_db_engine_get_scope_members_by_symbol_id_filtered), (symbol_db_engine_get_file_symbols), (symbol_db_engine_get_parent_scope_id_by_symbol_id), (symbol_db_engine_find_symbol_by_name_pattern_filtered): test: removed mutex (actually set it to NULL). Libgda rev 3186 have sqlite 3.6.0beta compiled inside, which has thread support. More: GdaLockable interface has been written and implemented for GdaConnection and GdaSqlParser. Actually symbol-db-engine works without mutexes, but before removing all the code associated this need some more wide tests. Completed implementation of searching symbols through session-enabled packages. A new feature on dynamic queries permits to filter up to 255 symbol kinds and up to 255 session packages. Was used a internal integer mapping. * plugins/symbol-db/symbol-db-engine.h: * plugins/symbol-db/symbol-db-system.c (on_engine_package_single_file_scan_end), (sdb_system_finalize), (sdb_system_do_engine_scan), (on_pkg_config_exit): * plugins/symbol-db/symbol-db-view-locals.c (sdb_view_locals_create_new_store): * plugins/symbol-db/symbol-db-view-locals.h: * plugins/symbol-db/symbol-db-view-search.c (sdb_view_search_dispose): disconnected signals
Created attachment 116739 [details] Symbol-db v 1.0 rc4 Hi guys, another must-have patch for symbol-db. Many bugfixes, a preferences addon and some new interfaces in this patch. 2008-08-16 Massimo Cora' <maxcvs@email.it> * libanjuta/anjuta-launcher.c (anjuta_launcher_send_ptyin): fixed a crasher, or at least added a warning. * libanjuta/interfaces/libanjuta.idl: * plugins/symbol-browser/an_symbol.c (anjuta_symbol_set_tag), (isymbol_get_id), (isymbol_iface_init): * plugins/symbol-browser/an_symbol_iter.c (isymbol_iter_first), (isymbol_iter_next), (isymbol_iter_previous), (isymbol_iter_foreach), (isymbol_iter_set_position), (isymbol_iter_get_position), (isymbol_iter_get_length): * plugins/symbol-browser/plugin.c (isymbol_manager_get_members), (isymbol_manager_get_class_parents), (isymbol_manager_get_parent_scope), (isymbol_manager_get_symbol_more_info), (isymbol_manager_get_symbol_by_id), (isymbol_manager_iface_init): * plugins/symbol-db/plugin.c (on_session_load), (on_importing_project_end), (on_project_root_added), (isymbol_manager_search), (isymbol_manager_get_members), (isymbol_manager_get_class_parents), (isymbol_manager_get_parent_scope), (isymbol_manager_get_symbol_more_info), (isymbol_manager_get_symbol_by_id), (isymbol_manager_iface_init): * plugins/symbol-db/symbol-db-engine-iterator-node.c (isymbol_get_id), (isymbol_iface_init): * plugins/symbol-db/symbol-db-engine.c (sdb_engine_clear_caches), (sdb_engine_get_statement_by_query_id), (sdb_engine_get_tuple_id_by_unique_name4), (sdb_engine_udpated_scope_gtree_populate), (sdb_engine_populate_db_by_tags), (sdb_engine_ctags_output_thread), (sdb_engine_timeout_trigger_signals), (sdb_engine_thread_monitor), (sdb_engine_ctags_output_callback_1), (sdb_engine_scan_files_1), (sdb_engine_init), (sdb_engine_finalize), (symbol_db_engine_new), (sdb_engine_connect_to_db), (sdb_engine_add_new_sym_type), (sdb_engine_add_new_symbol), (on_scan_update_buffer_end), (symbol_db_engine_update_buffer_symbols), (symbol_db_engine_get_class_parents_by_symbol_id), (symbol_db_engine_get_scope_members_by_symbol_id), (symbol_db_engine_find_symbol_by_name_pattern_filtered): * plugins/symbol-db/symbol-db-engine.h: many bugfixes, for instance on search queries. Added some new interface methods to let the use of IAnjutaSymbolManager easier. * plugins/symbol-db/anjuta-symbol-db.glade: * plugins/symbol-db/symbol-db-prefs.c (on_prefs_executable_changed), (on_check_button_toggled), (sdb_prefs_init1): added an option on preferences to let user the choice of let autopopulation work or not on project opening. * plugins/symbol-db/symbol-db-prefs.h: * plugins/symbol-db/symbol-db-view-locals.c (sdb_view_locals_get_iter_from_row_ref), (symbol_db_view_locals_clear_cache), (sdb_view_locals_init), (sdb_view_locals_finalize), (traverse_on_scan_end), (trigger_on_symbol_inserted), (prepare_for_adding), (consume_symbols_inserted_queue_idle_destroy), (consume_symbols_inserted_queue_idle), (on_scan_end), (on_symbol_scope_updated), (symbol_db_view_locals_recv_signals_from_engine), (symbol_db_view_locals_update_list): * plugins/symbol-db/tables.sql: corrected trigger and added a new index. Fixed a bug that broke locals view when two structs had the same member name inside.
I forgot to mention that due to some changes in tables.sql you should delete your .anjuta_sym_db.db files.
The last patch is commited now. Anyway, I still think it's a bad idea to do project population and global population at the same time because the system load is too high. I don't think your bugfix in anjuta-launcher.c is correct because the bug seems to be caused by a memory corruption in symbol-db and I still get it (because launcher is not NULL but some invalid memory). Another thing I noticed is that the population stalls at some files for a very long time. I don't think that this is caused by the file size because it happens in global population which only scans header files that shouldn't be too large in size. Further I think that the number of total files is sometimes computed incorrectly and the plugin waits for more files to be scanned even if all have already been scanned.
(In reply to comment #139) > The last patch is commited now. Anyway, I still think it's a bad idea to do > project population and global population at the same time because the system > load is too high. Ok, probably there's an easy solution without removing the code written: add a preference checkbox [defaulting to false] where the user can specify whether he wants the population in parallel or not. > > I don't think your bugfix in anjuta-launcher.c is correct because the bug seems > to be caused by a memory corruption in symbol-db and I still get it (because > launcher is not NULL but some invalid memory). hmm yes I've to investigate it deeper. Anyway IMHO it's a good thing to always check the values coming from external functions, while internal (static) functions' values should be trusted: this can avoid nasty bugs that can go deep in functions that work good but that do not check the input values. > > Another thing I noticed is that the population stalls at some files for a very > long time. I don't think that this is caused by the file size because it > happens in global population which only scans header files that shouldn't be > too large in size. Further I think that the number of total files is sometimes > computed incorrectly and the plugin waits for more files to be scanned even if > all have already been scanned. > ok this is another thing to fix. It would be a good thing to know which files are slower than others. thanks and regards, Massimo
Created attachment 117268 [details] Symbol-db v 1.0 rc5 Hi, this patch fixes two bugs for comment #139. I cannot reproduce the long-time required for some files so it's status is still unknown. 2008-08-23 Massimo Cora' <maxcvs@email.it> * plugins/symbol-browser/plugin.c (on_editor_destroy), (deactivate_plugin): fix #522032 * plugins/symbol-db/anjuta-symbol-db.glade: added two checkboxes: one is for parallel scan control, the other is for the buffer updating control. * plugins/symbol-db/plugin.c (on_editor_buffer_symbols_update_timeout), (on_editor_destroy), (on_editor_update_ui), (on_char_added), (on_editor_saved), (value_added_current_editor), (on_session_load), (value_removed_current_editor), (on_importing_project_end), (do_import_system_src_after_abort), (do_import_project_src_after_abort), (do_import_project_sources), (on_received_project_import_end), (on_project_root_added), (on_project_root_removed), (symbol_db_activate), (symbol_db_deactivate), (symbol_db_class_init), (on_prefs_buffer_update_toggled), (ipreferences_merge): * plugins/symbol-db/plugin.h: * plugins/symbol-db/symbol-db-engine-iterator-node.c: * plugins/symbol-db/symbol-db-engine-iterator-node.h: * plugins/symbol-db/symbol-db-engine-iterator.c: * plugins/symbol-db/symbol-db-engine-iterator.h: * plugins/symbol-db/symbol-db-engine.c (sdb_engine_populate_db_by_tags), (sdb_engine_ctags_output_thread), (sdb_engine_thread_monitor), (sdb_engine_ctags_output_callback_1), (on_scan_files_end_1), (sdb_engine_init), (sdb_engine_unref_removed_launchers), (sdb_engine_terminate_threads), (sdb_engine_finalize), (symbol_db_engine_set_ctags_path), (symbol_db_engine_close_db): * plugins/symbol-db/symbol-db-engine.h: fix #548606 now engine exits cleanly even if it's in scanning mode. It still remains the libgda bug anyway, bug #545979. Fixed bug #522032 also for symbol-db. * plugins/symbol-db/symbol-db-prefs.c (on_update_button_toggled), (sdb_prefs_init1), (sdb_prefs_class_init): * plugins/symbol-db/symbol-db-prefs.h: * plugins/symbol-db/symbol-db-view-locals.c (sdb_view_locals_init), (do_add_root_symbol_to_view), (do_add_child_symbol_to_view), (on_scan_end), (on_symbol_removed), (on_symbol_scope_updated), (on_symbol_inserted), (symbol_db_view_locals_display_nothing), (symbol_db_view_locals_update_list): * plugins/symbol-db/symbol-db-view-locals.h: * plugins/symbol-db/symbol-db-view-search.c: * plugins/symbol-db/symbol-db-view-search.h: * plugins/symbol-db/symbol-db-view.c: * plugins/symbol-db/symbol-db-view.h: headers update.
Thanks for the last patch, it seems to make everything a lot more stable. One other thing I noticed: We a file is added to the project manually (e.g. by editing the Makefile.am) symbol-db does not seem to detect/scan it.
(In reply to comment #142) > Thanks for the last patch, it seems to make everything a lot more stable. > > One other thing I noticed: We a file is added to the project manually (e.g. by > editing the Makefile.am) symbol-db does not seem to detect/scan it. > well, signal 'element-added' is not raised from project-manager. On plugin.c I already listen for it g_signal_connect (G_OBJECT (pm), "element_added", G_CALLBACK (on_project_element_added), sdb_plugin); Should project manager be fixed in that way?
Created attachment 117653 [details] Symbol-db v 1.0 rc6 Hi, a little patch that fixes some minor bugs. thanks and regards, Massimo 2008-08-30 Massimo Cora' <maxcvs@email.it> * plugins/symbol-db/anjuta-symbol-db.glade: fix bug #549518 * plugins/symbol-db/symbol-db-engine.c (sdb_engine_free_cached_queries), (sdb_engine_get_file_defined_id), (sdb_engine_finalize), (symbol_db_engine_new), (symbol_db_engine_file_exists), (symbol_db_engine_project_exists), (symbol_db_engine_add_new_project), (sdb_engine_add_new_language), (sdb_engine_add_new_file), (sdb_engine_add_new_sym_type), (sdb_engine_add_new_sym_kind), (sdb_engine_add_new_sym_access), (sdb_engine_add_new_sym_implementation), (sdb_engine_add_new_scope_definition), (sdb_engine_second_pass_update_scope_1), (sdb_engine_second_pass_update_heritage), (sdb_engine_add_new_symbol), (symbol_db_engine_update_project_symbols): * plugins/symbol-db/symbol-db-prefs.c (on_autoscan_button_toggled), (on_parallel_button_toggled), (sdb_prefs_init1): fixed on_parallel signal and preferences saving. used g_value_set_static_string () which is quicker than its brother g_value_set_string ().
The project-manager can't know if the Makefile.am has been changed (when anjuta is not running). So the symbol-db plugin should at least check on start-up if new source files have been added to the project.
(In reply to comment #145) > The project-manager can't know if the Makefile.am has been changed (when anjuta > is not running). So the symbol-db plugin should at least check on start-up if > new source files have been added to the project. > I'll activate that function after the release because there's some test needed.
Created attachment 118624 [details] Symbol-db v. 1.0 Hi, this is a release-ready patch. Now symbol-db works with latest svn-3204 libgda version. 2008-09-12 Massimo Cora' <maxcvs@email.it> * plugins/symbol-browser/an_symbol_iter.c (isymbol_iter_next): removed some debug messages. * plugins/symbol-db/anjuta-symbol-db.glade: converted the GtkFileChooser into GtkComboBoxEntry because of bug #551384. * plugins/symbol-db/plugin.c (g_list_compare), (on_session_load), (do_check_offline_files_changed), (on_project_root_added): * plugins/symbol-db/symbol-db-engine-iterator-node.c (symbol_db_engine_iterator_node_set_data): * plugins/symbol-db/symbol-db-engine-iterator.c (symbol_db_engine_iterator_new): * plugins/symbol-db/symbol-db-engine.c (sdb_engine_cache_lookup), (sdb_engine_insert_dyn_query_node_by_id), (sdb_engine_get_tuple_id_by_unique_name), (sdb_engine_get_tuple_id_by_unique_name2), (sdb_engine_get_tuple_id_by_unique_name3), (sdb_engine_get_tuple_id_by_unique_name4), (sdb_engine_ctags_output_callback_1), (sdb_engine_init), (symbol_db_engine_add_new_workspace), (symbol_db_engine_add_new_project), (sdb_engine_add_new_language), (sdb_engine_add_new_file), (sdb_engine_add_new_sym_type), (sdb_engine_add_new_sym_kind), (sdb_engine_add_new_sym_access), (sdb_engine_add_new_sym_implementation), (sdb_engine_add_new_heritage), (sdb_engine_add_new_scope_definition), (sdb_engine_add_new_tmp_heritage_scope), (sdb_engine_second_pass_update_scope_1), (sdb_engine_second_pass_update_scope), (sdb_engine_second_pass_update_heritage), (sdb_engine_add_new_symbol), (sdb_engine_detects_removed_ids), (sdb_engine_update_file), (on_scan_update_files_symbols_end), (symbol_db_engine_update_project_symbols), (symbol_db_engine_remove_file), (symbol_db_engine_remove_files), (sdb_engine_walk_down_scope_path), (symbol_db_engine_get_files_with_zero_symbols), (sdb_engine_prepare_file_info_sql), (symbol_db_engine_get_class_parents_by_symbol_id), (symbol_db_engine_get_class_parents), (symbol_db_engine_get_global_members_filtered), (symbol_db_engine_get_scope_members_by_symbol_id_filtered), (symbol_db_engine_get_scope_members_by_symbol_id), (symbol_db_engine_get_scope_members), (symbol_db_engine_get_current_scope), (symbol_db_engine_get_file_symbols), (symbol_db_engine_get_symbol_info_by_id), (symbol_db_engine_find_symbol_by_name_pattern), (symbol_db_engine_get_parent_scope_id_by_symbol_id), (symbol_db_engine_find_symbol_by_name_pattern_filtered), (symbol_db_engine_get_files_for_project): * plugins/symbol-db/symbol-db-engine.h: * plugins/symbol-db/symbol-db-prefs.c (on_prefs_executable_changed), (sdb_prefs_init1): * plugins/symbol-db/symbol-db-prefs.h: * plugins/symbol-db/symbol-db-view-locals.c (symbol_db_view_locals_update_list): * plugins/symbol-db/symbol-db-view.c (symbol_db_view_get_pixbuf): ported to libgda-svn 3204. Still some bugs due to libgda remain but they are being squashed.
Hi, I've just done my first commit on svn. I attach here the ChangeLog only. I was wondering if this bounty bug should remain opened for further comments or can be closed? thanks and regards, Massimo 2008-10-12 Massimo Cora' <mcora@svn.gnome.org> * plugins/symbol-db/symbol-db-engine-iterator.c (symbol_db_engine_iterator_first), (symbol_db_engine_iterator_last), (symbol_db_engine_iterator_set_position): * plugins/symbol-db/symbol-db-engine.c (sdb_engine_finalize), (sdb_engine_second_pass_update_scope_1), (sdb_engine_second_pass_update_scope), (sdb_engine_second_pass_update_heritage), (symbol_db_engine_update_project_symbols), (symbol_db_engine_get_files_with_zero_symbols): support libgda svn rev #3238, aka libgda-3.99.6.
Hmm, to me it's seem much much slower than before: Time per symbol: 0.1 seconds (increasing as more symbols are parsed). I remember that we had this problem before and solved it by using fewer transactions. Otherwise, you can close this bug and discuss things on anjuta-devel list if you like.
(In reply to comment #149) > Hmm, to me it's seem much much slower than before: > > Time per symbol: 0.1 seconds (increasing as more symbols are parsed). I > remember that we had this problem before and solved it by using fewer > transactions. > hmm that's really strange because I just used the new functions available on libgda apis... I didn't add any new logic. I confirm the slowness here too, and I suspect some problems again on libgda... > Otherwise, you can close this bug and discuss things on anjuta-devel list if > you like. > well, I'll leave this open then, and I'll close it when new release will be due.
(In reply to comment #149) > Hmm, to me it's seem much much slower than before: > > Time per symbol: 0.1 seconds (increasing as more symbols are parsed). I > remember that we had this problem before and solved it by using fewer > transactions. > > Otherwise, you can close this bug and discuss things on anjuta-devel list if > you like. > have a look at bug #556327
(In reply to comment #140) > > Another thing I noticed is that the population stalls at some files for a very > > long time. I don't think that this is caused by the file size because it > > happens in global population which only scans header files that shouldn't be > > too large in size. Further I think that the number of total files is sometimes > > computed incorrectly and the plugin waits for more files to be scanned even if > > all have already been scanned. > > > > ok this is another thing to fix. It would be a good thing to know which files > are slower than others. OK, I found out that the time simply increases linear with the number of symbol processes. The worst files have about 1000 symbols and take about 8 seconds which is too much. We are currently at about 0.01 s/symbol and to be really useful we have to get this down to 0.001-0.003 s/symbol.
> OK, I found out that the time simply increases linear with the number of symbol > processes. The worst files have about 1000 symbols and take about 8 seconds > which is too much. > they're too much if user enables real-time scan, yes. > We are currently at about 0.01 s/symbol and to be really useful we have to get > this down to 0.001-0.003 s/symbol. > I really *hope* to push down some millis with a memory pool system, but I also think that we cannot go after a specific bound, due to the architecture itself. I would have preferred infact to have a sort of an in-memory ctags system. We could have avoided then the flushing on file, the parsing and then re-read into structs by ctags library, even if this ctags is fast. Anyway we got this. I was wondering if we could push down the pool-time of anjuta_launcher to scan the output?
> I was wondering if we could push down the pool-time of anjuta_launcher to scan > the output? Don't know what you mean exactly but I see no problem in adding a getter/setter to adjust this time for certain launchers and leave the default.
Partially ported to gio. I need some functions from glib 2.18 and I'll use them as soon as gnome 2.24 enters in debian sid. I think it's ok to depend on glib 2.18, right? 2008-10-25 Massimo Cora' <mcora@svn.gnome.org> * plugins/symbol-db/plugin.c (on_editor_buffer_symbols_update_timeout), (on_project_element_added), (on_project_element_removed), (do_import_system_src_after_abort), (do_import_project_src_after_abort), (do_import_project_sources), (on_project_root_added): * plugins/symbol-db/symbol-db-engine-iterator-node.c: * plugins/symbol-db/symbol-db-engine.c (symbol_db_engine_update_project_symbols): * plugins/symbol-db/symbol-db-engine.h: ported to GIO. The only file that remains out is symbol-db-system.c: it'll fixed as soon as my distrib'll give out gnome 2.24 (glib 2.18)
This patch depends on the patch attached to bug #559183, which needs to be applied to libgda head before running. It shouldn't take that much to be committed on libgda module though. 2008-11-03 Massimo Cora' <mcora@svn.gnome.org> * plugins/symbol-db/symbol-db-engine.c (sdb_engine_get_file_defined_id), (sdb_engine_populate_db_by_tags), (sdb_engine_init), (symbol_db_engine_file_exists), (symbol_db_engine_project_exists), (symbol_db_engine_add_new_project), (sdb_engine_add_new_language), (sdb_engine_add_new_file), (symbol_db_engine_add_new_files), (sdb_engine_add_new_sym_type), (sdb_engine_add_new_sym_kind), (sdb_engine_add_new_sym_access), (sdb_engine_add_new_sym_implementation), (sdb_engine_add_new_heritage), (sdb_engine_add_new_scope_definition), (sdb_engine_add_new_tmp_heritage_scope), (sdb_engine_second_pass_update_scope_1), (sdb_engine_second_pass_update_heritage), (sdb_engine_add_new_symbol), (symbol_db_engine_update_project_symbols), (sdb_engine_walk_down_scope_path), (symbol_db_engine_get_class_parents_by_symbol_id), (symbol_db_engine_get_class_parents), (symbol_db_engine_get_global_members_filtered), (symbol_db_engine_get_scope_members_by_symbol_id_filtered), (symbol_db_engine_get_scope_members_by_symbol_id), (symbol_db_engine_get_scope_members), (symbol_db_engine_get_current_scope), (symbol_db_engine_get_file_symbols), (symbol_db_engine_get_symbol_info_by_id), (symbol_db_engine_find_symbol_by_name_pattern), (symbol_db_engine_get_parent_scope_id_by_symbol_id), (symbol_db_engine_find_symbol_by_name_pattern_filtered), (symbol_db_engine_get_files_for_project): Memory pool object system. Now the hundreds of GValue used in the population/queries are reused instead of being created/destroyed. This gains 5-10 ms in some cases but there's still some profiling work to do. * plugins/symbol-db/symbol-db-view-search.c (sdb_view_search_model_filter): fixed bug #558930.
(In reply to comment #154) > > I was wondering if we could push down the pool-time of anjuta_launcher to scan > > the output? > > Don't know what you mean exactly but I see no problem in adding a getter/setter > to adjust this time for certain launchers and leave the default. > I started some perform tests. First of all I used the performance-scritps that should output a png: http://www.gnome.org/~federico/hacks/index.html#performance-scripts but I had no luck because the resulting image was very large when loaded into memory [1.4 GB], and was of no help :/ The I tried with gprof. I used the gprof plugin of anjuta, after having compiled the files with -pg of course. I must say it wasn't of so much help too. 1. There's lack of documentation on how to use that plugin and its data. 2. the data itself wasn't coherent between a launch and another, i.e. I still have to understand where the flow could be improved. I think I'm gonna try with setter/getter into annjuta-launcher, to see if something can change to have a greater performance. BTW, after the mem pool system I can reach 10ms per symbol on average case with the test benchmark program.
gprof seems to be quite tricky to use - I haven't been able to figure out how it works myself. I personally use valgrind/callgrind to track performance which seems to work better!
Created attachment 122378 [details] callgrind output data I ran some sessions with valgrind and this is its output. I think its pretty but also it needs some deep analysis. btw, I don't see any evident bottle neck. What do you say?
Yes, there is no real bottleneck anymore - the trace looks far better than older version: 15% sqlite - this is were the real works happens -> OK 18% glib - these are things like memory allocations -> OK 27% gobject - I think this is the point were we can probably optimize a bit. The worst problems are the g_value_* methods. Maybe we can avoid some more g_value_copy() somewhere.
(In reply to comment #160) > 27% gobject - I think this is the point were we can probably optimize a bit. > The worst problems are the g_value_* methods. Maybe we can avoid some more > g_value_copy() somewhere. > I think it could be in the internals of libgda. On the symbol-db engine I removed most of the 'copy' actions about the GValues.
With this patch I think there's no left feature to add. We should now ask people for huge debug of symbol-db. 2008-11-25 Massimo Cora' <mcora@svn.gnome.org> * plugins/symbol-db/plugin.c (do_add_new_files), (on_project_element_added), (do_import_project_src_after_abort), (do_import_project_sources), (do_check_offline_files_changed), (on_project_root_added): * plugins/symbol-db/symbol-db-engine.c (symbol_db_engine_add_new_files), (symbol_db_engine_remove_file): code cleaning. Added check for offline Makefile.am changes. Fixed bug #558171.
1. Trying to use anjuta project to create an db data base file and it crashs all the time. 2. Don't have short cut goto tag definition and goto tag declaration, like the way did in symbol browser.
Do you work with latest trunk? There was a very nasty crasher fixed today and I imported the whole anjuta project (about 900) files without any problem here.
(In reply to comment #163) > 1. Trying to use anjuta project to create an db data base file and it crashs > all the time. > > 2. Don't have short cut goto tag definition and goto tag declaration, like the > way did in symbol browser. > please check also to have ctags "Exuberant Ctags 5.7, Copyright (C) 1996-2007 Darren Hiebert" installed.
No.1 is OK now.
2008-12-20 Massimo Cora' <mcora@svn.gnome.org> * plugins/symbol-db/symbol-db-engine-priv.h: I forgot to add this file. 2008-12-20 Massimo Cora' <mcora@svn.gnome.org> * plugins/symbol-db/Makefile.am: * plugins/symbol-db/plugin.c (symbol_db_activate), (on_prefs_package_remove): * plugins/symbol-db/plugin.h: * plugins/symbol-db/symbol-db-engine-core.c (sdb_engine_cache_lookup), (sdb_engine_insert_cache), (sdb_engine_clear_caches), (sdb_engine_init_caches), (sdb_engine_execute_unknown_sql), (sdb_engine_execute_select_sql), (sdb_engine_execute_non_select_sql), (sdb_engine_get_statement_by_query_id), (sdb_engine_get_dyn_query_node_by_id), (sdb_engine_dyn_child_query_node_destroy), (sdb_engine_insert_dyn_query_node_by_id), (sdb_engine_get_query_parameters_list), (sdb_engine_free_cached_queries), (sdb_engine_free_cached_dynamic_queries), (sdb_engine_disconnect_from_db), (sdb_engine_get_tuple_id_by_unique_name), (sdb_engine_get_tuple_id_by_unique_name2), (sdb_engine_get_tuple_id_by_unique_name3), (sdb_engine_get_tuple_id_by_unique_name4), (sdb_engine_get_file_defined_id), (sdb_engine_udpated_scope_gtree_populate), (sdb_engine_populate_db_by_tags), (sdb_engine_ctags_output_thread), (sdb_engine_timeout_trigger_signals), (sdb_engine_thread_monitor), (sdb_engine_ctags_output_callback_1), (on_scan_files_end_1), (sdb_engine_ctags_launcher_create), (sdb_engine_scan_files_1), (sdb_engine_init), (sdb_engine_unlink_shared_files), (sdb_engine_unref_removed_launchers), (sdb_engine_terminate_threads), (sdb_engine_finalize), (sdb_engine_class_init), (sdb_engine_get_type), (symbol_db_engine_set_ctags_path), (symbol_db_engine_new), (sdb_engine_set_defaults_db_parameters), (sdb_engine_connect_to_db), (sdb_engine_create_db_tables), (symbol_db_engine_db_exists), (symbol_db_engine_file_exists), (symbol_db_engine_close_db), (symbol_db_engine_open_db), (symbol_db_engine_add_new_workspace), (symbol_db_engine_project_exists), (symbol_db_engine_add_new_project), (sdb_engine_add_new_language), (sdb_engine_add_new_file), (sdb_engine_get_unique_scan_id), (symbol_db_engine_add_new_files), (sdb_engine_add_new_sym_type), (sdb_engine_add_new_sym_kind), (sdb_engine_add_new_sym_access), (sdb_engine_add_new_sym_implementation), (sdb_engine_add_new_heritage), (sdb_engine_add_new_scope_definition), (sdb_engine_add_new_tmp_heritage_scope), (sdb_engine_second_pass_update_scope_1), (sdb_engine_second_pass_update_scope), (sdb_engine_second_pass_update_heritage), (sdb_engine_second_pass_do), (sdb_engine_add_new_symbol), (sdb_engine_detects_removed_ids), (sdb_engine_update_file), (on_scan_update_files_symbols_end), (symbol_db_engine_get_sym_type_conversion_hash), (symbol_db_engine_fill_type_array), (symbol_db_engine_update_files_symbols), (symbol_db_engine_update_project_symbols), (symbol_db_engine_remove_file), (symbol_db_engine_remove_files), (on_scan_update_buffer_end), (symbol_db_engine_update_buffer_symbols): * plugins/symbol-db/symbol-db-engine-core.h: * plugins/symbol-db/symbol-db-engine-queries.c (sdb_engine_walk_down_scope_path), (sdb_engine_prepare_file_info_sql), (sdb_engine_prepare_symbol_info_sql), (symbol_db_engine_get_class_parents_by_symbol_id), (symbol_db_engine_get_class_parents), (symbol_db_engine_get_global_members_filtered), (symbol_db_engine_get_scope_members_by_symbol_id_filtered), (symbol_db_engine_get_scope_members_by_symbol_id), (symbol_db_engine_get_scope_members), (symbol_db_engine_get_current_scope), (symbol_db_engine_get_file_symbols), (symbol_db_engine_get_symbol_info_by_id), (symbol_db_engine_find_symbol_by_name_pattern), (symbol_db_engine_get_parent_scope_id_by_symbol_id), (symbol_db_engine_find_symbol_by_name_pattern_filtered), (symbol_db_engine_get_files_for_project): * plugins/symbol-db/symbol-db-engine-queries.h: * plugins/symbol-db/symbol-db-engine-utils.c (symbol_db_glist_compare_func), (symbol_db_gtree_compare_func), (symbol_db_engine_is_locked), (symbol_db_engine_get_full_local_path), (symbol_db_engine_get_file_db_path), (symbol_db_engine_get_files_with_zero_symbols): * plugins/symbol-db/symbol-db-engine-utils.h: * plugins/symbol-db/symbol-db-engine.c: * plugins/symbol-db/symbol-db-engine.h: * plugins/symbol-db/symbol-db-view-locals.c (symbol_db_view_locals_update_list): * plugins/symbol-db/symbol-db-view-search.c: * plugins/symbol-db/symbol-db-view-search.h: * plugins/symbol-db/symbol-db-view.c (on_scan_end), (sdb_view_init), (symbol_db_view_open): symbol-db-engine refactoring. Fixed #564048 - Split symbol-db-engine into several files. see comment http://bugzilla.gnome.org/show_bug.cgi?id=564048#c2 for more details about this split-patch.
Hmm, somehow I don't really get why it blocks UI so much. Which methods are really executed in the main thread? I only found sdb_engine_timeout_trigger_signals but it takes a maximum of 6ms so I guess that shouldn't be a problem.
(In reply to comment #168) > Hmm, somehow I don't really get why it blocks UI so much. Which methods are > really executed in the main thread? I only found > sdb_engine_timeout_trigger_signals but it takes a maximum of 6ms so I guess > that shouldn't be a problem. > yes I'm doing nothing special in main thread. I update the progress bar but not much more. Try to put the THREADS_MAX_CONCURRENT to 1. Here it seems more responsive. I fear that the worker threads eat all the cpu available in the processors, leaving nothing for gtk_main (). A way could be to renice the threads to a lower priority, but I don't know if such feature exists in glib.
Some stats for the project Anjuta: find out the number of items inside these tables: sqlite> select count(*) from symbol; 24184 sqlite> select count(*) from sym_type; 20574 sqlite> select count(*) from sym_kind; 13 sqlite> select count(*) from sym_access; 4 sqlite> select count(*) from sym_implementation; 2 sqlite> select count(*) from scope; 19287 we've correctly cached kind, access and implementations. We shouldn't cache sym_type because the probability of a new symbol is bigger than find an inserted one. That's because the sym_type is populated with a paradigm "insert first, select after if not found". Same for the scope.
On average I'm seeing a 13.7 ms population time per symbol. Test was executed on a machine with intel 1.8 GHz 32bit, 384 MB ram, debian sid. benchmark program was used.
Yeah. I was able to touch the 11.9 ms/symbol with the benchmark program. With Anjuta's gui I have no more freezes here. This seems quite fine and with rev 4503 I should be able to close at least two nasty bugs, #565773 and #565769. Btw: I removed some indexes, maybe the performances on searches are less impressive now, but just tell me if you see something weird that I try to re-enable them without affecting population time.
Hi Massimo, Now that Symbol DB is released with Anjuta, I think we can close the bounty. I will transfer the remaining bounty prize 500 USD to you. Please confirm it once received. Thanks and very good job. You may keep this bug open if you want, or close it and work with fresh bugs for other issues.
(In reply to comment #173) > Hi Massimo, Now that Symbol DB is released with Anjuta, I think we can close > the bounty. I will transfer the remaining bounty prize 500 USD to you. Please > confirm it once received. Thanks and very good job. You may keep this bug open > if you want, or close it and work with fresh bugs for other issues. > Hi Naba, thank you very much for the payment! I've received it. I think we can close this one because other specific issues are coming out and it's better to keep them separated. My plans are now to make symbol-db as stable as possible and later (for 2.28) to implement a completion-engine. thanks and regards, Massimo