GNOME Bugzilla – Bug 567135
Generate vim syntax highlighting files
Last modified: 2018-05-21 15:46:10 UTC
I've been maintaining vim syntax highlighting files for many Gnome libs http://www.vim.org/scripts/script.php?script_id=1000 for quite some time. However, this is something gtk-doc itself is best suited to generate. Namely because the distributed files do not seem to contain enough information, or do not contain it in a nice-enough form, so one has to rebuild the API documentation for each package to be able to generate the vim syntax file anyway. Here is the current script generating the vim syntax files: http://gwyddion.svn.sourceforge.net/viewvc/gwyddion/trunk/gwyddion/utils/vim-syn-gen.py?view=markup It is used together with a very simple configuration file that says where to find libfoo-decl-list.txt and defines some metadata for the output file: http://gwyddion.svn.sourceforge.net/viewvc/gwyddion/trunk/gwyddion/data/gwyddion-cfg.py?view=markup just python vim-syn-gen.py libfoo.py >libfoo.vim I do not submit a patch yet, because there are open questions: - should we do this at all - under what conditions (e.g., the package must explicitly request so...) - where to install the result, if anywhere, or just distribute it, or what ...
We could just ship this as a tool (under tools) and give an example for the makefile rule.
That would be nice. However, what I'm going to do in Gwyddion is to add the vim syntax generation rule to the end of scan-build. This (with some vim setup) ensures nicely highlighted source examples because the highlighter knows all the type, function, macro, constant, etc. names in the library when it's run. I wonder if something similar can be done for the other highlighters. At this moment this possibility makes vim clearly superior to the other two even though vim is used only as a fallback in configure (though a considerable part is lost in the default stylesheet because it does not highlight types -- probably due to the failure of the other two highlighters to actually highlight them) and cannot be explicitly selected as a configure option. source-highlight has language definition files with an include feature so it's possible to improve the highlighting by generating c-foo.lang files for libraries, distributing and using them. I haven't looked at highlight. So there's a clear advantage in generating syntax files as a part of the build process. Not just for vim.
Created attachment 150644 [details] syntax item extraction script
So a different approach is needed. The attached script takes FOO-decl.txt and it generates a list of symbols for syntax highlighting in a neutral and easy to consume form. Each line starts with the symbol kind and then all symbols of this kind follow. Deprecated symbols are listed separately. Unlike vim-syn-gen, it does not require any config file although you can have one to override some symbols kinds or other things. It should be inserted to the scan phase and the result should be distributed and installed (presumably somewhere to gtkdoc data dir). The next step is to get rid of the external highlighters because each produces different output which makes gtkdoc output styling unpredictable. Apparently, the current style is modelled for source-highlight catering to all its deficiencies such as the emission of useless "normal" class or failure to highlight types. The amount of code required to implement C source highlighting directly in gtkdoc-fixxref (using the generated neutral syntax files for FOO and other installed packages) will be smaller than or comparable to the code necessary to detect the highlighters, try to canonicalize their output and generate highlighter-specific syntax extensions. The output will be predictable and we will get rid of external dependencies. Vim syntax files, or other syntax files if anyone is interested, can be then generated rather trivially from the neutral syntax file -- and as it will be distributed, it will not be necessary to do this in gtk-doc.
Created attachment 150675 [details] syntax item extraction script New version of the syntax item extraction script with some vim-syn-gen cruft removed.
Created attachment 150676 [details] Perl module for C syntax highlighting The module is documented and it has a trivial interface: require "gtkdoc-highlight.pm"; import GtkDocHighlight; # Imports AddHighlightSymbols and HighlightC # Load symbol files for highlighting &AddHighlightSymbols("/paths/to/files/generated/by/gtkdoc-syntax", ...); # Highlight C code to HTML my $html_code = &HighlightC($c_code); It produces a bare HTML that should be put to the <pre>-programlisting with line numbers etc. Standard C keywords, types and constants are also in a symbol file, to be attached.
Created attachment 150677 [details] supplementary standard C symbol file
The next step is an integration patch, getting rid of the other highlighters and improving style.css. Nevertheless, some comments would be welcome first...
Created attachment 150678 [details] supplementary standard C symbol file
Created attachment 150755 [details] [review] integration patch The patch does not contain the new files because I cannot make git diff to do diff -N. Also, see the commented-out and commented linkification code in fixxref, marked with `(Yeti)'. It used to do strange things like attempting to linkify words of `normal' <span>s because syntax-highlight failed to recognize types (there is no `normal' <span> class any more, normal is simply normal). I'd personally prefer if there were no links in the code examples at all. And if there must be, that they would be as little visually disturbing as possible, ideally styled exactly as non-link things of the same kind.
Created attachment 150772 [details] syntax item extraction script gtkdoc-syntax rewritten in Perl The patch already uses this, I just forgot to attach it.
Regarding the linkification. The code should be passed to the highlighter untouched, with no links inserted. Prior linkification is IMO misguided because it means: - mixing HTML with C which is a sure way to confuse the C parser - the C highlighter then applies arbitrary <span>s to the resulting mess and you have to painfully correct it afterwards - if you give the highlighter nice C it will serve you symbols it in nice <span class="type">...</span> tags so you just need to take these and linkify them if you want links
Stefan, could you please comment on this? The patch is rather extensive so if it's going to be completely rejected I will at least waste no additional time keeping it applicable to HEAD.
Created attachment 150803 [details] [review] integration patch Small corrections.
Ping?
+1 for getting this into gtk-doc. Consistent results is an important end-user feature: confusing if I post a screenshot of a doc that used one formatter but a user generated the same doc from the same package but it looks "different". And doing as much as possible internally *especially* if it's no harder than doing it externally, makes my job as a package-manager developer easier: fewer dependencies. Especially source-highlight, which is claimed above to be the primary target but least standard and is also not a lightweight package on my system. Along those lines, if we do maintain support for multiple formatters, it would be nice if there were a configure flag that controlled which one(s) to support instead of uncontrollably picking first-available in a list of choices whose order I can't control.
Yeti, Sorry for the delay. I'll give this a go tomorrow - it sounds good after a quick glance. I had the vim highlighter as the last, as for me it was the slowest and produced less markup. One question - these *.syn files. Should those be installed in a way that vim finds them in general? Or are they just used for gtkdoc to add markup for types? patch contains a unchanged file -> a/tests/gobject/docs/tester.types (which we want to remove from git anyway).
(In reply to comment #17) > I'll give this a go tomorrow - it sounds good after > a quick glance. Note there are some unresolved likification issues (see 10 and 12), but once we agree what it should do, writing the code to do that will be easy. > One question - these *.syn files. Should those be installed in a way that vim > finds them in general? Or are they just used for gtkdoc to add markup for > types? The .syn file should be installed to the same directory as the HTML, index.sgml, devhelp files, etc., so that the gtk-doc built-in highlighter can find it. The modified gtk-doc.notmpl.make takes care of it and gtkdoc-fixxref already looks for it there. Regarding vim, I was thinking about directly reading .syn files in vim, but in the first step I will just modify vim-syn-gen to use them and generate normal Vim syntax files. The crucial difference is that if a package distributes the .syn file I no longer have to rebuild the package and its documentation to obtain package-decl.txt (which is not distributed). Instead, the vim syntax can be generated by a more or less trivial processing of package.syn. > patch contains a unchanged file -> a/tests/gobject/docs/tester.types (which we > want to remove from git anyway). Sorry, I still fight with git... Do you know how to generate an equivalent of svn diff -N ?
Created attachment 151537 [details] [review] proposed patch
Review of attachment 151537 [details] [review]: I'd like to merge this in the current cycle. some questions: * is this now abandon the other highlighter totally? - does this need any vim extensions? - for me using vim here was quite slow * why gtkdoc-highlight.pm if its only used in gtkdoc-fixxref? ::: gtkdoc-fixxref.in @@ +306,3 @@ + $entire_file =~ s%\<GTKDOCLINK\s+HREF=\"(.*?)\"\>(.*?)\</GTKDOCLINK\>%\<GTKDOCLINK\ HREF=\"$1\"\>$2\</GTKDOCLINK\>%gs; + + # XXX: I find the linkification of stuff in examples very disrupting. you've said that before already. Other people like the feature. What things break? Could you please file an extra bug for it? ::: gtkdoc-syntax.in @@ +7,3 @@ +my $filterout = '^_'; +my %overrides; +if (scalar @ARGV != 1 and scalar @ARGV != 2) { this should support --help and --version like the other tools
(In reply to comment #20) > * is this now abandon the other highlighter totally? No, this patch abandons *all* external highlighters. > - does this need any vim extensions? It does not use vim at all. In fact, the current version does not have anything to do with vim at all. Except that the neutral syntax file, used by the built-in highlighter, can also be easily used for generation of vim syntax files. > * why gtkdoc-highlight.pm if its only used in gtkdoc-fixxref? Because it's a good design principle to put self-contained code with a well-defined function and small surface to a separate module. I wish more of gtk-doc followed it. > + # XXX: I find the linkification of stuff in examples very disrupting. > > you've said that before already. Other people like the feature. What things > break? It is *visually* disturbing. I am not against the links being there. But I don't like the different look of the link and non-link kinds of the same syntax item and I don't like code cluttered by underscores or what. Note that now many more things (and of more symbol kinds) can be turned to links now so the visual clutter would become even worse. > Could you please file an extra bug for it? Well, I can but this patch must change the code anyway and this patch must change the CSS anyway, etc. It must be solved here somehow because the patch must contain some concrete code. > this should support --help and --version like the other tools OK.
Sorry for my ignorance. All that no makes a lot of sense. Basically there is c.syn that gives the basic c-syntax highlighting and the people just have to build all the docs to not only get xrefs, but also syntax highlighting for the types. Regarding the visual clutter. Your are not against the colors, right? We could just have one style - people will realize wheter things are xreffed when clicking them. Is that what you mean?
(In reply to comment #22) > Basically there is > c.syn that gives the basic c-syntax highlighting and the people just have to > build all the docs to not only get xrefs, but also syntax highlighting for the > types. Exactly. I tried to explain, as the idea of generation of vim syntax files gradually evolved to this but probably it is not easy to follow. I hope to fix at least the program options after the weekend. > Regarding the visual clutter. Your are not against the colors, right? We could > just have one style - people will realize wheter things are xreffed when > clicking them. Is that what you mean? For me, this would be fine as once this works all common symbols would be links. And of course there should be hover/focus CSS styles so that people can see what symbols are links when they point/move focus to them. I understand that others may prefer permanent visual differentiation, but then please please make it decent.
Created attachment 158291 [details] [review] patch gtk-doc.make as well I can look into finetuning the look once this patch is in. You could also add the commandline options to the new tool as a second patch. I just patched the gtk-doc.make as well so that I could test it with glib. Unfortunately I wasted some time to build glib several times until I realized that it still uses the tmpl stuff and you haven't patched that yet. Please merge that with the next iteration. The hilighting seems to work, but the symbols were not linked for me. From my understanding of the patch it should work, right?
Yeti, another though. Is vim using those syn files directly? Would it find them under /usr/share/gtk-doc/html/<libname>/<libname>.syn? I am wondering, as we have the content already in the .devhelp2 files. SO from gtk-doc perspective we could skip the syn files and use the .devhelp2 files.
(In reply to comment #24) > The hilighting seems to work, but the symbols were not linked for me. From my > understanding of the patch it should work, right? No, this part is commented out at present (and must be modified to create links for specific symbol types). (In reply to comment #25) > Yeti, another though. Is vim using those syn files directly? Would it find them > under /usr/share/gtk-doc/html/<libname>/<libname>.syn? > > I am wondering, as we have the content already in the .devhelp2 files. SO from > gtk-doc perspective we could skip the syn files and use the .devhelp2 files. Devhelp files are designed for devhelp and too idiosyncratic for general use. They use decorated symbol names such as `gtk_foo ()' or `enum GtkBar'. They use the same symbol class for constant-like #defines and code/function-like #defines so one could not highlight them differently. And of course they are XML and contain lots of stuff only devhelp is interested in. I am still undecided whether to use the syn files directly in vim or generate usual syntax files (I have to do some profiling), anyway, the syn files were designed for this, i.e. to be loaded in 2-3 lines of code in nice languages and a handful of lines in less nice languages as vim or C, using no external libraries. Using devhelp files requires considerable coding and/or XSLTing.
grep "keyword" tester.devhelp2 | egrep -v "(property|signal)" | sed 's/^.*name="\([a-zA-Z0-9_ ]*\).*$/\1/' maybe the .syn files could also be produced from the xslt. Then we don't need to modify the makefiles and there would be no extra tool to run.
(In reply to comment #27) > grep "keyword" tester.devhelp2 | egrep -v "(property|signal)" | sed > 's/^.*name="\([a-zA-Z0-9_ ]*\).*$/\1/' > > maybe the .syn files could also be produced from the xslt. Then we don't need > to modify the makefiles and there would be no extra tool to run. Except that > They use decorated symbol names such as `gtk_foo ()' or `enum GtkBar'. They > use the same symbol class for constant-like #defines and code/function-like > #defines so one could not highlight them differently. The first would require changing the .devhelp2 files, the latter makes the transform less simple. Anyway, I fail to see a difference between running xsltproc with a XSLT program and running perl with a Perl program. Except that the former will be slow.
That was just an idea for reuse, we should really have an open discussion about the way it is going to be done. Even tough I am not happy about the extra files, its definitely better then the external dependency on highlighters and handling their differences. So lets proceed. Can you work and enabling the links? Also please set you email in cat ~/.gitconfig so that there is something valid in the patch. The patch should then get in. Lets fix small things then incrementally (like the --help option on the tool).
Now that gtk-doc becomes more a python library with a bunch of tools, this could be done if there still is interest. Although, over the years, no one else join the discussion here of expressed interest. Yeti, shall we close this or are you interested implementing this?
With the python port, these patches would need to be redone. Closing this now.