GNOME Bugzilla – Bug 536928
have syntax highlghted and xrefs source code samples
Last modified: 2009-03-09 16:05:06 UTC
I have many sample programs in the source tree that are compiled and built by default, so they are never fail to build. I like to have a single-line way to include a sample in the docs. With hyperlinked symbols and, well, syntax highlighting. For C, I've used some little tool called prettyc to do syntax-highlighting to HTML before.
The is a easy way to includes samples source which I discovered when working on the gtk-doc code. Its now documented. It goes like this: -- This example shows how to frobnicate twice:[| gint times=2 gint result=frobnicate(param,times); |] -- the [| |] are expanded to <informalexample><programmlisting> and the respective closing tags. If you want to include sources use <xi:include>. Regarding syntax highlighting. That would need to be supported on the docbook level. I did some quick googling to cehck the status: * http://www.sagehill.net/docbookxsl/SyntaxHighlighting.html * http://tohtml.com/dbsy/ * http://sourceforge.net/projects/xslthl/ Xrefs would be nice to have too, but would require to parse the listings and autogenerate xrefs. Normally gtk-doc relies on the special markers (like #%()). I have some plans for this as I would like to (optinally) emit warnings if gtk-doc find words in comments that could be linked. Honestly this part might take sometime unless someone comes up with a patch.
Humm, how can I include unformatted source code using <xi:include>?
My first attempt was: <informalexample><programlisting><textobject> <textdata fileref="example.c" /> </textobject></programlisting></informalexample> but apparently i doesn't work (the listing is empty). The xsltprocessing gives no warning though :/ And there is an explanation: http://lists.oasis-open.org/archives/docbook-apps/200209/msg00005.html But then another try: <informalexample> <programlisting><xi:include xmlns:xi="http://www.w3.org/2003/XInclude" parse="text" href="../../examples/gobject.c" /></programlisting> </informalexample> And this works \o/. I added it to the gobject test.
Of course when you xinclude sources, we cannot xref then and neither syntax highlight them :/
I found that link, will check if it helps: http://xmlguru.cz/2006/07/docbook-syntax-highlighting This should be available in docbook-xsl >= 1.71 - e.g. Ubunto 8.10 has 1.73.2.dfsg.1-4 Main problem is that this only works with saxon. Saxxon is 20 times slower than xsltproc and it does not support xinlude :/ I'll check if it can be used from xsltproc in a simillar way.
We could also think about using javascript: http://mihai.bazon.net/projects/javascript-syntax-highlighting-engine http://softwaremaniacs.org/soft/highlight/en/
Behadad, I investigated more. I checked a lot of perl modules, but they are not packaged widely. But then I can accross GNU source-highlight (http://www.gnu.org/software/src-highlite/source-highlight.html) It seems widely available as well: Ubuntu 8.10: source-highlight-2.10-0ubuntu1 Debian-sid: source-highlight-2.4-5.2 Gentoo: source-highlight-2.11 Fedora 10: highlight-2.6.13-2 OpenSuSE-11.0: highlight-2.6.9-3.2 I tested it a bit and it looks good. I could run some post processing script after generating the html (it the tool is available), then extracts samples and colorize them. Alternatively I could alreay process them on the docbook level one step earlier. What do you think?
(In reply to comment #7) > Behadad, I investigated more. I checked a lot of perl modules, but they are not > packaged widely. The clearly most widely packaged tool for generating syntax-highlighted HTML from code is Vim (see syntax/2html.vim and :h 2html). Using vim would be slower than running specialized tools, but given the time taken by xsltprocing, maybe not noticeably.
Thanks Stefan for looking into this! I have no expertise whatsoever in this field, so I don't know which approach is better. Perhaps make gtkdoc rely on highlight in a devel version and see how many people/distros scream?
My plan right now, is to hack the tool that loops over the code snippets and pipes them through gnu highlight. I'll make it extensible for other tools. Maybe Yeti can contribute the vim support.
Yes, once there is a well defined hook for the highlighting postprocessor I can write a vim implementation.
If we do the syntax highlighting and xreffing for html only, I would be tempted to put it into gtkdoc-fixxref. Not sure if I should rename this to gtkdoc-postprochtml and install a symlink for gtkdoc-fixxref. I have commited what I have to gtk-doc svn for now.
What would the benefit of doing it at fixxref-time be?
Sorry, didn't mean to screw the priorities.
Its more like that fixxref is anyway rewriting the html. Main benefit would be that its not impacting performance too much in a negative way and we don't have another buildstep/tool. Btw. the change in svn assumes that people have gnu source-highlight installed, will fix that next.
The code in svn supports highlight and source-highligh as external tools. It also tries to xref everything inside the source (please not that enums and flags are only linked if you have rebuild the docs of the dependent package as well). @Yeti, if you want to give the vim integration a go. Everything is done in gtkdoc-fixxref. I have a few things still on my todo for the next days, mainly adding a way of specifying the language and protecting it from getting lost in the docbook->html transformation. One thing that I don't like is that if you select and copy an example, browsers will insert '#' chars for each line number. I can't fine a way to tell them that this sucks :( Anyway, please test: checkout svn trunk make && make check epiphany tests/gobject/docs/html/index.html (note that external xrefs only work once installed)
Stefan, ATM, I'm stuck on a foreign university without direct internet connection (won't give location to protect the guilty...) so I cannot do `svn up'. I will implement the vim highlighting, but if you are trying to release 1.12 soon don't wait for it and leave this bug open...
Created attachment 127081 [details] [review] proposed patch A first version of the patch. Note I had to add a different function for vim because the I do not dump the output to the standard output. Also, it is slow. Probably noticeably, if you have many code examples.
Now a few remarks to the output. I am sorry, but it is ugly. The attempt to emulate <pre> with <ol> causes some strange inconsistent font to be used (see attached screenshot). The line numbers get pasted with the code if you select the example in browser -- you have already noticed that, but IMO this is a show-stopper. And it introduces a new class -- why? I want the thing to be just oridnary <pre class="programlisting"> as the other literal code outputs, please, because that is what it is: a program listing.
Created attachment 127084 [details] screenshot
The idea behind the using-ol-experiment (that two two whopping days) was that you see line numbers, but you would not select & copy them. Interestingly all my browsers don't select them, but the copy them (or '#' instead), which totally sucks. Even worse it sems to be a feature and people complain that IE6 does not do it. I'll probably change it back to <pre> and style the 'span' for the line-number so that we get the colored column. I'll try to figure what e.g. http://code.google.com/p/google-code-prettify/source/browse/trunk/src/prettify.js is doing to get line numbers but avoid them beeing copied.
(In reply to comment #21) > I'll try to figure what e.g. > http://code.google.com/p/google-code-prettify/source/browse/trunk/src/prettify.js > is doing to get line numbers but avoid them beeing copied. They show the code as a two-column table, with the line numbers in the left column and code in the right column. Every line in its own <td>. No comment. Nonetheless I was thinking about something similar, except that the line numbers and code would be each in its own <div>, positioned alongside, and inside the <div>s we would just use ordinary <pre>. But then imagined getting this to display right in all versions of MSIE...
Created attachment 127101 [details] test file for code listing Yeah, the table works and I'd say its quite okay. Tested with FF2, FF2 and IE6. Comments?
Like before, the font is weird (i.e. different from other fonts). It's a sort of abuse of HTML. If you really want a table, would not a table with just two cells, line numbers (left) and code (right), each containing a <pre>, do? And please get rid of the colons, the numbers are already over-separated.
Created attachment 127124 [details] test file for code listing The font should be monospaced, I am not aware of any way of saying "format this like pre". Using <pre> inside the linenumber cells will cause extra spacing. But yep, the colons are superflous - I've updated the sample.
Created attachment 127131 [details] HTML example with two cells Where exactly do you see the spaces?
My example wasn't good. If I also split the code into tr's, the table with can shrink and long lines can wrap without line numbers getting off. But maybe we just don't allow that. Then just using two cells and <pre> in both is fine. If you take my example and put <pre>1</pre> <pre>2</pre> into the line number cells you see what I mean.
Does CSS2 text insertion syntax help with the numbering? Something like: #line1:before { content: "1"; } That's a lot of work that shouldn't be needed though, I agree...
Behdad, using CSS for the line numbering would make sense (IMO) if it resulted in shorter total content. Which does not seem to be the case here. And since you want to add anchors to each line which is not possible with CSS counters (you cannot add new elements), it is simplest to just generate the complete thing. Stefan, I would not allow line wrapping (the example should be formatted reasonably) so I'm fine with wrapping not possible.
Actually using CSS with :before selectors was my first attempt. Unfortunately its too early to get consitent results across the browsers yet. I also did not manage to style it differenty (e.g. have a different color for the numbers and the code). Finally as this becomes generated *content* atleast gecko based browsers also copy the numbers along with the code :/ Somehow its totaly missing to specify that something is just visual decoration. I would just misuse the tables like in the latest attachment. I'll rewrite that part soon.
Created attachment 127635 [details] [review] proposed patch A new version of the vim patch.
David, can vim recognice functions? The other two hightlighters would e.g. mark g_object_new with a span.
Vim can, but the default C syntax patterns do not highlight functions. IMO this is good because functions are not really different from other identifiers. Vim can do something much better, though. It can highlight GLib/Gtk+/.../project-specific functions, macros, constants, types, etc. in any context, see bug 567135. This means not only in g_object_new(...); the function name is highlighted, but it is also highlighted in var = &g_object_new; It also means that G_TYPE_OBJECT is highlighted as constant, that GObject is highlighted as type, that G_UNLIKELY is highlighted as macro (not as a function) because it is a macro, and that in var = g_new(GList, 1); GList is highlighted as a type because vim knows it is a type and is not confused by the whole thing looking like a function call. See bug 567135 for details.
David, I have committed this now, thanks.