After an evaluation, GNOME has moved from Bugzilla to GitLab. Learn more about GitLab.
No new issues can be reported in GNOME Bugzilla anymore.
To report an issue in a GNOME project, go to GNOME GitLab.
Do not go to GNOME Gitlab for: Bluefish, Doxygen, GnuCash, GStreamer, java-gnome, LDTP, NetworkManager, Tomboy.
Bug 536928 - have syntax highlghted and xrefs source code samples
have syntax highlghted and xrefs source code samples
Product: gtk-doc
Classification: Platform
Component: general
Other Linux
: Normal enhancement
: 1.12
Assigned To: gtk-doc maintainers
gtk-doc maintainers
Depends on:
Reported: 2008-06-06 03:08 UTC by Behdad Esfahbod
Modified: 2009-03-09 16:05 UTC
See Also:
GNOME target: ---
GNOME version: ---

proposed patch (4.10 KB, patch)
2009-01-23 11:56 UTC, Yeti
none Details | Review
screenshot (185.72 KB, image/png)
2009-01-23 12:06 UTC, Yeti
test file for code listing (1.55 KB, text/html)
2009-01-23 16:11 UTC, Stefan Sauer (gstreamer, gtkdoc dev)
test file for code listing (1.66 KB, text/html)
2009-01-23 20:33 UTC, Stefan Sauer (gstreamer, gtkdoc dev)
HTML example with two cells (984 bytes, text/html)
2009-01-23 21:46 UTC, Yeti
proposed patch (4.72 KB, patch)
2009-01-31 16:33 UTC, Yeti
committed Details | Review

Description Behdad Esfahbod 2008-06-06 03:08:22 UTC
I have many sample programs in the source tree that are compiled and built by default, so they are never fail to build.

I like to have a single-line way to include a sample in the docs.  With hyperlinked symbols and, well, syntax highlighting.

For C, I've used some little tool called prettyc to do syntax-highlighting to HTML before.
Comment 1 Stefan Sauer (gstreamer, gtkdoc dev) 2008-06-07 12:25:23 UTC
The is a easy way to includes samples source which I discovered when working on the gtk-doc code. Its now documented. It goes like this:

This example shows how to frobnicate twice:[|
gint times=2
gint result=frobnicate(param,times);


the [| |] are expanded to <informalexample><programmlisting> and the respective closing tags. If you want to include sources use <xi:include>.

Regarding syntax highlighting. That would need to be supported on the docbook level. I did some quick googling to cehck the status:

Xrefs would be nice to have too, but would require to parse the listings and autogenerate xrefs. Normally gtk-doc relies on the special markers (like #%()).
I have some plans for this as I would like to (optinally) emit warnings if gtk-doc find words in comments that could be linked. Honestly this part might take sometime unless someone comes up with a patch.
Comment 2 Behdad Esfahbod 2008-06-07 19:47:27 UTC
Humm, how can I include unformatted source code using <xi:include>?
Comment 3 Stefan Sauer (gstreamer, gtkdoc dev) 2008-06-08 06:16:53 UTC
My first attempt was:

  <textdata fileref="example.c" />

but apparently i doesn't work (the listing is empty). The xsltprocessing gives no warning though :/ And there is an explanation:

But then another try:
<programlisting><xi:include xmlns:xi="" parse="text" href="../../examples/gobject.c" /></programlisting>

And this works \o/. I added it to the gobject test.
Comment 4 Stefan Sauer (gstreamer, gtkdoc dev) 2008-06-09 08:46:06 UTC
Of course when you xinclude sources, we cannot xref then and neither syntax highlight them :/

Comment 5 Stefan Sauer (gstreamer, gtkdoc dev) 2009-01-05 15:29:44 UTC
I found that link, will check if it helps:

This should be available in docbook-xsl >= 1.71 - e.g. Ubunto 8.10 has  1.73.2.dfsg.1-4

Main problem is that this only works with saxon. Saxxon is 20 times slower than xsltproc and it does not support xinlude :/ I'll check if it can be used from xsltproc in a simillar way.
Comment 6 Stefan Sauer (gstreamer, gtkdoc dev) 2009-01-05 15:53:06 UTC
We could also think about using javascript:
Comment 7 Stefan Sauer (gstreamer, gtkdoc dev) 2009-01-07 08:31:50 UTC
Behadad, I investigated more. I checked a lot of perl modules, but they are not packaged widely. But then I can accross GNU source-highlight

It seems widely available as well:
Ubuntu 8.10: source-highlight-2.10-0ubuntu1
Debian-sid: source-highlight-2.4-5.2
Gentoo: source-highlight-2.11
Fedora 10: highlight-2.6.13-2
OpenSuSE-11.0: highlight-2.6.9-3.2

I tested it a bit and it looks good. I could run some post processing script after generating the html (it the tool is available), then extracts samples and colorize them. Alternatively I could alreay process them on the docbook level one step earlier.

What do you think?
Comment 8 Yeti 2009-01-07 08:53:16 UTC
(In reply to comment #7)
> Behadad, I investigated more. I checked a lot of perl modules, but they are not
> packaged widely.

The clearly most widely packaged tool for generating syntax-highlighted HTML from code is Vim (see syntax/2html.vim and :h 2html).  Using vim would be slower than running specialized tools, but given the time taken by xsltprocing, maybe not noticeably.
Comment 9 Behdad Esfahbod 2009-01-08 01:27:29 UTC
Thanks Stefan for looking into this!  I have no expertise whatsoever in this field, so I don't know which approach is better.  Perhaps make gtkdoc rely on highlight in a devel version and see how many people/distros scream?
Comment 10 Stefan Sauer (gstreamer, gtkdoc dev) 2009-01-08 17:43:14 UTC
My plan right now, is to hack the tool that loops over the code snippets and pipes them through gnu highlight. I'll make it extensible for other tools. Maybe Yeti can contribute the vim support.
Comment 11 Yeti 2009-01-09 08:48:02 UTC
Yes, once there is a well defined hook for the highlighting postprocessor I can write a vim implementation.
Comment 12 Stefan Sauer (gstreamer, gtkdoc dev) 2009-01-09 15:13:35 UTC
If we do the syntax highlighting and xreffing for html only, I would be tempted to put it into gtkdoc-fixxref. Not sure if I should rename this to gtkdoc-postprochtml and install a symlink for gtkdoc-fixxref.

I have commited what I have to gtk-doc svn for now.
Comment 13 Behdad Esfahbod 2009-01-09 15:24:19 UTC
What would the benefit of doing it at fixxref-time be?
Comment 14 Behdad Esfahbod 2009-01-09 15:28:00 UTC
Sorry, didn't mean to screw the priorities.
Comment 15 Stefan Sauer (gstreamer, gtkdoc dev) 2009-01-09 16:58:43 UTC
Its more like that fixxref is anyway rewriting the html. Main benefit would be that its not impacting performance too much in a negative way and we don't have another buildstep/tool.

Btw. the change in svn assumes that people have gnu source-highlight installed, will fix that next.
Comment 16 Stefan Sauer (gstreamer, gtkdoc dev) 2009-01-21 13:41:42 UTC
The code in svn supports highlight and source-highligh as external tools. It also tries to xref everything inside the source (please not that enums and flags are only linked if you have rebuild the docs of the dependent package as well).

@Yeti, if you want to give the vim integration a go. Everything is done in gtkdoc-fixxref.

I have a few things still on my todo for the next days, mainly adding a way of specifying the language and protecting it from getting lost in the docbook->html transformation.

One thing that I don't like is that if you select and copy an example, browsers will insert '#' chars for each line number. I can't fine a way to tell them that this sucks :(

Anyway, please test:
checkout svn trunk
make && make check
epiphany tests/gobject/docs/html/index.html
(note that external xrefs only work once installed)
Comment 17 Yeti 2009-01-21 15:08:34 UTC
Stefan, ATM, I'm stuck on a foreign university without direct internet connection (won't give location to protect the guilty...) so I cannot do `svn up'.  I will implement the vim highlighting, but if you are trying to release 1.12 soon don't wait for it and leave this bug open...
Comment 18 Yeti 2009-01-23 11:56:28 UTC
Created attachment 127081 [details] [review]
proposed patch

A first version of the patch.  Note I had to add a different function for vim because the I do not dump the output to the standard output.  Also, it is slow.  Probably noticeably, if you have many code examples.
Comment 19 Yeti 2009-01-23 12:05:22 UTC
Now a few remarks to the output.  I am sorry, but it is ugly.

The attempt to emulate <pre> with <ol> causes some strange inconsistent font to be used (see attached screenshot).

The line numbers get pasted with the code if you select the example in browser -- you have already noticed that, but IMO this is a show-stopper.

And it introduces a new class -- why?  I want the thing to be just oridnary <pre class="programlisting"> as the other literal code outputs, please, because that is what it is: a program listing.
Comment 20 Yeti 2009-01-23 12:06:37 UTC
Created attachment 127084 [details]
Comment 21 Stefan Sauer (gstreamer, gtkdoc dev) 2009-01-23 12:29:51 UTC
The idea behind the using-ol-experiment (that two two whopping days) was that you see line numbers, but you would not select & copy them. Interestingly all my browsers don't select them, but the copy them (or '#' instead), which totally sucks. Even worse it sems to be a feature and people complain that IE6 does not do it.

I'll probably change it back to <pre> and style the 'span' for the line-number so that we get the colored column.
I'll try to figure what e.g.
is doing to get line numbers but avoid them beeing copied.
Comment 22 Yeti 2009-01-23 12:40:25 UTC
(In reply to comment #21)
> I'll try to figure what e.g.
> is doing to get line numbers but avoid them beeing copied.

They show the code as a two-column table, with the line numbers in the left column and code in the right column.  Every line in its own <td>.  No comment.

Nonetheless I was thinking about something similar, except that the line numbers and code would be each in its own <div>, positioned alongside, and inside the <div>s we would just use ordinary <pre>.  But then imagined getting this to display right in all versions of MSIE...
Comment 23 Stefan Sauer (gstreamer, gtkdoc dev) 2009-01-23 16:11:59 UTC
Created attachment 127101 [details]
test file for code listing

Yeah, the table works and I'd say its quite okay. Tested with FF2, FF2 and IE6. Comments?
Comment 24 Yeti 2009-01-23 16:48:44 UTC
Like before, the font is weird (i.e. different from other fonts).  It's a sort of abuse of HTML.  If you really want a table, would not a table with just two cells, line numbers (left) and code (right), each containing a <pre>, do?

And please get rid of the colons, the numbers are already over-separated.
Comment 25 Stefan Sauer (gstreamer, gtkdoc dev) 2009-01-23 20:33:06 UTC
Created attachment 127124 [details]
test file for code listing

The font should be monospaced, I am not aware of any way of saying "format this like pre". Using <pre> inside the linenumber cells will cause extra spacing. But yep, the colons are superflous - I've updated the sample.
Comment 26 Yeti 2009-01-23 21:46:00 UTC
Created attachment 127131 [details]
HTML example with two cells

Where exactly do you see the spaces?
Comment 27 Stefan Sauer (gstreamer, gtkdoc dev) 2009-01-23 22:05:35 UTC
My example wasn't good. If I also split the code into tr's, the table with can shrink and long lines can wrap without line numbers getting off. But maybe we just don't allow that. Then just using two cells and <pre> in both is fine.
If you take my example and put <pre>1</pre> <pre>2</pre> into the line number cells you see what I mean.
Comment 28 Behdad Esfahbod 2009-01-23 22:10:07 UTC
Does CSS2 text insertion syntax help with the numbering?  Something like:

#line1:before { content: "1"; }

That's a lot of work that shouldn't be needed though, I agree...
Comment 29 Yeti 2009-01-24 08:57:34 UTC
Behdad, using CSS for the line numbering would make sense (IMO) if it resulted in shorter total content.  Which does not seem to be the case here.  And since you want to add anchors to each line which is not possible with CSS counters (you cannot add new elements), it is simplest to just generate the complete thing.

Stefan, I would not allow line wrapping (the example should be formatted reasonably) so I'm fine with wrapping not possible.
Comment 30 Stefan Sauer (gstreamer, gtkdoc dev) 2009-01-24 16:03:57 UTC
Actually using CSS with :before selectors was my first attempt. Unfortunately its too early to get consitent results across the browsers yet. I also did not manage to style it differenty (e.g. have a different color for the numbers and the code). Finally as this becomes generated *content* atleast gecko based browsers also copy the numbers along with the code :/ Somehow its totaly missing to specify that something is just visual decoration.

I would just misuse the tables like in the latest attachment. I'll rewrite that part soon.
Comment 31 Yeti 2009-01-31 16:33:41 UTC
Created attachment 127635 [details] [review]
proposed patch

A new version of the vim patch.
Comment 32 Stefan Sauer (gstreamer, gtkdoc dev) 2009-02-20 16:42:55 UTC
David, can vim recognice functions? The other two hightlighters would e.g. mark g_object_new with a span.
Comment 33 Yeti 2009-02-20 17:22:06 UTC
Vim can, but the default C syntax patterns do not highlight functions.  IMO this is good because functions are not really different from other identifiers.

Vim can do something much better, though.  It can highlight GLib/Gtk+/.../project-specific functions, macros, constants, types, etc. in any context, see bug 567135.  This means not only in


the function name is highlighted, but it is also highlighted in

var = &g_object_new;

It also means that G_TYPE_OBJECT is highlighted as constant, that GObject is highlighted as type, that G_UNLIKELY is highlighted as macro (not as a function) because it is a macro, and that in

var = g_new(GList, 1);

GList is highlighted as a type because vim knows it is a type and is not confused by the whole thing looking like a function call.  See bug 567135 for details.
Comment 34 Stefan Sauer (gstreamer, gtkdoc dev) 2009-03-09 16:05:06 UTC
David, I have committed this now, thanks.