GNOME Bugzilla – Bug 777443
gnome-desktop-testing: output TAP
Last modified: 2021-06-05 16:29:40 UTC
Created attachment 343709 [details] [review] Optionally produce TAP output (Sorry, I couldn't find a Bugzilla component for gnome-desktop-testing, and reasoned that a bug *somewhere* was better than private email.) https://wiki.gnome.org/Initiatives/GnomeGoals/InstalledTests suggests that tests should output the machine-readable TAP format, and many do. However, the reference runner for those installed-tests, gnome-desktop-testing, doesn't do the same. I think it should, so that non-GNOME-specific tools can report which specific sub-tests failed. If the test also declares that it outputs TAP, we can use the unofficial but widely-supported "subtest" syntax (which Perl's Test::More, one of the major TAP producers, has had since at least 2009). If not, we can escape the output of the sub-test so it isn't interpreted as part of our output, by prepending "#". The new --tap-output option implies --parallel=1, because screen-scraping output isn't very compatible with concurrency. (For context for this, there has been mention recently of adding finer-grained test result reporting to Debian's autopkgtest, and I am intending to recommend using TAP for that instead of inventing new wheels. Having GNOME/GLib libraries easily produce TAP from gnome-desktop-testing would be a good way to get finer-grained reporting in many packages at once.)
Created attachment 343710 [details] [review] If a test declares that it outputs TAP, print it as a sub-test The TAP sub-test syntax is non-standard, but Perl's Test::More produces it, and https://github.com/tapjs/tap-parser can parse it. Parsers that do not understand it will treat it as a non-TAP line to be ignored. (Extensive discussion which unfortunately contains many bike sheds: <https://github.com/TestAnything/Specification/issues/2>.)
Created attachment 343711 [details] Sample output: ./gnome-desktop-testing-runner --tap-output dbus
It would be more interoperable if this had ----8<---- -# Running test: dbus/test-sd-activation.test +# Subtest: dbus/test-sd-activation.test # random seed: R02S6e6108a40723e788e641ea2b9fd11aee 1..2 # Start of sd-activation tests # set DBUS_TEST_DATA to a directory containing valid-config-files/systemd-activation.conf ok 1 /sd-activation/activation # SKIP DBUS_TEST_DATA not set # set DBUS_TEST_DATA to a directory containing valid-config-files/systemd-activation.conf ok 2 /sd-activation/uae # SKIP DBUS_TEST_DATA not set # End of sd-activation tests ok 1 - dbus/test-sd-activation.test ----8<---- which is a popular non-standard syntax for sub-tests. Alternatively, or even in addition, g-d-t-r could take an argument naming a directory into which to write one .tap file per .test that declares TAP output, without the extra indentation (so each file in that directory is a single valid TAP stream).
(e.g. http://www.node-tap.org/tap-format/ documents that non-standard syntax)
(In reply to Simon McVittie from comment #0) > (Sorry, I couldn't find a Bugzilla component for gnome-desktop-testing, and > reasoned that a bug *somewhere* was better than private email.) I think they go under gnome-continuous.
Bug #788470 has a simpler version, which just diverts tests' stdout to g-d-t's stderr so it doesn't have to be interpreted as TAP. This is not done when using --report-directory or --log-directory, which might be preferable anyway. Capturing tests' stdout and writing it into the TAP stream, if done, would need to be rebased on what's in Bug #788470.
CCing Emmanuele since he has somehow avoided getting marked as a maintainer of the gnome-continuous Bugzilla product.
(In reply to Simon McVittie from comment #6) > Bug #788470 has a simpler version, which just diverts tests' stdout to > g-d-t's stderr so it doesn't have to be interpreted as TAP. This is not done > when using --report-directory or --log-directory, which might be preferable > anyway. > > Capturing tests' stdout and writing it into the TAP stream, if done, would > need to be rebased on what's in Bug #788470. ⇒ This needs rebasing, or closing. It probably is valuable to have tests’ output in the aggregate output TAP stream.
(In reply to Philip Withnall from comment #8) > ⇒ This needs rebasing, or closing. It probably is valuable to have tests’ > output in the aggregate output TAP stream. If it'll get reviewed eventually, I'll try to rebase. Is Bugzilla still the preferred route for gnome-desktop-testing contributions?
I’ll review it. While I’m not a maintainer of g-d-t, I think I can probably wrangle accepting some patches. g-d-t has had its git migrated to GitLab, but the old issues have not been migrated yet. Despite that, I think anyone who’s likely to review or care about this would prefer an MR on GitLab to patches on Bugzilla. Up to you. https://gitlab.gnome.org/GNOME/gnome-desktop-testing
gnome-continuous is not under active development anymore. Its codebase has been archived: https://gitlab.gnome.org/Archive/gnome-continuous Closing all its open tickets as part of housekeeping.