GNOME Bugzilla – Bug 412837
Need a single number to summarize test code coverage
Last modified: 2008-07-22 19:27:16 UTC
Orca needs a mechanism to gather test code coverage reports, for example, when runall.sh is run.
Created attachment 83504 [details] [review] First cut 2007-02-27 Lynn Monsanto <lynn.monsanto@sun.com> * src/orca/test_coverage.py, src/orca/coverage.py src/orca/orca.in, src/test/runone.sh, src/test/runall.sh: First cut at getting code test coverage for Orca. Coverage testing is done using the coverage.py package developed by Ned Batchelder. (http://www.nedbatchelder.com/code/modules/rees-coverage.html) I put src/orca/coverage.py under SVN control since it it not normally included in Python distributions. src/orca/test_coverage.py needs a lot of refinement! For example, the orca modules to test were manually entered into the source file. The code coverage reports need to be converted to a more friendly HTML format. To invoke code coverage testing: run runall.sh with the "-Z" flag.
Currently, in order to generate the coverage analysis and reports, it's necessary to manually terminate the python process running orca. Simply hit the quit button in the orca startup dialog. This needs to be fixed next.
Created attachment 84190 [details] [review] partial fix
Comment on attachment 84190 [details] [review] partial fix 2007-03-07 Lynn Monsanto <lynn.monsanto@sun.com> * test/harness/runone.sh, test/harness/runall.sh: Partial fix for bug #412837 - Orca needs test code coverage reporting. Run test/harness/runall.sh with the '-c' flag. This will cause test2html to generate a code coverage report in the test/coverage directory. The report can be viewed by opening index.html in the code coverage subdirectory, that is labelled by the date and time that runall.sh completed. NOTE: After runall.sh completes, the Orca will still be running. In order to generate the code coverage report for this run of runall.sh, it is necessary to quit Orca from the Orca dialog. This needs to be fixed.
Lynn - the majority of the work for this bug seems to be completed (thanks!). A nice feature to add might be a single number representing the total coverage across all files. This would give us a simple metric of where we are.
Created attachment 88600 [details] [review] Diffs to trace2html.py to keep running totals and output single coverage metric I spent about 30 minutes looking at trace2html.py. This patch seems to provide a single percentage of that represents the total coverage across all files. It might be worth testing -- if it works, maybe it would be something to give back to the trace2html author.
I'm keeping this bug open to refer to the requirement to have a single number to summarize text code coverage.
Marking this for 2.19.3. Lynn - Mr. Trace2HTML is very busy and will not be able to make a release for a while. So, we should just go with the patch for now. I suggest just putting it in the SVN repository under test/harness/trace2html-coverage-patch.txt (or something like that) and updating http://live.gnome.org/Orca/RegressionTesting to indicate what to do with it. Once that's done (and it actually works, of course ;-)), I say close this bug as fixed. Thanks!
Created attachment 89034 [details] [review] bug fix I applied Will's trace2html patch and ran the code coverage test. The generated HTML file displayed '50%' as the current test code coverage.
Closing as fixed. test/harness/trace2html-coverage-patch.txt has been put into the SVN repository. The Orca wiki still needs to be updated to show how to apply the trace2html patch, plus build and install the patched trace2html. It will take a bit of time to describe an easy-to-follow procedure, since it's a bit complicated to download, patch, build and install trace2html.
> The generated > HTML file displayed '50%' as the current test code coverage. Way cool and thanks. I'm assuming it's getting better. :-) Where did we start (i.e., if you go back in time to when you first started working on regression testing, what was our coverage)?