After an evaluation, GNOME has moved from Bugzilla to GitLab. Learn more about GitLab.
No new issues can be reported in GNOME Bugzilla anymore.
To report an issue in a GNOME project, go to GNOME GitLab.
Do not go to GNOME Gitlab for: Bluefish, Doxygen, GnuCash, GStreamer, java-gnome, LDTP, NetworkManager, Tomboy.
Bug 412837 - Need a single number to summarize test code coverage
Need a single number to summarize test code coverage
Status: RESOLVED FIXED
Product: orca
Classification: Applications
Component: general
0.9.x
Other Linux
: Normal normal
: 2.20.0
Assigned To: Lynn Monsanto
Orca Maintainers
Depends on:
Blocks:
 
 
Reported: 2007-02-27 23:22 UTC by Lynn Monsanto
Modified: 2008-07-22 19:27 UTC
See Also:
GNOME target: ---
GNOME version: 2.15/2.16


Attachments
First cut (51.33 KB, patch)
2007-02-27 23:24 UTC, Lynn Monsanto
needs-work Details | Review
partial fix (4.15 KB, patch)
2007-03-07 19:01 UTC, Lynn Monsanto
needs-work Details | Review
Diffs to trace2html.py to keep running totals and output single coverage metric (2.11 KB, patch)
2007-05-22 12:48 UTC, Willie Walker
none Details | Review
bug fix (23.90 KB, patch)
2007-05-30 01:30 UTC, Lynn Monsanto
committed Details | Review

Description Lynn Monsanto 2007-02-27 23:22:02 UTC
Orca needs a mechanism to gather test code coverage reports, for example, when runall.sh is run.
Comment 1 Lynn Monsanto 2007-02-27 23:24:56 UTC
Created attachment 83504 [details] [review]
First cut

2007-02-27  Lynn Monsanto <lynn.monsanto@sun.com>

        * src/orca/test_coverage.py, src/orca/coverage.py
        src/orca/orca.in, src/test/runone.sh, src/test/runall.sh:

        First cut at getting code test coverage for Orca. Coverage
        testing is done using the coverage.py package developed by
        Ned Batchelder.
        (http://www.nedbatchelder.com/code/modules/rees-coverage.html)
        I put src/orca/coverage.py under SVN control since it it
        not normally included in Python distributions.

        src/orca/test_coverage.py needs a lot of refinement! For example,
        the orca modules to test were manually entered into the source
        file.

        The code coverage reports need to be converted to a more
        friendly HTML format.

        To invoke code coverage testing: run runall.sh with the "-Z"
        flag.
Comment 2 Lynn Monsanto 2007-02-27 23:28:12 UTC
Currently, in order to generate the coverage analysis and reports, it's necessary to manually terminate the python process running orca.  Simply hit the quit button in the orca startup dialog. This needs to be fixed next.
Comment 3 Lynn Monsanto 2007-03-07 19:01:36 UTC
Created attachment 84190 [details] [review]
partial fix
Comment 4 Lynn Monsanto 2007-03-07 19:03:23 UTC
Comment on attachment 84190 [details] [review]
partial fix

2007-03-07  Lynn Monsanto <lynn.monsanto@sun.com>

        * test/harness/runone.sh, test/harness/runall.sh: Partial 
	fix for bug #412837 - Orca needs test code coverage reporting.
	Run test/harness/runall.sh with the '-c' flag. This will cause
	test2html to generate a code coverage report in the 
	test/coverage directory. The report can be viewed by opening
	index.html in the code coverage subdirectory, that is labelled
	by the date and time that runall.sh completed.

	NOTE: After runall.sh completes, the Orca will still be running.
	In order to generate the code coverage report for this run
	of runall.sh, it is necessary to quit Orca from the Orca dialog.
	This needs to be fixed.
Comment 5 Willie Walker 2007-05-22 12:46:44 UTC
Lynn - the majority of the work for this bug seems to be completed (thanks!).  A nice feature to add might be a single number representing the total coverage across all files.  This would give us a simple metric of where we are.
Comment 6 Willie Walker 2007-05-22 12:48:58 UTC
Created attachment 88600 [details] [review]
Diffs to trace2html.py to keep running totals and output single coverage metric

I spent about 30 minutes looking at trace2html.py.  This patch seems to provide a single percentage of that represents the total coverage across all files.  It might be worth testing -- if it works, maybe it would be something to give back to the trace2html author.
Comment 7 Lynn Monsanto 2007-05-25 00:05:09 UTC
I'm keeping this bug open to refer to the requirement to have a single number to summarize text code coverage. 
Comment 8 Willie Walker 2007-05-25 17:09:24 UTC
Marking this for 2.19.3.  Lynn - Mr. Trace2HTML is very busy and will not be able to make a release for a while.  So, we should just go with the patch for now.  I suggest just putting it in the SVN repository under test/harness/trace2html-coverage-patch.txt (or something like that) and updating http://live.gnome.org/Orca/RegressionTesting to indicate what to do with it.

Once that's done (and it actually works, of course ;-)), I say close this bug as fixed. Thanks!
Comment 9 Lynn Monsanto 2007-05-30 01:30:31 UTC
Created attachment 89034 [details] [review]
bug fix

I applied Will's trace2html patch and ran the code coverage test. The generated HTML file displayed '50%' as the current test code coverage.
Comment 10 Lynn Monsanto 2007-05-30 01:37:57 UTC
Closing as fixed. test/harness/trace2html-coverage-patch.txt has been put into the SVN repository.

The Orca wiki still needs to be updated to show how to apply the trace2html patch,  plus build and install the patched trace2html. It will take a bit of time to describe an easy-to-follow procedure, since it's a bit complicated to download, patch, build and install trace2html.
Comment 11 Willie Walker 2007-05-30 13:05:49 UTC
> The generated
> HTML file displayed '50%' as the current test code coverage. 

Way cool and thanks.  I'm assuming it's getting better.  :-)  Where did we start (i.e., if you go back in time to when you first started working on regression testing, what was our coverage)?