GNOME Bugzilla – Bug 519543
[pending] gtk-demo/role_icon.py regression test #7 produces different results on Solaris and Ubuntu.
Last modified: 2009-03-10 00:04:40 UTC
This is on latest Ubuntu Hardy: EXPECTED: "BUG? - we cannot get reliable output from this test since it depends on the contents of /", "BUG? - we do not announce selection of icons (e.g., 2 of 26 items selected).", ACTUAL: "BRAILLE LINE: 'gtk-demo Application GtkIconView demo Frame ScrollPane LayeredPane cdrom Icon'", " VISIBLE: 'cdrom Icon', cursor=1", "SPEECH OUTPUT: 'Icon panel'", "SPEECH OUTPUT: 'cdrom'", "SPEECH OUTPUT: '2 of 24 items selected'", "SPEECH OUTPUT: 'on item 3 of 24'", [FAILURE WAS EXPECTED - LOOK FOR BUG? IN EXPECTED RESULTS] Looks like this might be a case of the regression test needs amending for the second BUG? Not sure what to do about the first. It seems pointless having a regression test if we can't get reliant/consistent expected results.
Created attachment 106643 [details] [review] Revision #1. Patch adjusts the first "BUG?" to be a "KNOWN ISSUE", and adjust the rest of the test to be the correct output for a Ubuntu Hardy system.
Can you look to see if there is some other test we could run where the contents of the icon pane are consistent? That way, we could avoid this known issue. Thanks!
Well the only way to get consistency is if there was a directory on both Solaris and Ubuntu that contains the same file. One such directory might be /etc/brytty. The problem I'm finding is that when I try to navigate to it (either by double-clicking on the icon with the mouse or via arrow keys and Return) gtk-demo core dumps.
(In reply to comment #3) > Well the only way to get consistency is if there was a directory > on both Solaris and Ubuntu that contains the same file. One such > directory might be /etc/brytty. The problem I'm finding is that > when I try to navigate to it (either by double-clicking on the > icon with the mouse or via arrow keys and Return) gtk-demo core > dumps. Is there another test somewhere else that might work?
> Is there another test somewhere else that might work? I couldn't find one. :-(
Here's a thought, though maybe not perfect, and I haven't tried it to be sure it would work, but: Make a *.params file for the test that contains something like the following: mkdir -p ~/AAAAA/foo ~/AAAAA/bar ~/AAAAA/baz ~/AAAAA/snafu This should work regardless if the directory has been created or not. If you wanted to be really clever, you might make the contents of ~/AAAAA look like the contents of "/" that the tests were initially created against (I think it was Ubuntu). Then, as part of running the test: 1) First go to the toolbar (Shift+Tab when this particular window first appears) 2) Activate the "Home" button 3) Tab and then Down Arrow to get to the icon pane. 4) "AAAAA" was chosen to make it appear first in the directory listing, so it should now be selected 5) Press Return and you're now in a controlled environment, assuming you aren't prone to making directories named AAAAA in your home directory. Might work.
This might all work if it wasn't for the fact that: " ... The problem I'm finding is that when I try to navigate to it (either by double-clicking on the icon with the mouse or via arrow keys and Return) gtk-demo core dumps." Will, are you seeing this problem? Joanie, you are up to date with Ubuntu Hardy. Do you see this problem?
The other problem here is there might already be a directory in your home directory called " AA" (or several such directories with similar names) which will appear before AAAAA. You can certainly reduce the chance that this test isn't going to work, but the suggested approach above is not 100% guaranteed to be consistent on all platforms and for all users.
So, like, I have to change my autoclub membership? ;-) ;-) ;-) But even after I join JimBob's Tows 'R Us, I think we're still going to see problems with platform specific differences. Bug 519271 and the poor unsuspecting bug from which that bug spawned testify to that. Will lacked all the unexpected errors that Rich and I were seeing, and yet he had two for which we had successes. I just opened bug 520656. If we: * Minimize differences in our individual environments by our setup * Minimize platform-specific differences when possible (e.g. adding specific fonts to the FF profile) AND * allow the harness to accept a list of possible expected outputs that might help. <shrugs>
Oops, sorry to be spammy, but I forgot this:(In reply to comment #7) > Joanie, you are up to date with Ubuntu Hardy. Do you see this problem? Yes to the former; no to the latter. Sorry! :-(
Created attachment 106683 [details] [review] Example that uses 1ORCA_TEST Well...I thought I'd throw caution to the wind. :-) Here's an example that uses a directory called 1ORCA_TEST. It ends up opening the directory just fine and I don't see any core dumps. It certainly does seem to expose a number of problems with icon panes, though. I think we need to look at our output overall. It just seems generally wrong. Here's some examples: 1) When the layered pane gets focus, I think I saw "0 items" being presented when in fact it was not empty. 2) When we press "Return" to open the 1ORCA_TEST directory, I don't think we output anything. Should we? If so, what should it be? (Question for Mike, I think.) 3) The icons do not always have names via the AT-SPI. For example I saw the "bin" icon have a name in the 1ORCA_TEST directory, but Orca just said "icon" for everything else. This is strange. We may have just hit on a Shangri-La of bugs. >You can certainly reduce the chance that this test isn't going to >work, but the suggested approach above is not 100% guaranteed to >be consistent on all platforms and for all users. Agreed. But, we can increase the chance that it will produce expected results across platforms. In addition, it also puts things more under our control rather than relying upon the OS distribution to keep the contents of "/" constant. I agree it's not 100% reliable for all user on all platforms, but it is better than what we had. I think the only way to guarantee 100% consistency is to write our own test application. >" ... The problem I'm finding is that >when I try to navigate to it (either by double-clicking on the >icon with the mouse or via arrow keys and Return) gtk-demo core >dumps." I see it pretty frequently with the mouse and occasionally with the keyboard. I have yet to see it when I follow the exact steps for the suggested alternative, however. Maybe I've been lucky?
Adjusting the summary to better reflect the remaining problem with the role_icon.py regression tests.
Created attachment 108151 [details] [review] Patch to use regular expressions and limit motion in the icon pane As we've seen with all the comments on this bug, the Icon View Basics demo is just plain troublesome. It crashes if you breathe on it even lightly, it's not something found in the wild, and it depends upon the contents of /. But, we also don't want to invest time in writing a new test. :-) This patch still keeps our dependency on the contents of /, but limits the motion to the first two icons, "bin" and "boot", since they are common between Solaris and Ubuntu. It also uses regular expressions to account for the differences in the number of files in the icon pane. Seems to work OK and it accomplishes the main thing we want to accomplish with this test: are our generators and where am I stuff working for icons?
Pending until I see what the nightly tests say.