GNOME Bugzilla – Bug 658495
Create config of expected failures for samplemedia.linaro.org files
Last modified: 2011-10-29 15:12:56 UTC
I'm on it. I'm going for something like this in the default settings.py that we ship: if os.path.exists(SAMPLEMEDIA_ROOT): INSANITY_TEST_FOLDERS[SAMPLEMEDIA_ROOT] = {lalalala} This way, we can ship it at all times and it'll Just Work if samplemedia-minimal or samplemedia-full exists.
http://cgit.collabora.com/git/user/alsuren/gst-qa-system.git/?h=samplemedia-658495 should be pretty much ready for review. There are a lot of other things that went into this branch, but I've tried to keep it as short and self-contained as possible. Shaved yaks include: * cherry-picked my Subtest-naming support patches (which are also required for linaro integration) * Create a concept of "expected skipped test" * Slightly modified "expected-failures" arg format * Documented all known failures and skips. * I tried my best to remove false positives here (if you want the patch that added false-positive checking, I can show you, but it's nowhere near production quality) * Hide the "expected-failures" arg, so that it doesn't waste space in the database. * I might re-work this last commit to be a bit more general (this was hacked up at 9:30 at night) but I think that the approach is sound.
++ as of 21890029ccf8e1e51cea494151d1c77c652a2c07 Rework of the settings dict makes sense. A couple of minor nitpicks: * hardcoding of the sample media root is unfortunate; can it be derived somehow from the location of the Django app? or at least define SAMPLEMEDIA_ROOT like in comment #1 * use if '0' and 'None' in settings is kind of ugly, would be nicer if 0 and None could be used (cast them to str() in check?) in insanity/scenario.py: * in docstring "would nit be" -> "would not be" * although you did leave a comment there, having sub.getFullArgumentList() vs sub._getFullArgumentList() is somewhat confusing, maybe rename it to something reflecting the difference? or just make it a small helper function inside getArguments, if it's the only one that's going to use it? in insanity/test.py: if checkitem not in results: continue elif ..: ... since the first if is a guard and the rest is only executed if it passes, I think this is more readable: if checkitem not in results: continue if ..
(In reply to comment #2) > ++ as of 21890029ccf8e1e51cea494151d1c77c652a2c07 > > Rework of the settings dict makes sense. > > A couple of minor nitpicks: > > * hardcoding of the sample media root is unfortunate; can it > be derived somehow from the location of the Django app? or > at least define SAMPLEMEDIA_ROOT like in comment #1 I decided on SAMPLEMEDIA_ROOT. > > * use if '0' and 'None' in settings is kind of ugly, > would be nicer if 0 and None could be used (cast them > to str() in check?) > It's not in check that the problem lies: it's: 2011-09-15 17:46:23,752 0x7fde7f082700 ERROR connection.py:571:call_async: Unable to set arguments ('/home/alsuren/src/gst-qa-system/tests/ismedia.pyc', 'tests.ismedia', 'IsMediaTest', {'uuid': '4496cdbbe14d189d04135489f2914d65', 'uri': u'file:///usr/share/samplemedia/Audio/big_buck_bunny_AAC_2Channel_44.1k_128K.AAC', 'expected-failures': [ ... ], 'bus_address': 'unix:abstract=/tmp/dbus-NUv3JUQXUM,guid=cafdad50c6241b9955fcfec7000083cc', 'proxy': True, 'timeout': 15}) according to signature u'sssa{sv}': <type 'exceptions.TypeError'>: Expected a string or unicode object Traceback (most recent call last):
+ Trace 228461
reply_handler(*message.get_args_list(**get_args_opts))
self._introspect_execute_queue()
proxy_method(*args, **keywords)
**keywords)
message.append(signature=signature, *args)
I think that dbus-python expects homogeneous types (I assume that this is why you gave up on your previously planned idea of passing filter functions in?). I know that it's horrible. If you can think of a better solution, I'm all ears. To aid debugging, I added an assert in settings.py. > in insanity/scenario.py: > * in docstring "would nit be" -> "would not be" > > * although you did leave a comment there, having > sub.getFullArgumentList() vs sub._getFullArgumentList() > is somewhat confusing, maybe rename it to something > reflecting the difference? or just make it a small helper > function inside getArguments, if it's the only one > that's going to use it? > > in insanity/test.py: > > if checkitem not in results: > continue > elif ..: > ... > > since the first if is a guard and the rest is only executed if it passes, I > think this is more readable: > > if checkitem not in results: > continue > if .. Fixed and appended to the branch (I also snuck in one commit from lava-658912 because I knew it would conflict later). If you want me to rebase/squash anything, I can.
Merged to master by Edward.