Published by breki on 09 Jan 2009 at 12:56 pm
MbUnit: Inconclusive Test Results
UPDATE: Jeff Brown pointed me to a better way of doing some of the things discussed in this post, so I’ve updated the post.
One of the lesser known (and documented) features of MbUnit and Gallio is marking tests as inconclusive:
public void InconclusiveTest()
{
if (WeDeterminedTheTestCannotBeRun)
Assert.Inconclusive("Inconclusive message");
WeThrowAnExceptionButItDoesNotMatter()
}
By calling Assert.Inconclusive() we tell the test run it should mark this test case as inconclusive. Assert.Inconclusive() does not throw any exceptions, the tests continues to run, but even if the later code throws an exception or some assert fails, the test outcome will still be marked as inconclusive:
456 run, 455 passed, 0 failed, 1 inconclusive, 1 skipped (1 ignored)
The build will not fail if we have one or more inconclusive tests. How does this come in handy? Sometimes you have tests which access certain external resources like internet pages. You want to be able to run such tests without causing the build to fail if internet connection is temporarily not available (I’m not saying that this is a good pattern for writing tests, just giving an example). One way to do this would be to first check the internet connection and mark the test as inconclusive if the connection is not available.
The second scenario (which we actually use in our acceptance test framework) is when you rely on certain test facility methods in order to execute the tests. Since these methods are often implemented in parallel with actual test code, we want to be able to mark them as not available until they are finished. We do this again by invoking Assert.Inconclusive() inside such methods, which will cause all test code that use these methods to have an inconclusive test result.
An alternative would be throwing NotImplementedExceptions, but we want to separate tests which actually failed from those which are not fully implemented.
There is a better way: Gallio: Setting Test Outcome Anyway You Like
Jeff Brown on 15 Jan 2009 at 2:57 #
Hehe, try [Pending] for marking incomplete stuff.
breki on 15 Jan 2009 at 8:04 #
Jeff,
I wanted to do that, but I couldn’t find a way to mark the test as pending during the execution of the test (something like NUnit’s Assert.Ignore() – http://www.nunit.org/index.php?p=utilityAsserts&r=2.2). I’ll write more about why I need this in my next posts.
Anyway, thanks for stopping by
.
Jeff Brown on 15 Jan 2009 at 9:09 #
Here’s another trick then:
throw new SilentTestException(TestOutcome.Pending);
breki on 15 Jan 2009 at 10:08 #
Hehe, looks like I’m going to have to update one more post then…. Thanks
igorbrejc.net » Gallio: Setting Test Outcome Anyway You Like on 15 Jan 2009 at 11:09 #
[...] one of my previous posts I discussed using Assert.Inconclusive() method to mark tests as "not finished yet" during [...]