igorbrejc.net

When doing automatic testing of Web apps using unit testing frameworks, it can be a pain in the butt to pinpoint the proper HTML element. A lot of times tests will fail because you used a wrong locator, but since the browser will automatically close after the test, you don’t have an access to the HTML code of the page to look at what’s actually there.

Fortunately Gallio provides a class called TestContext which contains the current information about the running test and which you can use to determine if the latest test is successful or not. This can then be used to run your custom handling code during the test teardown:

        [TearDown]
        protected virtual void Teardown()
        {
            if (TestContext.CurrentContext.Outcome.Status == TestStatus.Failed)
            {
                using (TestLog.BeginSection("Failed web page HTML"))
                    TestLog.Write(WebDriver.PageSource);
            }
        }

In the above snippet, we record the current Web page’s HTML code into Gallio’s log (the TestLog class). To avoid spamming the log, we do this for failed tests only.

Gallio provides a powerful framework which I think is very much underused, mostly because the documentation is not very detailed (to say the least).

UPDATE (June 17th): I’ve updated the code, see the reasons for it at the end of the post.

In previous projects I worked on we made sure the Selenium Java server was running by manually starting it on our machines (both developers’ and build ones). This was cumbersome: restarting the build server meant we had to log on to the server after the reboot and run the Selenium server again. Of course, a lot of times we forgot to do this, which caused the build to fail.

This got me into thinking: is there a way in Gallio to specify some initialization (and cleanup) actions on the test assembly level? And of course, the answer is yes: using the AssemblyFixture attribute. This is what I like about Gallio/MbUnit: most of the time the feature requests I come up with are actually already implemented.

So anyway, you can specify this attribute on a class and then add FixtureSetUp and FixtureTearDown attributes on its methods. These will be executed on the test assembly-level: setup methods will be executed before any test fixtures have been run and teardown methods will be executed before the test assembly has been unloaded by the test runner.

I then used this nice feature to start the Selenium server and then dispose of it after tests:

[AssemblyFixture]
public class SeleniumTestingSetup : IDisposable
{
    [FixtureSetUp]
    public void Setup()
    {
        seleniumServerProcess = new Process();
        seleniumServerProcess.StartInfo.FileName = "java";
        seleniumServerProcess.StartInfo.Arguments =
            "-jar ../../../lib/Selenium/selenium-server/selenium-server.jar -port 6371";
        seleniumServerProcess.Start();
    }

    /// <summary>
    /// Performs application-defined tasks associated with freeing, releasing, or
    /// resetting unmanaged resources.
    /// </summary>
    public void Dispose()
    {
        Dispose(true);
        GC.SuppressFinalize(this);
    }

    /// <summary>
    /// Disposes the object.
    /// </summary>
    /// <param name="disposing">If <code>false</code>, cleans up native resources. 
    /// If <code>true</code> cleans up both managed and native resources</param>
    protected virtual void Dispose(bool disposing)
    {
        if (false == disposed)
        {
            if (disposing)
                DisposeOfSeleniumServer();

            disposed = true;
        }
    }

    private void DisposeOfSeleniumServer()
    {
        if (seleniumServerProcess != null)
        {
            try
            {
                seleniumServerProcess.Kill();
                bool result = seleniumServerProcess.WaitForExit(10000);
            }
            finally 
            {
                seleniumServerProcess.Dispose();
                seleniumServerProcess = null;
            }
        }
    }

    private bool disposed;
    private Process seleniumServerProcess;
}

Note that the class is disposable – this ensures the Selenium server is stopped even if you run tests in the debugger and then force the debugger to stop before finishing the work. The Dispose method calls DisposeOfSeleniumServer, which does the actual work of killing the process and disposing of the evidence.

NOTE: This is a second version of the code. I needed to update the old one because I noticed that when running the tests in CruiseControl.NET, the Selenium server java process was not stopped properly. The only way I could stop it is by killing it, which in general isn’t a good practice. The unfortunate side effect of this “killing” is that the CruiseContol.NET service cannot be stopped normally – it also has to be killed when you need to restart it. I’ll try to solve this problem in the future.

There are three attributes which function as filters when running tests using any of the Gallio’s test runners:

• Pending: tests which are in development and currently don’t run should be marked with the Pending attribute. This means the test runner will skip them when running the build script.
• Ignore: this attribute is used for marking tests which are never to be run (they are kept in the code as a history). In general, it is a good practice to avoid such tests – you can get the history from your source control.
• Explicit: tests marked with this attribute will only run when selected individually in the VS (Resharper, TestDriven.NET). They will not be run as part of the build script. Explicit tests are usually those which depend on a certain external system which cannot be guaranteed to be available at all times – we don’t want such tests to cause failures in our builds.

It is a good practice to supply these attributes with a string argument describing the reasons for marking the test.

UPDATE: Jeff Brown kindly provided some additional information about these attributes:

Tests marked [Ignored] and [Pending] will show as Warning annotations in the test report in addition to being skipped. In ReSharper they will also be shown with a yellow warning stripe to remind you that they are there and need to be looked at.

You can also add your own annotations to various code elements with the [Annotation] attribute.

This is my first article in the “guidelines” series I plan to write in the future. I want to maintain these guidelines separate from concrete project’s documentation since in the past I always had to copy this kind of stuff from one project’s wiki to another.

photo credit: Shahram Sharif

Introduction

Yesterday we finally managed to get our tests to run using our acceptance tests framework. I promise to write more about it some other time, but I’ll make a quick introduction now.

First let’s start with the name of the framework: Accipio. The idea of Accipio is to specify acceptance tests in an XML form which is then automatically translated into MbUnit test code. I guess you can call it a lightweight FitNesse – there’s no Wiki, all test specifications are stored in XML files (which are then source-controlled). The XML is quite simple (you can see some initial brainstorming samples here).

But that’s not what I wanted to talk about now.

Time Is Money, They Say

While running the tests we determined that the whole test process took too long. We had around 100 test cases, each of which had to wait for 10 seconds after the initial action before it could assert the test conditions, which means at least 20 minutes of test running time (and we expect much more test cases to be written in the future). We’ll try to refactor that code so that this wait period is not necessary, but nevertheless these tests are written in a way that should allow parallel execution without any negative effects. In fact the parallelization would be welcome since it mimics the "real" situation in production.

Luckily with little Googling we found a thread on gallio-dev forum called "best way to parallelize a test suite" in which Jeff Brown (Gallio arhictect) discusses a new experimental addition to Gallio – the Parallelizable attribute. It can be applied both to test fixtures and test methods. From what I discerned, the Parallelizable attribute applied on test fixtures means that two or more fixtures can run in parallel, while marking test methods Parallelizable means that two or more test methods in the same fixture run in parallel (I simplified description a little here, for more details please read the mentioned thread).

We needed the second option (parallelization of methods), so I downloaded the latest Gallio package, marked all of our test methods with Parallelizable attribute…

…and run tests with Gallio.Echo runner. So far so good – the test do run in parallel, although occasionally the runner throws some exceptions, we’ll need to investigate this further (after all, this is an experimental feature, so I’m expecting it to break once in a while .

You can set the rough number of concurrent threads that will process test code by setting DegreeOfParallelism value:

NOTE: I’ve added this code to the fixture setup method because I didn’t know any better place to put it. Also, it would be better not to hard-code it like I did, use a configuration file instead.

Conclusion

Not all tests can be executed in parallel, of course. Once we integrate Selenium into acceptance testing we will have to be careful when selecting which tests should be parallelized and which should not – until we give the Selenium Grid a try, we will have to run Web UI tests on a single thread. I guess the best thing to do is keep non-parallalizable tests in fixtures separate from parallelized ones.

photo credit: duimdog

In one of my previous posts I discussed using Assert.Inconclusive() method to mark tests as "not finished yet" during the execution of the test (as opposed to declaring test outcomes using attributes such as [Pending]).

It turns out there’s a better way to do this (which Jeff Brown kindly pointed me to):

SilentTestException allows you to mark the test with any outcome you like. Which is exactly what I was looking for.