parallel
Creative Commons License photo credit: Shahram Sharif

Introduction

Yesterday we finally managed to get our tests to run using our acceptance tests framework. I promise to write more about it some other time, but I’ll make a quick introduction now.

First let’s start with the name of the framework: Accipio. The idea of Accipio is to specify acceptance tests in an XML form which is then automatically translated into MbUnit test code. I guess you can call it a lightweight FitNesse – there’s no Wiki, all test specifications are stored in XML files (which are then source-controlled). The XML is quite simple (you can see some initial brainstorming samples here).

But that’s not what I wanted to talk about now.

Time Is Money, They Say

While running the tests we determined that the whole test process took too long. We had around 100 test cases, each of which had to wait for 10 seconds after the initial action before it could assert the test conditions, which means at least 20 minutes of test running time (and we expect much more test cases to be written in the future). We’ll try to refactor that code so that this wait period is not necessary, but nevertheless these tests are written in a way that should allow parallel execution without any negative effects. In fact the parallelization would be welcome since it mimics the "real" situation in production.

Luckily with little Googling we found a thread on gallio-dev forum called "best way to parallelize a test suite" in which Jeff Brown (Gallio arhictect) discusses a new experimental addition to Gallio – the Parallelizable attribute. It can be applied both to test fixtures and test methods. From what I discerned, the Parallelizable attribute applied on test fixtures means that two or more fixtures can run in parallel, while marking test methods Parallelizable means that two or more test methods in the same fixture run in parallel (I simplified description a little here, for more details please read the mentioned thread).

We needed the second option (parallelization of methods), so I downloaded the latest Gallio package, marked all of our test methods with Parallelizable attribute…

[Test]
[Metadata("UserStory", "SMSUi.AddSubs.Schedule")]
[Parallelizable]
public void AddScheduledSubs()
{
    …

…and run tests with Gallio.Echo runner. So far so good – the test do run in parallel, although occasionally the runner throws some exceptions, we’ll need to investigate this further (after all, this is an experimental feature, so I’m expecting it to break once in a while ;) .

You can set the rough number of concurrent threads that will process test code by setting DegreeOfParallelism value:

[FixtureSetUp]
public void FixtureSetup()
{
    Gallio.Framework.Pattern.PatternTestGlobals.DegreeOfParallelism = 20;
}

NOTE: I’ve added this code to the fixture setup method because I didn’t know any better place to put it. Also, it would be better not to hard-code it like I did, use a configuration file instead.

Conclusion

Not all tests can be executed in parallel, of course. Once we integrate Selenium into acceptance testing we will have to be careful when selecting which tests should be parallelized and which should not – until we give the Selenium Grid a try, we will have to run Web UI tests on a single thread. I guess the best thing to do is keep non-parallalizable tests in fixtures separate from parallelized ones.