Blog

NUnit can be used for a variety of test approaches, including writing unit tests and integration tests. Integration tests may take a non-trivial amount of time to complete, and so a large suite can lead to long overall test run times and delayed feedback of any bugs or issues. Situations like these may benefit from splitting up the test run into multiple “chunks” or shards, where each targets a subset of the total suite.

Overview and Usage

NUnit shards tests by partitioning them up into repeatable, deterministic buckets. The number of buckets and the bucket to target within a given run can be specified on the command line using the “partition” filter. For example, the following will allow dotnet test to split a run into 3 shards and target only the first:

dotnet test -- NUnit.Where="partition == 1/3"

This pattern can then allow for a large suite to be split up across multiple nodes by having each target a separate shard:

dotnet test -- NUnit.Where="partition == 1/3"
dotnet test -- NUnit.Where="partition == 2/3"
dotnet test -- NUnit.Where="partition == 3/3"

NUnit Console also supports this:

nunit3-console {Tests.dll} --where="partition == 1/3"
nunit3-console {Tests.dll} --where="partition == 2/3"
nunit3-console {Tests.dll} --where="partition == 3/3"

This sharding, if done in parallel, can reduce total suite run times to a half, third, or more depending on the number of parallel shards. This drastically shortens feedback loops and allows for quicker responses to any potential bugs or issues.

Support

Built-in support for test sharding was added in version 3.14.0 of the NUnit framework. If running the tests using dotnet test then version 4.6 or higher of the NUnit3TestAdapter is required. If running the tests using the NUnit console then version 3.17 or higher of the console runner is required.

Software systems often require the ability to centrally configure common features, or cross-cutting concerns, like logging and other observability hooks. Many application frameworks allow for this by providing hooks into their execution pipelines to allow for registration and execution of customized middleware code. This can also be desirable in test systems. In this post I’ll show how this can be done in NUnit using the Execution Hooks feature recently introduced in version 4.5.

Overview

NUnit defines a range of available hooks within the ExecutionHookAttribute class. The hooks can be implemented by defining a custom attribute which extends this class and overriding the desired method(s). These pipeline methods all follow the “BeforeX” and “AfterX” paradigm to allow for custom code to run before or after any of the NUnit test lifecycle states. Each defined before hook has a corresponding after hook to support scoped management of resources. Some examples of these hooks are BeforeEverySetUpHook, BeforeTestHook, and AfterTestHook.

One of the many use cases for using execution hooks may be that you have a large integration test suite and wish to log any uncaught exceptions that prevented tests from completing. This particular problem can be solved by overriding the AfterTestHook method and checking the Exception property of the HookData instance which gets passed in. The custom attribute can then be configured to target an “assembly” so that it only needs to be defined once for an entire suite. Other standard attribute targets in .NET which may be useful for other scenarios are “class” or “method”.

Example

Combining these concepts may look something like below.

// Allow for this behaviour to be defined once-per-assembly
[AttributeUsage(AttributeTargets.Assembly)]
public sealed class CentralizedExceptionLoggingAttribute : ExecutionHookAttribute
{
    static ExceptionLoggingHookAttribute()
    {
        // Setup the output file
        var dir = TestContext.CurrentContext.WorkDirectory;
        File.Delete(Path.Combine(dir, "FailedTests.txt"));
    }
    
    public override void AfterTestHook(HookData hookData)
    {
        if (hookData.Exception is not null)
        {
            // Write to output file
            var dir = hookData.Context.WorkDirectory;
            File.AppendAllLines(Path.Combine(dir, "FailedTests.txt"), [hookData.Context.Test.FullName]);
            // Write to standard output
            TestContext.Out.WriteLine($"Exception thrown in test ({hookData.Context.Test.Name}). Details:");
            TestContext.Out.WriteLine(hookData.Exception);
        }
    }
}

This will output error info to the standard output stream and separately write a list of failed tests to a file for later reference or rerunning. It can be easily modified to support more complex requirements by using a standard logging library like Serilog, NLog, or log4net.

Sample usage:

[assembly: CentralizedExceptionLogging]

This is just one of many possible uses. A full list of the 10 hooks can be found in the documentation, along with several other example use cases. More information on using hooks for exception handling can also be found here.

Conclusion

In conclusion, the execution hooks feature introduced in NUnit 4.5 offers a wide range of customization and uses to gain visibility into what is happening in your test suites. It can also be used as a basis to customize test behaviours, execution contexts, and many other features.

NUnit allows for data-driven parameterized tests and does so by incorporating the data points into the test name as method arguments. For example, something like this:

[TestCaseSource(nameof(GenerateData))]
public void MyTest(int cost)
{
    Assert.Pass();
}
private static int[] GenerateData()
{
    return [14, 140 1400];
}

Will translate into three distinct tests named (by default):

• “MyTest(14)”
• “MyTest(140)”
• “MyTest(1400)”

This is concise but it’s not clear what the data points represent, especially if the test runs or reports are being reviewed by someone unfamiliar with the test implementation. It would be clearer if the parameter name could be included so that it’s clear what the number represents. Ex:

• “MyTest(cost: 14)”
• “MyTest(cost: 140)”
• “MyTest(cost: 1400)”

The argument display in the test name can be customized if desired by updating the source of the data to return TestCaseData instances instead of the raw underlying data. The TestCaseData class supports a few different ways to include the parameter name in the test method. Each of the below are built-in ways to do that.

[TestCaseSource(nameof(GenerateData))]
public void MyTest(int cost)
{
    Assert.Pass();
}
private static TestCaseData[] GenerateData()
{
    return [
        new TestCaseData(14).SetArgDisplayNames("cost: 14"),
        new TestCaseData(140).SetName("{m}{a}"),
        new TestCaseData(1400).SetName("{m}{p}")
    ];
}

One downside of this is they must specified separately for each test case, and in some cases even duplicated within a single test case. This leaves the possibility of inconsistencies over time. While NUnit doesn’t support a centralized way to support this formatting today, a custom attribute could be developed to help. For example, something like this can be applied as desired to individual method parameters to format them. Note that it requires both a parameter-level attribute as well as a custom test case source attribute.

Usage:

[CustomTestCaseSource(nameof(GenerateData))]
public void MyTest([IncludeParamName]int cost)
{
    Assert.Pass();
}
private static TestCaseData[] GenerateData()
{
    return [
        new TestCaseData(14),
        new TestCaseData(140).SetArgDisplayNames("19"),
        // Setting the name here will cause the parameter-level pretty printing to get skipped
        new TestCaseData(1400).SetName("{m}{p}")
    ];
}

Implementation:

[AttributeUsage(AttributeTargets.Parameter, AllowMultiple = false)]
public class IncludeParamNameAttribute : Attribute { }
public class CustomTestCaseSource : TestCaseSourceAttribute, ITestBuilder
{
    private readonly NUnitTestCaseBuilder _builder = new();
    private const BindingFlags PrivateInternalBinding = BindingFlags.NonPublic | BindingFlags.Instance;
    private static readonly MethodInfo? GetTestCasesForMethod = typeof(TestCaseSourceAttribute)
            .GetMethod("GetTestCasesFor",
            PrivateInternalBinding,
            [typeof(IMethodInfo)]
            );
    public CustomTestCaseSource(string source) : base(source) { }
    IEnumerable<TestMethod> ITestBuilder.BuildFrom(IMethodInfo method, Test? suite)
    {
        var GetTestCasesFor = GetTestCasesForMethod!.CreateDelegate<Func<IMethodInfo, IEnumerable<ITestCaseData>>>(this);
        var argDisplayNamesProperty = typeof(NUnit.Framework.Internal.TestParameters)
            .GetProperty("ArgDisplayNames", PrivateInternalBinding);
        var methodParams = method.MethodInfo.GetParameters();
        var paramNameDisplayMap = methodParams.Select(o => o.GetCustomAttribute<IncludeParamNameAttribute>() is not null).ToArray();
        var doShowParameterNames = paramNameDisplayMap.Any(o => o);
        int count = 0;
        foreach (ITestCaseData parms in GetTestCasesFor(method))
        {
            // Check if TestName is set or not as ArgDisplayNames can not be used if TestName is set
            if (doShowParameterNames && parms.TestName is null && parms is TestCaseData tcd)
            {
                var displayNames = (string[]?)argDisplayNamesProperty?.GetValue(tcd) ?? new string[methodParams.Length];
                var hasChangedNames = false;
                for(var i = 0; i < displayNames.Length; i++)
                {
                    var displayName = displayNames[i] ?? parms.Arguments[i]?.ToString() ?? "null";
                    if (paramNameDisplayMap[i])
                    {
                        displayNames[i] = $"{methodParams[i].Name}: {displayName}";
                        hasChangedNames = true;
                    }
                    else
                    {
                        displayNames[i] = displayName;
                    }
                }
                if (hasChangedNames)
                {
                    argDisplayNamesProperty?.SetValue(tcd, displayNames);
                }
            }
            count++;
            yield return _builder.BuildTestMethod(method, suite, (TestCaseParameters)parms);
        }
        // BELOW WAS COPIED FROM THE BUILT-IN TESTCASESOURCE IMPLEMENTATION
        // -----------------------------
        // If count > 0, error messages will be shown for each case
        // but if it's 0, we need to add an extra "test" to show the message.
        if (count == 0 && method.GetParameters().Length == 0)
        {
            var parms = new TestCaseParameters();
            parms.RunState = RunState.NotRunnable;
            parms.Properties.Set(PropertyNames.SkipReason, "TestCaseSourceAttribute may not be used on a method without parameters");
            yield return _builder.BuildTestMethod(method, suite, parms);
        }
    }
}

Disclaimer: This was written as a PoC for a one-off purpose and may still have a few edge cases not accounted for but it should be stable enough for the majority of cases. Enjoy!

NUnit supports several ways to assert against data. The recommended one, constraint-based, allows for a more naturally-read syntax which supports the flexible chaining of conditions to prove simple or complex facets about the system.

It follows the “Assert that” format:

Assert.That(actualValue, Is.EqualTo(expectedValue));

With the following components (listed left to right):

• actualValue : The actual output of the system under test which you wish to validate
• Is : A starting clause. The most common is Is but NUnit also defines Has, Does, and others to allow for readable tests
• EqualTo() : A example function which returns the contraint which validates your data. EqualTo returns an instance of an EqualConstraint class which will internally contains validation logic. Other examples include LessThan() or Even.
• expectedValue : The data to compare actualValue to using the rules defined by the constraint.

NUnit also contains built-in operators. For example, checking inequality is a matter to prepending the Not operator in front of EqualTo():

Assert.That(actualValue, Is.Not.EqualTo(expectedValue));

Similarly, if a situation requires check a characteristic of a value instead of comparing then a unary constraint like Even could be used:

Assert.That(actualValue, Is.Even);

The built-in constraints will likely meet 99.9% of use cases, but there may be some domain-specific rules which aren’t covered out-of-the-box. For example, a math-oriented program may wish to validate that a number is prime. It would be very nice if a test could be written to read:

Assert.That(actualValue, Is.Prime);

NUnit supports this through custom constraints. Custom constraints are classes which extend NUnit’s own Constraint class.

A PrimeConstraint may look like:

public class PrimeConstraint : Constraint
{
    public override string Description => "A prime value";
    public override ConstraintResult ApplyTo<TActual>(TActual actualValue)
    {
        var actualInt = Convert.ToInt32(actualValue);
        ArgumentOutOfRangeException.ThrowIfLessThanOrEqual(actualInt, 0, nameof(actualInt));
        for (int i = 2; i <= (int)Math.Sqrt(actualInt); i++)
        {
            if (actualInt % i == 0)
            {
                // Not prime if we've found an evenly divisible factor
                return new ConstraintResult(this, actualValue, false);
            }
        }
        return new ConstraintResult(this, actualValue, true);
    }
}

In addition to extending from the Constraint class, the class must implement an ApplyTo<TActual>() method which will validate the actualValue originally passed into Assert.That(). Hooking this into the NUnit syntax tree is then very easy thanks to the new C# 14 extension members feature. Adding a new function onto NUnit’s static Is class and adding a new property onto the ConstraintExpression class can be achieved in one line each.

public static class ConstraintExtensions
{
    extension(NUnit.Framework.Is)
    {
        public static Constraint Prime => new PrimeConstraint();
    }
    extension(ConstraintExpression expression)
    {
        public Constraint Prime => expression.Append(new PrimeConstraint());
    }
}

And that’s it. They can be used in tests seamlessly afterwards as if they were part of NUnit itself.

[Test]
public void Test1()
{
    Assert.That(5, Is.Prime);
    Assert.That(4, Is.Not.Prime);
}

NUnit 4 has added support for cooperatively cancelling long-running tests in several scenarios. Cooperative cancellation is the preferred way to end any long-running operation in .NET as it allows for the graceful ending of operations, however it requires explicit coordination by calling code to do this. This coordination is handled, in part, by the passing of a CancellationToken into the potentially long-running method. The cancellation token can be signaled to a “cancelled” state to allow the long-running operation to react and gracefully end itself.

For example, the below code which must get an external resource over HTTP where a cancellationToken is passed in as the last parameter. This GetAsync() method will exit early when the operation is cancelled.

        var httpClient = new HttpClient();
        await httpClient.GetAsync("https://server", cancellationToken);

But where does this cancellationToken come from? It’s possible to construct the token and manage it from a CancellationTokenSource yourself, however frameworks will often have a mechanism to do this for you.

NUnit supports cooperative cancellation in a few ways, the simplest of which is through the CancelAfter attribute. This attribute will indicate to NUnit that it should manage a cancellation token on behalf of the test. The cancellation token itself can be used by the test in one of two ways:

Read from the test context:

        [Test]
        [CancelAfter(CooperativeTimeoutMilliseconds)]
        public async Task WithCooperativeCancellation_Context()
        {
            var delay = TimeSpan.FromMilliseconds(600);
            var timer = Stopwatch.StartNew();
            await Task.Delay(delay, TestContext.CurrentContext.CancellationToken);
            timer.Stop();

            var expectedDelay = Math.Min(delay.Milliseconds, CooperativeTimeoutMilliseconds);

            Assert.That(timer.ElapsedMilliseconds, Is.EqualTo(expectedDelay).Within(50));
        }

Passed by NUnit as an argument into the method:

        [Test]
        [CancelAfter(CooperativeTimeoutMilliseconds)]
        public async Task WithCooperativeCancellation_Argument(CancellationToken cancellationToken)
        {
            var delay = TimeSpan.FromMilliseconds(600);
            var timer = Stopwatch.StartNew();
            await Task.Delay(delay, cancellationToken);
            timer.Stop();

            var expectedDelay = Math.Min(delay.Milliseconds, CooperativeTimeoutMilliseconds);

            Assert.That(timer.ElapsedMilliseconds, Is.EqualTo(expectedDelay).Within(50));
        }

Both conventions are supported by NUnit and will allow a long-running test to respond to and gracefully cancel any long-running tasks in flight.

Tests, especially unit tests, should be reliable and reproducible. System or integration tests can, however, exercise many different parts of a codebase or even a network. This increase in the number of moving pieces can potentially lead to decreased test reliability. NUnit’s Retry attribute was added to support cases where a developer may believe a test could have a reasonable chance at passing if retried after an initial failure.

For example, the below test will fail about half the time it is run, but it will be run up to 3 times before being finally marked as failed in the run.

[Test, Retry(3)]
public static void TestRandomlyEven()
{
    Assert.That(Random.Shared.Next(), Is.Even);
}

These retried failures, however, have only been for when a test fails an assertion. Test failures due to unhandled exceptions are not retried. So the below test will also fail about half the time it is run, but those failures will not be retried and the test will immediately be treated as failed.

[Test, Retry(3)]
public static void TestRandomlyEven()
{
    if (Random.Shared.Next() % 2 == 0)
        throw new InvalidOperationException("Odd number.");
    Assert.Pass();
}

This may be desirable based on how a test is written and if you want to be alerted to any instability or potential flakiness. There may be other cases, such as detailed system integration tests, where one may want network or database exceptions to be retried. Writing a custom attribute to support this is quite easy as most of the pieces are already exposed as public types within NUnit. The below was written against NUnit 4.4 but should work on any lower version too.

[Test, RetryOnExceptionAttribute(3)]
public static void TestRandomlyEven()
{
    if (Random.Shared.Next() % 2 == 0)
        throw new InvalidOperationException("Odd number.");
    Assert.Pass();
}

public class RetryOnExceptionAttribute(int tryCount) : NUnitAttribute, IRepeatTest
{
    public TestCommand Wrap(TestCommand command)
    {
        return new RetryAttribute.RetryCommand(new FailOnExceptionCommand(command), tryCount);
    }

    private class FailOnExceptionCommand(TestCommand innerCommand) : DelegatingTestCommand(innerCommand)
    {
        public override TestResult Execute(TestExecutionContext context)
        {
            try
            {
                return innerCommand.Execute(context);
            }
            catch (Exception ex)
            {
                context.CurrentResult.SetResult(ResultState.Failure, ex.Message, ex.StackTrace);
                context.CurrentResult.RecordTestCompletion();
                return context.CurrentResult;
            }
        }
    }
}

Fortunately the ability to retry on specific exceptions will be available as part of NUnit 4.5. A new RetryExceptions property will be added to the attribute that can be given an array of Types of exceptions to retry.

[Test, Retry(3, RetryExceptions = [typeof(InvalidOperationException))]
public static void TestRandomlyEven()
{
    if (Random.Shared.Next() % 2 == 0)
        throw new InvalidOperationException("Odd number.");
    Assert.Pass();
}

The entries are treated as base classes so retrying all exceptions will become as simple as:

[Test, Retry(3, RetryExceptions = [typeof(InvalidOperationException))]
public static void TestRandomlyEven()
{
    if (Random.Shared.Next() % 2 == 0)
        throw new InvalidOperationException("Odd number.");
    Assert.Pass();
}

A big thank you to manfred-brands for having added this feature recently.

In my previous post I showed how to use awaitable TestCase or Value sources in NUnit 3.14. NUnit 4 continues the story of adding async support by also allowing TestCaseSource, ValueSource, or TestFixtureSource to return an IAsyncEnumerable.

public class AsyncTestSourcesTests
{
    [TestCaseSource(nameof(MyMethodAsync))]
    public void Test1Async(MyClass item)
    {
    }

    public static async IAsyncEnumerable<MyClass> MyMethodAsync()
    {
        using var file = File.OpenRead("Path/To/data.json");
        await foreach (var item in JsonSerializer.DeserializeAsyncEnumerable<MyClass>(file))
        {
            yield return item;
        }
    }

    public class MyClass
    {
        public int Foo { get; set; }
        public int Bar { get; set; }
    }
}

As with IEnumerable-backed sources, NUnit will lazily enumerate the collection to avoid bringing all the objects into memory at once, instead only generating the test or value sources when needed.

Many async enumerable operations also require async disposal of the underlying resource after enumerating the collection. NUnit will also take care of calling the dispose method to ensure everything is cleaned up properly. In the event that both DisposeAsync and Dispose methods are present then NUnit will only call the asynchronous DisposeAsync method and not the Dispose method.