Custom execution test started event #305

plioi · 2023-12-03T20:28:29Z

Most custom IExecution implementations will make use of the built-in convenience method test.Run(...), which emits a Test Started event and naturally handles any exceptions thrown by the test method in order to unambiguously emit a Pass or Fail result. This keeps the semantics of test running natural, while letting custom execution authors avoid littering their code with excessive exception handling.

However, some custom execution implementations need to take much more fine grained control over the testing lifecycle. For instance:

Someone replicating NUnit's ExpectedException will want to let a test method throw, catch the exception, and evaluate whether or not the exception should be treated as a Pass or Fail result.
Someone with a concept of an "inconclusive" result, or an acceptance testing concept of a "pending requirement" that should be displayed as a Skip-with-reason until the first time it begins to Pass, may want to allow some test methods to throw, catch the exception, and evaluate whether to emit a Pass or Skip result.

In either case, such a user might want to start with a custom IExecution that perfectly mirrors the built-in test.Run(...) behavior, and then further customize from there. Instead of calling test.Run(...), which handles exceptions and emits Pass or Fail, they would want to call the method directly while allowing for exceptions: test.Method.Call(...). Since they'd no longer be calling test.Run(...), they'd instead call the specific event emitting methods explicitly. Prior to this PR, the best you could do is the following:

class CustomExecution : IExecution
{
    public async Task Run(TestSuite testSuite)
    {
        foreach (var testClass in testSuite.TestClasses)
        {
            foreach (var test in testClass.Tests)
            {
                //OOPS! We never emit the Test Started event!

                try
                {
                    var instance = testClass.Construct();

                    await test.Method.Call(instance);

                    await test.Pass();
                }
                catch (Exception failureReason)
                {
                    await test.Fail(failureReason);
                }
            }
        }
    }
}

Since the user had no way to emit the Test Started event, IDE runs have no opportunity to display a spinner or similar indicator on long-running tests. If you have a long-running test, you would get no visual indicator of which test was running slowly.

test.Run(...) naturally emits the Test Started event, but since we are calling test.Method.Call(...) instead, we naturally miss out on it just like we miss out on automatic Pass and Fail events. A user taking control like this would prefer to take full control.

This PR adds the simple ability to emit Test Started events, useful in such high-customization IExecution scenarios:

class CustomExecution : IExecution
{
    public async Task Run(TestSuite testSuite)
    {
        foreach (var testClass in testSuite.TestClasses)
        {
            foreach (var test in testClass.Tests)
            {
                // Emit a Test Started event, allowing IDE test runners
                // to draw attention to long-running tests.
                await test.Start();

                try
                {
                    var instance = testClass.Construct();

                    await test.Method.Call(instance);

                    await test.Pass();
                }
                catch (Exception failureReason)
                {
                    await test.Fail(failureReason);
                }
            }
        }
    }
}

(Thanks to @Suremaker for discovering the missed IDE 'spinner' behavior and for diagnosing it so clearly during our discussion on #57.)

… in Test Explorer.

…) logic, by attempting to with a custom IExecution. Because the user cannot explicitly emit the TestStarted event, long running tests no longer display a spinner.

…, as a foundation for some further customization. Because the user can explicitly emit the TestStarted event, long running tests display a spinner.

… in supporting development of test.Start().

plioi added 5 commits December 3, 2023 13:06

Ensure one example test runs long enough to warrant a visible spinner…

a9de2b8

… in Test Explorer.

Reveal that end users cannot yet fully replicate the test.RunCore(...…

13fc7af

…) logic, by attempting to with a custom IExecution. Because the user cannot explicitly emit the TestStarted event, long running tests no longer display a spinner.

Ensure that end users can fully replicate the test.RunCore(...) logic…

12f2ae0

…, as a foundation for some further customization. Because the user can explicitly emit the TestStarted event, long running tests display a spinner.

Remove temporary custom IExecution now that it has served its purpose…

086bf5f

… in supporting development of test.Start().

Remove temporary sleep.

ceb241b

plioi merged commit f816 70E3 e79 into main Dec 3, 2023

plioi deleted the custom-execution-test-started-event branch December 3, 2023 20:55

plioi mentioned this pull request Dec 6, 2023

Feature Request: Manually "Start" a test #295

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Custom execution test started event #305

Custom execution test started event #305

Uh oh!

Uh oh!

Uh oh!

Custom execution test started event #305

Custom execution test started event #305

Uh oh!

Conversation

Uh oh!

Uh oh!

Uh oh!