Enhancements

This is a minor maintenance release in response to the release of the .NET 9 SDK, the net9.0 target framework, and C# 13.

Enhancements

Add target framework net9.0 as officially supported.
string and char serialization in parameterized case names respects the \e escape sequence introduced in C# 13. For earlier C# versions, continue to use the supported sequence \u001B. #359

Housekeeping

Builds use the .NET 9 SDK.
Remove the "terminal logger" command line flag from the build script, as it is the default behavior starting in the .NET 9 SDK.
Drop detailed Target Framework Change instructions from CONTRIBUTING.md, as we do not recommend third parties attempt this kind of change and because the content of the instructions always lags the real work encountered for each such change.

This introduces stability to the test runner under parallel execution.

Prior to this release, even users intending single threaded execution could experience severe failure for seemingly-harmless uses of System.Console. Console capture has been phased out. As tempting as it is to request support for console capture, the concept is broken by definition and is no longer supported in the interest of stability. Standard output will still appear when running dotnet fixie at the command line, and so will still appear in CI build logs, but will not be captured per test nor echoed to reports' individual test results. You are still free to redirect System.Console.Out within your tests to capture strings and assert on them or witness them under the debugger.

Prior to this release, if a user's custom IExecution attempted to run tests 8000 in parallel, the test runner would become clearly unstable resulting in symptoms like unexecuted tests, never-ending spinner icons in visual IDE test runners, or unpredictable exceptions as mutable state was accessed across threads. You can now opt into parallelism either with ParallelExecution or a custom IExecution modeled after ParallelExecution.

Whether you opt for sequential execution or parallel execution, reports now run in a dedicated reporting thread. This allows custom report authors to write their report code with confidence that they can process well-ordered test result events, and improves throughput even for sequential test runs.

Warning

Parallel test execution is not a zero-cost "Make It Go Faster" button. Choosing to run tests in parallel is a serious architectural decision affecting the design of sound tests. We cannot debug your interdependent tests. We cannot debug clever custom IExecution implementations.

Note

In highly customized parallel IExecution implementations, you may find the need to perform test duration measurements yourself, passing them to the new optional TimeSpan parameters, as the meaning of "How long did this test take?" becomes extremely debatable in this context.

Breaking Changes

Remove support for deprecated target frameworks netcoreapp3.1, net6.0, net7.0 #307 #309 #310
Drop the Shuffle convenience method in favor of the new .NET 8 Shuffle implementation. #314
Modernize test assembly main method #319
- Although technically a breaking change, this is extremely unlikely to affect real users.
Stability Under Parallelism
- Drop deprecated console capture #326
  - See the PR for rationale.
  - TestCompleted reporting events (TestFailed, TestPassed, TestSkipped) lose their Output parameter.
- Dedicated reporting thread #327
  - Although technically a breaking change, in that your custom report now runs in a dedicated thread distinct from test execution, this is extremely unlikley to affect real users.
- Parallelism stability #328
  - Custom IExecution is now allowed to run tests in parallel. See ParallelExecution for a working example.
  - Test methods Pass, Skip, and Fail now have optional TimeSpans so that highly custom/parallel IExecution can provide their own opinion on what it means to measure a test's duration.
    - Usages of the commonly used Test.Run still have a natural test duration calculated automatically.
Reports
- Improve target framework reporting #330
  - Improves the value of TestEnvironment.TargetFramework when possible. Atypical custom reports relying on specific string values may experience the change.
- Remove deprecated AppVeyor integration. #343
- Drop XML Reports #344
  - This legacy built-in report has already been superceded by guidance in the documentation around textual formats like JSON and XML.

Fixes

Refine case names #355
- Fixes escaping of unlikely control/whitespace characters in parameterized test names.
Modernize CommandLine.Partition #347
- Fixes handling of unlikely usages with multiple bare -- parameters.

Enhancements

Improve the readability of console output by bringing the final tally line closer to the "Running " line, so that it is easier to pair them when scanning a console log of multiple test projects in series. Prior to this change it was too easy to visually pair the tally line of test project N to the opening line of test project N+1. 8e5593f
Modernize TeamCity Report #345
Method invocation error signatures #349
When dotnet fixie provokes a build of the test project, display Building Example.Tests the same way we later display Running Example.Tests, for consistency. f549418
Promote ParallelExecution to a built-in option. #356

Third-Party Contribution Management

GitHub Actions workflows skip deployment attempts (which would safely fail anyway) for forks #350
Updated Contribution Guidelines, PR Template, Issue Templates
The publish script signals warning/error states when publishing is skipped for a logged reason. cb4f296
Configure warning/error severity in a dedicated PropertyGroup with a comment explaining the reasoning, to reduce surprise changes in third party contributions. f33b695

Preparation for `Fixie.Assertions` Library

Fixie has never and will never have a built in assertion library. Choice of assertion library is orthogonal to choice of test framework. During the old .NET Core conversion, our dependency on a .NET Frameork-only assertion library needed to be phased out, and the solution was a minimal internal-use assertion library within our self-testing Fixie.Tests assembly. With recent additions to C# and .NET, this matured to a point where it could be polished and extracted to an independent library of its own, Fixie.Assertions. At the time of this writing, Fixie.Assertions is nearing a public prerelease but is not yet available or recommended for production use.

Phase out assertion params keyword #331
Descriptive assertions #332
Improve string assertions #334
String assertion messaging #335
Collection assertions #336
Assertion failure readability #338
Flexible assertions #340
Test coverage and improved messaging for ShouldThrow<TException>. #341
Reorganize assertions in preparation for extracting a library #351
Adopt Fixie.Assertions #353
Upgrade Fixie.Assertions #354

Housekeeping

C# / .NET Modernization
- Apply File-Scoped Namespaces #311
- Using Directive Placement #312
- Apply global and implicit using directives #313
- Apply collection literal [] for all empty collections. #316
- Apply collection literal [...] for all compatible populated collection initialization. #317
- Apply collection literals in nontrivial situations. #318
- Modernize pipe message models' nullability declarations. #321
- Non null pipe deserialization #322
- Convert dotnet fixie to Top Level Statements #323
Parallel self testing #329
Update actions/setup-dotnet to the latest version. #337
Simplify test comparison fakery #339
Modern supply chain management #342
Calculate TestEnvironment.RootPath instead of relying on the caller to pass it in correctly. #346
Simplify runner filtering #348

Add Target Framework net8.0 and Adapt to Modern Tooling

#298 While consumers could already target net8.0 themselves, officially targeting this lets us adapt to a few changes in .NET, primarily around our ability to provide test failure stack traces that omit distracting test runner implementation details.
#297 Enable "Artifacts Output Layout". While this only affects use of this repo, it helped to prove that end users can opt into this feature as well without breaking Fixie's ability to locate and run your test assemblies in their new location.
295925f NuGet packages include the README document.
#303 Modernize NuGet Packages to meet the latest guidance in the .NET 8 era:
- Enable Source Link.
- Publish a separate symbols package, using the modern supported snupkg format.
- Avoid redundantly embedding symbols in the dlls.
- Enable "Deterministic Builds" at CI time.

Stability for 3rd Party Test Runners

Third party test runners such as those provided by ReSharper or Rider can have varying support for the full VSTest API, and may encounter regressions of their own over time. In this version, we phased out some insufficient workarounds and improved the ability to detect and react to known problem areas.

#300 Phase out in-process Test Adapter runs. Our use of in-process Test Adapter runs was always a workaround for problematic third party test runners who fail to attach a debugger, and was in fact insufficient in real world projects. End users using JetBrains test runners could experience wildly inconsistent behavior around loading dependent assemblies with no clear path to success. We removed the workaround since it was not in fact working around anything.
#302 Instead of the poor workaround removed in #300, the Test Adapter now detects when a third party fails to attach a debugger, and presents the user with feedback in the test results tree:
1. Exactly what went wrong, preventing their breakpoint from being hit.
2. Exactly what to do right now to hit their breakpoint and get back to the task at hand.
3. Exactly what evidence to provide to the third party test runner's development team, to guide them towards complete compatibility with the API defined by the VSTest team.
- This should help to keep end users unblocked and to prevent Issues from being submitted to the wrong repo.
0d813f5 Phase out deprecated use of the difficult-to-format ProcessStartInfo.Arguments string, instead using its replacement ArgumentList property. Reduces the risk of command line argument serialization errors across varying operating systems.

Other Fixes

--tests Command Line Argument:
- dfcbc35 Rename the environment variable which supports the --tests command line argument, for consistency with other environment variables and to avoid any debate over whether ":" is a valid environment variable name character in cross-platform scenarios.
- 7e5da96 When the --tests filtering argument is provided but matches zero tests, run zero tests and accurately report that as a failure instead of accidentally running ALL tests.
- #304 Fix --tests pattern matching semantics.
17d5c63 Change fatal error exit code from -1, which is nonstandard/mishandled/misreported on some platforms, to 2. -1 can be received by the calling process as 255 on Linux, for instance, because the platform has rules around exit codes being treated as unsigned and having only certain bits treated as the exit code.
76d926b When resolving generic arguments for parameterized tests, support passing a concrete value to Nullable<T> parameters by inferring T to be that concrete value's type.
#305 Custom IExecution can emit "Test Started" events. This is useful for custom IExecution implementations that take fine-grained control over the test lifecycle, allowing IDE runs to display a spinner on long-running tests just like you already get for other IExecution.

This release adds targets for net6.0 and net7.0. Earlier releases worked for consumer solutions that targeted those frameworks. This release includes explicit targets for those frameworks and includes quality of life enhancements related to officially supporting them.

#267 XML documentation comments are now included in the package.
#273 Include Directory.Build.props to the solution items for easier access.
#284 When running from the command line, the fatal error "No tests found." is highlighted in red.
#289 Upgrade MinVer.
#290 Add target framework net6.0.
#291 Update vscode tasks for realistic usage.
#292 Simplify serialization as used cross-process in IDE runs.
#293 Demonstrate how consumers can integrate with GitHub Actions Job Summaries.
#294 Add target framework net7.0.
Expose extension method Exception.StackTraceSummary() so that custom reports can use the same cleaned-up stack trace as built-in reports, so that custom reports can benefit from stack trace presentation depicting the test method itself as the start of execution.
Upgrade dependencies: Mono.Cecil, Microsoft.NET.Test.Sdk

Fixes:

Earlier 3.x releases used a Test Adapter that would load and run the test assembly in-process. In practice, this worked for simple consumers but could cause problems for users whose system-under-test generated and used assemblies of their own at runtime.

Version 3.2.0 reverts back to the tried-and-true two-process approach we used during all ov the 2.x line. The Test Adapter kicks off the test assembly as a separate process, capable of loading all its dependencies naturally as any console application would. An extra challenge this time, though, is that non-Visual Studio test runs must fall back to the in-process approach when running tests under the debugger, as the VsTest-provided LaunchProcessWithDebuggerAttached is only reliable inside Visual Studio runs.

Additions:

New property TestEnvironment.TargetFramework provides convenient access to the test assembly's [TargetFramework] name (such as ".NETCoreApp,Version=v3.1"), so that reports can display this context during runs of multi-targeted test assemblies.

@garoyeri

Even before this release, Fixie implemented a "Test Adapter" which made it compatible with Visual Studio Test Explorer. However, the adapter was atypical compared to those of other frameworks, which limited the ability for other test runners to detect it.

By better-matching the intended patterns of all .NET Test Adapters, Fixie is now supported in the following IDE test runners:

Visual Studio Test Explorer
JetBrains ReSharper
JetBrains Rider
Visual Studio for Mac
Visual Studio Code via dotnet test-compatible extensions.

(For JetBrains tools, be sure to go into their respective settings under Unit Testing > VSTest and check "Enable VSTest adapters support" followed by a restart of the IDE.)

Special thanks to @garoyeri, @khalidabuhakmeh, and @estrizhok for guidance and troubleshooting.

Fixie 3.0 is a major technology refresh and API overhaul, meant to address many long-standing pain points and limitations.

Major Breaking Changes

Dropped support for legacy frameworks net4x and netcoreapp2.x.
Separated the Fixie package into Fixie and Fixie.TestAdapter. Fixie.TestAdapter depends on Fixie.
The Fixie.Console package becomes a "local tool" providing the dotnet fixie command.
No longer invoke IDisposable.Dispose() on customization types or test classes. If you want to use IDisposable beyond its intended use such as for test "tear down" operations, do so explicitly in your custom IExecution.
The default discovery behavior now only considers instance methods, excluding static methods. Custom discovery can still continue to consider static and nonstatic as you see fit.
New convention API for discovery and execution phases, giving far more control over the test project lifecycle.
Convention reuse: begin your customization with an ITestProject implementation, which can register conventions defined in any assembly.

The `dotnet fixie` Local Tool

dotnet fixie can run multiple test projects by name from the root folder using * wildcards.
dotnet fixie command line arguments support single letter abbreviations where unambiguous. In particular you can now say -c instead of --configuration just like other dotnet commands.
To enable end users to parse custom command line arguments however they see fit, such as with the library of their choice or when taking part in the IOptions pattern, we simply pass the custom arguments array to their customizations rather than attempting to bind them to some model via reflection.
New --tests argument allows running a subset of tests from the command line using patterns. Use a * wildcard, with an implicit wildcard at the start, end, and between consecutive uppercase letters: dotnet fixie MyTestProject --tests MTC.*Validate will match tests such as "MyNamespace.MyTestClass.ShouldValidateStuff()"

Other Fixes and Enhancements

Applied Nullable Reference Type hints.
Fixed reporting of test case duration in Test Explorer.
Accurately report the start of each test, giving the Test Explorer UI an opportunity to show spinners on long running tests.
Speed and memory optimizations for large test assemblies, when reporting line numbers to Test Explorer.
Improved generic type argument resolution on parameterized tests.
Support for test methods declared to return ValueTask.
Reports which need to write to the standard output stream or rely on the a consistent notion of the test assembly's directory are safe from tests or systems-under-test which redirect standard output or change the current directory.
Improve the presentation of test failures when the test's exception originates within an F# Async<T'> operation, by presenting the original exception rather than a wrapping of that original exception.
Skip reasons are marked non-null. If you try to thwart this, a default reason string will be substituted.
Custom reports.

Fixes #246 "Enable RollForward = Major to run on later .NET Core versions": For the 2.x line, the Fixie.Console installed by DotNetCliToolReference respects major version roll-forward, so that consumers no longer have to install and old SDK just for invoking tests compiled with more recent SDKs.

The previous release supported .NET Core 3 test projects for most use cases, but as a happy coincidence. No action had to be taken in Fixie's implementation, as the existing support for .NET Core 2 carried forward naturally.

Fixie 2.2.1 includes official support for such test projects. However, because netcoreapp3.0 has already reached End-of-Life, netcoreapp3.0 is not supported. As a Long-Term-Support release, netcoreapp3.1+ is supported. If you need to target netcoreapp3.0, we recommend sticking with Fixie 2.2.0 until you can upgrade to a target framework supported by Microsoft.

Enhancements for .NET Core Test Projects:

Improved readability of test failure stack traces for netcoreapp2.1+.
When consumers target netcoreapp3.1+, internal messaging serialization uses System.Text.Json.JsonSerializer instead of the deprecated DataContractJsonSerializer, for its simpler API and improved performance.
Update VS Test Platform dependencies to take advantage of their performance improvements.
Fixed the ability to run tests under the debugger in Visual Studio's Test Explorer for netcoreapp3.1+ test projects.

Enhancements when running on Azure DevOps:

When a test assembly completes, the test run as a whole is marked as completed. Doing so lights up several elements in the Azure DevOps "Tests" screen: assembly-level success/failure icons, assembly-level durations, and high-level statistics.
When posting skipped test results to Azure DevOps, use the 'outcome' value 'Warning'. This allows the Azure DevOps UI to properly categorize the result, to display a warning icon by the result, and to pass through the skip reason as a visible message. Don't skip tests.
#211: Improved logging and error handling around Azure DevOps API failures, allowing test runs to complete with full details in the build log and an accurate exit code even when the Azure DevOps API fails completely.

Other Enhancements:

Because AppVeyor relies on test case name + the assembly FileName to distinguish unique results, include the target framework in that FileName. This way, test results are correctly distinguished from each other during test runs of multitargeted test assemblies.
Modernize deprecated elements in NuGet packages.

When running under Azure DevOps, if you have configured your Pipeline to allow access to the Azure DevOps test reporting API, Fixie will report each test result so that they will appear on the build's "Tests" screen. If you forget that configuration, the console output will provide guidance based on the Azure DevOps documentation (https://docs.microsoft.com/en-us/azure/devops/pipelines/build/variables#systemaccesstoken)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Enhancements

Housekeeping

Uh oh!

Breaking Changes

Fixes

Enhancements

Third-Party Contribution Management

Preparation for `Fixie.Assertions` Library

Housekeeping

Uh oh!

Add Target Framework net8.0 and Adapt to Modern Tooling

Stability for 3rd Party Test Runners

Other Fixes

Uh oh!

Uh oh!

Uh oh!

Contributors

Uh oh!

Major Breaking Changes

The `dotnet fixie` Local Tool

Other Fixes and Enhancements

Uh oh!

Uh oh!

Enhancements for .NET Core Test Projects:

Enhancements when running on Azure DevOps:

Other Enhancements:

Uh oh!

Uh oh!

Releases: fixie/fixie

Add Target Framework `net9.0`

Enhancements

Housekeeping

Uh oh!

Stability Under Parallel Execution

Breaking Changes

Fixes

Enhancements

Third-Party Contribution Management

Preparation for Fixie.Assertions Library

Housekeeping

Uh oh!

Add `net8.0`, Stability for 3rd Party Test Runners

Add Target Framework net8.0 and Adapt to Modern Tooling

Stability for 3rd Party Test Runners

Other Fixes

Uh oh!

Add Target Frameworks `net6.0` & `net7.0`

Uh oh!

Assembly Loading Fixes for IDE Runs

Uh oh!

IDE Integrations

Contributors

Uh oh!

Fixie 3.0

Major Breaking Changes

The dotnet fixie Local Tool

Other Fixes and Enhancements

Uh oh!

2.2.2

Uh oh!

Enhancements for .NET Core 3.1 Test Projects, Azure DevOps

Enhancements for .NET Core Test Projects:

Enhancements when running on Azure DevOps:

Other Enhancements:

Uh oh!

Azure DevOps Integration

Uh oh!

Preparation for `Fixie.Assertions` Library

The `dotnet fixie` Local Tool