Description
This is a (multiple allowed):
- bug
- enhancement
- feature-discussion (RFC)
Maintaining and improving the current fixture system is challenging as it is complex has many hidden features and relies on deprecated PHPUnit APIs.
Problems
- The various ways we currently manage schema results in even more complexity as handling constraints requires workarounds and hacks.
- We rely on deprecated PHPUnit APIs which will be removed. Potentially in PHPUnit 10.
- The replacement APIs in PHPUnit are not suitable for our use case.
As there are a few orthogonal problems we could address them separately, but addressing them together could unlock a better developer experience.
Integration with PHPUnit
Currently the test suite uses PHPUnit\Framework\TestListener to listen for tests being started. The test suite and test case start events are used to create schema and populate records before a test is run. This interface is deprecated, and extensions are recommended instead. Unfortunately, extensions do not have access to the testsuite or the test case objects. While they do have access to the class and method names, this data is not ideal and limits any future fixture APIs to being static.
We have had reasonable success using traits that leverage the @before annotation to do setup work. Another potential approach is to use the TestCase::setUp() to collaborate with shared global that wraps the database 'state'.
Managing Schema
Currently the test suite drops and manages schema at the table level. As fixtures are loaded/used tables are created. At the end of the test run all tables are dropped. This incurs some overhead as we need to track which tables have been created, and whether the table needs to be dropped before it is created. Compounding this complexity is the variety of ways we allow developers to define schema:
- As abstract schema in fixture classes via the
$schemaproperty. This form is not the actual schema, but an abstract representation, that can be leaky or incorrect. Having schema in fixtures, creates complexity for developers, as there is another place to manage schema as it changes. - As an 'import' from a table or a model. Using the reflection API's schema is read and then reverse engineered back into SQL. This conversion can be lossy or inaccurate.
- As an unmanaged table that does not import or generate a schema but does truncate table rows.
These various schema generation approaches require us to have a comprehensive schema generation library in CakePHP adding complexity to the database abstractions.
Potential Solutions
Managing Schema
There are several potential options when it comes to schema management.
- No schema management - We could take a path where the test suite expects all of the required database schema to exist before tests are run. The test suite would still use fixtures to create/drop rows, but all schema management would be external to tests. This option also enables users to use their chosen migration tool to create the necessary schema.
- Leverage migrations - Similar to how Laravel manages databases in tests, the migrations plugin could provide a trait that allows database schema & seed data to be 'reset' by dropping the database and running migrations again.
- In addition to option 2, offer basic SQL schema loading in CakePHP core.
- Something else?
Load schema during test bootstrap
Previous revisions of this issue proposed loading schema via the PHPUnit extension and leveraging different extensions to provide support for migrations. PHPUnit extensions have limited configuration options, and upon further consideration we can have more expressive APIs if we load schema during test bootstrap.
Test bootstrap is an ideal time to build the required schema for a test run. Datasource plugins like elastic-search can provide their own adapters and applications can use multiple adapters. Using tests/bootstrap.php to initialize schema also lets application developers extend schema creation to suite their needs if they have more custom requirements.
The general behavior of a schema loader would be:
- Drop the test database.
- Load the required schema.
An example of this approach can be found in the cakephp-test-migrator. If we adapt that interface to the migrations plugin we could have an API like:
\Migrations\TestSuite\Migrator::migrate();This method would accept additional parameters to define which connection and which plugins need migrations run:
// Each element defines a set of migrations to run.
\Migrations\TestSuite\Migrator::migrate([
// For the test connection, read migrations from the migrations/TestFolder
['connection' => 'test', 'source' => 'TestFolder'],
// Run the FooPlugin migrations on the FooConnection connection.
['plugin' => 'FooPlugin', 'connection' => 'FooConnection'],
// Run migrations in migrations/BarFolder as well.
['source' => 'BarFolder'],
...
]);For applications that want to load schema from a SQL file the Cake\TestSuite\Schema\SchemaManager class can be used. It exposes a create() method to create schema:
\Cake\TestSuite\Schema\SchemaManager::create('test', 'path/to/schema.sql');For CakePHP core tests we could use a small wrapper around phinx and a single migration that can be run on all database vendors. This would remove the burden of having to maintain 4 schema files that will inevitably diverge.
Loading Rows
In all of the above options we need to integration points. One for loading and resetting rows, and the other for potentially managing schema. Ideally loading rows can continue to use the same interfaces we use today. Rows would continue to be loaded during setUp() and tables truncated during tearDown(). Any table that had a fixture loaded during a test (either via $fixtures, loadFixtures(), or fixtureManager->loadSingle()) would be automatically truncated during teardown.
Similar to the current fixture system, data fixtures would load rows from within the extension. Using the BeforeTest hook the fixture extension can create an instance of the test case and use getFixtures() to get the fixture list. Likewise, the AfterTest hook would be used to truncate rows.
When rows are loaded foreign keys would be disabled or deferred. While this will allow invalid states to be created with some database vendors it greatly aids test suite performance, and is a behavior of our current fixture system.
Data only fixture manager
In order to maintain backwards compatibility a simpler fixture manager (Cake\TestSuite\DataFixtureManager) that implements a common interface as the current FixtureManager would be required. The required public methods for common fixture manager interface are:
loadSingle(string $name, ?ConnectionInterface $connection = null): voidLoad a single fixture based on the name.load(TestCase $test): voidLoad all fixtures for the provided test.loaded(): arrayGet a list of loaded fixtures.fixturize(TestCase $test)Would be a no-op on the data fixture manager.
DataFixtureManager would use a global registry, with a single instance per connection (managed through static methods).
Resetting state
Because data and schema will be separate we can no longer rely on the fixture manager knowing precisely which tables are used by a given test. This complicates resetting the state of the database, as the tables that need to be truncated are unknown. There are a few potential options around resetting data:
- Wrapping each test in a transaction that is rolled back during teardown. This approach is used by Symfony, Laravel and Django. It can create issues with older versions of MySQL where nested transactions are poorly supported, and can fail if tests mutate schema.
- Applying triggers to all tables and using those triggers to track which tables have mutations. This approach is used by the test suite light package to enable factory based flows. This approach requires more complex code within CakePHP as triggers and a 'tracking table' need to be installed so that cleanup can be done.
- Resetting all tables with
truncate. This approach while simple and mostly free of complications is the least performant and most likely to scale poorly for large applications.
In any of the above approaches we should aim to make the truncation strategy separate/contained from other aspects of fixture data management so that user land fixture extensions can customize/replace the approach used.
Opt-in process
Applications would opt into using the new fixture manager by using one the new PHPUnit extensions (see above). The new extensions would set constants that enable TestCase to get a reference to the new fixture manager in a setupBeforeClass hook instead of relying on the filter setting the public property. This would mean that having both the new and old PHPUnit extensions would end up with the old fixture system.
Questions
- Can the same extension be used multiple times in PHPUnit?
- How do plugins provide database schema to their users?
- Do plugins ship with migrations, or SQL files?
- How common are applications that use multiple test database connections?
Changes
- Added
env://to file loading - Added transactions around tests to rollback changes.