The aim of this project is to provide a set of compatibility acceptance tests for libraries that implement the GraphQL specification. Since GraphQL server implementations are written in different programming languages, this project defines test cases in a language independent format (YAML).
Using this test suite has following advantages:
- The users and the author of a GraphQL library can be more confident that a library is compliant to the GraphQL specification and if not, which parts of the library are not compliant
- It reduces amount of work a library implementor needs to do
- Makes it much easier for existing implementations to keep up with the specification changes
GraphQL implementations that have already implemented or currently working on the implementation of graphql-cats driver:
If you are working on the driver for graphql-cats for your project, we would appreciate if you could create a PR and add it into this list.
At the moment this project finds itself in an infant phase. Any contributions are very welcome! Please join us in a gitter chat. If you would like to join this effort and be part of the project, please let us know.
There are still a number of open questions that we need to figure out together:
scenarios folder contains a set of *.yaml and *.graphql files. Every subfolder represents a particular part of GraphQL specification.
*.yaml files contain scenarios with a list of test cases. *.graphql files contain SDL schema definitions that are used in some of the
scenarios (they are explicitly referenced).
Every scenario is a YAML file with following structure:
scenario- String - the name of this scenariobackground- Object (optional) - common definitions used by all of the testsschema- String (optional) - inline GraphQL SDL schema definitionschema-file- String (optional) - SDL schema definition file path relative to the scenario filetest-data- Object (optional) - test data used for query execution and directivestest-data-file- String (optional) - test data file path relative to the scenario file. File can be in either JSON or YAML format.
tests- Array of Objects - list of testsname- String - a name of the testgiven- Object - input information for the testquery- String - the GraphQL query to execute an action againstschema- String (optional) - inline GraphQL SDL schema definitionschema-file- String (optional) - SDL schema definition file path relative to the scenario filetest-data- Object (optional) - test data used for query execution and directivestest-data-file- String (optional) - test data file path relative to the scenario file. File can be in either JSON or YAML format.
when- Object - action that should be performed in the test. See the Actions section for a list of available actions.then- Object | Arrays of Objects - assertions that verify result of an action. See the Assertions section for a list of available actions.
Definitions in the given part of a test may override definitions defined in the background section.
- Query parsing
parse- Boolean - just parses the query
- Query validation
validate- Array of Strings - the list of validation rule names to validate a query against. This action will only validate query without executing it.
- Query execution
execute- Boolean | Object - executes a queryoperation-name- String (optional) - the name of an operation to execute (in case query contains more than one)variables- Object (optional) - variables for query executionvalidate-query- Boolean (optional) -trueif query should be validated during the execution,falseotherwise (trueby default)test-value- String (optional) - the name of a field defined in thetest-data. This value should be passed as a root value to an executor.
- Validation/Parsing is successful
passes- Boolean - verifies that validation was successful. Only applicable in conjunction with query validation/parsing action
- Parsing syntax error
syntax-error- Boolean - query contains a syntax error. Only applicable in conjunction with query parsing action (No text matching takes place, it just verifies the fact that there is a syntax error)
- Data match
data- Object - compares thedataobject with the result of a query execution. Only applicable in conjunction with query execution action
- Error count
error-count- Number - number of the errors in execution/validation results
- Error code match
error-code- String - execution/validation results contains an error with provided code. Each error code must be described in the error mapping. See Error Mapping section for more details.args- Object (optional) - arguments for an error code (might be used as the placeholders in a human-readable error message representation)loc- Array of Objects | Array of Arrays of Numbers (optional) - a list of error locationsline- Numbercolumn- Number
- Error contains match
error- String - execution/validation results contain provided error message (provided error message may contain only part of the actual message)loc- Array of Objects 8000 | Array of Arrays of Numbers (optional) - a list of error locationsline- Numbercolumn- Number
- Error regex match
error-regex- String - execution/validation results contain provided error message (uses provided regular expressions to match an error message)loc- Array of Objects | Array of Arrays of Numbers (optional) - a list of error locationsline- Numbercolumn- Number
- Execution exception contains match
exception- String - execution may throw an exception during the execution (for instance, if operation name is not provided, but query contains more than one named operation). This assertion verifies the message of this exception (provided error message may contain only part of the actual message). Only applicable in conjunction with query execution action
- Execution exception regex match
error-regex- String - execution may throw an exception during the execution (for instance, if operation name is not provided, but query contains more than one named operation). This assertion verifies the message of this exception (uses provided regular expressions to match an error message). Only applicable in conjunction with query execution action
Error mapping is located in scenarios/error-mapping.yaml. It is a YAML file which defines a mapping between
an error code and the human-readable message which can be seen in the reference implementation for this code. It also provides
additional meta-information for every individual error code, like links to the specification and reference implementation which define
these errors.
Every error code has following structure in the mapping:
- message - String - human-readable message which can be seen in the reference implementation for this code. If error code takes arguments,
then arguments might be used in the message with as the placeholders (placeholder syntax:
${argumentName}). For example (fieldNameandtypeare the arguments):Field "${fieldName}" must not have a selection since type "${type}" has no subfields. - references
- spec - String - link to a place in the specification which defines this error.
- implementation - String - link to a place in the reference implementation which defines this error.
All test cases and scenarios are self-contained. This means that they contain the schema definition, query and test data for an execution.
Schema definition expected to be materialized in executable form. By default, fields' resolve function must return the values provided
with the test-data/test-data-file and test-value properties which are defined in a scenario file. Some fields may have special behaviour
which is defined via schema directives which you can find in the next section.
directive @resolveString(value: String!) on FIELD_DEFINITIONResolves String field with provided value. Value may contain a placeholders which refer to the field arguments and must be replaced during the field resolution.
Example:
for given schema definition:
type Query {
article(title: String): String @resolveString(value: "Test article with title '$title'")
}
schema {
query: Query
}and query:
{
article(title: Foo)
}result of an execution should produce following JSON:
{
"data": {
"article": "Test article with title 'Foo'"
}
}directive @argumentsJson on FIELD_DEFINITIONResolves String field with compact JSON representation of all field arguments.
directive @resolvePromiseString(value: String!) on FIELD_DEFINITIONResolves String field with provided value which may contain argument placeholders. Internally field should be resolved with successful
promise or an equivalent of promise. Ideally short delay should be applied before actual resolution. If implementation does not support
promises, then it may return an eager value.
directive @resolveEmptyObject on FIELD_DEFINITIONResolves ObjectType field with an empty object.
directive @resolveTestData(name: String!) on FIELD_DEFINITIONResolves field with a value provided via test-data property.
directive @resolvePromiseTestData(name: String!) on FIELD_DEFINITIONResolves field with a value provided via test-data property. Internally field should be resolved with successful
promise or an equivalent of promise. Ideally short delay should be applied before actual resolution. If implementation does not support
promises, then it may return an eager value.
directive @resolvePromise on FIELD_DEFINITIONResolves field with context value (as in the default case). Internally field should be resolved with successful promise or an equivalent of promise. Ideally short delay should be applied before actual resolution. If implementation does not support promises, then it may return an eager value.
directive @resolveError(message: String!) on FIELD_DEFINITIONField resolution fails with provided error message.
directive @resolveErrorList(values: [String!]!, messages: [String!]!) on FIELD_DEFINITIONField of type [String] fails with provided errors messages. Even though field resolution fails, it still must produce a value which contains
provided values.
directive @resolvePromiseReject(message: String!) on FIELD_DEFINITIONField resolution fails with provided error message. Internally field should be resolved with rejected
promise or an equivalent of promise. Ideally short delay should be applied before actual resolution. If implementation does not support
promises, then it may return an eager error.
directive @resolvePromiseRejectList(values: [String!]!, messages: [String!]!) on FIELD_DEFINITIONField of type [String] fails with provided errors messages. Even though field resolution fails, it still must produce a value which contains
provided values. Internally field should be resolved with rejected
promise or an equivalent of promise. Ideally short delay should be applied before actual resolution. If implementation does not support
promises, then it may return an eager error.
Test data is a JSON/YAML object where every top-level field may be referenced by name within a scenario (with test-value property),
schema directive (@resolveTestData, @resolvePromiseTestData) or test data itself (with {"$ref": "name"} value).
Let's look at small scenario example that takes advantage of the test data and references:
scenario: "Execute: Handles basic execution tasks"
tests:
- name: test scenario
given:
schema: |
type Basket {
name: String
content: [Fruit]
}
type Fruit {
name: String
inBasket: Basket
}
type Query {
apples: Basket @resolveTestData(name: "tasty")
}
test-data:
green-apple: {name: "Green Apple!", inBasket: {$ref: tasty}}
tasty:
name: Tasty fuits!
content: [{name: "Red Apple!", inBasket: {$ref: tasty}}, {$ref: green-apple}]
query: |
query {
apples {
content {
name
inBasket {
name
}
}
}
}
when:
execute: true
then:
data:
apples:
content:
- name: Red Apple!
inBasket:
name: Tasty fuits!
- name: Green Apple!
inBasket:
name: Tasty fuits!The test defines 2 data elements: "green-apple" and "tasty". These JSON objects are then referenced by name from the @resolveTestData(name: "tasty")
directive and withing the test data itself.
In order to use this test suite in your GraphQL library, you need to implement a small driver that reads scenario files and executes the tests. An example driver can be found in Sangria (scala GraphQL implementation):
Main entry point that generates the tests can be found in the generateTests function.
In 5F01 general, driver executes following steps:
- Load scenario data from the YAML file (CatsScenarioData)
- Generate an executable schema based on the SDL provided in the scenario ("given" part of the scenario) (CatsScenarioExecutor)
- Execute a test query against the generated schema ("when" part of the scenario) (CatsScenarioExecutor.executeAction)
- Assert results of the execution ("then" part of the scenario) (CatsAssertions)