Data formats
This section documents input and output formats of data used by spaCy, including the training config, training data and lexical vocabulary data. For an overview of label schemes used by the models, see the models directory. Each trained pipeline documents the label schemes used in its components, depending on the data it was trained on.
Training config v3.0
Config files define the training process and pipeline and can be passed to
spacy train. They use
Thinc’s configuration system under the
hood. For details on how to use training configs, see the
usage documentation. To get started with the
recommended settings for your use case, check out the
quickstart widget or run the
init config command.
explosion/spaCy/master/spacy/default_config.cfg
nlp section
Defines the nlp object, its tokenizer and
processing pipeline component names.
| Name | Description |
|---|---|
lang | Pipeline language ISO code. Defaults to null. str |
pipeline | Names of pipeline components in order. Should correspond to sections in the [components] block, e.g. [components.ner]. See docs on defining components. Defaults to []. List[str] |
disabled | Names of pipeline components that are loaded but disabled by default and not run as part of the pipeline. Should correspond to components listed in pipeline. After a pipeline is loaded, disabled components can be enabled using Language.enable_pipe. List[str] |
before_creation | Optional callback to modify Language subclass before it’s initialized. Defaults to null. Optional[Callable[[Type[Language]], Type[Language]]] |
after_creation | Optional callback to modify nlp object right after it’s initialized. Defaults to null. Optional[Callable[[Language],Language]] |
after_pipeline_creation | Optional callback to modify nlp object after the pipeline components have been added. Defaults to null. Optional[Callable[[Language],Language]] |
tokenizer | The tokenizer to use. Defaults to Tokenizer. Callable[[str],Doc] |
batch_size | Default batch size for Language.pipe and Language.evaluate. int |
components section
This section includes definitions of the
pipeline components and their models, if
available. Components in this section can be referenced in the pipeline of the
[nlp] block. Component blocks need to specify either a factory (named
function to use to create component) or a source (name of path of trained
pipeline to copy components from). See the docs on
defining pipeline components for details.
paths, system variables
These sections define variables that can be referenced across the other sections
as variables. For example ${paths.train} uses the value of train defined in
the block [paths]. If your config includes custom registered functions that
need paths, you can define them here. All config values can also be
overwritten on the CLI when you run
spacy train, which is especially relevant for data paths
that you don’t want to hard-code in your config file.
corpora section
This section defines a dictionary mapping of string keys to functions. Each
function takes an nlp object and yields Example objects. By
default, the two keys train and dev are specified and each refer to a
Corpus. When pretraining, an additional pretrain
section is added that defaults to a JsonlCorpus.
You can also register custom functions that return a callable.
| Name | Description |
|---|---|
train | Training data corpus, typically used in [training] block. Callable[[Language], Iterator[Example]] |
dev | Development data corpus, typically used in [training] block. Callable[[Language], Iterator[Example]] |
pretrain | Raw text for pretraining, typically used in [pretraining] block (if available). Callable[[Language], Iterator[Example]] |
| … | Any custom or alternative corpora. Callable[[Language], Iterator[Example]] |
Alternatively, the [corpora] block can refer to one function that returns
a dictionary keyed by the corpus names. This can be useful if you want to load a
single corpus once and then divide it up into train and dev partitions.
| Name | Description |
|---|---|
corpora | A dictionary keyed by string names, mapped to corpus functions that receive the current nlp object and return an iterator of Example objects. Dict[str, Callable[[Language], Iterator[Example]]] |
training section
This section defines settings and controls for the training and evaluation
process that are used when you run spacy train.
| Name | Description |
|---|---|
accumulate_gradient | Whether to divide the batch up into substeps. Defaults to 1. int |
batcher | Callable that takes an iterator of Doc objects and yields batches of Docs. Defaults to batch_by_words. Callable[[Iterator[Doc], Iterator[List[Doc]]]] |
before_to_disk | Optional callback to modify nlp object right before it is saved to disk during and after training. Can be used to remove or reset config values or disable components. Defaults to null. Optional[Callable[[Language],Language]] |
before_update v3.5 | Optional callback that is invoked at the start of each training step with the nlp object and a Dict containing the following entries: step, epoch. Can be used to make deferred changes to components. Defaults to null. Optional[Callable[[Language, Dict[str, Any]], None]] |
dev_corpus | Dot notation of the config location defining the dev corpus. Defaults to corpora.dev. str |
dropout | The dropout rate. Defaults to 0.1. float |
eval_frequency | How often to evaluate during training (steps). Defaults to 200. int |
frozen_components | Pipeline component names that are “frozen” and shouldn’t be initialized or updated during training. See here for details. Defaults to []. List[str] |
annotating_components v3.1 | Pipeline component names that should set annotations on the predicted docs during training. See here for details. Defaults to []. List[str] |
gpu_allocator | Library for cupy to route GPU memory allocation to. Can be "pytorch" or "tensorflow". Defaults to variable ${system.gpu_allocator}. str |
logger | Callable that takes the nlp and stdout and stderr IO objects, sets up the logger, and returns two new callables to log a training step and to finalize the logger. Defaults to ConsoleLogger. Callable[[Language, IO, IO], [Tuple[Callable[[Dict[str, Any]], None], Callable[[], None]]]] |
max_epochs | Maximum number of epochs to train for. 0 means an unlimited number of epochs. -1 means that the train corpus should be streamed rather than loaded into memory with no shuffling within the training loop. Defaults to 0. int |
max_steps | Maximum number of update steps to train for. 0 means an unlimited number of steps. Defaults to 20000. int |
optimizer | The optimizer. The learning rate schedule and other settings can be configured as part of the optimizer. Defaults to Adam. Optimizer |
patience | How many steps to continue without improvement in evaluation score. 0 disables early stopping. Defaults to 1600. int |
score_weights | Score names shown in metrics mapped to their weight towards the final weighted score. See here for details. Defaults to {}. Dict[str, float] |
seed | The random seed. Defaults to variable ${system.seed}. int |
train_corpus | Dot notation of the config location defining the train corpus. Defaults to corpora.train. str |
pretraining sectionoptional
This section is optional and defines settings and controls for
language model pretraining. It’s
used when you run spacy pretrain.
| Name | Description |
|---|---|
max_epochs | Maximum number of epochs. Defaults to 1000. int |
dropout | The dropout rate. Defaults to 0.2. float |
n_save_every | Saving frequency. Defaults to null. Optional[int] |
objective | The pretraining objective. Defaults to {"type": "characters", "n_characters": 4}. Dict[str, Any] |
optimizer | The optimizer. The learning rate schedule and other settings can be configured as part of the optimizer. Defaults to Adam. Optimizer |
corpus | Dot notation of the config location defining the corpus with raw text. Defaults to corpora.pretrain. str |
batcher | Callable that takes an iterator of Doc objects and yields batches of Docs. Defaults to batch_by_words. Callable[[Iterator[Doc], Iterator[List[Doc]]]] |
component | Component name to identify the layer with the model to pretrain. Defaults to "tok2vec". str |
layer | The specific layer of the model to pretrain. If empty, the whole model will be used. str |
initialize section
This config block lets you define resources for initializing the pipeline.
It’s used by Language.initialize and typically
called right before training (but not at runtime). The section allows you to
specify local file paths or custom functions to load data resources from,
without requiring them at runtime when you load the trained pipeline back in.
Also see the usage guides on the
config lifecycle and
custom initialization.
| Name | Description |
|---|---|
after_init | Optional callback to modify the nlp object after initialization. Optional[Callable[[Language],Language]] |
before_init | Optional callback to modify the nlp object before initialization. Optional[Callable[[Language],Language]] |
components | Additional arguments passed to the initialize method of a pipeline component, keyed by component name. If type annotations are available on the method, the config will be validated against them. The initialize methods will always receive the get_examples callback and the current nlp object. Dict[str, Dict[str, Any]] |
init_tok2vec | Optional path to pretrained tok2vec weights created with spacy pretrain. Defaults to variable ${paths.init_tok2vec}. Ignored when actually running pretraining, as you’re creating the file to be used later. Optional[str] |
lookups | Additional lexeme and vocab data from spacy-lookups-data. Defaults to null. Optional[Lookups] |
tokenizer | Additional arguments passed to the initialize method of the specified tokenizer. Can be used for languages like Chinese that depend on dictionaries or trained models for tokenization. If type annotations are available on the method, the config will be validated against them. The initialize method will always receive the get_examples callback and the current nlp object. Dict[str, Any] |
vectors | Name or path of pipeline containing pretrained word vectors to use, e.g. created with init vectors. Defaults to null. Optional[str] |
vocab_data | Path to JSONL-formatted vocabulary file to initialize vocabulary. Optional[str] |
Training data
Binary training format v3.0
The main data format used in spaCy v3.0 is a binary format created by
serializing a DocBin, which represents a collection of Doc
objects. This means that you can train spaCy pipelines using the same format it
outputs: annotated Doc objects. The binary format is extremely efficient in
storage, especially when packing multiple documents together.
Typically, the extension for these binary files is .spacy, and they are used
as input format for specifying a training corpus and for spaCy’s
CLI train command. The built-in
convert command helps you convert spaCy’s previous
JSON format to the new binary format. It also supports conversion
of the .conllu format used by the
Universal Dependencies corpora.
Note that while this is the format used to save training data, you do not have to understand the internal details to use it or create training data. See the section on preparing training data.
JSON training format deprecated
Example structure
Here’s an example of dependencies, part-of-speech tags and named entities, taken from the English Wall Street Journal portion of the Penn Treebank:
explosion/spaCy/v2.3.x/examples/training/training-data.json
Annotation format for creating training examples
An Example object holds the information for one training
instance. It stores two Doc objects: one for holding the
gold-standard reference data, and one for holding the predictions of the
pipeline. Examples can be created using the
Example.from_dict method with a reference Doc and
a dictionary of gold-standard annotations.
| Name | Description |
|---|---|
text | Raw text. str |
words | List of gold-standard tokens. List[str] |
lemmas | List of lemmas. List[str] |
spaces | List of boolean values indicating whether the corresponding tokens is followed by a space or not. List[bool] |
tags | List of fine-grained POS tags. List[str] |
pos | List of coarse-grained POS tags. List[str] |
morphs | List of morphological features. List[str] |
sent_starts | List of boolean values indicating whether each token is the first of a sentence or not. List[bool] |
deps | List of string values indicating the dependency relation of a token to its head. List[str] |
heads | List of integer values indicating the dependency head of each token, referring to the absolute index of each token in the text. List[int] |
entities | Option 1: List of BILUO tags per token of the format "{action}-{label}", or None for unannotated tokens. List[str] |
entities | Option 2: List of (start_char, end_char, label) tuples defining all entities in the text. List[Tuple[int, int, str]] |
cats | Dictionary of label/value pairs indicating how relevant a certain text category is for the text. Dict[str, float] |
links | Dictionary of offset/dict pairs defining named entity links. The character offsets are linked to a dictionary of relevant knowledge base IDs. Dict[Tuple[int, int], Dict] |
spans | Dictionary of spans_key/List[Tuple] pairs defining the spans for each spans key as (start_char, end_char, label, kb_id) tuples. Dict[str, List[Tuple[int, int, str, str]] |
Examples
Lexical data for vocabulary
This data file can be provided via the vocab_data setting in the
[initialize] block of the training config to pre-define the lexical data to
initialize the nlp object’s vocabulary with. The file should contain one
lexical entry per line. The first line defines the language and vocabulary
settings. All other lines are expected to be JSON objects describing an
individual lexeme. The lexical attributes will be then set as attributes on
spaCy’s Lexeme object.
First line
Entry structure
Here’s an example of the 20 most frequent lexemes in the English training data:
explosion/spaCy/master/extra/example_data/vocab-data.jsonl
Pipeline meta
The pipeline meta is available as the file meta.json and exported
automatically when you save an nlp object to disk. Its contents are available
as nlp.meta.
| Name | Description |
|---|---|
lang | Pipeline language ISO code. Defaults to "en". str |
name | Pipeline name, e.g. "core_web_sm". The final package name will be {lang}_{name}. Defaults to "pipeline". str |
version | Pipeline version. Will be used to version a Python package created with spacy package. Defaults to "0.0.0". str |
spacy_version | spaCy version range the package is compatible with. Defaults to the spaCy version used to create the pipeline, up to next minor version, which is the default compatibility for the available trained pipelines. For instance, a pipeline trained with v3.0.0 will have the version range ">=3.0.0,<3.1.0". str |
parent_package | Name of the spaCy package. Typically "spacy" or "spacy_nightly". Defaults to "spacy". str |
requirements | Python package requirements that the pipeline depends on. Will be used for the Python package setup in spacy package. Should be a list of package names with optional version specifiers, just like you’d define them in a setup.cfg or requirements.txt. Defaults to []. List[str] |
description | Pipeline description. Also used for Python package. Defaults to "". str |
author | Pipeline author name. Also used for Python package. Defaults to "". str |
email | Pipeline author email. Also used for Python package. Defaults to "". str |
url | Pipeline author URL. Also used for Python package. Defaults to "". str |
license | Pipeline license. Also used for Python package. Defaults to "". str |
sources | Data sources used to train the pipeline. Typically a list of dicts with the keys "name", "url", "author" and "license". See here for examples. Defaults to None. Optional[List[Dict[str, str]]] |
vectors | Information about the word vectors included with the pipeline. Typically a dict with the keys "width", "vectors" (number of vectors), "keys" and "name". Dict[str, Any] |
pipeline | Names of pipeline component names, in order. Corresponds to nlp.pipe_names. Only exists for reference and is not used to create the components. This information is defined in the config.cfg. Defaults to []. List[str] |
labels | Label schemes of the trained pipeline components, keyed by component name. Corresponds to nlp.pipe_labels. See here for examples. Defaults to {}. Dict[str, Dict[str, List[str]]] |
performance | Training accuracy, added automatically by spacy train. Dictionary of score names mapped to scores. Defaults to {}. Dict[str, Union[float, Dict[str, float]]] |
speed | Inference speed, added automatically by spacy train. Typically a dictionary with the keys "cpu", "gpu" and "nwords" (words per second). Defaults to {}. Dict[str, Optional[Union[float, str]]] |
spacy_git_version v3.0 | Git commit of spacy used to create pipeline. str |
| other | Any other custom meta information you want to add. The data is preserved in nlp.meta. Any |