Tokenizer is a fast, generic, and customizable text tokenization library for C++ and Python with minimal dependencies.
By default, the Tokenizer applies a simple tokenization based on Unicode types. It can be customized in several ways:
- Reversible tokenization
Marking joints or spaces by annotating tokens or injecting modifier characters. - Subword tokenization
Support for training and using BPE and SentencePiece models. - Advanced text segmentation
Split digits, segment on case or alphabet change, segment each character of selected alphabets, etc. - Case management
Lowercase text and return case information as a separate feature or inject case modifier tokens. - Protected sequences
Sequences can be protected against tokenization with the special characters "｟" and "｠".
See the available options for an overview of supported features.
The Tokenizer can be used in Python, C++, or command line. Each mode exposes the same set of options.
pip install pyonmttok>>> import pyonmttok
>>> tokenizer = pyonmttok.Tokenizer("conservative", joiner_annotate=True)
>>> tokens, _ = tokenizer.tokenize("Hello World!")
>>> tokens
['Hello', 'World', '￭!']
>>> tokenizer.detokenize(tokens)
'Hello World!'See the Python API description for more details.
#include <onmt/Tokenizer.h>
using namespace onmt;
int main() {
Tokenizer tokenizer(Tokenizer::Mode::Conservative, Tokenizer::Flags::JoinerAnnotate);
std::vector<std::string> tokens;
tokenizer.tokenize("Hello World!", tokens);
}See the Tokenizer class for more details.
$ echo "Hello World!" | cli/tokenize --mode conservative --joiner_annotate
Hello World ￭!
$ echo "Hello World!" | cli/tokenize --mode conservative --joiner_annotate | cli/detokenize
Hello World!See the -h flag to list the available options.
- (optional) SentencePiece
- (optional) ICU
- (required by clients) Boost (
program_options)
CMake and a compiler that supports the C++11 standard are required to compile the project.
mkdir build
cd build
cmake -DCMAKE_BUILD_TYPE=<Release or Debug> ..
make
It will produce the dynamic library libOpenNMTTokenizer and tokenization clients in cli/.
- To compile only the library, use the
-DLIB_ONLY=ONflag. - To compile with the ICU unicode backend, use the
-DWITH_ICU=ONflag.
Tests are using the Google Test framework. Once installed, simply compile the project and run:
test/onmt_tokenizer_test ../test/data