🎤 vibrato: Viterbi-based accelerated tokenizer
-
Updated
Apr 11, 2025 - Rust
8000
🎤 vibrato: Viterbi-based accelerated tokenizer
Sudachi in Rust 🦀 and new generation of SudachiPy
CodeChain's official implementation in Rust.
🛥 Vaporetto: Very accelerated pointwise prediction based tokenizer
Simple NLP in Rust with Python bindings
The Implementation of The Ledger of Things Node. Layer 1 decentralized blockchain platform for the tokenization of objects. Proof of Scan protocol. Useful smart-contracts and dApps.
🛥 Vaporetto is a fast and lightweight pointwise prediction based tokenizer. This is a Python wrapper for Vaporetto.
TI Tools is a CLI tool designed for converting 8xp files (used by TI-83 and TI-84 calculators) to text files and vice versa. It also supports various other features for working with 8xp files.
3D object recognition CLI tool for Linux
A compiled mecab-ipadic-neologd dictionary for vibrato
3D object recognition tool for WASM. HASH ID sustainable identity creation and its verification.
Security Tokens on Lightning Network
A cli program that calculates the derivative of a function
A WebAssembly binding for the charabia multilingual text tokenizer used by Meilisearch.
Add a description, image, and links to the tokenization topic page so that developers can more easily learn about it.
To associate your repository with the tokenization topic, visit your repo's landing page and select "manage topics."