High performance Chinese tokenizer with both GBK and UTF-8 charset support based on MMSEG algorithm developed by ANSI C. Completely based on modular implementation and can be easily embedded in other programs, like: MySQL, PostgreSQL, PHP, etc.

A Japanese morphological analyzer Sudachi as a Solr plugin.

This repository contains codes related to the experiments in "An Experimental Evaluation of Japanese Tokenizers for Sentiment-Based Text Classification" presented at https://www.anlp.jp/nlp2021/. Authors: Andre Rusli and Makoto Shishido (Tokyo Denki University).

A japanese tokenizer and morphological analyzer

A Helper gem for download and store Sudachi jar and dict files written with Ruby ( for combination with `rudachi` )

Async Japanese Tokenizer Native Plugin for React Native for iOS and Android