Description
Text tokenization models can be trained and applied from prebuilt SentencePiece command-line tools. It is for developers and researchers preparing NLP datasets and vocabularies.
FICHA · AUR
sentencepiece-bin
Unsupervised text tokenizer for Neural Network-based text generation
- tokenizer CLI tools
- CLI
- NLP
- TOKENIZATION
- Runs in terminal
official+codex · reviewed · Jun 4, 2026 description in
en Permissions
Permissions not analysed for this source yet.