Tiktoken

Name: Tiktoken
Rating: 4.3 (8500 reviews)
Author: OpenAI

by OpenAI8.5k

Featured

Fast BPE tokenizer for OpenAI models

About

Fast BPE tokenizer for OpenAI models Provides comprehensive functionality for seamless integration.

pip install tiktoken

Hugging Face

Fast tokenizers library from Hugging Face

Google

Unsupervised text tokenizer and detokenizer

Community

Standalone GPT-2 tokenizer implementation