tiktoken

Fast BPE tokenizer for use with OpenAI's models.

Stars

19,619

Forks

1,669

Language

Python

Get the code

# Clone the repository
git clone https://github.com/openai/tiktoken

What is tiktoken?

Fast BPE tokenizer for use with OpenAI's models.

Is tiktoken safe to use?

tiktoken is an open-source project by openai on GitHub. Review the source code and license before using it in production.

How do I get started with tiktoken?

git clone https://github.com/openai/tiktoken

What language is tiktoken written in?

openSource.faqWhatLanguageAnswer

The model-definition framework for state-of-the-art ML models across text, vision, and audio.

State-of-the-art diffusion models for image, video, and audio generation in PyTorch.

Build and share machine learning demos and apps entirely in Python.