Develop Weave's Platform Tokenizers
Different LLM has different set of tokenizers. To streamline the token calculation method on our platform, it's best to develop our own set of tokenisers.
Reference: https://github.com/huggingface/tokenizers
Status: PRD in Progress
Log in to comment and vote
No comments yet
Be the first to share your thoughts.