Develop Weave's Platform Tokenizers
Different LLM has different set of tokenizers. To streamline the token calculation method on our platform, it's best to develop our own set of tokenisers.
Reference: https://github.com/huggingface/tokenizers
Please authenticate to join the conversation.
Upvoters
Status
PRD in Progress
Board
💡 Request a feature
Date
Almost 2 years ago
Author
Shuwei Li
Subscribe to post
Get notified by email when there are changes.
Upvoters
Status
PRD in Progress
Board
💡 Request a feature
Date
Almost 2 years ago
Author
Shuwei Li
Subscribe to post
Get notified by email when there are changes.