[Backport M84] Replace tiktoken with gpt-tokenizer #7667
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
## Changes In my memory profiling and local stress testing I always see tiktoken leaking memory and contributing to a huge spikes in a CPU usage. If I keep hammering IDE with text changes (like keep pressed Enter for a minute) it can even lead into 100% CPU utilisation which **never recovers**. I decided to give some alternatives a try, and I change is easy to see with a bare end. Namely I replaced tiktoken with [gpt-tokenizer](https://github.com/niieani/gpt-tokenizer) which claims to be: > most feature-complete, open-source GPT tokenizer on NPM. This package is a port of OpenAI's [tiktoken] It is also much more performant and less memory hungry. Additionally because it initialises much faster I deleted a bunch of code for lazy initialization of tiktoken, and I see no downside.   In a practical test with Cody, I tried the scenario with enter key being pressed for a minute (adding 1-2k lines). After the test I looked at CPU utilisation (if it persists) and memory usage: **Old version with tiktoken:** 100% CPU utilisation, 1.5G+ of memory used by agent process **New version with gpt-tokenizer:** **_0% CPU utilisation, less than 400MB of memory used by agent process_** ## Test plan All unit tests still works without any changes to the logic or values. **Scenario 1:** 1. Open big file (like few MB log file) 2. Make sure it is excluded from the context **Scenario 2:** 1. Open any file 2. Keep appending text to that file (e.g. by keeping `Enter` pressed) until ~1000 lines are added 3. After that check CPU utilisation and memory usage - CPU usage should be 0% and memory utilisation should not increase more than few % (cherry picked from commit 16c0d79)
hitesh-1997
approved these changes
Apr 4, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Changes
In my memory profiling and local stress testing I always see tiktoken leaking memory and contributing to a huge spikes in a CPU usage. If I keep hammering IDE with text changes (like keep pressed Enter for a minute) it can even lead into 100% CPU utilisation which never recovers.
I decided to give some alternatives a try, and I change is easy to see with a bare end.
Namely I replaced tiktoken with gpt-tokenizer which claims to be:
> most feature-complete, open-source GPT tokenizer on NPM. This package is a port of OpenAI's [tiktoken]
It is also much more performant and less memory hungry. Additionally because it initialises much faster I deleted a bunch of code for lazy initialization of tiktoken, and I see no downside.
In a practical test with Cody, I tried the scenario with enter key being pressed for a minute (adding 1-2k lines).
After the test I looked at CPU utilisation (if it persists) and memory usage:
Old version with tiktoken:
100% CPU utilisation, 1.5G+ of memory used by agent process
New version with gpt-tokenizer:
0% CPU utilisation, less than 400MB of memory used by agent process
Test plan
All unit tests still works without any changes to the logic or values.
Scenario 1:
Scenario 2:
Enterpressed) until ~1000 lines are addedBackport 16c0d79 from Replace tiktoken with gpt-tokenizer #7662