Don't perform memory check if client sets use_mmap true. #8895
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This was referenced Feb 7, 2025
|
Any chance that will be merged in one regular human lifetime? |
|
This one I circumvent with generating a large zram swap, which is useful anyway. I normally load the models into a ramdrive anyway on a live linux. Then the models are already in memory, and duplication is avoided with mmap. I actually need to modify line 213 as well to not get no_mmap. In this scenario line 213 acts insane: If the model is so small that it fits again into memory, it works, but uses mmap, and it actually does not duplicate and does not end up using that extra memory. If the model is so big that it does not fit again, it does not use mmap, and, with the zram swap, runs out of memory. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
If the client overrides
use_mmap, don't prevent the model from loading due to apparent over-commit.On Linux, a mmap'd file doesn't use swap backing store unless modified, so there's no need for the check. Windows has dynamic swap and so falls i 8000 n to the same bucket as darwin. Inference on deepseek-r1:671b-1.5b runs at ~0.15 t/s where the model requires swap on SSD, ~0.3 t/s with mmap instead of swap on the the same SSD, and ~1.4 t/s when the model is mapped on an NVME drive.
Also add
OLLAMA_USE_MMAPfor global configuration.