Support for MLX models in LLM.
Read my blog for background on this project.
Install this plugin in the same environment as LLM. This plugin likely only works on macOS.
llm install llm-mlxTo install an MLX model from Hugging Face, use the llm mlx download-model command. This example downloads 1.8GB of model weights from mlx-community/Llama-3.2-3B-Instruct-4bit:
llm mlx download-model mlx-community/Llama-3.2-3B-Instruct-4bitThen run prompts like this:
llm -m mlx-community/Llama-3.2-3B-Instruct-4bit 'Capital of France?' -s 'you are a pelican'The mlx-community organization is a useful source for compatible models.
The following models all work well with this plugin:
mlx-community/Qwen2.5-0.5B-Instruct-4bit- 278MBmlx-community/Mistral-7B-Instruct-v0.3-4bit- 4.08GBmlx-community/DeepSeek-R1-Distill-Qwen-32B-4bit- 18.5GBmlx-community/Llama-3.3-70B-Instruct-4bit- 40GB
MLX models can use the following model options:
-o max_tokens INTEGER: Maximum number of tokens to generate in the completion (defaults to 1024)-o unlimited 1: Generate an unlimited number of tokens in the completion-o temperature FLOAT: Sampling temperature (defaults to 0.8)-o top_p FLOAT: Sampling top-p (defaults to 0.9)-o min_p FLOAT: Sampling min-p (defaults to 0.1)-o min_tokens_to_keep INT: Minimum tokens to keep for min-p sampling (defaults to 1)-o seed INT: Random number seed to use
For example:
llm -m mlx-community/Llama-3.2-3B-Instruct-4bit 'Joke about pelicans' -o max_tokens 60 -o temperature 1.0You can use this plugin in Python like this:
from llm_mlx import MlxModel
model = MlxModel("mlx-community/Llama-3.2-3B-Instruct-4bit")
print(model.prompt("hi").text())
# Outputs: How can I assist you today?Using MlxModel directly in this way avoids needing to first use the download-model command.
If you have already registered models with that command you can use them like this instead:
import llm
model = llm.get_model("mlx-community/Llama-3.2-3B-Instruct-4bit")
print(model.prompt("hi").text())The LLM Python API documentation has more details on how to use LLM models.
To set up this plugin locally, first checkout the code. Then create a new virtual environment:
cd llm-mlx
python -m venv venv
source venv/bin/activateNow install the dependencies and test dependencies:
llm install -e '.[test]'To run the tests:
python -m pytest