Glad to help, if there's anything else you can think of that would be helpful to put here or find any mistakes, please create an issue request so I can make sure it gets taken care of.

I'm also right now looking into performance issues with oobabooga and will likely incorporate their latest release which will support LLaMa2 models and more. I will first deploy it to the test branch over there for people to preview and make comments, and find bugs.

My primary focus now is to make sure all the standard packages work with oobabooga on macOS and use Apple Silicon GPU acceleration. The first LLM and encoding I want to get fully working is the llama-cpp-python models working. These will have LLaMa2 support in them.

I'm usually over on the oobabooga mac-setup channel on their Discord.

Thanks for letting me know it has helped you out!

Glad to help with testing. Honestly I don't care about cuda, that is important for windows/linux environments with x86-64+nvidia hardware. With your optimized version is it possible to run quantized gptq/awq models ? If so, did you see a noticeable performance difference (compared to gguf with cpu inference + gpu layers) ? Which Apple Silicon cpu/gpu combination have you tested ?

There is no CUDA and I don't test for it and don't really care either. Would you mind if I stripped the one click installer out? It has caused issues for me sometimes and it just adds bulk. Docker configs also cannot use the GPU/NE either, so that could go as well.

Get me a GPTQ model and I'll load it up and try working through the code to make it work. I might just toss CUDA altogether.

Was hoping to have a complete install, and been testing that, but I could get you manual install instructions and you could do it by hand, it's not very difficult, just following the steps and getting the local pip recompile's in the right order with the right flags.

If you have a small GPTQ model I could load and test, I'll be happy to do that as well. Would like to see that running myself, been using GGUF only so far, but no reason why I couldn't get it working I can see.

For GPU layers, it shouldn't matter the number as long as it's 1 our more, you're using Unified Memory, so what's really important is setting mlock on and while I haven't found anything for sure on mmap, my inclination would be to leave it off. You want the whole model in memory, block will do that for you. Even though it's only 1 GPU layer, I sometimes set the same as the number of layers in the model, not that I see any difference, it just makes me feel good I guess.

I have a simple stress test python script which hammers pretty hard on the system with Torch and NumPy , which hardly registers on the CPU or GPU, with the build using the Accelerate Framework, it appears to run on the NE almost completely. It is really dramatic for it's eigenvalue calculation coming in about 50% of the time for the PyPi packages.

Tested on Apple MacBook Pro 16" M2 Max 96GB RAM macOS - Sonoma 14.4.1

Let me know what your priority is @robik72 and I'll adjust to that, you are number one in the queue. It will be great to have some feedback and someone else banging on it other than me. Let m know if you have any other questions too.

Be well...

Thanks a lot ! There are many questions and ideas coming to my mind ...

• for gptq, if you want to start "small" i would suggest a quantized llama-3 like this one: https://huggingface.co/astronomer/Llama-3-8B-Instruct-GPTQ-8-Bit
• does the mac os version already use pytorch mps ? would it be a good idea to use mlx ?
  

@robik72 Ok, have got an update, I am working on getting my repo sorted out, but an initial copy is here:

https://github.com/unixwzrd/text-generation-webui-macos/tree/macOS-dev

You may have some issues with it until I get the main and head updated, but give it a try. This does use the latest PyTorch if you install as above plus it uses NumPy optimizations, which are really much more important and are not built into the PyPi module for NumPy.

You will need the latest CMake, but if you can download and install Conda, go into the directory where you put the repo and after doing:

pip install -r requirements.txt 

Do the following the order is important:

# PyTorch, NumPy, Llama.cpp.python and CTransformers
# Llama.cpp.python
export CMAKE_ARGS="-DLLAMA_METAL=on"; \
export FORCE_CMAKE=1; \
pip install llama-cpp-python --force-reinstall --no-cache --no-binary :all: --compile --no-deps

# Pip install from daily build
pip install --pre torch torchvision torchaudio \
    --index-url https://download.pytorch.org/whl/nightly/cpu \
    --force-reinstall \
    --no-deps

# NumPy Rebuild with Pip
export CFLAGS="-I/System/Library/Frameworks/vecLib.framework/Headers -Wl,-framework -Wl,Accelerate -framework Accelerate"; \
  pip install numpy --force-reinstall --no-deps --no-cache --no-binary :all: --compile \
    -Csetup-args=-Dblas=accelerate \
    -Csetup-args=-Dlapack=accelerate \
    -Csetup-args=-Duse-ilp64=true

 # CTransformers *UNTESTED, may or may not work*
export CFLAGS="-I/System/Library/Frameworks/vecLib.framework/Headers -Wl,-framework -Wl,Accelerate -framework Accelerate"; \
export CT_METAL=1; \
  pip install ctransformers --no-binary :all: --no-deps --compile --force-reinstall
unset CMAKE_ARGS FORCE_CMAKE CFLAGS CT_METAL

Yes, I know I am exporting the environment variables, but I am working on some functions which will preserve a history of any Conda or Pip modifications to the VENV's and have yet to work everything out from my other project, so no worries about them. This is all advance of any additional testing and instructions, so it may or may not work in this case, but let me know if you run into any problems and I'll try to address them as soon as possible.

To start it up, switch to the VENV you installed everything in and make sure you are in the directory where you cloned the repo, then:

python server.py

Add any flags you wish after it such as --verbose or --listen

I've just made the switch over to using GitHub Work Trees, it kinda forced me, so just getting used to it. There may be bits missing, but wanted to get this to you sooner rather than later. Have not got any GPTQ working yet, had other things I was finishing up.

Yeah, an all MLX version would be nice, but that will also be a major departure from the oobabooga base code, I'll need to figure a way to do that and have a few ideas, but it may take a bit of time to get there. I'd like to get to that if possible, but again, that will take a bit of work to accomplish and the end result may no longer be able to remain on a parallel development path with oobabooga, but I might be able to get creative.

Let me know how that works out for you, and let me know if you rn into any problems. I'll be continuing to work on improvements over the coming days. Life as usual gets in the way sometimes, but that, life.

Be well...

@robik72 I cannot get packages to build correctly it seems after the macOS 14.5 update, also GitHub forcing me into work trees it seems I lost some of my edits, but putting them back together. Have the GPTQ model downloaded and will be testing with that today if I can get my packages to build now.. Tracking that down right now and Think it was 14.5 update as the cause.