Apple GPU on macOS
Run Ollama natively on a Mac with Apple Silicon and connect it to DeepFellow Infra to use the GPU.
Docker on macOS gives containers no access to the GPU, so the ollama service runs on the CPU there. An Ollama installed natively on a Mac with Apple Silicon uses the GPU (Metal). To use it, run Ollama on your Mac and connect it to DeepFellow Infra with the ollama-external service. The ollama service has no url field, so it does not connect to an Ollama that already runs on your Mac.
For supported GPUs and example configurations, see Hardware Requirements.
Before You Begin
Make sure you have the following:
- A Mac with Apple Silicon
- DeepFellow CLI and a running DeepFellow Infra, see Installation
Connect Ollama to DeepFellow Infra
Install Ollama on your Mac and start it. By default, it listens on 127.0.0.1:11434.
Install the ollama-external service and point it at your Mac:
deepfellow infra service install ollama-external --spec '{"url": "http://host.docker.internal:11434"}'DeepFellow Infra runs in Docker, so localhost points at the Infra container and not at your Mac. With the default url, http://localhost:11434, the installation fails with the error Cannot connect to Ollama at http://localhost:11434.
Install a model through DeepFellow Infra. The Ollama on your Mac downloads it. For example, to install the embedding model that deepfellow suite install sets up:
deepfellow infra model install ollama-external mxbai-embed-largeA model that you already pulled with ollama pull needs no installation. DeepFellow Infra lists the models of your Ollama as soon as you install the service.
If You Already Ran Quick Install
deepfellow suite install installs the ollama service with its models. When ollama and ollama-external both have a model with the same name, DeepFellow Infra serves requests for that model from the ollama service, which runs on the CPU.
To use the GPU, install the model in ollama-external first, then uninstall it from ollama:
deepfellow infra model install ollama-external mxbai-embed-large
deepfellow infra model uninstall ollama mxbai-embed-largeRepeat both commands for each model that the two services share. After that, DeepFellow Infra serves the model from ollama-external.
Use the Models in DeepFellow Server
Models of ollama-external appear in DeepFellow Server like any other model. The --models option of deepfellow server project create controls a project's access to them. See Create a Project.
Check That the GPU Is Used
Send a request to the model through DeepFellow Server, then run this command on your Mac:
ollama psThe PROCESSOR column shows 100% GPU for the loaded model.
We use cookies on our website. We use them to ensure proper functioning of the site and, if you agree, for purposes such as analytics, marketing, and targeting ads.