Gonna be presenting a demo of teaching local a to search wikipedia with "Function Calling"

@mauve Hey #ollama is avalliable with GPU acceleration the AMD Ryzen 7640, just use the rocm docker container (no need for further drivers)

@mauve Recomend using podman, to acess the GPU inside the container use sudo setsebool container_use_devices=true to make SE Linux comply with it.

If its a GPD device you can allocate more VRAM in the BIOS

@mauve Personally recommend strongly to not install the ROCm drivers on your device but using them in a container instead, they are not that stabile on those chips and it can lead to your device crashing. Also officially only like an LTS Ubunut and Cent OS and a couple of GPU's are supported.

Container on the other hand is one command (use the amd rcom version further info here:
hub.docker.com/r/ollama/ollama
ollama.com/blog/amd-preview
There is no substantial performance lose using a container

Follow

@fredy_pferdi Sweet just followed this guide to install it in my ubuntu distrobox container and it's working great :o

reddit.com/r/steamdeck_linux/c

· · Web · 1 · 0 · 0

@fredy_pferdi Spoke too soon, ollama dies when I try to load the model. Will need to mess with it another day :) TY again for the tip.

@mauve Distrobox is just an interface interface for Podman i think just running the already made images or building them yourself is way easier then to recreate the install manually with Distrobox.

@mauve
first allow podman to use GPU
`sudo setsebool -P container_use_devices=true`

and then just run this to start the container
`podman create --name=ollama --security-opt seccomp=unconfined --device /dev/dri --device /dev/kfd -e HSA_OVERRIDE_GFX_VERSION=10.3.0 -e HCC_AMDGPU_TARGETS=gfx1035 -e OLLAMA_DEBUG=1 -v .ollama:/root/.ollama:U,rw -p 11434:11434 -i --tty --restart unless-stopped docker.io/ollama/ollama:rocm`

@mauve with the Ryzen 7600u you may have to use -e HSA_OVERRIDE_GFX_VERSION=11.0.0 instead of -e HSA_OVERRIDE_GFX_VERSION=10.3.0 and make sure that you allocated enough RAM to the graphics card in the BIOS of your #GPD WIn 4 (in the advanced option)

@fredy_pferdi Cool TY, I found the HSA_OVERRIDE online and it ended up working great in my ubuntu container. 😁 Wish I had this for last night's demo! Also I don't have nearly enough RAM on this thing with 16 GB. TT_TT

@mauve for testing you could allocate 8gb that should be enough to run a small model while still using the os.

Screenshot of the option to allocate more vram:
reddit.com/r/gpdwin/comments/y

@fredy_pferdi Yeah the issue is my Matrix client ends up eating way too much RAM and then I start eating swap. Might also have a memory leak somewhere in my OS wasting RAM after going out of sleep mode

@mauve Yeah there are suspend issues with those GPD devices.

@fredy_pferdi Alas! It's still worth it for me to not have to use Windows or a regular laptop :P

Sign in to participate in the conversation
Mauvestodon

Escape ship from centralized social media run by Mauve.