Gonna be presenting a demo of teaching local a #llm to search wikipedia with "Function Calling"

Source for my demo code: https://github.com/RangerMauve/mind-goblin

Video of my talk about making an #OpenSource #LLM perform function calling on my machine.

https://www.youtube.com/watch?v=GztPt4Yhl8M

#ai

@mauve Hey #ollama is avalliable with GPU acceleration the AMD Ryzen 7640, just use the rocm docker container (no need for further drivers)

@mauve Recomend using podman, to acess the GPU inside the container use sudo setsebool container_use_devices=true to make SE Linux comply with it.

If its a GPD device you can allocate more VRAM in the BIOS

@fredy_pferdi Interesting I may be able to get it running without a container too. https://github.com/rocm-arch/rocm-arch

@mauve Personally recommend strongly to not install the ROCm drivers on your device but using them in a container instead, they are not that stabile on those chips and it can lead to your device crashing. Also officially only like an LTS Ubunut and Cent OS and a couple of GPU's are supported.

Container on the other hand is one command (use the amd rcom version further info here:
https://hub.docker.com/r/ollama/ollama
https://ollama.com/blog/amd-preview
There is no substantial performance lose using a container

@fredy_pferdi Sweet just followed this guide to install it in my ubuntu distrobox container and it's working great :o

https://www.reddit.com/r/steamdeck_linux/comments/102hzav/guide_how_to_install_rocm_for_gpu_julia/

@fredy_pferdi Spoke too soon, ollama dies when I try to load the model. Will need to mess with it another day :) TY again for the tip.

@mauve Distrobox is just an interface interface for Podman i think just running the already made images or building them yourself is way easier then to recreate the install manually with Distrobox.

@mauve
first allow podman to use GPU
`sudo setsebool -P container_use_devices=true`

and then just run this to start the container
`podman create --name=ollama --security-opt seccomp=unconfined --device /dev/dri --device /dev/kfd -e HSA_OVERRIDE_GFX_VERSION=10.3.0 -e HCC_AMDGPU_TARGETS=gfx1035 -e OLLAMA_DEBUG=1 -v .ollama:/root/.ollama:U,rw -p 11434:11434 -i --tty --restart unless-stopped docker.io/ollama/ollama:rocm`