Just ran a lil LLM on my #SteamDeck. 7b with 4bit quantization so it's pretty small and dumb. I'm calling it stink-demon.
I'm gonna try to get it to do simple formatting tasks from text I've selected with my mouse based on the speech to text I'm adding to my deck via Vosk and steam-dictation.
Follow
As expected, this thing can't code for shit. :P
Cursed cURL request:
curl -X POST -H "Content-Type: application/octet-stream" -X PUT -X Content-Type: application/octet-stream -X Accept: */* -H "Authorization: Bearer <YOUR_API_KEY>" --get-header "Content-Type: application/octet-stream" --get-header "Content-Disposition: form-data; name=\"site.tar\"" $tar https://api.distributed.press/site/$site_url
ai, llm
Gonna try this one next.
https://huggingface.co/autobots/alpaca-7b-native-enhanced-4bit/blob/main/README.md
It's the same thing but apparently the quantization is "better" according to this huggingface post: https://huggingface.co/8bit-coder/alpaca-7b-nativeEnhanced/discussions/5