Mauve 👁💜 @mauve@mastodon.mauve.moe

@skryking Nice. I've been wanting to get into Rust for years but didn't have much of a use case. Now with the candle library from HuggingFace and my latest adventures with LLMs I've had an actual reason to write something in it. :) https://github.com/huggingface/candle/

@skryking This post by @simon is what exposed me to the idea for the first time: https://til.simonwillison.net/llms/python-react-pattern

I also have a slightly improved prompt here: https://gist.github.com/RangerMauve/19be7dca9ced8e1095ed2e00608ded5e

I'll likely be publishing any new work as open source on Github. :) Probably with Rust.

@jonny Are you gonna play the pokemon with guns game?

feels kinda satisfying to prolapse this function to return an iterator instead of its heavy for loop.

@skryking it has less innate knowledge of facts but it is pretty good at "reasoning". I'm gonna teach it to make function calls and traverse datasets + summarize stuff. 😁

@skryking Nice. I only do CPU workloads. Try running phi 2 some time! It's super low in resurce usage. Particually the Q4 quantized models.

@laskov Oh yeah, I read their release but haven't used it yet. Was there anything specific they excelled at?

@staltz I've already got the programming socks. :P

@skryking What have you been using to run the models? I find LM Studio really nice for tinkering. https://lmstudio.ai/

I find Q4 quantized models work pretty well on my steam deck.

Hmm, after testing the raw Phi 2 within LM Studio instead of the examples provided by HuggingFace candle, I think it's actually pretty decent after all.

Specifically I got phi 2 Q4_K_S gguf working from TheBloke.

Can't get that model running with candle since it can't seem to load the model weight.

Having tested a bunch of #OpenSource #LLM projects, I gotta say that OpenHermes 2.5 is the most helpful out of the ones I can run locally.

I recently wasted a bunch of time getting Phi-2 to do some summarization work, and it just couldn't stay focused for more than a sentence or two.

Woot, I have finally written enough #rust code to be unable to avoid the lifetime specifications / borrow checker stuff.

@beka_valentine RIP. What software are you using? I generally find OBS does everything I want.

K. I swallowed my pride and gave up on trying to run all my build tools as "close to the metal" as possible. I'm now going to pretend I'm in an Ubuntu environment via distrobox :P

Ugh C++ dependencies bring me so much pain. I'd take a hundred weird react-native version mismatches over rustc being unable to find OpenSSL for a library I didn't think would even need rustc in the first place. This is not an invitation to help because my setup is far far away from the happy path of debugging.

I think I'd enjoy living in a wizard tower.