@mauve looks like Replicate are running it on a A100 40GB, which is a ~$7,000 GPU!

@simon Quality of results. The same query on the ggml model via LLM vs the raw model on replicate has vastly different results. It feels like the replicate instance is more stable and has less jitter? I feel like the parameters for max tokens and heat might be the culprit.

@simon I'm on a steam deck and the speed is actually great!

@mauve yeah it would be interesting to figure out if they are using different parameters

@simon I'll dig around. Are there already llm plugins that specify how to change parameters? Might have a pr up my sleeve if there's time this weekend.

@mauve this one does https://github.com/simonw/llm-llama-cpp/blob/main/README.md#options

@simon Looked into this, I think the top_p and top_k are the main differences. The default in gpt4all is way more "loose".

Would a PR that sets different defaults be welcome? Or would you prefer to just have the flags exposed like your llama-cpp example?

https://github.com/simonw/llm-gpt4all/blob/main/llm_gpt4all.py#L112

https://docs.gpt4all.io/gpt4all_python.html#the-generate-method-api

I'll try hardcoding some values and running a generation again to see if it's "better" in the meantime.

@mauve exposing options like llama-cop does would be fantastic

@simon PR: https://github.com/simonw/llm-gpt4all/pull/17

Gonna need to mess with the parameters more another day though. But my gut feeling is we can up the quality of output significantly by turning down the temperature a bit and reducing the top_p to 1 and top_k to 4 like in the replicate.com demo

@mauve my plan at the moment is to make it much easier for people to experiment with and share alternative configurations for different models

@simon Nice, like a file format for the configs so folks could pass them around and track changes in git?