@cwebber Is this from personal experience and finding out or second hand from what others have said?
@mauve From reading multiple papers studying the effects of code assistants
@cwebber Interesting. My main frustratiom is thatthe code it write just isn't anywhere near the quality I need and it generates stuff that barely makes sense. I've mostly been exploring this as an alternative to speech to text coding and for doing some quick formatting but so far it's just sucked. Even very junior devs need less hand holding and reviews
absolutely same experience.
i can only imagine that some ppl doing more mainstream or generic programming tasks get some off-the-shelf templates that kinda help?
I even went through various exercises with and without the help of chatgpt to see if there is any improvement in quality or speed, but for the life of me cant make it produce anything useful that would be in any way an improvement over doing it all on my own.
It feels more like me teaching and babysitting chat gpt
@fleeky I only use locally runnable models so I have no clue how well it compares to the big corpo ones. 🤷 Sadly the code Ibwas using this on is a proprietary codebase so I can't share it. I ended uo deleting and rewriting myself like three times. It was some pretty basic cli arg parsing code tho
@mauve *edit* have a feels that quantizing a model down definitely impacts it's effectiveness.. I definitely have suboptimal results with local models unfortunately. I think there are definitely ways to improve it but that is it's own super deep research project...
@mauve i havent been able to try it personally but this person made there own fine tuned model for godot coding and supposedly it has good results.
basically if you want good local results you probably have to fine tune some model , boo ..
@fleeky yeah sadly fine tuning needs way more resources than I have available. :( Maybe if I get an egpu?
@mauve are you using ollama? There are many coding models you could easily pull and try out. It also features lots of Integrations into IDEs and text editors.
@hermeticvm Yeah I'm using continue.dev with ollama and phi3
Previously I used some 7B models that were just too slow.
The starcoder model has been giving okay suggestions once in a while but is super slow still. Deepseek seems to be too dumb to do anything useful.
@mauve I built a simple python app taking various file inputs in format x,y,z and convert it to a clean epub with the partial help of ChatGPT and codellama2 to covert to a specific open format and it worked pretty well. I didn't use any IDE integration but instead prompted them to be a guidance instead of writing the entire code autonomously.
Great to learn and I don't end up with code I don't understand and thus can't fix.
@hermeticvm Cool to hear ty for sharing. Is your code published somewhere I may read?
Today I was trying to get it to generate some cli commands in Node.js (this is with phi3+continue via ollama) but t has trouble adhering to the specific syntax and structure I wanted it to use.
I found a big issue for me has been hallucinated packages or APIs within packages. Maybe doing multi step stuff where inject context from api docs could help there.
@mauve Based on what I've read from others who have tried using copilot to generate code, the problem is that because it's an LLM AI with no actual knowledge of coding practices, it simply sucks ate generating working code if it can't find a known working piece of code to crib from.
@mauve For languages I know the generators only slow me down. When I don't know the language and I just need a snippet or two I've had some luck getting a generator to suggest a basic strategy and useful syntax but then I end up rewriting it. So, not useless but also not terribly helpful.
@mauve they're great for generating bugs you won't know how to fix later