I let my local model read whatever it wants and use a bunch of "non destructive" bash commands, but I review everything else it tries to do. So far so good. Maybe my closet box is too dumb to bother doing fancy sandbox escapes.
@mauve Can I ask you which model you are using? I'm curious if there's a good one I can run on my laptop locally.
@futureisfoss Hmm tbh it depends on what you're trying to do. I'd say for multi round coding taskd with longer contexts anything less than 27B won't cut it. And even then Qwen 3.6+ seems to be one of the few that actually cut it. For basic stuff like zero shot bash scripts or code snippets 8b models can work well enough, even the 4b gemma models can handle zero shot tasks if you keep their context small. Use vllm or llama.cpp instead of ollama for running inference.
@futureisfoss the meta right now is buying those mac minis with the "unified memory" so you can use the RAM with their GPU acceleration. Something like a 64 gig box should do the trick. I ended up getting a 128 GB DGX spark alternative cause I didn't want to worry about ram for a few years. 😝