I really want to love this model, particularly since it's good to have an open weight model on my team so to speak, but I think I might be holding it wrong.
What are you using it for?
I really want to love this model, particularly since it's good to have an open weight model on my team so to speak, but I think I might be holding it wrong.
What are you using it for?
I'm also using it with my own harness (you mentioned "in my harness"), so I'll share two things I found about using open weight model providers. The first is that getting the configuration right isn't a per model thing, its a per model per provider per point in time thing. The point in time thing comes from middleware updates as those introduce or fix defects that can very seriously impact inference quality. The second thing is inference quality varies a lot between providers, even when they are ostensibly using the same quant of the same model.
If you want to try it something else for comparison I open sourced my harness a couple of weeks ago: https://fluximp.org
I recommend trying deepseek v4.1 flash from together.ai (not the 1017 one the one released last month). deepinfra.com is cheaper for that but its noticeably slower and the inference from together.ai feels like it might be better (not quantified; just vibes).
What type of ongoing tuning do you do for your model+provider+harness combo, and when do you know it's time to redo it? Or more specifically, when you do the tuning, how do you actually go about doing it?
I am mostly using omp (omp.sh) which is very tunable, and I did some tuning for one of the GLM models in the past (mostly based on vibes because people said it was getting dumb like a brick after some context lengths so I had omp compress sooner).
On an overall speed/cost/quality comparison, it seems unmatched in my experience these days.
For tasks like crawling and real time log analysis etc., the costs really add up at scale.