Blue_Cosma

yes, private AI is not financially viable right now vs. APIs. The main reason to invest in such hardware is privacy and security.

Thanks a lot for sharing. Haven't tested Strix Halo myself. Did you consider DGX Spark as well?

What is your target use case? Curious what feels suboptimal about llama.cpp + Vulkan so far.

Our main driver and hypothesis was to work with regulated industry. We worked with a few large enterprise clients in defence and industry for R&D and IP use cases mostly.

Our stack changes per project, adapting to client needs and infra: Llama 70B on a Mac Studio M1 with Ollama in 2024, vLLM on 4xH100 private cloud for larger deployments. Most recently, we've been working on a custom workstation with 2x RTX PRO 6000 Blackwell Max-Q + 1.1TB DDR5 to run larger models locally using SGLang and KTransformers.

The question isn't rhetorical, I'm trying to understand if the demand we see in regulated sectors is the whole market or if there's broader adoption I'm missing.

Excellent point. How do you evaluate a developer’s ability to learn and integrate new tools (like AI assistants) into their workflow?

This is indeed during interviews (question updated) Thanks! I assume the 1-hour coding session is done live. From your experience, do candidates seem comfortable using AI tools as naturally as they would on their own? Do you also pay attention to how they interact with these tools — for example, prompting, reviewing, or correcting suggestions?

AI coding tools (Cursor, Claude Code, etc.) are now part of most developers’ daily workflow, they speed up prototyping, planning, implementation but also change how we think and debug.

I’m curious how you assess developers’ ability to leverage these tools efficiently during the recruiting process. Any tips to share? Any return on experience?

Hi, Constant here. I'm the Chief Mobster!

On hallucinations, larger models like Llama 3.1 70B still have a clear advantage, but they're also too costly to run at the moment on most API for those use cases. We built our API specifically for that reason.

I will try to make the disclaimer even more visible. We're not trying to hide anything here !

Post here if you have any comments or questions.

Let me know if you have any questions about the article