dev_tools_lab

This is exactly why single-model evaluation is dangerous. Benchmarks are gamed, but disagreement between models is harder to fake. Multi-model consensus catches what individual benchmarks miss.

Thanks for this project. Prioritizing MoE models and adding an intelligent NVMe cache could improve efficiency, especially on the M4 Max where bandwidth makes usage more realistic.

True, Claude and Gemini aren’t local yet — I mostly meant running all available local models in parallel.

Even with just open-source LLMs, you can see interesting differences in flagged issues when cross-validating outputs.

Agreed. The practical implications are often more interesting than the math anyway — smaller models running locally means you can afford to run multiple models in parallel for cross-validation, which changes how you approach tasks like code analysis or bug detection.

Nice work on the scheduler. Have you benchmarked parallel inference across multiple models? Running GPT, Claude and Gemini simultaneously on the same input is where latency becomes a real constraint.

Makes sense for now. Thresholding becomes critical at scale though — good luck with the next iteration!

Good reminder to pin dependency versions and verify checksums. SHA256 verification should be standard for any tool that makes network calls.

Nice use of native video embedding. How do you handle cases where Gemini's response confidence is low? Do you have a fallback or threshold?

One pattern I've noticed: the apps that work best combine multiple models rather than relying on one. Single-model outputs have too much variance for production use cases.