ericcurtin

Time to use some local ai with Docker Model Runner :)

No cloudflare no problem

https://github.com/docker/model-runner

I'm one of the devs, happy to answer any questions

Will be around the next few hours

Hi everyone, we're the maintainers.

We're rebooting the model-runner community and wanted to share what we've been up to and where we're headed.

When we first built this, the idea was simple: make running local models as easy as running containers. You get a consistent interface to download and run models from different backends (llama.cpp being a key one) and can even transport them using familiar OCI registries like Docker Hub.

Recently, we've invested a lot of effort into making it a true community project. A few highlights:

- The project is now a monorepo, making it much easier for new contributors to find their way around.

- We've added Vulkan support to open things up for AMD and other non-NVIDIA GPUs.

- We made sure we have day-0 support for the latest NVIDIA DGX hardware.

- There is a new "docker model run" UX: https://www.docker.com/blog/docker-model-run-prompt/

Keep opening pull requests and issues, we need these things, you are right!

Any new features you think we should add to further enhance your usage? Glad you find it useful

Is there overlap with "Show HN: docker/model-runner – an open-source tool for local LLMs" posted today, we encapsulate AI workloads in containers using docker model runner. Some clear encapsulations advantages at a minimum. Could we work together?

What did you like? Anything stand out?

Thanks very much. It worked well for you? Which hardware? :) Any other feedback, keep it coming!

Hi everyone, we're the maintainers.

We're rebooting the model-runner community and wanted to share what we've been up to and where we're headed.

When we first built this, the idea was simple: make running local models as easy as running containers. You get a consistent interface to download and run models from different backends (llama.cpp being a key one) and can even transport them using familiar OCI registries like Docker Hub.

Recently, we've invested a lot of effort into making it a true community project. A few highlights:

- The project is now a monorepo, making it much easier for new contributors to find their way around.

- We've added Vulkan support to open things up for AMD and other non-NVIDIA GPUs.

- We made sure we have day-0 support for the latest NVIDIA DGX hardware.