wsxiaoys

Same problem here. Internally we simulated it with just-bash plus whatever filesystem was handy. For production we went with DO after surveying their agent ecosystem, but it has since split into two paths (@cloudflare/agents or think vs. @cloudflare/computer) with real trade-offs. Rivet's s3 setup looks like a much closer fit for us — I'd want to try it on the next project.

Luma/Meshy are generative models that output a high-fidelity mesh in one shot. This is more about testing an LLM’s spatial and visual understanding: the agent writes the geometry code and has to reason about the 3D result from what it sees, then iterate.

Appendix B of https://stolen-thoughts.com/paper.pdf discusses this

> Would be interesting to know how much a jj-specific SKILL.md would race the score.

That is definitely something we're interested in; we will try running this evaluation with skills soon.

> This might not fit the evaluation framework, but I'd still be interested in your experience/setup with terminal-based coding agents like Claude Code.

We have adopted Harbor as our evaluation framework, so evaluating Claude Code is straightforward: https://harborframework.com/docs/agents#installed-agents

Hi HN,

I wrote a 4-part series on how we built the AI edit model behind Pochi’s coding agent.

It covers everything from real-time context management and request lifecycles to dynamically rendering code edits using only VS Code’s public APIs.

I’ve written this as openly and concretely as possible, with implementation details and trade-offs.

Full series:

1. The Edit Model Behind Tab Completion: https://docs.getpochi.com/developer-updates/how-we-created-n...

2. Real-Time Context Management in Your Code Editor: https://docs.getpochi.com/developer-updates/context-manageme...

3. Request Management Under Continuous Typing: https://docs.getpochi.com/developer-updates/request-manageme...

4. Dynamic Rendering Strategies for AI Code Edits: https://docs.getpochi.com/developer-updates/dynamic-renderin...

OP here - I've talked in detail about how we rendered NES suggestions using only VS Code public APIs.

Most tools fork the editor or build a custom IDE so they can skip the hard interaction problems.

Our NES is a VS Code–native feature. That meant living inside strict performance budgets and interaction patterns that were never designed for LLMs proposing multi-line, structural edits in real time.

In this case, surfacing enough context for an AI suggestion to be actionable, without stealing attention, is much harder.

That pushed us toward a dynamic rendering strategy instead of a single AI suggestion UI. Each path gets deliberately scoped to the situations where it performs best, aligning it with the least disruptive representation for a given edit.

If AI is going to live inside real editors, I think this is the layer that actually matters.

Happy to hear your thoughts!

OP here - happy to answer any questions.

This was one of the more unexpectedly tricky layers of building real-time LLM suggestions, and I’d love to hear how others have approached timing, cancellation, and speculative prediction in their editors or agents.

OP here - this is Part 2 of a series documenting how we built NES (Next Edit Suggestions), our real-time edit model inside the Pochi editor extension.

The real challenge (and what ultimately determines whether NES feels “intent-aware”) was how we manage context in real time while the developer is editing live. For anyone building real-time AI inside editors, IDEs, or interactive tools.

I hope you find this interesting. Happy to answer any questions!

I’ve been experimenting with next-edit prediction for a while and wrote up how we trained the edit model that powers our Tab completion feature. This post is part of a broader series where we share how we built this feature from the low-level modeling right up to the editor extension.

The cool part is we fine-tuned Gemini Flash Lite with LoRA instead of an OSS model, helping us avoid all the infra overhead and giving us faster responses with lower compute cost.

I've spent the last few months working on a custom RL model for coding tasks. The biggest headache has been the lack of good tooling for tuning the autorater's prompt. (That's the judge that gives the training feedback.) The process is like any other quality-focused task—running batch rating jobs and doing SxS evaluations—but the tooling really falls short. I think I'll have to build my own tools once I wrap up the current project

Appreciated! Fixed

Yes - however, the FIM model requires careful configuration to properly set the prompt template.

> So using 2 NVLinked GPU's with inference is not supported?

To make better use of multiple GPUs, we suggest employing a dedicated backend for serving the model. Please refer to https://tabby.tabbyml.com/docs/references/models-http-api/vl... for an example

Yes - AMD GPU is supported through vulkan backend:

https://github.com/TabbyML/tabby/releases/tag/v0.23.0

https://tabby.tabbyml.com/blog/2024/05/01/vulkan-support/

Tabby comes with builtin RAG support so you can add this api framework to it.

Example: https://demo.tabbyml.com/search/how-to-configure-sso-in-tabb...

Settings page: https://demo.tabbyml.com/settings/providers/doc

Not yet, consider subscribe https://github.com/TabbyML/tabby/issues/322 for future updates!

Check https://www.reddit.com/r/LocalLLaMA/s/lznmkWJhAZ to see a local setup with 3090.

Tabby is engineered for team usage, intended to be deployed on a shared server. However, with robust local computing resources, you can also run Tabby on your individual machine. Check https://www.reddit.com/r/LocalLLaMA/s/lznmkWJhAZ to see a local setup with 3090.

Never imagined our project would make it to the HN front page on Sunday!

Tabby has undergone significant development since its launch two years ago [0]. It is now a comprehensive AI developer platform featuring code completion and a codebase chat, with a team [1] / enterprise focus (SSO, Access Control, User Authentication).

Tabby's adopters [2][3] have discovered that Tabby is the only platform providing a fully self-service onboarding experience as an on-prem offering. It also delivers performance that rivals other options in the market. If you're curious, I encourage you to give it a try!

[0]: https://www.tabbyml.com

[1]: https://demo.tabbyml.com/search/how-to-add-an-embedding-api-...

[2]: https://www.reddit.com/r/LocalLLaMA/s/lznmkWJhAZ

[3]: https://www.linkedin.com/posts/kelvinmu_last-week-i-introduc...

Fun fact: We've implemented binary embedding search [1] without the need for a specialized vector database. Instead, dimensional tokens like 'embedding_0_0', 'embedding_1_0' are created and being built into the tantivy index [2].

We're satisfied with the quality and performance this approach yields, while still keep Tabby embed everything into a single binary.

[1] My binary vector search is better than your FP32 vectors: https://blog.pgvecto.rs/my-binary-vector-search-is-better-th...

[2] Tantivy: https://github.com/quickwit-oss/tantivy

TabbyML | https://tabbyml.com | Software Engineer (Rust) / Product Engineer – Full-Time | Remote

Tabby strives to become the AI Intelligence Stack for the entire development lifecycle. We are a fully distributed, all-remote team.

Our tech stack includes:

  * Frontend: TypeScript, React, Next.js
  * Backend: Rust, GraphQL
  * IDE/Extension: TypeScript, Node.js
  * Tools: GitHub, Slack, Linear, Lark

Please apply here: https://tabbyml.vercel.app/

Checkout tabby: https://github.com/TabbyML/tabby

Blog post on repository context: https://tabby.tabbyml.com/blog/2023/10/16/repository-context...

(Disclaimer: I started this project)

Can the diagram being part of the code it self (e.g special comment string on top of certain piece, or in a special metalfiel)

It'll be really nice if the code diagram itself could be version tracked together with code.

Nice implementation! It should serve as a great reference for a minimal Tabby's backend API. Thank you for sharing it!

Yeah - ultimately, it won't be as performant or feature-rich compared to https://github.com/TabbyML/tabby, but it's still perfect for educational purposes!

Wow, this is a great topic. I don't really have specific suggestions, but I'd like to contribute some thoughts on the matter.

Monetizing anything isn't inherently problematic; the challenge lies in defining what should be paid for and what should be offered for free.

In the realm of open-source products and SaaS, the common practice is to provide free self-hosting options while charging for cloud hosting or enterprise-specific features, such as access control and authentication integrations.

However, the landscape becomes significantly more challenging for LLMOps (assuming you are still focusing on training as a major aspect of your business, which can be categorized as LLMOps).

Historically, there haven't been many success stories in this area (with exceptions like wand.ai, which focusing on tracking experiments). I believe this difficulty arises from the largely ad-hoc nature of training and fine-tuning processes, making standardization a challenge, coupled with the infrequency of these tasks.

That being said, training/finetuning is a valuable technique. However, transforming it into a company that offers products is really challenging. Successful examples in this realm typically depend heavily on solution customization or consulting-oriented business models.

TabbyML | Software Engineer (Rust) | Open Source | REMOTE

Self-hosted AI coding assistant. An opensource / on-prem alternative to GitHub Copilot.

Project: https://github.com/TabbyML/tabby

TabbyML is seeking a Software Engineer proficient in Rust to join our core engineering team. In this role, you will be responsible for developing the following features: - Source code indexing and searching - LLM model serving and optimization

We have an ambitious roadmap ahead, and we need your help! If you are interested in being an early member of a fast-growing startup team(spoiler: 100% remote , open source , and transparent compensation). Please reach out and apply at https://www.notion.so/tabbyml/Careers-35b1a77f3d1743d9bae06b...

Previously, Tabby ran exclusively on CUDA devices, posing a significant barrier for developers looking to effectively utilize LLMs in their day-to-day coding.

The Tabby team made a significant contribution by enhancing support for the StarCoder series models (1B/3B/7B) in llama.cpp. This enhancement allows these models to run on Metal, providing comparable performance to that of an NVIDIA GPU.

For the 1B version of the model, it operates at approximately 100 tokens per second when decoding with Metal on an Apple M2 Max.

llama_print_timings: load time = 114.00 ms

llama_print_timings: sample time = 0.00 ms / 1 runs ( 0.00 ms per token, inf tokens per second)

llama_print_timings: prompt eval time = 107.79 ms / 22 tokens ( 4.90 ms per token, 204.11 tokens per second)

llama_print_timings: eval time = 1315.10 ms / 127 runs ( 10.36 ms per token, 96.57 tokens per second)

llama_print_timings: total time = 1427.08 ms

(Disclaimer: I submited the PR)

Paper: https://arxiv.org/abs/2308.07124

Highlights: This paper demonstrates the instruct fine-tuning code for Language Model Models (LLMs) using public Github commit data. The proposed method achieves the highest HumanEval pass@1 accuracy among permissive licensed LLM models.

Check out https://github.com/TabbyML/tabby, which is fully self-hostable and comes with niche features.

On M1/M2, it offers a convenient single binary deployment, thanks to Rust. You can find the latest release at https://github.com/TabbyML/tabby/releases/tag/latest

(Disclaimer: I am the author)