abcdabcd987

Related discussion on serving finetuned LLMs: https://news.ycombinator.com/item?id=38196661

Thanks for your encouragement! We are working on quantization as well. We recently submitted a paper, Atom [1], that uses 4-bit quantization, delivering 7.73x throughput compared to FP16 and 2.53x compared to INT8. Atom is able to maintain a perplexity (i.e., model accuracy) close to FP16, outperforming existing quantization approaches.

We are polishing the 4-bit code. It will be added to Punica code base soon. Please stay tuned :)

[1] https://arxiv.org/abs/2310.19102

Certainly! We'd like our good designs to be picked up by frameworks and serve all users. Currently, Punica is built on top of PyTorch and HuggingFace Transformers ecosystems. Therefore, vLLM and LMDeploy, which are also in the PyTorch ecosystem, should have a smooth adaption. As for Nvidia Triton and TensorRT-LLM, since our kernels are written in CUDA, I believe it will also work seamlessly.

We call for the open source community to help us integrate Punica with all frameworks, thus the whole society can benefit from the efficiency improvement!

Thank you! We are also very excited about combining the fast fine-tuning and efficient serving. In fact, what you just said is very related to one of our very first motivations. In my previous blog post [1], I call this scheme "Just-in-time Fine-tuning". Our previous measurement is that, for a medium-sized webpage (~10K tokens), it takes around 30 seconds to 2 minutes to finetune a LoRA model. Another good side of this JIT fine-tuning scheme is that, we can turn any model into a long-context model.

We'll keep doing more research on finetuning. And hopefully, we'll see the results soon.

[1] https://le.qun.ch/en/blog/2023/09/11/multi-lora-potentials/

Thanks for the question. Currently Punica is on the ecosystem of PyTorch and HuggingFace Transformers. So PyTorch users can start to use Punica now.

Look forward to collaboration with TVM and MLC to reach more users :)

On a related note, Google also uses optical circuit switches in their datacenter network. See the paper form SIGCOMM'22 [1].

[1] Jupiter Evolving: Transforming Google’s Datacenter Network via Optical Circuit Switches and Software-Defined Networking. https://research.google/pubs/pub51587/

I'm using 27-inch 4K monitors with Gnome. I found it more practical to use no scaling but 1.25x text size (settings in gnome-tweaks). The problem for me with 2x scaling or fractional scaling is that it scales UI (icons, margins) as well. As someone who appreciates functionality (i.e., displaying text) over overdesigned white spacing, scaling the whole UI is just wasting my workspace size whereas scaling text-only is a perfect balance between good-looking text and workspace size.

But anyway, good to see that Linux desktop is gaining fractional scaling support!

I use Automate[1] to do this. Just draw an infinite loop that does sms_received[2] and email_send[3].

[1] https://play.google.com/store/apps/details?id=com.llamalab.a...

[2] https://llamalab.com/automate/doc/block/sms_received.html

[3] https://llamalab.com/automate/doc/block/email_send.html

> If you find a really good tutorial by programmers who are both excellent teachers and experienced in that particular field, it beats JIT learning on a personal project in one important respect. You get exposure to the One True Way of doing things.

That's exactly my feeling when I was watching Jon Gjengset's Rust tutorials. I like his real reactions to unexpected problems. Really learned a lot from this kind of lengthy but realistic videos. https://www.youtube.com/c/JonGjengset/featured

Just mentioning two related programs in my mind: Franz (https://meetfranz.com/) and nativefier (https://github.com/jiahaog/nativefier).

Just want to mention bitbar[1]. Write anything to stdout, it'll make it show up in the menu bar.

[1] https://github.com/matryer/bitbar

Check out the HotOS'19 paper from our lab :) https://danyangzhuo.com/papers/HotOS19-Stone.pdf

I think I did almost the same thing three years ago. See: https://www.v2ex.com/t/121272 (in Chinese only, sorry. I should translate it to English when I'm free)

Well, it's not a big problem. This case won't exist in most BSTs. Even if it appears in some BST, splay tree for example, it will be amortized.

I'm confused. I can follow the logic for O(sqrt n) searching. yeah, that's cool. But how to maintain the structure when inserting or deleting?

Insertion for example, if I want to insert a value which is smaller than the smallest element in the last heap, then it'll have to be inserted into some heap in the middle, right? And since the heap is in the middle, say heap K, its size already reaches its upper limitation K^2. Then where should the new value go? If I insist on pushing it into heap K, then the heap will violate the size limitation. Should I split the oversized heap into several heaps some time? If it does split, will the O(sqrt n) searching time still holds?

Or maybe I just have not caught the author's idea yet. :-(

BTW, why not BST, for everything is O(log n)?

Update: @nikic solved my questions. Thanks!