Hacker News Reader

Pokémon is increasingly used to evaluate modern large language models, but current practices lack standardization, and depend heavily on game-specific harness. The Pokémon Red involves three major tasks—navigation, combat control and training a competitive Pokémon team. We find they come with limitations: navigation tasks are too hard, combat control is too simple, and Pokémon team training is too expensive. We address these issues in Lmgame Bench, a new framework offering standardized evaluations and initial results across diverse games.

zhisbugon [dead]

where other models tops out in a few moves

zhisbugon [dead]

More details here: https://x.com/haoailab/status/1909712259326394519

zhisbugon [dead]

We find that spatial perception and spatial reasoning remain very difficult even for the strongest models like o3 or Claude 3.7

zhisbugon Can LLMs play real-time games like supermario (other than Pokemon red)?

gaming agent code here: https://github.com/lmgame-org/GamingAgent/tree/main

zhisbugon Sliding Tile Attention: A New Method That Speeds Up HunyuanVideo's Outputs by 3x

https://hao-ai-lab.github.io/blogs/sta/

zhisbugon Fast Video Generation with Sliding Tile Attention

Sliding tile attention accelerates Hunyuan video generation by 3x with no quality drop and no need for training

zhisbugon More Efficient Chain-of-Thought Reasoning Through Certainty Probing

Try our demo and let us know

zhisbugon Evaluating LLM Reasoning Through Live Computer Games

This is pretty clever and seems to have high potential, but it still relies on humans. What if some day all humans cannot outsmart AI?

zhisbugon AI Space Escape: Playing Games While Evaluting LLM Reasonsing

please try and give us feedback!

zhisbugon AI Space Escape: Playing Games While Evaluting LLM Reasonsing

We hope to redefine ai evaluation via our gamified AI evaluation platform: game arena!

zhisbugon Efficient LLM Scheduling by Learning to Rank

We study how to approximate the famous shortest-job-first scheduling in LLM inference!

zhisbugon FastVideo: a lightweight framework for accelerating large video diffusion models

Hugginface model and data link: https://huggingface.co/FastVideo

zhisbugon Throughput Is Not All You Need: Maxing Goodput in LLM Serving via Disaggregation

New work from the vLLM team that disaggregates prefill and decoding to maximize goodput (throughput subject to latency constraints) in LLM serving

zhisbugon Break the Sequential Dependency of LLM Inference Using Lookahead Decoding

New parallel decoding algorithm that trades flops for latency reduction

zhisbugon vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention

oops typo

zhisbugon vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention

yep, it is agonistic to 4-bit. You can deploy a 4-bit model and still use vllm + pagedattention to double or even triple your serving throughput.

zhisbugon vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention

This really depends on what GPUs you use. If you GPUs has very small amount of memory, vLLM will help more.

vLLM addresses the memory bottleneck for saving KV caches and hence increases the throughput.

zhisbugon Chatbot Arena: Benchmarking LLMs in the Wild with Elo Ratings

could you provide any reference? Is it a variant of ELO?

zhisbugon Fastchat-T5: 4x smaller but more powerful than Dolly-v2, commercial use ready

Released by Lmsys.org who made Vicuna

zhisbugon State-of-the-Art Chatbot, Vicuna-7B, now runs on MacBook with GPU acceleration

And I think the problem of taking the roles of users in vicuna is caused by this bug: https://github.com/lm-sys/FastChat/commit/1bb234265d16bdfd50...

which has been fixed recently.

Lmsys are launching new training jobs after this patch, please stay tuned.

zhisbugon State-of-the-Art Chatbot, Vicuna-7B, now runs on MacBook with GPU acceleration

My point is that I am not aware of any official 4-bit quantization version (delta or weights) by lmsys so it might too early to draw your conclusion that vicuna finetuned llamas degenerates a lot of performance at 4 bit but others are fine.

zhisbugon State-of-the-Art Chatbot, Vicuna-7B, now runs on MacBook with GPU acceleration

it is really a matter of having faith on pytorch (or JAX) or on third-party cross-platform supports like llama-cpp. Apparently pytorch reduces a lot of complexity and grows extremely faster on cross-platform supports.

And, PyTorch does so well on GPUs!

zhisbugon State-of-the-Art Chatbot, Vicuna-7B, now runs on MacBook with GPU acceleration

no, the requirement on a particular HF commit has been fixed. It is no longer needed.

zhisbugon State-of-the-Art Chatbot, Vicuna-7B, now runs on MacBook with GPU acceleration

Lmsys hasn't released any official 4-bit version. It might be a better idea to wait for the official 4-bit version. But it is interesting to learn that the third-party 4bit version has performance degeneration.

zhisbugon State-of-the-Art Chatbot, Vicuna-7B, now runs on MacBook with GPU acceleration

No llama.cpp nor any compilation complexity. Run with two Python commands!

zhisbugon State-of-the-art open-source chatbot, Vicuna-13B, just released model weights

https://github.com/facebookresearch/llama/pull/184