p12tic

> Only a matter of time for local models to reach Opus level. We are 1 or at most 2 years behind that and Anthropic knows that.

Can confirm. Kimi K2.5 is pretty intelligent and most of the time there's no difference between Opus and Kimi.

> In the last few years California runs 100% renewable on many days (and growing) every year.

How many is "many days"? Gas is still used for at least one fifth of electricity. https://app.electricitymaps.com/map/zone/US-CAL-CISO/5y/mont...

> not ... web/db servers, lightweight stuff like that.

They scale very well for web and db servers as well. You just put lots of containers/VMs on a single server.

AMD EPYC has a separate architecture specifically for such workloads. It's a bit weaker, runs at lower frequency and power and takes less silicon area. This way AMD can put more such cores on a single CPU (192 vs 128 for Zen 5c vs 5). So it's the other way round - web servers love high core count CPUs.

Depends on a server. This test got 79W idle for _two socket_ E5 2690-V4 server.

https://www.servethehome.com/lenovo-system-x3650-m5-workhors...

The problem is with the form factor, not the server hardware per-se. If one buys regular ATX motherboard that accepts server CPUs and fits it in regular ATX case, then there's lots of space for a relatively silent CPU air cooler. 2690 v4 idles at less than 40W which is not much more than a regular gaming desktop with a powerful GPU.

The only problem in practice is that server CPUs don't support S3 suspend, so putting whole thing to sleep after finishing with it doesn't work.

Better build a single workstation - less noise, less power usage and the form factor is way more convenient. A budget of $3000 can buy 128 cores with 512GB of RAM on a single regular EATX motherboard, a case, a power supply and other accessories. Power usage is ~550W at maximum utilization which not much more than a gaming rig with a powerful GPU.

> Today it's a bit more complicated when you have servers with 100+ cores as an option for under $30k (guestimate based on $10k CPU price).

If one can buy used, then previous generation 128C 256T epyc server is less than $5k. For homelabs that can accept non-rackmount gear it's less than $3k.

That's just an artifact of Intel disabling ECC on consumer processors.

There's no reason for ECC to have significantly higher power consumption. It's just an additional memory chip per stick and a tiny bit of additional logic on CPU side to calculate ECC.

If power consumption is the target, ECC is not a problem. I know firsthand that even old Xeon D servers can hit 25W full system idle. On AMD side 4850G has 8 cores and can hit sub 25W full system idle as well.

We both agree. Batch size 1 is only relevant to people who want to run models on their own private machines. Which is the case of OP.

All of this is for batch size 1.

State of the art of local models is even further.

For example, look into https://github.com/kvcache-ai/ktransformers, which achieve >11 tokens/s on a relatively old two socket Xeon servers + retail RTX 4090 GPU. Even more interesting is prefill speed at more than 250 tokens/s. This is very useful in use cases like coding, where large prompts are common.

The above is achievable today. In the mean time Intel guys are working on something even more impressive. In https://github.com/sgl-project/sglang/pull/5150 they claim that they achieve >15 tokens/s generation and >350 tokens/s prefill. They don't share what exact hardware they run this on, but from various bits and pieces over various PRs I reverse-engineered that they use 2x Xeon 6980P with MRDIMM 8800 RAM, without GPU. Total cost of such setup will be around $10k once cheap Engineering samples hit eBay.

Incorrect. https://en.wikipedia.org/wiki/USB_hardware#USB_Power_Deliver... is a good start about the subject: "PD-aware devices implement a flexible power management scheme by interfacing with the power source through a bidirectional data channel and requesting a certain level of electrical power <...>".

For all intents and purposes cache may not exist when the working set is 17B or 109B parameters. So it's still better that less parameters are activated for each token. 17B parameters works ~6x faster than 109B parameters just because less data needs to be loaded from RAM.

Most laptops are severely limited by heat dissipation. So it's normal that performance is much worse. The CPU cannot stay in turbo as long and must drop to lower frequencies sooner. On longer benchmarks they CPU starts throttling due to heat and becomes even slower.

Container security boundary can be much stronger if one wants.

One can use something like https://github.com/google/gvisor as a container runtime for podman or docker. It's a good hybrid between VMs and containers. The container is put into sort of VM via kvm, but it does not supply a kernel and talks to a fake one. This means that security boundary is almost as strong as VM, but mostly everything will work like in a normal container.

E.g. here's I can read host filesystem even though uname says weird things about the kernel container is running in:

  $ sudo podman run -it --runtime=/usr/bin/runsc_wrap -v /:/app debian:bookworm  /bin/bash
  root@7862d7c432b4:/# ls /app
  bin   home            lib32       mnt   run   tmp      vmlinuz.old
  boot  initrd.img      lib64       opt   sbin  usr
  dev   initrd.img.old  lost+found  proc  srv   var
  etc   lib             media       root  sys   vmlinuz
  root@7862d7c432b4:/# uname -a
  Linux 7862d7c432b4 4.4.0 #1 SMP Sun Jan 10 15:06:54 PST 2016 x86_64 GNU/Linux

One can use something like https://github.com/google/gvisor as a container runtime for podman or docker. It's a good hybrid between VMs and containers. The container is put into sort of VM via kvm, but it does not supply a kernel and talks to a fake one. This means that security boundary is almost as strong as VM, but mostly everything will work like in a normal container.

E.g. here's I can read host filesystem even though uname says weird things about the kernel container is running in:

  $ sudo podman run -it --runtime=/usr/bin/runsc_wrap -v /:/app debian:bookworm  /bin/bash
  root@7862d7c432b4:/# ls /app
  bin   home            lib32       mnt   run   tmp      vmlinuz.old
  boot  initrd.img      lib64       opt   sbin  usr
  dev   initrd.img.old  lost+found  proc  srv   var
  etc   lib             media       root  sys   vmlinuz
  root@7862d7c432b4:/# uname -a
  Linux 7862d7c432b4 4.4.0 #1 SMP Sun Jan 10 15:06:54 PST 2016 x86_64 GNU/Linux

Gvisor let's one have strong sandbox without resorting to WASM.

Seems like that's true: https://learn.microsoft.com/en-us/sharepoint/redirect-known-...

It's complicated, memory accesses can really block for relatively long periods of time.

Consider that regular memory access via cache takes around 1 nanosecond.

If the data is not in top-level cache, then we're looking at roughly 10 nanoseconds access latency.

If the data is not in cache at all, we are looking into 50-150 nanoseconds access latency.

If the data is in memory, but that memory is attached to another CPU socket, it's even more latency.

Finally, if the data access is via atomic instruction and there are many other CPUs accessing the same memory location, then the latency can be as high as 3000 nanoseconds.

It's not very hard to find NVMe attached storage that has latencies of tens of microseconds, which is not very far off memory access speeds.

That would be a significant downgrade. Threadripper CPUs top out at 64 cores / 128 threads.

I agree that just buying the most expensive monitor is waste of resources. On the other hand, I think price shouldn't even come into the picture when deciding on basic things, such as the number of monitors or whether to choose 4K monitor or just FHD.

I guess the most reasonable approach is to select a set of models that pass the requirements and only then think about the price.

Some requirements are really expensive though. If one insists on OLED and wants to fit 3 4K monitors side by side the full setup cost is around $10k.

It's strange when monitor costs are being discussed while they are the primary tool for software development. If software developer produces $100k/year value, then it's worth spending several thousands of dollars for every percent of increased effectiveness. This can buy almost any monitor on the market.

I've found the following worth considering:

- It makes sense maximizing usable screen real estate, both in terms of pixels and area. Complex problems involve a lot of information and being able to view everything at the same time makes it easier to understand them. Personally I've found three 4K 32 inch monitors work well.

- OLED allows working comfortably at low lightning conditions.

- Many larger higher-end monitors support picture-by-picture, which allows splitting the monitor into multiple virtual monitors for separate inputs (2x or 4x per monitor). This is useful in case of multiple external test devices which can then be integrated into existing display setup.

It uses a custom widget toolkit. Adding touchpad gesture support is certainly doable, but it would benefit only single application, so we haven't prioritized that so far.

Touchpad gesture implementation in X server has been released in version 21.1

Wayland touchpad gesture implementation in Qt widget framework has been released in version 6.2.0

X11 touchpad gesture implementation in Gtk widget framework has been released in version 4.5.0

X11 touchpad gesture implementation it Qt widget framework will be released in version 6.3.0 (March 2022)

Touchpad gesture implementation for XWayland will be released in version 22.1 (early 2022)

Fedora and Gnome is only one desktop environment and one widget toolkit. For example, Qt-based apps didn't have touchpad gestures at all anywhere until my work on Wayland gestures landed this year.

It is true though that if one limits oneself to Wayland and only to Gtk-based applications that touchpad gestures mostly worked before. We now switched focus to adding touchpad gesture support to more applications, so there will be measurable progress even for this case too.

Interesting, this feature may only be for touchscreens because two-finger swipes are registered as scrolls on Wayland. This will indeed need further work.

What does work though is two-finger pinch gesture to zoom in/out of a web page.

Most applications will use either Gtk or Qt widget libraries, so a lot of similarity of how applications behave already exists.

I don't think it's possible to make Gtk and Qt themselves behave identically at this point. For example Qt is a commercial project and an effort that would break backwards compatibility is not worth it. Gtk on the other hand has strong opinions about how things should work, so it would be hard to change that too.

For now we only focus on touchpads. I think that if we're successful in delivering touchpad improvements then we will gain credibility and trust that could be useful when working on other input devices.

In short - there are no binaries and it's relatively hard to compile these manually, so I recommend to wait until the Linux distributions picks these projects up. This usually takes around 6 to 12 months.

You need to wait up to a year for the distributions to pick up the code into their default installs. Installing bleeding edge window managers, widget toolkits and similar software yourself is more difficult than it's worth for a normal user.

Hi, I'm the developer behind this effort. I can answer any questions you have.