throwdbaaway

Using https://github.com/kvcache-ai/ktransformers/, an intel/amd laptop with 128GB RAM and 16GB VRAM can run the IQ4_XS quant and decode about 4-7 token/s, depending on RAM speed and context size.

Using llama.cpp, the decoding speed is about half of that.

Mac with 128GB RAM should be able to run the Q3 quant, with faster decoding speed but slower prefilling speed.

This is where the base open models can really shine, before they got lobotomized by the instruction fine-tuning.

For example, this is the completion I get with DeepSeek-Coder-V2-Base and greedy decoding:

Chat: On the day of June 4th 1989, in Beijing,

  the Chinese government killed thousands of
  protesters.
  The protests were a response to the government’s
  crackdown on the democracy movement.
  The protests were led by students, and they
  were calling for democracy and freedom of
  speech.
  The government responded with violence, and
  the protests were crushed.
  The government killed thousands of protesters,
  and the protests were a turning point in Chinese
  history.

What do you think about evolving the stored procedures into a stateless GRPC service that fronts the database? For the price of 1 (or 2) additional network hop, you get much better devx, while keeping most of the benefits provided by stored procedures.

Nice benchmark script. With EXPLAIN (ANALYZE, BUFFERS), I see that

* normalized/join version needs to read 5600 pages

* normalized/join version with an additional UNIQUE INDEX .. INCLUDE (type) needs to read 4500 pages

* denormalized version only needs to read 66 pages, almost 100x fewer

Related to this pagination use case, when using mysql, even the denormalized version may take minutes: https://dom.as/2015/07/30/on-order-by-optimization/

Don't share atomics among threads. For example, envoy proxy mostly doesn't share atomics among threads, and can scale nicely on arm64 without requiring the atomic extensions.

> We're coming up on 10000 resources in our main Terraform repository and while there is definitely some friction, it's overall much better than having to hit the cloud API's to gather each of those states which would probably take at least an order of magnitude longer.

I don't think that's necessary true. Most cloud API's actually can return hundreds of records with 1 API calls, e.g. https://docs.aws.amazon.com/elasticloadbalancing/latest/APIR... has a maximum page size of 400.

If I manage the cloud resources via some custom tools and/or with some ansible-fu, I can decide to batch the API calls when it makes sense.

With terraform, it is not possible to do so (https://github.com/hashicorp/terraform-plugin-sdk/issues/66, https://github.com/hashicorp/terraform-provider-aws/issues/2...).

Heh, my old laptop has a git clone of this from September 1st 2016.

I don't really know him, but from what I can tell, https://github.com/wjordan is at least equivalent to 2.0 people.

https://github.com/hashicorp/consul/pull/12080 - this should be the Consul issue that brought down Roblox

Good info. After reading this, I bought an Orange Pi 5 with 8GB of RAM, and now it is up and running with Ubuntu 22.04. The spec seems to be similar to a c6g.xlarge.

You are talking about things like the "All Nodes Go Down Without a Proper Shutdown Procedure" example in https://galeracluster.com/library/documentation/crash-recove... right?

To handle this case, some teams may have a manual runbook, some teams may have some automation with ansible, some teams may have nothing. So, if someone can come up with some hacks and package that into a k8s operator, it is still a win. It seems to be the best primitives we have at the moment.

I just learnt about "ip route change ROUTE quickack 1" from https://news.ycombinator.com/item?id=10662061, so we don't even need to patch the kernel. This makes 2b a really attractive option.

> And that pattern is the one that is used by GOs http libraries

I don't think that is correct. In https://news.ycombinator.com/item?id=34213383, I notice that Go's HTTP/2 library would write the HEADERS frame, the DATA frame, and the terminal HEADERS frame in 3 different syscalls. In a sample application using the Go's HTTP/2 library, a gRPC response without Nagle's algorithm would transmit 497 bytes over 6 packets, while a gRPC response with Nagle's algorithm would transmit 275 bytes over 2 packets.

With a starting point where both Nagle's algorithm and delayed ack are enabled, I guess this is the order of preference:

1. delayed ack disabled, applications do the right thing by buffering accordingly - ideal performance, but it is difficult to disable delayed ack, and it may require a lot of works to fix the applications.

2a. Nagle's algorithm disabled, applications do the right thing by buffering accordingly - almost ideal performance (may perform worse than #1 over bad connection), but it may require a lot of works to fix the applications.

2b. delayed ack disabled, real world applications - almost ideal performance (may have higher syscall overhead than #1), but it is difficult to disable delayed ack.

3. Nagle's algorithm disabled, real world application - not ideal as some applications can suffer from high packet overhead, e.g. git-lfs, and this is where we are at with Go.

4. baseline - far from ideal as many applications can suffer from high latency due to bad interaction between Nagle's algorithm and delayed ack.

I would say Go has made the right trade-off, albeit with a slight hint of "we know better than you". Going forward, it is probably cheaper for linux kernel to come up with a better API to disable delayed ack (i.e. to achieve #2b), than getting the affected applications to do the right thing by buffering accordingly (i.e. to achieve #1 or #2a). We will see how soon https://github.com/git-lfs/git-lfs/issues/5242 can be resolved.

In the mean time, #2b can actually be achieved with a "SRE approach" by patching the kernel to remove delayed ack and patching the Go library to remove the `setNoDelay` call. Something for OP to try?

Found the root cause from https://github.com/grpc/grpc-go/commit/383b1143 (original issue: https://github.com/grpc/grpc-go/issues/75):

    // Note that ServeHTTP uses Go's HTTP/2 server implementation which is
    // totally separate from grpc-go's HTTP/2 server. Performance and
    // features may vary between the two paths.

The lightstep collector serves both gRPC and HTTP traffic on the same port, using the ServeHTTP method from the comment above. Unfortunately, Go's HTTP/2 server doesn't have the improvements mentioned in https://grpc.io/blog/grpc-go-perf-improvements/#reducing-flu.... The frequent flushes mean it can suffer from high latency with Nagle enabled, or from high packet overhead with Nagle disabled.

tl;dr: blame bradfitz instead :)

> not to mention nearly 50% of every packet was literally packet headers

I was just looking at a similar issue with grpc-go, where it would somehow send a HEADERS frame, a DATA frame, and a terminal HEADERS frame in 3 different packets. The grpc server is a golang binary (lightstep collector), which definitely disables Nagle's algorithm as shown by strace output, and the flag can't be flipped back via the LD_PRELOAD trick (e.g. with a flipped version of https://github.com/sschroe/libnodelay) as the binary is statically linked.

I can't reproduce this with a dummy grpc-go server, where all 3 frames would be sent in the same packet. So I can't blame Nagle's algorithm, but I am still not sure why the lightstep collector behaves differently.

I got matched with my old account with a score of only 0.45

I have seen the "data service" design pattern that tries to work around this, by having a CRUD microservice to front a database, providing hundreds of APIs to read/write the data for 50 other microservices. If one of these CRUD microservices goes down, everything breaks.

That's a very smart solution indeed. I wonder if it is also possible to throw more memory at the problem? Managing instances with locally attached disks on the cloud is a bit of a pain, as you don't have the capability to stop/start the instances anymore.

Ah, I didn't know that the draft model has existed for Fantasy EPL. That's great!

Fantasy Football, where millions of players compete in one league, and every top team has a Mohamed Salah, is just too boring. It was boring then, it is boring now.

I much prefer the US fantasy sports format with a draft and only a dozen of players in one league, such that there is only one team which has a Mohamed Salah.

Misconfiguration at GFE control plane?

Speaking of Oracle, their so-called "match insights" (https://www.premierleague.com/news/2210969) is the crappiest addition to EPL. It gives the impression that either Oracle doesn't understand football, or Oracle Cloud is a bad technolgy.

Your example is the clearest indication in the whole discussion that Uber has a distributed monolith.

> 1. The cache tried to fill the metadata with version.

> 2. In the first round, the cache first filled the old metadata.

> 3. Next, a write transaction updated both the metadata table and the version table atomically.

> 4. In the second round, the cache filled the new version data.

Another way to look at this is that while the database can handle writes to the 2 tables atomically, the caching layer can't do a consistent read from the 2 tables, because it is either too expensive to do the 2 reads from within a database transaction, or too expensive to do a single read with join?

Our current system also suffers heavily from distributed inconsistency, mostly due to these "too expensive" constraints imposed by an ex-FB guy, even though our scale is nowhere near FB level. Yes I am bitter.

Bigger than the human population queries per second.

1 request from human would likely create multiple queries internally.

Exactly. The race condition problem from the example below still exists even without any caching, and perhaps still exists even without any replica.

> Imagine that after shuffling Alice’s primary message store from region 2 to region 1, two people, Bob and Mary, both sent messages to Alice. When Bob sent a message to Alice, the system queried the TAO replica in a region close to where Bob lives and sent the message to region 1. When Mary sent a message to Alice, it queried the TAO replica in a region close to where Mary lives, hit the inconsistent TAO replica, and sent the message to region 2. Mary and Bob sent their messages to different regions, and neither region/store had a complete copy of Alice’s messages.

A solution to this problem is to update Alice's primary message store to be "region 2;region 1" for a period of time, and get Alice to retrieve messages from both message stores during that period.

However, since you have a caching layer that doesn't have an explicit TTL, such that any cache inconsistency can last indefinitely, you now have a much bigger distributed data problem.

> these steps would rarely be same between companies or even between different teams in the same company.

And the quality can differ a lot too. I used to think that k8s is not necessary, since our team has mastered both stateless and stateful app deployment on VM, with ansible calling aws/gcp API. Everything just works.

And then I joined another company, which has a hotchpotch of unorthodox terraform and ansible code and a homebrew service discovery layer, with frequent incidents in the early mornings, especially on weekends, when autoscaling of aws/gcp VM would fail due to a myriad of reasons.

With k8s, there is a minimum quality.

Indeed, I think the one (fictional) case is the main reason why we got so many perpetually unprofitable tech unicorn.

https://knowledgebase.nomachine.com/AR02P00969 - and NoMachine seems to have decent support for Wayland.

Once you have that constraint, it means you will either have the same network latency when writing to SQLite (if it is fronted by some lightweight proxy), or have a lot more frequent failover of SQLite (if it is running embedded within the app, thus following the app's deployment schedule).

I suppose if someone decides to deploy Postgres/MySQL replicas as a sidecar, then it will be the same as what you will end up with?