rfoo

I'd recommend having a "gemm with a twist" [0] example in the README.md instead of having an element-wise example. It's pretty hard to evaluate how helpful this is for AI otherwise.

[0] For example, gemm but the lhs is in fp8 e4m3 and rhs is in bf16 and we want fp32 accumulation, output to bf16 after applying GELU.

What if I prefer to have a clone of me doing my coding, and then I throw my clone under the bus and start to (angrily) hyperfocus explore and change every piece to be beautiful? Does this mean I love coding or I hate coding?

It's definitely a personality thing, but that's so much more productive for me, than convincing myself to do all the work from scratch after I had a design.

I guess this means I hate coding, and I only love the dopamine from designing and polishing my work instead of making things work. I'm not sure though, this feels like the opposite of hate coding.

Meanwhile, in a certain modern OS, unloading a library is too broken to the point that people are discouraged to do so... Try to unload GLib [0] from your process :p

[0] https://docs.gtk.org/glib/

`Any` is the correct call.

It could be:

  def f(i=0) -> None:
    if i is None:
      do_something()
    else:
      do_something_else()

Yeah, I know it's retarded. I don't expect high quality code in a code base missing type annotation like that. Assuming `i` is `int` or `float` just makes incrementally adoption of a type checker harder.

> so much from your macbook

At least on cloud I can actually have hundreds of GiBs of RAM. If I want this on my Macbook it's even more expensive than my cloud bill.

There are a lot of examples about SQL in comments. In the SQL case you want something like:

  def process_template(template: Template) -> tuple[str, tuple]:
    sql_parts = []
    args = []
    for item in template:
      if isinstance(item, str):
        sql_parts.append(item)
      else:
        sql_parts.append("?")
        args.append(process_value(item.value))
    return "".join(sql_parts), tuple(args)

(of course it would be more nuanced, but I hope you get the point)

Google has been doing this since forever for recaptcha. And, to be fair, it seems to be fairly effectively for bot detection.

https://github.com/neuroradiology/InsideReCaptcha

> bots seem to be going a very different route

If the "very different route" means running a headless browser, then it's a success for this tech. Because the bot must run a blackbox JS now, and this gives people a whole new street of ways to run bot detection, using the bot's CPU.

> I've already seen this story play out with low-code. The pitch was the same

That's the point. Unlike low-code or no-code bullshit, this somehow worked. People "magically" want to design solutions, and are able to, now.

> People executing small scripts all over the place sounds to me like the stuff of nightmares.

Me too. However, watching how eager people want to (and they indeed can!) make progress with these LLM generated horror, I believe it's time to give up and start designing secure systems despite the existence of massive slightly broken scripts.

And I pay Cursor only for autocomplete - this explains the difference I guess.

I do sometimes use Composer (or Agent in recent versions), but it's being increasingly less useful in my case. Not sure why :(

> Cursor isn't providing their own models

For use cases demanding the most intelligent model, yes they aren't.

However, there are cases that you just can't use best models due to latency. For example next edit prediction, and applying diffs [0] generated by the super intelligent model you decided to use. AFAIK, Cursor does use their own model for these, which is why you can't use Cursor without paying them $20/mo even if you bring your own Anthropic API key. Applying what Claude generated in Copilot is just so painfully slow to the point that I just don't want to use it.

If you tried Cursor early on, I recommend you update your prior now. Cursor had been redesigned about a year ago, and it is a completely different product compared to what they first released 2 years ago.

[0] We may not need a model to apply diff soon, as Aider leaderboard shows, recent models started to be able to generate perfect diff that actually applies.

Codeium's completion model is better than whatever GitHub Copilot has. For me it's Cursor > Codeium >>> Copilot. Yes, Copilot is that bad.

And yes Codeium/Windsurf focuses on enterprise customers more. As GP said they have an on-prem [0], a hybrid SaaS offering and enterprise features that just make sense (e.g. pooled credits). Their support team is more responsive (compared to Anysphere). Windsurf also "feels" more finished than Cursor.

[0] but ultimately if you want to "vibe-coding" you have to call Claude API

Plus that they finally have someone else to blame, and it seems like people want to pretend it's all someone else's fault too. Probably don't even need to use their authoritarian cards.

In the same way how you manage a group of brilliant interns.

> I think 15M random read IOPS is actually quite fast

680x NVMe SSDs over 68 storage servers (so 68 CPUs) for just 15M (or 25M, tuned) random read IOPS is pretty underwhelming. The use cases where 3FS (or some other custom designs) shine are more like, 200M random read IOPS with 64 servers each with 8 PCIe gen 4 NVMe SSDs (512x SSDs in total).

I'd estimate that there would be two orders of magnitude of difference in 4K random IOPS. If not three.

IMO they serve similar at a glance, but actually very different use cases.

SeaweedFS is more about amazing small object read performance because you effectively have no metadata to query to read an object. You just distribute volume id, file id (+cookie) to clients.

3FS is less extreme in this, supports actual POSIX interface, and isn't particularly good at how fast you can open() files. On the other hand, it shards files into smaller (e.g. 512KiB) chunks, demands RDMA NICs and makes reading randomly from large files scary fast [0]. If your dataset is immutable you can emulate what SeaweedFS does, but if it isn't then SeaweedFS is better.

[0] By scary fast I mean being able to completely saturate 12 PCIe Gen 4 NVMe SSD at 4K random reads on a single storage server and you can horizontally scale that.

3FS isn't particularly fast in mdbench, though. Maybe our FDB tuning skill is what to blame, or FUSE, I don't know, but it doesn't really matter.

The truly amazing part for me is combining NVMe SSD + RDMA + supports reading a huge batch of random offsets from a few already opened huge files efficiently. This is how you get your training boxes consuming 20~30GiB/s (and roughly 4 million IOPS).

> If you're in the European Economic Area, Switzerland, or the United Kingdom, the terms under "How Google uses Your Data" in "Paid Services" apply to all Services, including Google AI Studio and unpaid quota in the Gemini API, even though they are offered free of charge.

See, living in the EU is a perk!

Before this change, 1000 512KB GET requests cost $0.0002 ($2e-4).

After this change, 1000 512KB GET requests cost $0.00003 + $0.0006 / 1024*3 * 512 * 1024 * 1000 = $0.00003 ($3e-5) + $0.00029296875 (2.93e-4, dominates the cost) = $0.00032296875, so roughly $0.0003 ($3e-4).

It's because before this change all data transfer charge was waived as long as requests are smaller than 512KB.

The announcement did mention it explicitly, but it tried to downplay it, by starting the sentence with a price reduction:

> In addition, S3 Express One Zone has reduced the per-GB charges for data uploads and retrievals by 60 percent, and these charges now apply to all bytes transferred rather than just portions of requests greater than 512 KB.

This "and" should be a "BUUUUUUUUUUUUUUUUUUUUUUUT".

> Codebase Divergence: Our engine is based on an early fork of vLLM from over a year ago

If you are in the same boat you'll see how much changed in vLLM compared to one year ago. Also, this meant that they haven't rebased for over a year, I don't believe that's because they don't want, it's because they effectively can't.

Yeah, surely they can maintain it as-is. But it will be increasingly hard to port over anything community has.

If all your GET requests are 512KB in response, this "price reduction" is effectively a price raising. GET requests go up to $0.000323 per 1K from $0.0002 per 1K.

tl;dr "we had our vLLM fork and it's unmaintainable now; guess we are going to rebuild it, in the public this time"

> a "no malicious actors" assumption in its design in order to make the performance/scaling guarantees it does

Didn't think deep into it, could this be solved with billing designs with more nuance?

It is well possible by changing 50 lines of code in a 10GB project you have to rebuild the entire project, if everything (indirectly) depends on what you just changed.

I think the joke here is that somehow management refused to sell Colossus (which is such an obvious nice product just like BigQuery) before and it takes "AI" to convince them.

I don't. I have no problem not running open-weight models myself because there's an efficiency gap of two orders of magnitude between "pretend-I-can" solution and running them on hundreds of H100s for high thousands of users.

> see o3-mini's release date & pricing vs R1's

This backfires though, if OAI released o3-mini before DeepSeek-R1, R1 would be a lot less impactful.

That's about the same number for DeepSeek-V3. If you count in fp8 MFU is about 20%. MoEs are hard.

That could also be why they did fp8. If we use theoretical performance of bf16 as baseline (I know this makes few sense, but for compare with previous trainings it's convenient) the about 40% MFU, not too bad.

IOW, MoE kills training MFU and they had to do fp8 to make it not looking funny. Both DeepSeek and Meta GenAI.

From model cards, suggested system prompt:

> You are Llama 4. Your knowledge cutoff date is August 2024. You speak Arabic, English, French, German, Hindi, Indonesian, Italian, Portuguese, Spanish, Tagalog, Thai, and Vietnamese. Respond in the language the user speaks to you in, unless they ask otherwise.

It's interesting that there's no single one of CJK languages mentioned. I'm tempted to call this a racist model even.

The point is, it isn't marginally better. I agree the setup is not a demonstration of intelligence, but the difference is pretty significant. Not to mention that on conventional benchmarks Llama 405B is usually worse than GPT-4o.