6keZbCECT2uB

I did not spot any errors in my skim of the architectures I'm familiar with where I would expect an LLM to go wrong, and it has a nicely scoped overview of topics that people in the space should familiarize themselves with. I don't think I've seen a better primer.

I can see how this makes sense as a default behavior for cost conscious users. I would prefer to have the option for my company to pay more to rehydrate the cache than to have there be a model performance difference when having idled for an hour.

"We tried a few different approaches to improve this UX:

1. Educating users on X/social

2. Adding an in-product tip to recommend running /clear when re-visiting old conversations (we shipped a few iterations of this)

3. Eliding parts of the context after idle: old tool results, old messages, thinking. Of these, thinking performed the best, and when we shipped it, that's when we unintentionally introduced the bug in the blog post."

I see how these interventions help users reduce their token burn rate, but they don't address the need for an enterprise user to maintain quality.

A common workflow for me is kick off a prompt, commute home, eat dinner, follow up on prompt. Frequently 80K tokens or less in the context, frequently > 3 hours. Or when running multiple sessions it's easy to let a session idle for a few hours while I focus on one. Or many meetings might mean idle time for an hour.

Also, for enterprise users, I don't think education on X is a great place. There are people upskilling on this that never intentionally go on X.

First thing that comes to mind would be a weekly tip feed of footguns and underutilized functionality published to an anthropic website. "The Old New Thing" "Guru of the Week" "Abseil tips of the week" all have that format.

I remember when the computer crashed and the user hadn't saved recently, we blamed the user.

"On March 26, we shipped a change to clear Claude's older thinking from sessions that had been idle for over an hour, to reduce latency when users resumed those sessions. A bug caused this to keep happening every turn for the rest of the session instead of just once, which made Claude seem forgetful and repetitive. We fixed it on April 10. This affected Sonnet 4.6 and Opus 4.6"

This makes no sense to me. I often leave sessions idle for hours or days and use the capability to pick it back up with full context and power.

The default thinking level seems more forgivable, but the churn in system prompts is something I'll need to figure out how to intentionally choose a refresh cycle.

Your reasonable options are: 1. I stop sharing the software I write 2. You take responsibility for the software you use

Any software you use with this clause, "THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE."

Already attests that the authors do not offer guarantees that the software will have the features you need, supply chain security or otherwise.

Once your cache hit ratios for some data structure go < .1%, I'd rather have 75% less tail latency even if it reduces cache hit rate further.

I like the project: taking it from refresh-induced tail latency to racing threads assigned to addresses that are de-correlated by memory channel. Connecting this to a lookup table which is broadcasted across memory channels to let the lookup paths race makes for a nice narrative, but framing this as reducing tail latency confused me because I was expecting this to do a join where a single reader gets the faster of the two racers.

From a narrative standpoint, I agree it makes more sense to focus on a duplicated lookup table and fastest wins, however, from an engineering standpoint, framing it in terms of channel de-correlated reads has more possibilities. For example, if you need to evaluate multiple parallel ML models to get a result then by intentionally partitioning your models by channel you could ensure that a model does reads on only fast data or only slow data. ML models might not be that interesting since they are good candidates for being resident in L3.

In Miami, there are several competing companies like Coco Robotics which employ human "pilots" to monitor a small fleet of robot delivery boxes where the restaurant deposits the food in the box and the box unlocks with integration into the app.

Just figured you'd want to know anytime soon was at least a year ago.

There's prior work: https://www.brendangregg.com/FlameGraphs/offcpuflamegraphs.h...

There are a few challenges here. - Off-cpu is missing the interrupt with integrated collection of stack traces, so you instrument a full timeline when they move on and off cpu or periodically walk every thread for its stack trace - Applications have many idle threads and waiting for IO is a common threadpool case, so its more challenging to associate the thread waiting for a pool doing delegated IO from idle worker pool threads

Some solutions: - Ive used nsight systems for non GPU stuff to visualize off CPU time equally with on CPU time - gdb thread apply all bt is slow but does full call stack walking. In python, we have py-spy dump for supported interpreters - Remember that any thing you can represent as call stacks and integers can be converted easily to a flamegraph. eg taking strace durations by tid and maybe fd and aggregating to a flamegraph

I've been meaning to look at Iceoryx as a way to wrap this.

Pytorch multiprocessing queues work this way, but it is hard for the sender to ensure the data is already in shared memory, so it often has a copy. It is also common for buffers to not be reused, so that can end up a bottleneck, but it can, in principle, be limited by the rate of sending fds.

Partly in jest, you can often find a Perl / bash available where you can't find a Python, Ruby, or Cargo.

Conda unifies by using a sat solver to find versions of software which are mutually compatible regardless of whether they agree on the meaning of semver. So, both approaches require unifying versions. Linking against C gets pretty broken without this.

The issue I was referring to is that in Javascript, you can write code which uses multiple versions of the same library which are mutually incompatible. Since they're mutually incompatible, no sat-solve or unifyer is going to help you. You must permit multiple versions of the same library in the same environment. So far, my approach of ignoring some Javascript libraries has worked for my backend development. :)

I agree. In my opinion, if you can keep the experience of Bazel limited to build targets, there is a low barrier to entry even if it is tedious. Major issues show up with Bazel once you start having to write rules, tool chains, or if your workspace file talks to the Internet.

I think you can fix these issues by using a package manager around Bazel. Conda is my preferred choice because it is in the top tier for adoption, cross platform support, and supported more locked down use cases like going through mirrors, not having root, not controlling file paths, etc. What Bazel gets from this is a generic solution for package management with better version solving for build rules, source dependencies and binary dependencies. By sourcing binary deps from conda forge, you get a midpoint between deep investment into Bazel and binaries with unknown provenance which allows you to incrementally move to source as appropriate.

Additional notes: some requirements limit utility and approach being partial support of a platform. If you require root on Linux, wsl on Windows, have frequent compilation breakage on darwin, or neglect Windows file paths, your cross platform support is partial in my book.

Use of Java for Bazel and Python for conda might be regrettable, but not bad enough to warrant moving down the list of adoption and in my experience there is vastly more Bazel out there than Buck or other competitors. Similarly, you want to see some adoption from Haskell, Rust, Julia, Golang, Python, C++, etc.

JavaScript is thorny. You really don't want to have to deal with multiple versions of the same library with compiled languages, but you have to with JavaScript. I haven't seen too much demand for JavaScript bindings to C++ wrappers around a Rust core that uses C core libraries, but I do see that for Python bindings.

Most of my time coding is spent on none of: elegant solutions, complex problems, or precise specifications.

In my experience, LLMs are useful primarily as rubber ducks on complex problems and rarely useful as code generation for such.

Instead, I spend most of my time between the interesting work doing rote work which is preventing me from getting to the essential complexity, which is where LLM code gen does better. How do I generate a heat map in Python with a different color scheme? How do I parse some logs to understand our locking behavior? What flags do I pass to tshark to get my desired output?

So, I spend less time coding the above and more time coding how we should redo our data layout for more reuse.

How does this work with things which are expressible only in C? For example, Pascal strings with flexible array members?

I guess since you said header, you keep everything opaque and create a header for that which gets translated to Zig.

In all seriousness, this seems to carry risk of never doing anything deep or hard. In particular, I've been programming for long enough, that I can be casual about many programming languages until I hit something which is actually new, such as in Rust or Prolog.

Promiscuous doesn't have to mean having a low tolerance for difficulty, but everything else you wrote seems to support that. So, are you saying that enduring difficulty is unnecessary, or did you mean something different?

Fowler's implementation is written in Java which has a different memory model from C++. To see another example of Java memory model vs a different language, Jon Gjengset ports ConcurrentHashMap to Rust

How does the tool ingest task sentiment? As a developer, I would never put in writing that I'm less than enthusiastic about any task.

The choice between replacing and making it an overlay is up to your editor. I think it would be pretty to handle either choice as a plugin in your editor given the returned json.

I was surprised it wasn't combined with clangd.

Spending 8 months when the business value at the end was mostly improved velocity doesn't sound likely to be a good tradeoff, especially if this is done as a big bang effort which either succeeds holistically or fails. You might have better success in the future by finding ways to integrate maintenance improvements incrementally.

This comes into tension with making names as clear as possible while still being short. Frankly, I don't find coming up with short names for things to be easy, let alone good names even when taken to the extreme.

Whether something is too short depends on your context. requests.get might be ambiguous for someone who has never seen the code base before, but will quickly get obviously with a little exposure, so does the code base have dedicated maintainers?

The skill of good naming isn't distributed evenly and OP's names are pretty good, so I'd be happy for them to come along to my code base and rename things as long as it was within other constraints (not consuming too much time, doesn't break api, etc).

The tensor core accelerates mostly matrix operations and is the big block you can see has 4 per SM. Cuda core refers to the thread per SM, which you can see as FP32 or INT32 units, so there are (32*4) per SM on that diagram.

Like you said, tensor core is similar to a special purpose ALU and is at a lower level of abstraction than something with an instruction pointer.

I'm not sure about the 4090, but most of the GPUs I use have a warp size of 32, and warp divergence affects only up to those 32 threads. If you have a branch and all threads agree, you only walk down one branch.

My mental model is a bit more like you have collections of warps in a block, and all warps in a block get scheduled onto an SM. Different GPU architectures allow for different numbers of warps to be simultaneously active or inactive, and each warp has its own instruction pointer and can be suspended while waiting for things like memory. I found the picture on pg 22 here really helpful: https://images.nvidia.com/aem-dam/en-zz/Solutions/data-cente...

Note that although there's 4 schedulers, on the A100, they don't dispatch every cycle iirc.

It is a pretty different way of programming, and that's part of what makes it so fun. Within constraints, you get a really interactive way to design algorithms which are much more parallel than what is feasible to write for a CPU. Debugging is so much better these days with some support for debuggers and printf (at least on CUDA). Maybe the same facilities aren't available for WebGPU?

Conditions and loops are largely fine as long as you avoid warp divergence and such.

Not sure about lsp, but I think if you defined your language in tree sitter, you might be able to define a basic autoformatter generically on tree sitter to accelerate bootstrapping your language.

You could also use an existing language agnostic package manager like nix, guix, or conda to bootstrap your language package manager.

Lsp is something I don't know of a way to make that easy without overly constraining the design space.

I can't relate to the other things, but I dislike the majority of things that constitute going on vacation. I don't like planning the vacation including what to do, where to stay or how to get there. I don't like having the vacation planned and looming over my head preventing me from making long term plans.

However, after all that, there are moments throughout which make up for it and make me glad I didn't just stay at home.

It's a common sentiment that others don't know how computers work. I'm sure you personally are an exception, but I think that most people with this sentiment live in glass houses and don't realize how little they understand about how computers work. Since I don't know how they work, I can't provide comprehensive examples, but I remember being surprised by details around dynamic linking and relocatable binaries, around how vfs maps to more specific implementations, the implementation of hard disk drive firmware, the on-drive caches and the firmware that runs that, the implementation in terms of magnetic polarity for storing bits, and many more layers and details besides that I have yet to discover.

Even if when emacs isn't your editor of choice, everyone benefits from exploration of advancing our vocabulary and idioms for navigating parse trees.

I think that ChatGPT could be a big accelerator for creative activity, but I wouldn't trust any output that I've only partially verified. That limits it to human scale problems in its direct output, but there are many ways that human scale output like code snippets can be useful on computer scale data.

I've wondered about the same thing of generating a permutation with a minimum number of samples. I'm not sure how Gray codes helps you here, and the multiplication sounds wrong since many outputs aren't reachable from any input.