sophiabits

What's the difference between "firing" Claude vs moving to a model from a different provider? The latter seems very analogous to firing an employee for performance and backfilling with someone new.

Re the rest, it's just not my experience that models become incapable of making good decisions in cases where input token count > the context window, but ymmv based on domain.

A very extreme example of this: a couple years ago when GPT 4 was state of the art and the 32K context variant was gated to design partners I worked at an EdTech company in the college admissions space that wanted to produce quarterly reports on student progress for parents. That involved crunching a LOT of data (multiple hours of meeting transcripts per week, very detailed notes about student activities, their general profile - UK and US admissions function very differently!)

It was a difficult problem, but we _did_ manage to produce these reports 4K output tokens at a time at a level of quality that exceeded what humans could do internally, and models+the surrounding tooling have only gotten better since then.

I wonder whether this Meta employee felt the same way about privacy a month ago, before they were personally impacted by this new initiative.

Of course the side effect is that now it's much harder to pull packages for legitimate reasons :/

[1] https://en.wikipedia.org/wiki/Npm_left-pad_incident

For example, the comment you're citing is claiming that because minimum wage has increased only 3x over the same period of time in which inflation has eroded the relative value of a dollar by 6x, that wages overall have increased at half the rate of inflation. But minimum wage is a measurement of a minimum, while inflation is a measurement of /average/ price increase so they can't be compared 1:1 in this way.

The housing argument also seems odd. In New Zealand (where I'm from -- I'm not familiar with the US' housing market, so the commenter could be right about that geo!) house prices have increased by far more than 20x since the 70s, but the houses available are of substantially higher quality due to improved regulations (e.g. all newer homes are subject to healthy homes rules which mandate insulation) so just comparing inflation-adjusted home prices vs income doesn't tell the full story

(Aside from that, a whole heap of items like food, electronics, transportation are all both far cheaper AND higher quality today than in the 70s)

The main problem with this approach at the moment is it busts your prompt cache, because LLMs expect all tool definitions to be defined at the beginning of the context window. Input tokens are the main driver of inference costs and a lot of use cases aren't economical without prompt caching.

Hopefully in future LLMs are trained so you can add tool definitions anywhere in the context window. Lots of use cases benefit from this, e.g. in ecommerce there's really no point providing a "clear cart" tool to the LLM upfront, it'd be nice if you could dynamically provide it after item(s) are first added.

An example of this: assume you live in a world where the formula for the circumference of a circle has not been derived. You end up deriving the formula yourself and write a function which returns 2piradius. This is as simple as it gets, not hacky at all, and you would /definitely/ want to include a comment explaining how you arrived at your weird and arbitrary-looking "3.1415" constant.

Fwiw my subjective experience has been that non-technical stakeholders tend to be more impressed with / agreeable to longer AI outputs, regardless of underlying quality. I have lost count of the number of times I’ve been asked to make outputs longer. Maybe this is just OpenAI responding to what users want?

[1] https://sophiabits.com/blog/new-llms-arent-always-better#exa...

LLMs are theoretically pretty fungible, because you send English and get English back--but in practice you still need to do some amount of technical due diligence before swapping model. These things are benchmarked on tasks which rarely resemble your specific use case. Blindly swap models at your own risk!

Something that has become very clear since the advent of GPT-3.5 is that LLMs are far from magic, and using them does not remove the need for good engineering fundamentals. It's important to have a solid eval suite so you can quickly benchmark your system against different LLMs, because the APIs we're all building on are constant moving targets.

This is from the first half of 2023 or so; maybe things are more stable now, but looks like the Python implementation is still pre-v1.

I’ve seen a few colleagues jump to writing off all technical debt as being inherently bad, but in cases like this it’s a sign of success and something that’s largely impossible to avoid (the EV of building for 10-100x current scale is generally negative, factoring in the risk of the business going bust). There’s a kind of entropy at play here.

Big fan of tidying things up incrementally as you go [1], because it enables teams to at least mitigate this natural degradation over time

[1] https://sophiabits.com/blog/be-a-tidy-kiwi

Imo the same mentality is good to have in software, and I’ve always appreciated being in a team that makes codebase improvements alongside feature additions. It makes things a lot more pleasant

As you say, I'm skeptical this counts as AGI. Although I admit that I don't have a particularly rock solid definition of what _would_ constitute true AGI.

e.g. If I am parsing a string to a number via Number.parseInt, I don’t need a “: number” annotation because I can just call the variable “myNumber” and use that.

Branding a string is in many ways an extension on the idea of “branding” my “myNumber” variable as “: number” rather than leaving it as “: any”. Even if the TS type system is easy to bail out of, I still want the type annotations in the first place because they are useful regardless. I like reducing the number of things I need to think about and shoving responsibility off to my tools.

A user is free to leave and rejoin a stream, and we want to retain old data. So each user_stream has columns id, user_id, stream_id (+ others)

Issues occur when people write code like the following:

streamsService.search({ withIds: userStreams.map((stream) => stream.id), });

The issue is easily noticed if you name the “stream” parameter “userStream” instead, but this particular footgun came up _all_ the time in code review; and it also occurred with other junction tables as well. Branded types on the various id fields completely solve this mistake at design time.

  type Target = 'currentNode' | (string & {});
  const targets: Target[] = [
    'currentNode', // you get autocomplete hints for this!
    'somethingElse', // no autocomplete here, but it typechecks
  ];

The numbers are obviously very rough; there’s a LOT of factors to consider and which vary from one node to another. But the methodology is quite comprehensive, e.g. they factor in power consumption of the end user’s device while waiting for the data to transfer over the wire

Green software engineering is interesting to me because it seems like a good way to impact greenhouse gases _without_ requiring consumers to change anything about their lifestyle (which is the hard part of climate change…) There’s a cool presentation from Rasmus Lerdorf from around the time when PHP7 released where he estimated a 50% adoption rate of PHP7 would result in a saving of 3.5B kg CO2/year iirc, purely because of the compute efficiency gains from PHP6 -> PHP7.

I used their numbers to calculate that swapping our CI/CD clones at my day job over to shallow clones saved (maybe) 5.5 kg/week of CO2 emissions [2]. Not quite as impressive as the PHP7 figures, but I still think it’s kinda neat.

[1] https://theshiftproject.org/en/lean-ict-2/

[2] https://sophiabits.com/blog/the-cost-of-bandwidth

At that point—why bother with the mock at all? Cutting out the intermediary and running integration tests of your application against a real database will be faster.

You can’t use SQLite here either because all SQLite isolation levels are serializable anyway, which is _very_ different from how PostgreSQL works. Testing against SQLite could end up giving you a false sense of security that your code is safe against race conditions, whereas in reality it’s vulnerable when connected to a PostgreSQL server because of the difference in default isolation levels

At this level of testing detail the only real option is to test against a real database that matches what you’re running in production. Otherwise you’re just testing a mock.

I’m pretty confident there will be situations where you can measure a statistically significant performance improvement by offering a tip or telling the model you have no hands, but I’m not convinced that it’s a universal best practice.

A big issue is that a lot of the advice you see around prompting is (imo) just the output of someone playing with GPT for a bit and noticing something cool. Without actual rigorous evals, these findings are basically just superstitions

[1]: https://aider.chat/docs/unified-diffs.html

We originally started out building features with LangChain (loading chains from YAML sounded good—it felt like it would be easy to get non-engineers to help with prompt development) but in practice it’s just way too complicated. Nice idea, but the execution feels lacking.

It also doesn’t help that LangChain is evolving so rapidly. When we first started using it a lot of code samples on the internet couldn’t be copy/pasted because of import paths changing, and at one point we had to bump by ~60 patch versions to get a bug fix, which was painful because it broke all kinds of stuff

ODATA [1] is probably the main one, and it’s adopted by e.g. Microsoft’s Azure APIs. JSON:API [2] is another, although I’ve never seen it used anywhere personally.

They’re both pretty heavy specifications so I’ve never bothered implementing anything that 100% follows them. But I have picked bits and pieces from time to time—in particular ODATA’s idea for filtering is nice and comprehensive imo, and there are a number of libraries in various languages which handle parsing its filter strings into structured data.

[1] https://www.odata.org/ [2] https://jsonapi.org/

The dprint plugin wraps Prettier under the hood so compatibility is good. The perf wins come from formatting files in parallel and incrementally.

[1] https://david.deno.dev/posts/faster-prettier-with-dprint/

I _wanted_ to drop the plugin and have Prettier automatically run on as a pre-commit hook (as you suggested) but in the end I lost :) We wound up keeping the plugin and eating the slower pipeline time.

Of course there’s more overhead than just Prettier in this case—the ESLint plugin itself will be contributing towards the added runtime!—but depending on your codebase and CI machines, Prettier can definitely slow things down quite a bit. CI taking over a minute or two is a big drag on developer productivity imo.

Ideal world is that everyone is on board with code formatting as a pre-commit task that only touches modified files so it doesn’t bloat CI unnecessarily, but it’s not always possible :(

Visualizing it all as a tree structure feels like an improvement. I’ll definitely give it a spin :)

It’s a small gripe, but I need to copy IDs multiple times per day and it adds up. Other ID formats like cuid or KSUID don’t have hyphens in their canonical representation, and it makes them far more pleasant to work with

HN’s UI definitely feels to me like it serves a similar purpose, whether intentional or not :)

Lots of other languages take the same approach. Rust’s must_use attribute is basically the same as nodiscard.

It’s a nice way of solving the problem imo.

[1] https://en.cppreference.com/w/cpp/language/attributes/nodisc...

I started a few years ago but only found a good rhythm in the past ~year. I am 75% of the way to my (arbitrary) goal of hitting 100k written words.

Lots of posts on architecture, AWS, and performance with a little bit of engineering leadership mixed in.

Some recent posts:

* https://sophiabits.com/blog/understanding-secrets-manager (did you know Secrets Manager “staging labels” can be used to colocate related secrets?)

* https://sophiabits.com/blog/using-terraform-plan-to-write-ia... (write Terraform for your infra without learning Terraform)

* https://sophiabits.com/blog/object-ids-for-humans (ID formats have surprising performance and DX impacts!)

* https://sophiabits.com/blog/evaluating-a-new-technology (my checklist for when a team member proposes adding something new to our tech stack)

RSS: http://sophiabits.com/feed.rss.xml