Vgoose

I wish I could write this well. What an enjoyable read.

I think there's a simple explanation for why the models aren't good at "maintainable" code: they're not explicitly trained on it in post-training. It being hard to quantify what maintainable is and there not being a popular benchmark for maintainability are probably some reasons why.

Thanks for sharing, I've never heard of variety before and that was an interesting read. It makes me think of a broader term I've been using to describe the run time state of the system: Entropy. I've given up on measuring that for now and I'm settling for a compile time proxy through measuring the _semantic_ cardinality of types as this seems a bit more approachable for a POC.

One drawback I ran into was that the variety/entropy of scalars tends to dominate everything else. I'm clamping them to 1 for now but that's also not great as it punishes more descriptive systems.

Ive have a definition of state space that is calculated by just the types of the system. It’s a rough approximation of the true state space but it’s convenient (and more reflective of what we actually mean by system state IMO) and types are inherently accessible to the model. I recently put down some thoughts around working with types to improve communication w/ AI that kind of sets this framing up: https://www.alecvo.org/blog/types-with-ai/. The actual definition is still WIP.

Nice! I actually ran across this paper+benchmark recently, too. It's the first I've found that start to aim at some of the non-functional and longitudinal requirements that I think have always been an important part of writing production code.

It's especially relevant now that models are good enough to solve ~most point-in-time problems.

Some relevant but disconnected thoughts:

- deterministic scores are so nice

- what "maintainable" is is probably some high dimensional space described by these signals; it'd probably require some human labeling to figure out where this space is

- another signal I've been thinking about and I'm seeing increasingly get brought up is the state space of a system; I'm seeing formal methods pop up a lot recently

My architect introduced Architecture Decision Records to my company and it's made a world of difference. Are there other tools and mechanisms people have found helpful to facilitate and scale group thinking?

This is my go to diagramming tool. It takes a lot of little tricks to get things to turn out exactly how you want, but they're not required. It communicates information just fine. The tricks are only required if you care about aesthetics.

I've actually seen more of her name than Akita itself. In a way, that's great marketing.

It used to be around every few weeks I would read something cool about Elixir. This past two weeks it's down to every few days.

I think someone's trying to tell me something.