tetha

"Molten" to me implies it is still liquid. Molten salt reactors, molten magma from a volcano, molten sand, molten steel, dipping something into molten cheese. All fluid.

If I was to nitpick, "melted" is kind of inaccurate and not entirely natural in this context. Technically, molten sand is also melted sand, because that's how you get it to that state? Usually, you'd hear about solidified magma, crystalized sand, cast iron, air-cast steel, unevenly settled corium... to make a better point on how it turned back into a solid and what to expect from it - something like "The molten sand crystalized into an unusual structure" would be clearer.

I'd usually rather hear "melted" if it is important to note that this had a phase change and back. Plastic on an electrical device may look melted, indicating heat. A hardened steel part may look melted, which may damage the hardening. Rubber on a hydraulic line may look melted, also indicating heat. A plastic container looking melted in the context of chemicals may indicate some compromise.

Now the words sound weird in my head. Thank you.

But in a perfect world, the question would be: Is it reasonable to expect an outage by sending a few single TCP packet to a system? Or, were you flooding the system unreasonably?

It is a huge security risk to treat systems as ancient eggshells you must not touch ever. A certain amount of touching has to be reasonable, because that is what foreign actors will do if they need to cause trouble. Apparently you could cause this company major operational harm with a pi zero. Why is that protected by professional ruin and jail time?

Yeah.

And I do think that security research should have some regulation about it, but it should be more about responsible handling of the privileged access you gained, or a responsibility to disclose found vulnerabilities in private and/or to a government entity. You know, "If you have gained access to a system, and you saw a button <Turn off cooling pump 2> and you pressed it, you are on the hook for the damages". That is common practice with paid pentesters already.

But we're at a point where a court had do decide if discovering an endpoint on an API without authorization is a "circumvention of a security boundary" or not. Luckily, we now have a ruling that accessing API endpoints without authorization logic is no circumvention of a security boundary, due to a lack of a security boundary like authorization.

That's the level we are at. I don't want to know what happens if foreign nation state actors start acting on this seriously.

> There are times when this is good, there are times when actively trying introduce an improvement is the best way forward. A good senior is able to recognise when those times are.

This is what I was thinking - I'd say the biggest step up a developer can make is to recognize that sometimes you need a bit of one approach, sometimes a bit of another one.

Sometimes minimalism is the way, and you need to wonder if the pain, workload or lacking capabilities and features are problematic. Or, sometimes adding the smallest possible thing is a good way, as long as we don't paint ourself into a corner and enable learning and accumulating information of what we actually need.

Sometimes buying a thing is a good way, if you can find a good vendor and a tool fitting your use case and especially if the effort of doing it on your own is high. This commonly occurs in security, because keeping up to date with the ongoing vulnerability and threat landscape can be a full job on its own.

And sometimes adding something bigger is the way, if the effort of maintaining it are less than the effort and pain incurred by not having it. Or if we can ramp up the effort of the thing incrementally, while reaping benefits along the way. This can be validated often by doing a small thing.

What the AI will do in my opinion is to push the bar more in this direction. Cozily hacking CRUD-Code in a web server together most likely won't be enough in a year or two for the average development job.

How do you define flawless though?

The CVEs here have their fair share of silly C problems, but also more rigid input validation and handling. These more rigid validations exclude stuff which may even be valid by the spec, but entirely problematic in practice.

As examples, take a look how many valid XML documents are practically considered unsafe and not parsed, for example due to recursive entity expansion. This renders the parsers not flawless and in fact not in spec.

Or, my favorite bait - there should be a maximum length limit on passwords. Why would you ever need a kilobyte sized password?

I've also grown somewhat sensitive to duplication, maybe to a painful level. But, the memmove-call from the AI writeup has duplication in there:

    memmove(args->begin_argv + extend,
            args->begin_argv + consume,
            args->endp - args->begin_argv + consume);   // ← bug

If both `args->begin_argv + consume` are supposed to be the same concept and thus the same value, I'd have a variable for it by now. Some people hate it with a passion, but something like this removes the precendence thinking, prevents modification of one and not the other and makes it easier to follow, for me at least:

    retained_tail_begin = args->begin_argv + consume
    memmove(args->begin_argv + extend,
            retained_tail_begin,
            args->endp - retained_tail_begin);

Though at that point one might also encode the entire intent (as far as I understand it) in variables as well:

    space_to_replace_end = args->begin_argv + extend
    retained_tail_begin = args->begin_argv + consume
    memmove(space_to_replace_end,
            retained_tail_begin,
            args->endp - retained_tail_begin);

Sure we can golf the names somewhat, but that code has my head spin a lot less about math and precedence.

We at work are currently going through the kernel modules available on Debian by default and deactivating things, yes.

And sorry, but I am ... frustrated by this. Why do my Debian 11 servers (currently upgrading, yes) have support for phone infrastructure from the 90s (ATM), or really obscure file systems like "Andrews File System" or support to run IP across amateur radios (AX.25) by default? We recently joked that we should start a pot you add a euro to whenever you find ancient discontinued tech you never heard about our systems support so we can have some nice dinner after this.

I do understand that going full Gentoo or Arch as a generally available distro is not feasible. I am also personally intimidated by compiling my own kernel with just what we need. But the amount of strange ancient things supported by default is also quite ridiculous.

> In the extreme I think there's a decent chance projects like Debian might have to radically overhaul or just shut down completely - the whole philosophy of slow and steady with old code just won't work.

It may actually be the opposite.

Debians steady and professional approach on shipping security patches with very little to no functional difference actually enables us to consider and work on automated, autonomous weekly or faster patches of the entire fleet. And once that's in place and trusted, emergency rollouts are very possible and easy.

We have other projects that "move fast and break things" and ship whatever they want in whatever versions they want and those will require constant attention to ship any update for a security topic. These projects require constant human attention to work through their shenanigans to keep them up to date.

Out of this dataset of 2-3 vulnerabilities, I'm noticing a pattern: All of those are in older and/or niche kernel modules. That raises two thoughts:

Maybe the more regularly used kernel code has a lot of low-hanging security topics shaken out of it already.

And second, I'm indeed wondering what a good path to minimize the loadable kernel code is on a system looks like. My container hosts for example have a fairly well defined set of requirements, and IPSec certainly is not in there. So why not block everything solely made to support IPSec? I'm sure there is more than that.

After all, the most reliable way to higher security is to do less things.

I can't even imagine that scenario with the remoteness of burning man.

Wacken got really bad a few years ago. Like, it's normal to rain here, and it's normal for cars to not get off campground, so a dozen of farmers or two are around with their tractors to evacuate people back to asphalt. Except that year, the rain escalated to badly that cars sunk deep enough into the mud that their undercarriage sat on the ground and the mud started to seep into the belly and the engine area.

At that point, dragging the car out has a decent risk of ripping rather important resources out of the rig, and then you got a scrapping job left. That was a fucking mess. They also closed off the Autobahn near Wacken that year, because the massive amount of mud the cars dragged onto the Autobahn turned into a rather slippery affair -- and hitting slippery mud at 100km/h, 60mph without expecting it can easily turn into a life-changing ad-hoc roller coaster.

Doing all of that at your distances in the middle of fucking nowhere would not be enjoyable or fun. Folks drowning in mud in northern Germany is now mostly a funny story among metal heads and rescue folks.

Monitor your disks to see if they grow full, and have an idea what your storage baseline should be. Storage in /var/lib/docker/overlay2 can also leak, even if you prune regularly.

It does. It's also very nice that this moves storage usage from /var/lib/docker over to /var/lib/containerd.

Due to that, a careless installation of a few new dev-systems under the new docker version immediately blew up storage usage on the root-disk, while happily ignoring hundreds of gigabytes on a volume on /var/lib/docker.. because that's where it needs the storage, right? A few older systems also were upgraded but didn't, which was quite confusing at first.

Sorry for being salty, but that was a pretty hectic afternoon with those new agents trashing builds, and now we have a pretty annoying migration plan to plan for the rest. And yes yes it's just a reinstallation, but we have other things to do as well.

SecuROM back in the day caused plenty of legitimately purchased copies to not work. You'd have a physical disc with the game on it from the store, and SecuROM decided it won't work on your computer for unknown, undebugable reasons. .

Piracy may be a problem, but that's a problem to customer who were willing to give a company money. We stopped buying anything with SecuROM on it after 1-2 of those situations.

I also have a Dungeon Crawl: Stone Soup with my first 3 runes around somewhere.

I'm aware I will probably lose it, but I'm also anxious to touch it. Maybe I should just get myself some good coffee tomorrow and get over with it. Biggest learning of that save is also how careful and defensive you have to play if you want to consistently get further.

Do have way too much fun with EICAR:

https://www.youtube.com/watch?v=cIcbAMO6sxo

This guy put the EICAR test string into a barcode and started to scan it on various systems, with rather funny effects.

And a lot of these older tools are not meant to be fed untrusted, unvetted input. The patch shown there confused me for quite a bit.

Or, more snarky: tee is also a huge security problem if you pipe untrusted input into `tee -a /etc/passwd`, such as `curl | tee -a /etc/passwd`. Not many things are safe with a `curl |` in front of them. I think yes might be?

The owner rug-pulls, or Broadcom buys the owner and starts squeezing.

Lifetime is the underlying issue.

For example, it is possible to create a vault lease for exactly one CI build and tie the lifetime of secrets the CI build needs to the lifetime of this build. Practically, this would mean that e.g. a token, some oauth client-id/client-secret or a username/password credential to publish an artifact is only valid while the build runs plus a few seconds. Once the build is done, it's invalidated and deleted, so exfiltration is close to meaningless.

There are two things to note about this though:

This means the secret management has to have access to powerful secrets, which are capable of generating other secrets. So technically we are just moving goal posts from one level to another. That is fine usually though - I have 5 vault clusters to secure, and 5 different CI builds every 10 minutes or so, or couple thousand application instances in prod. I can pay more attention to the vault clusters.

But this is also not easy to implement. It needs a vault cluster, dynamic PostgreSQL users take years to get right, we are discovering how applications can be terrible at handling short-lived certificates every month (and some even regress. Grafana seems to have with PostgreSQL client certs in v11/v12), we've found quite a few applications who never thought that certs with less than a year of lifetime even exists. Oh and if your application is a single-instance monolith, restarting to reload new short-lived DB-certs is also terrible.

Automated, aggressive secret management and revocation imo is a huge problem to many secret exfiltration attacks, but it is hard to do and a lot of software resists it very heavily on many layers.

Yeah that was an interesting discovery in a development meeting. Many people were chasing after the next best model and everything, though for me, Sonnet 4.6 solves many topics in 1-2 rounds. I mainly need some focus on context, instructions and keeping tasks well-bounded. Keeping the task narrow also simplifies review and staying in control, since I usually get smaller diffs back I can understand quickly and manage or modify later.

I'll look at the new models, but increasing the token consumptions by a factor of 7 on copilot, and then running into all of these budget management topics people talk about? That seems to introduce even more flow-breakers into my workflow, and I don't think it'll be 7 times better. Maybe in some planning and architectural topics where I used Opus 4.6 before.

> The real question isn't "do you need a database" but "do you need state" — and often the answer is no.

We have a bunch of these applications and they are a joy to work with.

Funny enough, even if you have a database, if you wonder if you need caches to hold state in your application server, the answer is, kindly, fuck no. Really, really horrible scaling problems and bugs are down that path.

There are use cases to store expensive to compute state in varnish (HTTP caching), memcache/redis (expensive, complex datastructures like a friendship graph), elasticsearch/opensearch (aggregated, expensive full-text search), but caching SQL results in an application server because the database is "slow" beyond a single transaction brings nothing but pain in the future. I've spent so much energy working around decisions born out of simple bad schema design decisions and tuning...

The main drawback is that you will need to be able to patch quick in the next 3-5 years. We are already seeing this in a few solutions getting attention from various AI-driven security topics and our previous stance of letting fixes "ripen" on the shelf for a while - a minor version or two - is most likely turning problematic. Especially if attackers start exploiting faster and botnets start picking up vulnerabilities faster.

But at that point, "fighting fire with fire" is still a good point. Assuming tokens are available, we could just dump the entire code base, changesets and all, our dependent configuration on the code base, company-internal domain knowledge and previous upgrade failures into a folder and tell the AI to figure out upgrade risks. Bonus points if you have decent integration tests or test setups to all of that through.

It won't be perfect, but combine that with a good tiered rollout and increasing velocity of rollouts are entirely possible.

It's kinda funny to me -- a lot of the agentic hype seems to be rewarding good practices - cooperation, documentation, unit testing, integration testing, local test setups hugely.

Any `return c` for some constant is a valid and correct hash function. It just has a lot of collisions and degenerates hash-maps to terrible performance. That was in fact my first thought when I read "simplest hash functions".

This. Sometimes during hectic times, my bosses need to get the right summary of everything in one phone screen. So I generally try to have mails with a first line summarizing everything in one glance, a short section you can read in an elevator ride, and then everything else.

For example, during Log4Shell, our Group Director of Operations was helping Support and Account Management with scared customers and was just bouncing customer meeting to meeting. Hence, line one of the mail was "Rollout of mitigation 1 to applications estimated by 12:13; delays expected at only internally reachable middleware". After that, a 3-4 sentence paragraph with more details and then a longer explanation in detail, ideally with keywords highlighted and such. Those would be discussed in some peace later on.

> (We also instituted pairing on production proxy maintenance. I'm not a fan of pair programming but pair maintenance is great.)

It's a great opportunity to share knowledge and techniques and I very much recommend doing so. It's an important way to get people familiar and comfortable with what the documentation says. Or, it's less scary to failover a database or an archiving clutser while the DBA or an archive admin is in a call with you.

Also reminds me of an entirely funny culture shock for a new team member, who was on a team with a much worse work culture and mutual respect beforehand. Just 2-3 months after she joined, we had a major outage and various components and clusters needed to be checked and put back on track. For these things, we do exactly this pilot/copilot structure if changes to the system must go right.

Except, during this huge outage, two people were sick, two guys had a sick kid, one guy was on a boat on the northern sea, one guy was in Finland and it was down to 3 of the regulars and the junior. Wonderful. So we shoved her the documentation for one of the procedures and made her the copilot of her mentor and then we got to work, just calmly talking through the situation.

Until she said "Wait". And some combined 40 - 50 years of experience stopped on a dime. There was a bit of confusion of how much that word weighed in the team, but she did correctly flag an inaccuracy in procedure we had to adress, which saved a few minutes of rework.

Ah, but then you choose the wrong language or language runtime and distros ship old versions for 10+ years :)

(compare: polkit. Both sides have their point, but I've been annoyed by this standoff a few times).

It depends on the complexity and if your LLM-driven changes fight the architecture of the project.

Some changes are in the area of "Well no one did that yet because no one needed it or had time for it", or "Well shit, no one thought of that". If Claude Code did these changes with good documentation and good intent and guidance behind it, why not? It is an honest and valid improvement to the library.

Some other changes rip core assumptions of the library apart. They were easy, because Claude Code did the ripping and tearing. In such a case, is it really a contribution to improve the library? If we end up with a wasteland of code torn apart by AI just because?

That ship has sailed, but I consider Unicode a good thing, yet I consider it problematic to support Unicode in every domain.

I should be able to use Ü as a cursed smiley in text, and many more writing systems supported by Unicode support even more funny things. That's a good thing.

On the other hand, if technical and display file names (to GUI users) were separate, my need for crazy characters in file names, code bases and such are very limited. Lower ASCII for actual file names consumed by technical people is sufficient to me.

At $work we're hosting business knowledge databases. Interestingly enough, if you need to revert a day or two of edits, you're better off to do it asap, over postponing and mulling over it. Especially if you can keep a dump or an export around.

People usually remember what they changed yesterday and have uploaded files and such still around. It's not great, but quite possible. Maybe you need to pull a few content articles out from the broken state if they ask. No huge deal.

If you decide to roll back after a week or so, editors get really annoyed, because now they are usually forced to backtrack and reconcile the state of the knowledge base, maybe you need a current and a rolled-back system, it may have regulatory implications and it's a huge pain in the neck.

I agree. I generally don't ask "Do we need to future proof this?", I rather ask: "Do we paint ourself into a corner?"

For example, if you implement a job framework of things calling a REST API to do stuff async, a sufficient first implementation is a simple monolith doing the stuff and some retries or periodic calls. Because, if it does not scale, we can start thinking about replacing that with something to put things into queues and scaling stuff-doers horizontally and such. But often enough, these simple things scale quite the distance.

On the other hand, if you start introducing in-memory caches into a singular instance to taper around database performance... that's a huge issue. That always leads to pain.

Additionally though, I"ve started to keep notes about people doing simple, efficient things without fanfare. This way, if my boss asks me what Bob did over the year, I can tell them that Bob made these problems disappear and how Bob is starting to handle this area of topics more and more. Suddenly Bob is growing responsible for this area, and if my boss asks Bob about these topics they did well in an annual review, Bob can shine. Hope they learn though.

Also, I have been called a bit of a hard-ass for this, but if the junior author of some piece of code is not able to explain to me why it is written that way or how they would extend it in a few reasonable cases, I consider that a problem.

This is orthogonal to both if it is well thought-out/naive/really strange code, or LLM generated/LLM assisted/hand written code. If there is a good understanding of the task and the goals behind it, the tools become secondary. If skills are lacking, it will end up a mess no matter the tools and it needs teaching.

Most of us could run stable servers with just ssh and vi. Would suck a lot though.