HNHacker News
TopNewBestAskShowJobs

peterwwillis

1 karma · joined March 4, 2021

submissionscomments
peterwwillis··on Literate DevOps
Some kinds of declarative automation are just bad. Case in point: configuration management.

Ansible was written before we realized immutable infrastructure is the best way to go. Configuration Management tools craft system state dynamically like a drunk sculpting a Roman bust with a Louisville Slugger. You can get them to do what you want, but it takes a lot of work, and even then the outcome is uncertain. CM tools are often complex because a system whose state is constantly shifting requires complexity to handle it.

If instead you make immutable artifacts, you don't need complexity. A simple series of straight-forward commands in a single version-controlled file does everything you need. Dockerfiles seem immature at first because of how simple they are, but in practice it's much more reliable than Ansible, not to mention easier to support. Thus, non-declarative automation, when used with immutable infrastructure as code, trumps declarative.

Ansible is also only optionally declarative, which people miss just because it has that bastardized form of YAML for a config file. The simplest tasks, roles and playbooks work when executed sequentially, but not necessarily when out of sequential order. When you do make it super-duper-declarative, it can involve tons of confusing logic that makes it nearly impossible to understand, and is much more verbose (and complex) than a simple script. As soon as automation is more work and cost than the alternative, it should be ditched.

peterwwillis··on Literate DevOps
I like the concept, but the problem with embedding random snippets of code into a document is there are too many assumptions about the state of the system they're running in. The operating system, the revision, the installed software, how it's configured, where things are located, orders of operation, what network you're on, what access your user has to what systems, etc.

Rather than embed snippets, write a single simple script that automates your steps, and create an immutable artifact. The simplest way to do this is to use Docker with a base image that mimics the target system, and performs all necessary steps in one go. This allows you to perform a 'test run' in an empty container to make sure the steps work. And you can embed in-line documentation and generate it with Doxygen or some other tool. (I highly recommend you get all your teams to standardize on a tool like this, and use it literally everywhere)

(Also, pet peeve, but what does DevOps have to do with this? Can we stop over-using this term please? Not everything that's "sysadminny" is DevOps)

peterwwillis··on Level 3 Global Outage
I have spent hours and hours banging my head against Erlang distributed system bugs in production. I am absolutely mystified why anyone thought just using a particular programming language would prevent these scenarios. If it's Turing-complete, expect the unexpected.
peterwwillis··on Don't trust default timeouts
Different timeouts exist for different purpose. Sometimes infinite is the only sane default. Sometimes the default must be dynamic. And sometimes you just pick something that sort of makes sense with all the other system components in mind.

There are a dozen or more timeouts just for a TCP connection. There's the initialization timeout, the 3-way handshake timeout, the half-closed timeout, the time-wait timeout, the unverified reset timeout, the established connection timeout, the retransmission timeout, the timed wait delay, the delayed ack timer, the arp cache timeout, the arp cache minimum reference timeout, the keep-alive timeout, and more.

Every single person in the world depends upon default timeouts, so of course they matter. When they are picked intelligently, they improve the default behavior of the majority of system interactions. So we can trust default timeouts, when they are useful. But if we're building a system, it makes sense for us to determine what the appropriate timeout is for our system.

peterwwillis··on I thought I would have accomplished a lot more today and also before I was 35
Imagine a takeout pizza restaurant run entirely by one woman. She opens the store in the morning, receives shipments, does food prep, takes orders, makes pizzas, delivers them, closes at night, files paperwork, and cleans up. By the end of the day she only gets a few dozen pizzas to customers. And as she's going to sleep, she tosses and turns, worrying that she didn't get enough done.

Now imagine her sole task for the day was to prep ingredients. How do you think she'd sleep?

peterwwillis··on The World of Legacy IT Systems
Something I'm not sure is clear from the article is that not all legacy is bad. The definition of the word 'legacy', essentially, is "something that is handed down [through generations]". You can hand down good things and bad.

Banks often have incredibly old systems because their products don't change. A savings account in 1940 is pretty much the same as a savings account today. They basically have no need for DevOps, because they have no need to provide fast-moving, competitive products, and silos are a natural fit in a world of sensitive financial information. So some really old systems are pretty much fine as-is, as long as the hardware can keep running, and a few over the hill programmers are on call.

Another example of useful legacy is old physical infrastructure. The Frankford Avenue Bridge in Philadelphia is a 73-foot long stone bridge that's 323 years old, and carries a substantial amount of car traffic. But it's also not being maintained like it should be, and the more-than-expected wear over time is slowly eroding it. That's the bad part of legacy systems: when you know they need maintenance and they don't get it, so they rot.

Modern infrastructure can suffer the same fate. Concrete bridges, wooden houses, cars... they all need maintenance. So do modern computer systems, where they need to get regular security patches. So the problem is really just a maintenance problem, and we happen to let older systems go to hell more often than we replace newer ones.

peterwwillis··on Zoom is now critical infrastructure – that’s a concern
Most of the internet that everyone in the world uses is constructed and maintained by companies that you don't give a single cent to, and who don't owe you anything if it goes down. This applies to a lot (maybe most?) things in the world. Say freight (trucks, trains, ships, etc) stops moving, for example. Within a month or two all the grocery stores would be empty. We'd have Mad Max-style convoys of cars driving around trying to find food and raid stores. It wasn't but a hundred or so years ago that most people survived on local food suppliers, and today almost none of our food is local.
peterwwillis··on Architecture Playbook
I really like the intent behind this, but it needs a few more years to cook, and a lot more docs.

If you want to build a house, there's a whole lot of ways to do it. You can use a lot of different tools and materials and techniques, or a few. Doesn't really matter. What matters is what the purpose for the house was, where you build it, how much money you have, etc. There's no one set of tools that can tell you how to architect any house. Architecture should spring from the creating of the house, not the other way around.

peterwwillis··on Blanked-out spots on China's maps helped us uncover Xinjiang's camps
Some options:

1. Resolve to not care if anyone is hurt in the process of providing you with consumables. Come up with excuses why it's not your responsibility, avoid learning more about it, ignore it, live in denial. The moral dread is gone.

2. Acquire your goods somewhere else, which may involve more cost and difficulty. The moral dread may not be gone if you realize that simply not supporting the oppression does not stop the oppression (avoiding German products does not stop the Holocaust)

3. Consume less. Same issue as #2 though.

4. Actively work towards ending the oppression which is giving you moral dread. This may end the moral dread, even if you continue to consume their products.

#1 is what most people go with.

peterwwillis··on Willscott/go-NFS: Golang NFSv3 server
Most internet applications use HTTP because real networking is a pain, designing a custom protocol correctly is difficult, and HTTP provides enough functionality and flexibility to munge network traffic any way you want. They use JSON because it's the least pain in the ass of all the data formats.

Custom protocol over UDP is the fastest, which NFS supports, so NFS is basically the fastest (unless you get into wonky multiplex-streamed custom apps). However, UDP apps do not often work well through firewalls.

peterwwillis··on Willscott/go-NFS: Golang NFSv3 server
Almost all WebDAV implementations are a big pile of crap, and network filesystems in general are much more tricky than people realize. If somebody invents a brand new protocol that that actually works it would probably be adopted, but it would face at least as many problems as NFS does.

I don't really understand why people want network filesystems most of the time. If you try to pretend it's like real local i/o, and it's not on a perfect network, you will have a bad time. Protocols designed specifically just to transfer a file tend to be more resilient and also not expose people to the problems of programs trying to do file i/o.

peterwwillis··on Non-obvious remote work techniques
I can tell you it won't work with Zoom. For some reason it takes over all of your sound, so if I play quiet music I can't hear it.

This would also be incredibly wasteful of bandwidth and CPU. Bandwidth is not unlimited, and if everyone in the Zoomy world kept a silent conference on in the background all day, the internet pipes [Zoom's especially] would be quite a bit more full, and my CPU would continue to churn unnecessarily.

Then there's the etiquette. What if people just start water-cooler-chatting? Maybe at a low volume it's OK, but I could see people leaving just because of too much laughter [yes laughter is annoying and yes I am a horrible person].

peterwwillis··on Want people to do the right thing? Save them the guilt trip
In different countries, the same law with the same penalty has different rates of law-breaking, so you can't necessarily draw a clear line of "this works, this doesn't" in every country/case. Yes, being harsh can have results. Does that mean its results are better than a different tact? The evidence of one method alone doesn't show you that, you have to experiment or study the same situation with both methods.

There is evidence that, for example, treating prisoners more humanely and giving them trust and responsibility can lower recitivism rates, while there's also ample evidence to suggest that harsh fines and imprisonment for minor crimes creates greater recitivism. But this may vary by country and implementation.

peterwwillis··on Docker to rate limit image pulls
Luckily though, companies have cash to pay for registrations, and those that won't probably have engineers who can set up Squid proxies.
peterwwillis··on Please read the paper before you comment
Twitter is a machine for removing intelligence from communication
peterwwillis··on What coronavirus will do to our offices and homes
Amazingly, no. Most commercial offices have fiberglass inside the ducts because it saves them more money. But the sealant breaks down over time and air turbulence makes the fiberglass fly. Larger particles cause symptoms like an allergic reaction, smaller particles can cause asbestosis if they are found in large numbers, though estimates so far indicate this is unusual. But it still seems stupid that we're pumping fiberglass throughout our offices. (To say nothing of mold/mildew if the fiberglass gets wet...)
peterwwillis··on What coronavirus will do to our offices and homes
> Laila makes a coffee in the kitchen area

Seriously? You're worried about viruses and yet you're still gonna allow the "coffee petri dish," where hundreds of grubby dirty hands fondle communal cups, bowls, coffee machine buttons, creamer, sugar, stirrers? To say nothing of the giant "catered lunch" buffets with no sneeze guards.

> The air conditioning system uses UV light to kill pathogens. It also reduces the humidity to help prevent germs multiplying, responding to a stream of data from sensors fixed around the building, and wearable sensors used by staff.

Yet it's still got 50-year-old unmaintained fiberglass insulation inside the ducts, which, because the overlayment breaks down over time, streams tiny particles of glass throughout the office to be inhaled. (this has probably been happening in your office for some time if it's an older building)

> The staff got fed up with plastic everywhere a couple of years ago. They said it made the office feel like a hospital. Laila prefers the plants.

The plants are plastic now. Too many finicky potential problems like mold/mildew, they're hard to clean, not maintenance-free.

> It's face-to-face with colleagues - but all at a safe distance.

We don't even practice social distancing now. It's not gonna happen at work with people who feel comfortable around each other. And honestly, what's the point indoors? You cough and the overhead A/C pushes microscopic water particles way farther than 6 feet.

> It's 16:00 and time to go home. Laila moved to the suburbs with a friend after the lockdown in 2020. It's a longer commute, but she doesn't mind as it's only once a week.

There a lot of roommates in the suburbs? Most middle-class young people working in offices like this will still need several roommates to afford rent, what with cost of living constantly increasing but pay not adjusting as much (news flash: covid-19 has not fixed income inequality). The suburbs aren't actually much cheaper in most parts, because people who work in cities live in them. People will continue to live in cities because there's still more access to everything in them.

> she knew she would need a home office [..] There's a new standing desk, a properly adjustable office chair and storage for documents

I work from home now and I don't need a home office. All you need is a table and a chair in a corner of a room. With kids, I get it, but otherwise, what's all that extra space going toward? Every "document" I own fits in a single box. If anything, apartments will just become "Tokyo-ified".

> She never realised, before, how important it was to have good lighting, so she had spotlights installed in the ceiling and bought a proper desk lamp.

Ok, now you're jumping the shark. Nobody's installing track lighting just to do computer work. One good desk lamp is all you need.

> She's near a big road and she needs to concentrate hard when she's working - sound-proofing is an issue she hadn't anticipated.

Headphones worked for the office, they work at home too.

peterwwillis··on US Border Patrol says they can create central repository of traveler emails
> What's a reasonable way to protect yourself here?

Contact your representatives in government and tell them to make legislation banning this practice. Donate to the ACLU.

peterwwillis··on US Border Patrol says they can create central repository of traveler emails
2 out of 3 people in the U.S. live within 100 miles of a border zone. About 200 million people. https://www.aclu.org/other/constitution-100-mile-border-zone
peterwwillis··on FBI release a report on Russian Linux Malware called Drovorub [pdf]
Here's a breakdown of the rootkit: https://www.splunk.com/en_us/blog/security/a-little-splunk-m...
peterwwillis··on How to contact Google SRE by dropping a shell in Cloud SQL
All you need is write access to the pty fd (or in the case of a reverse shell, just the fd of the tcp socket). The SREs could talk to the hackers and the hackers could just echo stuff in their terminal which the SREs could read. Writing a file to disk is less l33t, but more straightforward :)

Edit: I think I was wrong; you can't manipulate network socket fds this way, you'd have to use ptrace() on the process. If it were a real shell with a pty I believe what I suggest could work, but reverse shells don't open ptys.

peterwwillis··on Mail-order chicks are arriving dead, costing Maine farmers thousands of dollars
At some point, probably when it's too late, people will realize that Trump appointing people who intentionally destroy the agencies they oversee was an intentional plan to destabilize the government.
peterwwillis··on Lost Advanced Civilizations
So, assume this is all true. We came from Mars and advanced civilizations came and went, completely undetected. Who the fuck cares? Isn't not like we can dig up their ancient technology and use it now, can we? It's a fantasy for people who don't have the imagination to believe in dieties. Super-advanced-yet-dead civilizations are a cheap imitation of dead Gods.
peterwwillis··on Do I Need Kubernetes?
Option 1: Everyone develops on the CI/CD platform.

Ops gives devs "disposable environments" to do their testing in. Basically, GitHub + Jenkins + Terraform + AWS. I've used this to stand up real infrastructure every time a PR is opened, and pipelines run against the infra. Code and infra match each other because they both exist in the same branch/repo (monorepo; you can of course do multi-repo, just takes more coordination). It's all destroyed as soon as the PR is closed, but you can also keep it up to do dev work against. You can also keep one copy of the latest master branch up at all times (the "dev" or "cert" or "test" infrastructure) as a shared test environment. Downside: you have to have access to the network (which is in AWS, so that's not so difficult). Upside: the dev environment always mimics production, devs don't need to do anything to stand it up.

Option 2: A bunch of mocks, and scripts to stand up stuff in Docker, using docker-compose or something else. This becomes a bit of a problem to manage with lots of devs, though, and docker-compose is not a good model for production deployments.

peterwwillis··on Do I Need Kubernetes?
It's a distributed decentralized database using self-signed certs. Just by itself it requires maintenance: upgrading the software, upgrading the host it runs on, rotating keys, networking, access control, key space maintenance, backup, etc. Here are the docs you need to know to run it: https://etcd.io/docs/v3.4.0/op-guide/ And there's another dozen docs not written there that the admin just sort of finds out over time.

But it's part of other systems too, making the overall thing a system of systems. Interactions between systems of systems are complex and cause unexpected behavior. At some point you will run into an error in K8s that you can't resolve that will require you to debug Etcd. And "Bob" help you if the database gets corrupt or overwritten, or incompatible versions of software screw up what's in the database, etc. (My original analogy was inaccurate... Etcd is more like the engine than just A/C, because if it stops working, everything stops working)

Do you know what happens if an Etcd certificate's SAN field does not include domain names but only IP addresses? The client requests HelloInfo with an empty ServerName so it doesn't trigger a TLS reload on handshake, making it more difficult to replace expired certs. That is a single random quirk in a single component of this software which underpins all of Kubernetes. I cannot sit here and explain every single reason why Etcd is complex; it must suffice to say that the software just is complex, and that this reality means that while it may sometimes be simple for some people to operate, it will definitely not always be simple to operate, and there will come a time that the true cost will emerge.

Now, most people don't need to pay for that high cost of complexity. They can use a SaaS/PaaS product like AWS ECS/Fargate or others, where somebody at some other company is dealing with the cost of complexity for you. All you have to do at that point is run some API calls and everything just works. Not only is it easier, it's immensely cheaper, less time-consuming, and more reliable.

...But you might not even need ECS! There's a lot of work just to get a simple PoC up on Fargate with an NLB, ACM cert, RDS instance, security groups, VPCs, cluster, service, task, etc. Compare that to just spinning up a micro instance and running MariaDB and a Python on it, and the latter you can have done in 20 minutes. If you can avoid complexity and still meet your SLOs, do that.

peterwwillis··on Do I Need Kubernetes?
What happens when your car's A/C stops working? Most people think, I'll just get a little can of R134a, fill up the system, and it'll be good as new. Somebody said they did that once and it worked just fine, so it should work for you too, right? I mean, it says so right on the can, and there's YouTube videos of it and everything.

The trouble is, A/C is a complex system. There are moving pieces with specialized oils that oxidize and break down over time. There's a sealed system of pressurized gas. There's a pump, clutch, coils, fans, filter, thermostat, drain, belt, and electronics, Any of those parts could fail in a number of ways. Just to inspect it you need a custom gauge set, a tank of R134a, and a vacuum pump.

Etcd is about as complex as an A/C system. That is one of a dozen components of a Kubernetes system, before we get into custom integrations, which you will need about another dozen of.

The million dollars is to pay for everything needed to set up and maintain all of that, create the custom integrations that do not come turn-key from the community, create the custom integrations the community doesn't even have, integrate it with your development and deployment systems, business requirements, application-specific needs, and so on.

A million is an average. You can get away with less, just like you can get away with pumping a pre-pressurized A/C system with extra coolant: if you're lucky it won't break. When it does, I hope you have either a lot of time, or a lot of money to pay a consultant.

peterwwillis··on Do I Need Kubernetes?
Here's a quick reference:

1) Are you on AWS? Then you don't need Kubernetes. Use Fargate.

2) Are you on Google Cloud? Then you don't need Kubernetes. Use Cloud Run.

3) Are you on Azure? Then you don't need Kubernetes. Use Azure Container Instances.

4) Are you on a PaaS like Heroku? Then you don't need Kubernetes.

5) Are you on a random VPC provider / bare metal machines? You could probably still do without Kubernetes using Docker Swarm (apparently it's not dead!), Nomad, Mesos DC/OS, or a standard Linux box and systemd (or some other process or cluster scheduler).

6) Do you need to solve the bin-packing problem? Do you need to self-host a service mesh of microservices in multiple colocated regions? Do you need a fully automated redundant fault-tolerant network of disposable nodes to constantly reschedule different versions of applications with stringent RBACs, scheduled tasks, dynamic resource allocation, and do you have about a million dollars to spend on building and maintaining it all? Then you need Kubernetes.

peterwwillis··on A layman's guide to understanding the Cloud
I don't think it's a problem. Every complex system is really an automated network of someone else's stuff employing some idiosyncratic tech to manage it. Mass-produced textiles, highways, factory farming, bee hives, air conditioners, coffee, rain. We don't always own the thing and don't always have control over how it works. But we still get a lot of value out of it and continue to use it.
peterwwillis··on The brilliance of All Gas No Brakes
> 98% of the people at area 51 (admittedly a lower percentage than most events) were completely normal,

Is it "normal" now to go to a place where conspiracy theories suggest the government is keeping alien remains and try to storm its gates?

peterwwillis··on Mutt email client 25 years old
Sure, but the problem of storing messages indefinitely isn't from the protocol, it's from the client. The client may need to verify if it's actually seen a message before and keep a local index of them; it may need to double check the message still is what it expects before it tries to delete it; it needs to present you with options on how to handle message deletion; etc.

POP3 is actually less reliable at keeping messages indefinitely than IMAPv4 is, due to limitations of the protocol. Some clients have defaults which simply delete the remote copy after download (or reading, which is different), but again that's not the protocol's fault.

← PreviousPage 6 of 34Next →