4,390 karma · joined November 6, 2013
1. We build our own custom build system, because there is no CI that can do actual DAGs (maybe a few). A custom Kubernetes operator that parses Jsonnet files to create 100s of CRDs and pods to achieve extreme parallelization. EKS was 144$/mo (now 72$) but no info on master node types. Using watch endpoints with hundreds of pods did not scale well. They had to bump up the master node instances to c5.18xlarge, but same price for managed. But figuring out it was needed to do just scale-up took days. One c5.18xlarge is 2k$ month, and EKS runs at least 3 for HA. So it's a horror story for them. But we also had 100s of worker nodes so it might offset some of them.
2. Similar to CI, we allowed devs to deploy all microservices (~80) from any branch so that they can port-forward and use them. All of them had Ingress endpoints. Days after headaches and frustrations, it turns out nginx ingress generates megabytes of configuration whenever a new deployment occurs, forks a new subprocess with new cfg, kills the other connections. When it's done often, it takes 30GB of memory when 50 developers use it (~4000 pods) and it often dies and restarts. Similar story for Prometheus, kube-state-metrics; they do not like short-lived containers and hug on memory.
Seriously though, what happened to that debate?
I have not found a way/tool to replace this. Many of the tools fail at dynamic data with cardinality. Wanted to use Clickhouse like this, adding columns as they are discovered but it did not go well, but it has been a while. Also adding replicas is not as easy as Elasticsearch.
Does anyone have a similar use case implemented with Clickhouse when data is not known before hand and unfolds as time goes by?
You have a great history of availability and would like to say congrats.
> A customer came in today to Resmo and wrote some queries. But they insist it does not work. We tried explaining to them they are continuously writing the wrong query and that is not the way system is supposed to work. They are abusing and they say we paid for it and I'll write whatever query I like! And I banned him. Am I the asshole?
It wrote:
> NTA
They sound like an annoyance, and actually pretty rude. It's not your job to teach them the correct way to use the system.
> YTA
You can’t just ban people for writing the wrong query. If you truly wanted them to write a certain way, have someone proof read their work before they start running queries against it. This seems like some sort of bait and switch tactic where you pay for an X amount of accesses in hopes that people abuse it and then get mad at them when they don’t meet your unrealistic expectations
> Toss-up
NTA, he was being abusive and you banned him to prevent further abuse. Why? Did he leave a contact email?
Can Atlassian’s RDS backups be used to roll back changes? We cannot use our RDS backups to roll back changes. These include changes such as fields overwritten using scripts, or deleted issues, projects, or sites. This is because our data isn’t stored in a single central database. Instead, it is stored across many micro services, which makes rolling back changes a risky process.
That all being said, LXD is great way to run non-ephemeral containers that behave more like a VM. Also checkout multipass, by Canonical that also makes spinning up Ubuntu VMs as easy as Docker.
Oh okay, risk is relatively low. Then:
> or have specifically searched using a query that would return the analysis in the search results.
Well, this is not good. Of course, it depends on what you'd publish on a private page, but its discomforting.
- Playwright is more lightweight. Can be good or bad on what you expect. But I definitely prefer Playwright.