317 karma · joined December 29, 2016
I'm wondering if anybody has any better recommendations given the requirement of being able to add storage capacity without having to completely recreate the FS.
So e.g. I have hyper+a for Alacritty, hyper+b for Browser etc.
Of course this is a model of how electricity should work, but it is still useful and they document the differences from "real" electricity decently in the books.
In any case its a super fun toy and I highly recommend getting it!
Then you progressively clean it and model it into some format that is easily usable for dashboards or self service analytics.
To do that you build a DAG of SQL scripts and dbt makes that easy through templating, macros and automatically generated docs.
You still have to execute the DAG somehow and here you either use their OSS version and schedule it with anything from Github Actions to Airflow, OR you buy their Cloud offering
We package all the code in 1-2 different Docker images and then create the DAG. We've faced many issues (logs out of order, missing, random race conditions, random task failures, etc.)
But what annoys me the most is that for that 1 big DAG, the UI is completely useless, tree view has insane dupplication, graph view is super slow and hard to navigate through and answering basic questions like, what exactly failed and what nodes are around it are not easy.
> Bad programmers worry about the code. Good programmers worry about data structures and their relationships.
The problem with dogmatic OOP is that it mixes data structures and code into one. This leads to hard to modify code, unexpected side effects, poor parallelism. Imperative or FP code does the minimum to define what the data should be shaped like and then has a bunch of functions that either modify it or do some IO with it.
That's a lot easier to trace through and reason about. That in turn leads to more malleable code, which lets you evolve it faster as your understanding of the problem domain evolves.
We have chosen k8s and I would again, because its nice to use. Its not necessarily easier, as you point out, the complexity of managing the cluster is considerable. But if you use a managed cluster like EKS or DO's k8s offering, you don't have to worry too much about the nodes and the unit of worry is the k8s config and then for deployment you can use Docker.
I like Docker, because its nice. Its nice to have the same setup locally as you have remotely.
In my experience the tooling around k8s is nice to manage declaratively, I never liked working with machines directly because even tools like Chef or Ansible feel very flimsy.
The other thing you can do is run on ECS or similar, but there the flexibility is a lot lower. So k8s for me offers the sweet spot of being able to do a lot quickly with a nice declarative interface.
I'd be interested to hear your take on how to best run a small cluster though.
My problem is that this is super inconsistent. Some things are done as a method call on an object, others by passing the object to a pandas function and others yet by passing a function to a method on an object. This is the major source of frustration for me.
Maybe there is some logic to that, but I haven’t found it yet and I think that is a sign of bad API design. Its like PHP to me. All nice and documented but useless without Googling everything
In some sense this is true, but lets not get carried away. The American Startupism is all about the flash, big growth, big impact, big IPO. In EU you have companies slowly grinding a hard problem for 20 years that also have huge impact but no flashy news. Look at things like vertical farming, ASLR, fintech. Thats all thanks to low risk and stability.