simo7

Many people seem to think the same but I don't get it.

If you only want to change lambda's configuration without touching the code, Terraform will do exactly that since the source_code_hash hasn't changed.

Opposite is also true.

I completely agree with your point about modules.

My rule of thumb is: only use a module if you want to enforce a very opinionated way to create resources across multiple projects. Most of the times you don't want that.

Regarding lambda, I also run a "make build" before "terraform apply". After all terraform is not a build tool. But I do make the zip with data.archive_file so that I can pass the output_base64sha256 straight to the lambda resource.

> They actually do serve the same purpose. The purpose is communication

You just said that a name, definition, question and assertion all serve the same purpose and can be used interchangeably...I mean what else can I say?

Exactly.

It was a nice exchange nonetheless but at this point it really looks like you hit a dead end.

> We call function names a "name" but it can also be called a function "sentence" or "function phrase"

I mean...we can also say that pigs are birds if you wish, everything is possible XD

Who in English would ever name anything as "A function that finds the x y coordinates of dogs cats and baboons in a picture". Does that sound like a name to you?

Do you actually speak like this in your daily life? "Please can you turn on the device for remotely visualising entertainment shows, news and sport events?"

Ah I forgot names don't exist: a name is a phrase, a definition is a phrase, a question is a phrase, an assertion is a phrase...can all be used interchangeably, they all serve the same purpose...ssure.

I think at this point you're lost in your own sophism. Good luck getting out of it!

You're mostly arguing on why your naming preferences are better. You're missing the point, I'm not addressing that.

Instead you seem to agree people name things in the same (irrational according to you) way both in English and Python. Which is exactly my point and what you previously claimed not to be the case.

Once again I am addressing this comment:

    A function that finds the x y coordinates of dogs cats and baboons in a picture.

> This is perfectly ok, but only because it's english.

No, it's not ok only because it's English. It's ok because it's a definition, it is not a name. In English as in Python we tend to prefer more concise names.

A function's name is...a name (duh), the comparison to prose makes no sense. Once you actually compare English and Python names you'll see they both tend to be more concise.

> See past your bias and ignore pointless aesthetics.

Ironically I find your style more poetic (we really have opposite tastes :P). But as you saw we both keep the same preferences independently of the language. No incoherence/bias there. That's the only point I'm making.

NB: I'm answering to the claim that we behave differently when naming things in programming languages vs natural languages. Otherwise I think it's mostly a matter of preferences.

I understand you prefer to be more detailed in your naming, that's fine, but in natural languages your names would sound unusual/verbose as much as they do sound unusual/verbose in Python.

You say everybody understands what a "washing machine" is therefore a short name.

Are you saying that when washing machines were still a novelty they should have been called "clothes-washing machines" instead? Unusual naming right? People do seem to have a distaste for long and overly-detailed names in spoken languages as well, don't you think?

And what's the point of a dictionary if names embed a full definition that leaves nothing to be guessed?

> Sure and I'm saying most people are wrong about where this "balance" actually lies. People place too much emphasis on conciseness.

Where to draw the line can be a matter of preferences, no intention of debating that, but people do tend to draw the line the same way whether they speak English or Python. No incoherent behaviour there.

The problem with your analogy to English is that you're comparing the wrong things.

A function is a "thing", you should rather compare it to an entity in the English language.

Why do we say washing machine instead of a home appliance powered by electricity used to wash laundry through the use of centrifugal force?

When naming things you need to balance descriptiveness with conciseness, a name is not a definition.

As a rule of thumb, avoid specifying things that can be easily guessed, especially if they are right there in your function signature (!):

     def find_xy_coordinate_of_dogs_cats_and_baboons_in_picture(picture: Picture) -> List[XYCoordinates]:

I'd refactor it to:

    def find_animals_in_pic(picture: Picture) -> List[XYCoordinates]:

> Hint: good PMs won't just base it on gut feelings or anecdotal evidence.

Sure you want some structure, but people often get this wrong.

Intuition is surprisingly important and it's easy to get into some pseudo-scientific approach and call it data-driven (eg. biased questions/scales in surveys are very common).

Ultimately to be approximately right is still better than to be precisely wrong.

I work with sensor data and although not explicitly mentioned I thought you could locate it in the "event streaming" and "stream processing" boxes.

What piece of architecture you think is left out?

For us it's the following reasons:

* It's more cost effective to place all the services on one cluster. As containers can share the instance resources it's easier to increase resource utilisation (not true w/ Fargate though).

* Support for new features on EB can be quite delayed. Eg. we need support for ALPN policies, it's a recent feature and it's not even in CloudFormation yet. With ECS we just manage the LB directly and we can do everything the API allows us to do.

* More granular control. Eg. during rolling updates we can decide a floor to how many containers are active at a given time (many others things like this).

* Integration w/ SSM and Secrets Manager.

* Better IaaS support (with EB everything happens in configuration files outside of Terraform or Cfn resources).

Of course...some people seem to have an obsession for race, they connotate everything with it.

> You can totally design your tasks to be idempotent

Yes of course, I mean Airflow is not a good fit for the tasks you don't want to be idempotent (I think most but not all tasks should be idempotent).

> I've never ran into issues with cross dag dependencies

I believe Airflow docs advice against them when possible. I see why from my experience: less visibility and more complexity, especially for backfills.

I think it's better to place everything in one DAG if that solves the problem. If it doesn't then sensors are ok I guess, but I would try to avoid them otherwise.

Correct, I use operators only to delegate the actual workload to an external service.

I would add these gotchas/recommendations:

- Airflow the ETL framework is quite bad. Just use Airflow the scheduler/orchestrator: delegate the actual data transformation to external services (serverless, kubernetes etc.).

- Don't use it for tasks that don't require idempotency (eg. a job that uses a bookmark).

- Don't use it for latency-sensitive jobs (this one should be obvious).

- Don't use sensors or cross-DAG dependencies.

So yeah unfortunately it's not a good fit for all the use cases, but it has the right set of features for some of the most common batch workloads.

Also python as the DAG configuration language was a very successful idea, maybe the most important contributor to Airflow success.

Understand but the quote from the article is wrong.

Edit: actually it seems in Italy is also reported every day.

Actually in Italy (as well as France) deaths outside hospitals are counted.

Yes, cases like Beretta, to name one, are more interesting.

Founded in 1526, privately owned, never changed its name nor got acquired.

It makes more than 200ml€/year in revenue and it seems it's still owned by somebody whose last name is Beretta.

I'm wondering how this technology could work for OLAP cubes.

An OLAP cube that is automatically & incrementally kept in sync with the changes in the source data sounds promising.

Is that a potential use case?

I think the article is a bit simplistic.

It's true that _often_ OLAP cubes are not needed. That's simply because the amount of data and the latency requirements are _often_ not too demanding.

Also, materialized views don't solve the major issue with OLAP cubes: the need of maintaining data pipelines.

I wonder if a solution to this problem could come from a different way of caching result sets: new queries that would produce a subset of a previously cached result could be run against the cached result itself. Of course this opens up a new set of problems, cache invalidation etc..

Two false statements in this article:

> ...Amazon, Airbnb, Uber and Google have rejected the data cube...

Airbnb uses Druid which is essentially an OLAP cube.

> BigQuery, for instance, doesn’t allow you to update data at all

It's not like that anymore since several years.

As in not phasing out products? That's more of a problem with Google than GCP itself.

GCP has its problems (heard bad things about CloudSQL support for Postgres), but it does seem to be willing to innovate when it sees an opportunity for it.

My point is about AWS being reluctant to embrace new technologies that could damage the ones it's deeply invested in. See Athena vs Redshift or Kubernetes vs ECS (the latter having much better support). That's what I meant by "Kodak moment".

Yet I have the feeling GCP is the one which has been improving the most the quality of its offering.

It's interesting to monitor what's happening with data-warehouses for instance.

It almost feels AWS is having its "Kodak moment" with Redshift: a very remunerative but old technology in which they are deeply invested in.

It prevents them to put more effort behind solutions that decouple compute from storage (see Athena) and as a result they are offering a subpar competitor to BigQuery.

I'm not convinced the idea could never possibly turn into a successful business.

Maybe he just tried to sell to the wrong people, maybe pharmaceutical companies could have been interested, maybe re-packaging the product differently...who knows...

The fact that it could provide real value for many people already sets it apart from most ideas.

Pronouns are meant to be a finite set, contrary to names which are an infinite (or unlimited) set.

That way you can group many multiple names under few pronouns.

If you want to allow an infinite set of pronouns you might just as well remove pronouns altogether as they would stop serving the purpose they are meant to serve.

> Think of special pronouns as names.

That's exactly the problem: pronouns are not names for a reason.

Pronouns are meant to be a finite set of words that can group an infinite set of other words, names.

In many languages gender is the criteria to form those groups. It 's a natural choice if you think of gender as a binary category.

Once you introduce the concept of gender fluidity the most logical solution is to introduce a residual pronoun that includes everybody who doesn't fit in the male/female dichotomy.

To adopt instead a inifinite set of pronouns means to poorly re-engineer a language. It's an extremely important and complex common good, we shouldn't be tinkering with it so lightheartedly.

True, big organisations are always dysfunctional in a way or another. It's surprising to see how successful companies can be, simply by not being too dysfunctional.

That being said, I do believe some of the German automakers are pretty high on the inefficiency scale.

> ...thus gets to the heart of the actual empirical disagreements between them.

I'm not sure the disagreement (between bears and bulls) is necessarily to be found in a different outlook on the future.

One can conclude all those conditions are likely to be met in the future and yet be bearish on the stock.

I'm overall optimistic on Tesla as a business, yet it seems my optimism is already fully factored in...no real upside, which is surprising given the risks.

I think he/she meant something different: it's not about how big/small are the chances, but about how much we know on the probability distribution.

You assume you know the probability distributions, you compare them and determine that nuclear will kill less than coal burning.

Except that you might have never observed a black swan event before (eg. massive sabotage of nuclear plants).

So the probability distributions you inferred from past events might be wrong.

The case of a lottery is different as the probability distribution of outcomes can be determined with certainty.

This should not come at a surprise as money really is debt.

Say you get a loan of 100$: now you have 100$ more to spend...but your creditor can still factor his/her credit (sell it at a discount) and spend that sum.

That initial 100$ can be spent twice (minus discounting factor).

Of course that loan needs to be repaid at one point. But what if you keep on making new debt to repay the previous debt? And what if the credits you generate have a particular utility for your creditors (eg. government-bonds for banks) so that you get 0 or even negative interest rate?

You got really close to creating money. The difference between different types of debts and money is rather quantitative (interest rate) than qualitative: https://en.wikipedia.org/wiki/Near_money.