P.S. I have at least twice ordered missing ingredients mid cooking and luckily both those times, delivery was under 10 minutes.
464 karma · joined April 15, 2022
P.S. I have at least twice ordered missing ingredients mid cooking and luckily both those times, delivery was under 10 minutes.
I thought of doing something similar for our data warehouse with AWS Fargate and Postgres but the cold starts and limited disk space required too much engineering on top to make it work.
Moving to Snowflake comes at the cost of losing so many Posgtres features in exchange for speed. Things like foreign keys, constraints, extensions etc which requires so much engineering to replace in Snowflake. I would be happy to pay 25x the price for a 10x speed increase for a specific query.
Sometimes adding a type to a single object (e.g. `df: pd.DataFrame = ...`) only once at the top of a 20-50 line function, is usually sufficient for Pycharm to infer all other types downstream without ambiguity.
> Sure that's only a 6.5% increase in temperature w.r.t. to 46C ... but that's an incorrect way of looking at it.
I've added an edit to clarify what I was aiming at.
for anyone who thinks 49C in Delhi is some extreme temperature it's not really that far from usual maximum (2019 and 2020 both May max. 46C).
That's a 3C increase!Sure that's only a 6.5% increase in temperature w.r.t. to 46C but that's an incorrect way of looking at it. If you were to set your air-conditioning to 32C, then the energy required to cool from 49C to 32C is 21% more than the energy required to cool from 46C to 32C.
32C is also the wet-bulb temperature, so when thinking about tolerability, at least use 32C as the baseline (and not 0C) when comparing temperatures.
P.S. Humidity is a major factor. I'm assuming humidity is the same this year as it was the previous years.
Edit: To clarify, my comment was about not giving into your insticts and underestimating the effects of 3C rise w.r.t. 46C. Thinking of 3C rise in terms of energy increase or comparing it to a baseline of wet-bulb temperature give you better intuitions.
[1] https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/postg...
A tidbit from the book is that humans works well in a tribe size of ~150 and anything larger requires a cohesive belief system that ties tribes together. According to the author, examples of such belief systems are religions and corporations. The author goes much deeper and no part of the book was boring. I highly recommend reading the book, it's a much more distilled version of the comment section.
It's hard to put to words what the GIL does for Python. I think the best I can do is to point to Rust and ask you to consider what Rust's number one claim to fame is? A more than significant part of Rust's bible ("the book") is about memory ownership. Python's GIL lets me be completely ignorant of these issues and all my brain cells just focus on the problem at hand.
My long term livelihood and sanity is at stake here and I really appreciate that some of core maintainers of Python are taking their time and thinking deeply about the implications of removing the GIL.
Python with the big bad GIL has become the most popular language in recent years. And it's gotten there slowly and steadily over 20 years. The most popular language just so happens to be the only (?) language that is single threaded.
I discovered pydantic-core [1] which is a backend of sorts for pydantic written in Rust. It is yet to be integrated into pydantic, but once the integration is done, the author expects a potential 10x speed improvement.
There's already an FDW for DuckDB [1]! But it's not available from within AWS RDS because they only allow a few extensions. So, I would launch an EC2 instance and launch a postgres instance on it. Then I would install the DuckDB FDW on this postgres instance. Then, on the AWS RDS postgres instance, I would use the postgres_fdw extension to connect to the postgres instance running on EC2 and indirectly gain access to it's DuckDB FDW. I'm not sure if this chained FDW strategy will work out. But here's why I think it's a great idea:
1. Any data that I am willing to put into DuckDB is guaranteed to have backups and/or an ETL already setup to sync data. Therefore I don't need this EC2 setup to be "reliable". If it goes down, I'll just rerun the ETL.
2. I need to pay a small DevOps cost to have some performance gain but the analytics complexity remains unchanged because at the end of the day it's just Postgres. I get to continue to use DBT against the original AWS RDS.
The only thing I need to see is if there are any performance gains to be had using this setup. I won't take that for granted until it's tested.
[1] https://en.m.wikipedia.org/wiki/Clathrate_gun_hypothesis
I'm from the south of India and now live in Europe. My body tolerates heat quite well compared to my European friends. I remember tolerating 38-40C indoor temperatures and sleeping through it with heavy sweating. But air temperature of 42C is where I draw the line. It is almost biologically impossible to survive this.
So this story isn't just about record high temperatures, it's about record high temperatures that coincide with biological limits. If air temperature reaches 44C, it's impossible to survive this without artificial cooling and I predict a state of emergency will be declared every summer starting 2030.
There's one additional factor. The hottest months in the equator are June to August. This happens to coincide with the monsoon season in India. So I remember dreading May but looking forward to the monsoon in June. If climate change messes with the monsoon season, the Indian subcontinent is f'ed.
I know this is a big ask, but if DuckDB can be in lock step with Postgres on JSONB handling :french-chef-kiss:
As far as I am aware (haven't researched), the standard way to get logs or metadata out of AWS Cloudwatch is in a JSON lines [1] format. Therefore, even if I were to write a custom JSON output format for my custom service, I would try my best to stick to the JSON lines format since I would have already written parsers and data connectors for AWS Cloudwatch. That's just me though.
So my metrics of success in this scenario are based on that fact that I have to deal with 10-100 such queries in a project in my day job, so I would choose SpyQL to write and maintain a simple and readable 5 line query in under 5 minutes with decent performance to solve a trivial use case of computing an average.
P.S. I know the article is about performance and your response about Python being is slow is beyond accurate and yet I will always choose to use it because it is not ashamed to sit on the shoulders of the fast and ugly.
[2] https://github.com/simdjson/simdjson/blob/master/doc/iterate...