HNHacker News
TopNewBestAskShowJobs

iknownothow

464 karma · joined April 15, 2022

submissionscomments
iknownothow··on The age of the fast self-checkout lane is over
Yes it has. I order groceries from my couch and I pay 2€ extra for delivery and my door bell rings under 15 minutes 95% of the times, sometimes in under 7 minutes. I live in a major German city.

P.S. I have at least twice ordered missing ingredients mid cooking and luckily both those times, delivery was under 10 minutes.

iknownothow··on Neon – Serverless Postgres
I knew my bet to sticking with Postgres would pay off! This looks super exciting.

I thought of doing something similar for our data warehouse with AWS Fargate and Postgres but the cold starts and limited disk space required too much engineering on top to make it work.

Moving to Snowflake comes at the cost of losing so many Posgtres features in exchange for speed. Things like foreign keys, constraints, extensions etc which requires so much engineering to replace in Snowflake. I would be happy to pay 25x the price for a 10x speed increase for a specific query.

iknownothow··on Study: Curcumin spice boosts memory by 30%, eases depression
In that case I would be curious if these differences show themselves in people from different cultures that consume Turmeric en masse. For example, my aging parents consume significant amounts of turmeric (approx. 250 mg) for 2 out of 3 meals every day.
iknownothow··on Useful Python decorators for data scientists
I don't exactly know what Pycharm does but I only add type hints when Pycharm's default type checker raises a complaint or is unable to infer the type. To me that's just the perfect middle ground.

Sometimes adding a type to a single object (e.g. `df: pd.DataFrame = ...`) only once at the top of a 20-50 line function, is usually sufficient for Pycharm to infer all other types downstream without ambiguity.

iknownothow··on 9-Euro-Ticket
Sources for claims? Genuinely curious.
iknownothow··on Delhi suffers at 49°C as heatwave sweeps India
Sure, I agree that it's incorrect. I should have used 0K as base.
iknownothow··on Delhi suffers at 49°C as heatwave sweeps India
For some reason you seem to have ignored the second half of the sentence I wrote.

> Sure that's only a 6.5% increase in temperature w.r.t. to 46C ... but that's an incorrect way of looking at it.

I've added an edit to clarify what I was aiming at.

iknownothow··on Delhi suffers at 49°C as heatwave sweeps India

  for anyone who thinks 49C in Delhi is some extreme temperature it's not really that far from usual maximum (2019 and 2020 both May max. 46C).
That's a 3C increase!

Sure that's only a 6.5% increase in temperature w.r.t. to 46C but that's an incorrect way of looking at it. If you were to set your air-conditioning to 32C, then the energy required to cool from 49C to 32C is 21% more than the energy required to cool from 46C to 32C.

32C is also the wet-bulb temperature, so when thinking about tolerability, at least use 32C as the baseline (and not 0C) when comparing temperatures.

P.S. Humidity is a major factor. I'm assuming humidity is the same this year as it was the previous years.

Edit: To clarify, my comment was about not giving into your insticts and underestimating the effects of 3C rise w.r.t. 46C. Thinking of 3C rise in terms of energy increase or comparing it to a baseline of wet-bulb temperature give you better intuitions.

iknownothow··on My experience with exporting Postgres RDS partitioned tables to S3
As far I can tell, exporting query results from Postgres RDS to S3 uses Postgres's Copy and only supports CSV and Text [1]. Can you provide a link for parquet exports?

[1] https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/postg...

iknownothow··on Ask HN: Is it true that any community that grows big enough, gets ruined?
The underlying causes of the problem are both mathematical and human. While the mathematical causes are reasonably well understood by experts, I think the book Sapiens covers the human factors quite well.

A tidbit from the book is that humans works well in a tribe size of ~150 and anything larger requires a cohesive belief system that ties tribes together. According to the author, examples of such belief systems are religions and corporations. The author goes much deeper and no part of the book was boring. I highly recommend reading the book, it's a much more distilled version of the comment section.

iknownothow··on Python Language Summit: Python Without the GIL
As someone who only knows Python, this gives me existential angst. Like some other people have pointed out, I think the GIL is secret sauce to Python's success. I would be vary of the religious fervor behind taking the GIL to the gallows.

It's hard to put to words what the GIL does for Python. I think the best I can do is to point to Rust and ask you to consider what Rust's number one claim to fame is? A more than significant part of Rust's bible ("the book") is about memory ownership. Python's GIL lets me be completely ignorant of these issues and all my brain cells just focus on the problem at hand.

My long term livelihood and sanity is at stake here and I really appreciate that some of core maintainers of Python are taking their time and thinking deeply about the implications of removing the GIL.

Python with the big bad GIL has become the most popular language in recent years. And it's gotten there slowly and steadily over 20 years. The most popular language just so happens to be the only (?) language that is single threaded.

iknownothow··on Ballerina: A Data-Oriented Programming Language
This pales in comparison to Pydantic in terms elegance, flexibility and batteries being included. But pydantic is quite slow.

I discovered pydantic-core [1] which is a backend of sorts for pydantic written in Rust. It is yet to be integrated into pydantic, but once the integration is done, the author expects a potential 10x speed improvement.

[1] https://github.com/samuelcolvin/pydantic-core

iknownothow··on Postgres and Parquet in the data lake
After reading a few comments here, here's the experiment I'm going to run this weekend to couple Postgres (AWS rds) and DuckDB

There's already an FDW for DuckDB [1]! But it's not available from within AWS RDS because they only allow a few extensions. So, I would launch an EC2 instance and launch a postgres instance on it. Then I would install the DuckDB FDW on this postgres instance. Then, on the AWS RDS postgres instance, I would use the postgres_fdw extension to connect to the postgres instance running on EC2 and indirectly gain access to it's DuckDB FDW. I'm not sure if this chained FDW strategy will work out. But here's why I think it's a great idea:

1. Any data that I am willing to put into DuckDB is guaranteed to have backups and/or an ETL already setup to sync data. Therefore I don't need this EC2 setup to be "reliable". If it goes down, I'll just rerun the ETL.

2. I need to pay a small DevOps cost to have some performance gain but the analytics complexity remains unchanged because at the end of the day it's just Postgres. I get to continue to use DBT against the original AWS RDS.

The only thing I need to see is if there are any performance gains to be had using this setup. I won't take that for granted until it's tested.

[1] https://github.com/alitrack/duckdb_fdw

iknownothow··on Upcoming feature in PostgreSQL 15 – MERGE statement
I'm not denying the importance of performance and query planning here, but merging two statements into one is what I as a (specific kind of) data engineer care about more. Letting postgres take care of consistency while keeping the SQL readable is just pure value addition.
iknownothow··on Nearly the hottest place on Earth, India swelters under intense heat wave
Thanks, that was a fun read.

[1] https://en.m.wikipedia.org/wiki/Clathrate_gun_hypothesis

iknownothow··on Nearly the hottest place on Earth, India swelters under intense heat wave
Just a personal anecdote to provide some context of how hot it really is.

I'm from the south of India and now live in Europe. My body tolerates heat quite well compared to my European friends. I remember tolerating 38-40C indoor temperatures and sleeping through it with heavy sweating. But air temperature of 42C is where I draw the line. It is almost biologically impossible to survive this.

So this story isn't just about record high temperatures, it's about record high temperatures that coincide with biological limits. If air temperature reaches 44C, it's impossible to survive this without artificial cooling and I predict a state of emergency will be declared every summer starting 2030.

There's one additional factor. The hottest months in the equator are June to August. This happens to coincide with the monsoon season in India. So I remember dreading May but looking forward to the monsoon in June. If climate change messes with the monsoon season, the Indian subcontinent is f'ed.

iknownothow··on Show HN: Hatch 1.0.0 – Modern, extensible Python project management
How do you keep your builds reproducible? I understand that once it's built, a docker image makes that specific build immutable and you can distribute it without worry. However, if you were to run the build stage again i.e. rebuild the docker image and redownload the python packages, without changing any package versions, the resulting docker image *may* be quite different from the previous docker image, even though you haven't changed a single line in the entire repository. This is because the dependencies of dependencies may change. So while this works for you, it's not a general solution for the community. Tools like poetry have solved this problem to a large extent.
iknownothow··on The fastest tool for querying large JSON files is written in Python (benchmark)
I decided not to introduce DuckDB for my org, 6 team members and <10TB of data solely based on lack of full support for Postgres like JSONB. Currently, we are happy using Postgresql JSONB with a small performance penalty . I would be loathe to introduce a tool which requires a new syntax for handling JSONs.

I know this is a big ask, but if DuckDB can be in lock step with Postgres on JSONB handling :french-chef-kiss:

As far as I am aware (haven't researched), the standard way to get logs or metadata out of AWS Cloudwatch is in a JSON lines [1] format. Therefore, even if I were to write a custom JSON output format for my custom service, I would try my best to stick to the JSON lines format since I would have already written parsers and data connectors for AWS Cloudwatch. That's just me though.

[1] https://jsonlines.org/

iknownothow··on The fastest tool for querying large JSON files is written in Python (benchmark)
The example of processing and querying a 1GB "JSON Lines" [1] file, where each line is a json document 0.1-10KB in size with a varying schema on every line is a very common use case in data engineering. On top of that, there are additional constraints where we might only be allowed to allocate 1vCPU to the task, there's additional IO overhead of downloading the file from S3 and finally, even though there's TBs of the same data we only ever need to process a few GBs per hour or day. How well can simdjson perform under these circumstances [2]? Probably quite well but not as fast as having to serialize a single 1TB json file.

So my metrics of success in this scenario are based on that fact that I have to deal with 10-100 such queries in a project in my day job, so I would choose SpyQL to write and maintain a simple and readable 5 line query in under 5 minutes with decent performance to solve a trivial use case of computing an average.

P.S. I know the article is about performance and your response about Python being is slow is beyond accurate and yet I will always choose to use it because it is not ashamed to sit on the shoulders of the fast and ugly.

[1] https://jsonlines.org/

[2] https://github.com/simdjson/simdjson/blob/master/doc/iterate...

← PreviousPage 5 of 5