HNHacker News
TopNewBestAskShowJobs

MrPowers

1,637 karma · joined November 3, 2012

Data blog: https://mungingdata.com/

Personal blog: https://neapowers.com/

Github: https://github.com/mrpowers

From New York, spend most of my time in Colombia & Brazil.

Speak Spanish fluently, learning Portuguese.

submissionscomments
MrPowers··on Pandas 2.0 and the Arrow revolution
Yes, I agree with your assessment that technologies that can deal with larger than memory datasets (e.g. Polars) can be used to filter data so there are less rows for technologies that can only handle datasets a fraction of the data (e.g. pandas).

Another way to solve this problem is using a Lakehouse storage format like Delta Lake so you only read in a fraction of the data to the pandas DataFrame (disclosure: I am on the Delta Lake team). I've blogged about this and think predicate pushdown filtering / Z ORDERING data is more straightforward that adding an entire new tech like Polars / DuckDB to the stack.

If you're using Polars of course, it's probably best to just keep on using it rather than switching to pandas. I suppose there are some instances when you need to switch to pandas (perhaps to access a library), but think it'll be better to just stick with the more efficient tech normally.

MrPowers··on Pandas 2.0 and the Arrow revolution
Dask sidesteps the problem by splitting up the data into smaller pandas DataFrames and processing them individually, rather than all at once. But the fundamental issue of pandas storing strings as objects in DataFrames (which is really inefficient) is also problematic with Dask (because a Dask DataFrame is composed of a lot of smaller pandas DataFrames). The technologies that really address this problem are Polars/PySpark - technologies that are a lot more memory efficient.
MrPowers··on Pandas 2.0 and the Arrow revolution
The Arrow revolution is particularly important for pandas users.

pandas DataFrames are persisted in memory. The rule of thumb was for RAM capacity / dataset size in memory to be 5-10x as of 2017. Let's assume that pandas has made improvements and it's more like 2x now.

That means you can process datasets that take up 8GB of RAM in memory on a 16GB machine. But 8GB of RAM in memory is a lot different than what you'd expect with pandas.

pandas historically persisted string columns as objects, which was wildly inefficient. The new string[pyarrow] column type is around 3.5 times more efficient from what I've seen.

Let's say a pandas user can only process a string dataset that has 2GB of data on disk (8GB in memory) on their 16GB machine for a particular analysis. If their dataset grows to 3GB, then the analysis errors out with an out of memory exception.

Perhaps this user can now start processing string datasets up to 7GB (3.5 times bigger) with this more efficient string column type. This is a big deal for a lot of pandas users.

MrPowers··on How to write Python extensions in Rust with PyO3
PyO3 is being used to expose the Python bindings to the delta-rs project: https://github.com/delta-io/delta-rs

It's a great way to expose Python bindings because it "feels" Pythonic. Most users run pip install deltalake and are completely unaware that the backend is implemented in Rust.

This is quite a different user experience than Python bindings for Java backends exposed via py4j. The py4j interfaces have the Java feel and require Java to be installed, which most Python users don't like.

MrPowers··on Most data work seems fundamentally worthless
I've been working in the data space for 8 years and find it incredibly fulfilling. Open source community is great (I've built and contributed to a lot of projects), LinkedIn has tons of great data content, meetups are fun.

The work is interesting and the problems are important.

It's funny cause the parts the author is mentioning are easy are the hard parts for me. The author says "The pressure is non-existent"... That hasn't been the case for me. Deadlines are tight, jobs fail, running jobs overnight, checking if they're still running before going to bed - it's not great for a restful sleep.

I've always given talked, created new open source libraries, and blogged whenever my day job gets a little boring. It's such a fast growing, exciting field. I feel like it's hard to keep up, let alone get bored!

MrPowers··on Ask HN: How do you test SQL?
Sure, this blog post explains the dependency injection design pattern with Spark: https://mrpowers.medium.com/dependency-injection-with-spark-...

You can structure your code to read from paths when run in the production environment, but inject DataFrames you build in memory for your test suite. Spark is designed to read multiple files in parallel, so it's not optimized to read a single tiny file. That's why it's best to avoid I/O in Spark test suites whenever possible.

MrPowers··on Ask HN: How do you test SQL?
Spark makes it easy to wrap SQL in functions that are easy to test. I am the author of the popular Scala Spark (spark-fast-tests) and PySpark (chispa) testing libraries. Some additional tips to speed up Spark tests (can speed up tests between 70-90%):

* reuse the same Spark session throughout the test suite

* Set shuffle partitions to 2 (instead of default which is 200)

* Use dependency injection to avoid disk I/O in the test suite

* Use fast DataFrame equality when possible. assertSmallDataFrameEquality is 4x faster than assertLargeDataFrameEquality. Some benchmarks here: https://github.com/MrPowers/spark-fast-tests#why-is-this-lib...

* Use column equality to test column functions. Don't compare DataFrames unless you're testing custom DataFrame transformations. See the spark-style-guide for definitions for these terms: https://github.com/MrPowers/spark-style-guide/blob/main/PYSP...

Spark is an underrated tool for testing SQL. Spark makes it really easy to abstract SQL into unit testable chunks. Configuring your tests properly takes some knowledge, but you can make the tests run relatively quickly.

MrPowers··on Apache Hudi vs. Delta Lake vs. Apache Iceberg Lakehouse Feature Comparison
You're understandably confused by the industry terminology that's ambiguous and morphing over time.

Data lakes are typically CSV/JSON/ORC/Avro/Parquet files stored in a storage system (cloud storage like AWS S3 or HDFS). Data lakes are schema on read (the query engine gets the schema when reading the data).

A data warehouse is something like Redshift that bundles storage and compute. You have to buy the storage and compute as a single package. Data warehouses are schema on write. The schema is defined when the table is created.

And yes, I'd say that Spark is generally considered the "standard" distributed data processing engine these days although there are alternatives.

MrPowers··on Apache Hudi vs. Delta Lake vs. Apache Iceberg Lakehouse Feature Comparison
Some high level context for those less familiar with the Lakehouse storage system space. For various reasons, several companies moved from data warehouses to data lakes starting around 7-10 years ago.

Data lakes are better for ML / AI workloads, cheaper, more flexible, and separate compute from storage. With a data warehouse, you need to share compute with other users. With data lakes you can attach an arbitrary number of computational clusters to the data.

Data lakes were limited in many regards. They were easily corrupted (no schema enforcement), required slow file listings when reading data, and didn't support ACID transactions.

I'm on the Delta Lake team and will speak to some of the benefits of Delta Lake compared to data lakes:

* Delta Lake supports ACID transactions, so Delta tables are harder to corrupt. The transaction log makes it easy to time travel, version datasets, and rollback to earlier versions of your data.

* Delta Lake allows for schema enforcement & evolution

* Delta Lake makes it easy to compact small files (big data systems don't like an excessive number of small files)

* Delta Lake lets readers get files and skip files via the transaction log (much faster than a file listing). Z ORDERING the data makes reads even faster.

The Delta Lake protocol is implemented in a Scala library and exposed via PySpark, Scala Spark, and Java Spark bindings. This is the library most people think of when conceptualizing Delta Lake.

There is also a Delta Lake Java Standalone library that's used to build other readers like the Trino & Hive readers.

The Delta Rust project is another implementation of the Delta Lake protocol that is implemented in Rust. This library is accessible via Rust or Python bindings. Polars just added a Delta Lake reader with delta-rs and this library can also be used to easily read Delta Lakes into other DataFrames like pandas or Dask.

Lots of DataFrame users are struggling with data lakes / single data files. They don't have any data skipping capabilities (unless Parquet file footers are read), their datasets are easily corruptible, and they don't have any schema enforcement / schema evolution / data versioning / etc. I expect the data community to accelerate the shift to Lakehouse storage systems as they learn about all of these advantages.

MrPowers··on Apache Hudi vs. Delta Lake vs. Apache Iceberg Lakehouse Feature Comparison
I think vendor published benchmarks are fine if the dataset is open / accessible, the benchmark code is published, all software versions are disclosed, and the exact hardware is specified. I definitely wouldn't consider an audited TPC benchmark that's based on industry standard datasets / queries worthless in the data space. Disclosure: I work for Databricks.
MrPowers··on Ask HN: What's the best technology for a CRUD website to minimize maintenance
Web Framework versions go EOL and then you're forced to upgrade. So constant maintenance is required with Rails, even if you just want to put the app in "security patch only mode".
MrPowers··on The evolution of the data engineer role
> The trend has been to shift as much work possible to the current generation of Data Warehouses, that abstract the programming model that Spark on columnar storage provided with only a SQL interface, reducing the space where you'd use Spark.

I feel like there there are some data professionals that only want to use SQL. Other data professionals only want to use Python. I feel like the trend is to provide users with interfaces that let them be productive. I could be misreading the trend of course.

MrPowers··on The evolution of the data engineer role
I am working on some blogs / videos that will hopefully help clarify the differences. I'm working on a Delta Lake vs Parquet blog post right now and gave a 5 Reasons Parquet files are better than CSV talk last year: https://youtu.be/9LYYOdIwQXg

Most of the content that I've seen in this area is really high-level. I'm trying to write posts that are a bit more concrete with some code snippets / high level benchmarks, etc. Hopefully this will help.

MrPowers··on The evolution of the data engineer role
> A predilection for SQL, yielding "get the answer right once" big-ball-of-SQL solutions which are infeasible to debug or modify without causing regressions.

Yea, thankfully some frameworks have Python / Scala APIs that let you abstract "SQL logic" into programatic functions that can be chained and reused independently to avoid the big-ball-of-SQL problem. The best ones also allow for SQL because that's the best way to express some logic.

> Poor support for unit testing.

I've written pandas, PySpark, Scala Spark, and Dask testing libs. Not sure which framework you're referring to.

> Poor support for version control.

The execution platform should make it easy to package up code in JAR files / Python wheel files and attach them to your cluster, so you can version control the business logic. If not, yea, that's a huge limitation.

> Frameworks over libraries (because the vendors want to lock you in)

Not sure what you're referring to, but interested in learning more.

MrPowers··on The evolution of the data engineer role
Data warehouse: bundles compute & storage and comes at a comparatively high price point. Great option for certain workflows. Not as great for scaling & non-SQL workflows.

Data lake: typically refers to Parquet files / CSV files in some storage system (cloud or HDFS). Data lakes are better for non-SQL workflows compared to data warehouses, but have a number of disadvantages.

Lakehouse storage formats: Based on OSS files and solve a number of data lake limitations. Options are Delta Lake, Iceberg, and Hudi. Lakehouse storage formats offer a ton of advantages and basically no downsides compared to Parquet tables for example.

Lakehouse architecture: An architectural paradigm to store data in a way such that's it's easily accessible for SQL-based and non-SQL-based workflows, see the paper: https://www.cidrdb.org/cidr2021/papers/cidr2021_paper17.pdf

There are a variety of tradeoffs to be weighed when selecting the optimal solution for your particular needs.

MrPowers··on The evolution of the data engineer role
Great article.

> data engineers have been increasingly adopting software engineering best practices

I think the data engineering field is starting to adopt some software engineering best practices, but it's still really early days. I am the author of popular Spark testing libraries (spark-fast-tests, chispa) and they definitely have a large userbase, but could also grow a lot.

> The way organizations structure data teams has changed over the years. Now we see a shift towards decentralized data teams, self-serve data platforms, and ways to store data beyond the data warehouse – such as the data lake, data lakehouse, or the previously mentioned data mesh – to better serve the needs of each data consumer.

I think the Lakehouse architecture is the real future of data engineering, see the paper: https://www.cidrdb.org/cidr2021/papers/cidr2021_paper17.pdf

Disclosure: I am on the Delta Lake team, but joined because I believe in the Lakehouse architecture vision.

It will take a long time for folks to understand all the differences between data lakes, Lakehouses, data warehouses, etc. Over time, I think mass adoption of the Lakehouse architecture is inevitable (benefits of open file formats, no lock in, separating compute from storage, cost management, scalability, etc.).

MrPowers··on The single market is looking wobbly
There is an excellent Real Life Lore video on "How Russia is Crippling Germany with Gas" that has quite a different take: https://www.youtube.com/watch?v=Xi74gPrn24Y&ab_channel=RealL...

Peter Zeihan also has quite a different perspective.

> those in Europe’s biggest economy will be able to behave as if not much is going on

It seems like the shut down of the Nord Stream will have a big impact on German manufacturing. I'm not sure how they'll be able to "behave as it not much is going on".

MrPowers··on Ask HN: What open source projects have great documentation?
I personally like the Polars docs because they have a detailed user guide that makes it so easy to get started with the project: https://pola-rs.github.io/polars-book/user-guide/
MrPowers··on Ask HN: How to level up your technical writing?
Blogging & measuring average time on page is a great way to see if you're able to write content that's relevant and engaging for readers. Average time on page should be at least 5 minutes for a blog post.

Improving open source project READMEs and documentation is another great way to practice writing.

I am writing an O'Reilly book now and having a professional editor will help you learn the common errors you're making.

You should try to write short paragraphs, short sentences, and at a 4th-6th grade reading level. Good writing for literature is a lot different that good technical writing.

A good novelist may write at a 12th grade reading level, may use complicated words, and will use literary devices like allegory and foreshadowing.

A good technical writer should explain a concept in the most simple way possible. They should explicitly avoid literary devices like foreshadowing - their goal is to explain the concept in a straightforward manner. They should also avoid big words and long sentences. A large portion of technical readers are not native English speakers, so only the most basic words should be used.

MrPowers··on Ask HN: Is the stock market's growth largely anything more than inflation?
The biggest factor influencing long term stock returns in real growth in earnings.

Companies don't pay out all profits to shareholders as dividends. They can also reinvest in their own business (e.g. build a new factory) or buy back shares.

The go-forward nominal rate of return on stocks should be higher than the inflation rate + the dividend yield. The nominal rate of return could be closer to shareholder yield + inflation (shareholder yield is the dividend yield + share buybacks). I think the real earnings growth is the best predictor of long term returns. Earnings growth is correlated with population growth & productivity growth, but those aren't the only important variables.

MrPowers··on Ask HN: Is the stock market's growth largely anything more than inflation?
Yes, stock market growth is largely driven by the real growth in earnings and dividends.

Nominal figures are not adjusted for inflation. "Real" numbers are adjusted for inflation (in economics-speak).

The stock market (e.g. S&P 500 index) has real earnings that have consistently grown over time (although earnings are quite volatile). The real dividends paid by the companies that make up the stock market have also grown over time.

Jeremy Siegel, a finance professor at Wharton, wrote a great book called Stocks for the Long Run that shows stocks have grown about 7% per year (inflation adjusted) over the last 200 years.

> What would the market look like if we corrected for the money supply?

I think it's better to correct for inflation. The money supply can grow and it doesn't necessarily cause inflation (see the 2008 monetary response to the Great Financial Crisis as an example).

> It seems like the stock markets (USA) growth is strongly correlated to inflation

I'm not sure this is true. In the 70s, inflation was high and stock returns are low. In the 90s, inflation was low and returns were high. In 2021, inflation was high and returns were high.

MrPowers··on The end of big data
Yea, agreed, Spark definitely isn't a database, it's a query engine.

I agree that the traditional Spark notebook query interface was a lot easier for people with pandas (or DataFrame experience) to understand. But their new SQL query interface is easily understandable for DBAs.

MrPowers··on The end of big data
The author is suggesting that Databricks should sell a product with infinitely scalable storage and is queryable via SQL or Python... isn't that exactly what they're doing?

Databricks has a product called Delta Lake that covers the infinitely scalable storage part. Here's a talk from a Delta Lake user that's writing hundreds of terabyes / up to a petabyte of data a day to Delta Lake: https://youtu.be/vqMuECdmXG0

Databricks recently rewrote the Spark query engine in C++ (called Photon) and provides a new SQL query interface for data warehouse type developers.

Databricks recently set the TPC-DS benchmark record, see this blog post: https://databricks.com/blog/2021/11/02/databricks-sets-offic...

This article doesn't align with my view of reality.

MrPowers··on Ask HN: Have we screwed ourselves as software engineers?
The new data tools I've seen are complex under the hood, but offer elegant user experiences, giving the best of both worlds.

You referenced a 500 line Python script being refactored with Rust and make me think of the Polars project: https://github.com/pola-rs/polars

Polars uses Rust to make DataFrame operations lightning fast. But you don't need to use Rust to use Polars. Just use the Polars Python API and you have an elegant way to scale on a single machine and perform analyses way faster.

I'm working on Dask and our end goal is the same. We want to provide users with syntax they're familiar with to scale their analyses locally & to clusters in the cloud. We also want to provide flexibility so users can provide highly custom analyses. Highly custom analyses are complex by nature, so these aren't "easy codebases" by any means, but Dask Futures / Dask Delayed makes the distributed cluster multiprocessing part a lot easier.

Anyways, I've just seen the data industry moving towards better & better tools. Delta Lake abstracting all the complications of maintaining all the complications of plain vanilla Parquet lakes is another example of the amazing tooling. Now the analyses and models... those seem to be getting more complicated.

MrPowers··on Why isn’t there a decent file format for tabular data?
HDF5 has some limitations that make it suboptimal for cloud based storage systems.

Zarr overcomes these limitations for array data and Parquet overcomes these limitations for tabular data.

MrPowers··on Why isn’t there a decent file format for tabular data?
Parquet is a wonderful file format and is a dream to work with compared to CSV. Parquet embeds the schema in the footer metadata, so the query engines don't need to guess what the column names / data types are.

Parquet used to be poorly supported, but now it's well supported by almost all languages. You can even view Parquet files in text editors now, but that's not something I've ever needed (https://blog.jetbrains.com/blog/2020/02/25/update-on-big-dat...).

Parquet column pruning & predicate pushdown filtering allow for great query performance improvements, see this blog post I wrote for benchmarks: https://coiled.io/blog/parquet-file-column-pruning-predicate...

Delta Lake (Parquet files + a transaction log) makes it quite pleasant to manage a large lake of data stored in Parquet. There are tons of other features Delta allows for like time travel, schema enforcement, schema evolution, etc.

MrPowers··on I need to stop being boring
> I lost my creativity and the sense of wonder I used to have in childhood. I am no longer curious because I am settled for the world as it is. I accept things the way they are instead of imagining how they could be.

The best way to expand your perspective of the world is move to a new country with a different culture, learn a new language, and make local friends.

I recently moved to Brasil and am learning Portuguese. The language barrier is difficult. Learning how to communicate indirectly is hard. Understanding the different social norms requires you to question some of your core beliefs. I don't think you can learn any of the core lessons without learning the language.

Travel will put you out of your comfort zone, but you should also start having a lot of fun... the type of fun you used to have with friends when you were younger. Not the buttoned up type of fun you have at a dinner party with acquaintances.

MrPowers··on Black, the uncompromising Python code formatter, is stable
I like Black for normal Python code, but it seems to mangle Pandas / Dask code at times. I still use it extensively cause it doesn't seem like there are other good alternatives.

I wrote a blog post on how to use Black in Jupyter Lab notebooks if anyone is interested: https://coiled.io/blog/code-formatting-jupyter-notebooks-wit...

It's really nice to format a notebook with the click of a button.

MrPowers··on Kinesis Advantage 360
Kinesis Advantage 2 is great, but it's massive and hard to travel with. Looks like the 360 might be a bit easier to travel with.

The Kinesis Advantage 2 fits a Magic Trackpad perfectly right in the middle of the keyboard. This lets you perform mouse actions with you right and left hands.

Emacs keybindings never made sense to me till I tried them with the Kinesis. I feel like Emacs sucks on a standard Mac keyboard and is awesome with a Kinesis. See here for a blog post on the topic if you're interested in learning more: https://mungingdata.com/emacs/learning-emacs-keybindings-aft...

It takes a while to get used to the keyboard layout, but only takes a week to get fully productive. You're eventually able to train your brain to operate fluently on Kinesis & Macbook key layouts without any extra thinking. It's a great investment if you're willing to put in a bit of effort.

MrPowers··on Bulgaria's new eGov minister is a software engineer
Look into technocracy if you're interested in learning more about appointing government officials based on their technical knowledge: https://en.wikipedia.org/wiki/Technocracy

Anyone know of any studies of famous techocrats and the overall view on the effectiveness of appointing ppl that aren't career politicians?

← PreviousPage 2 of 10Next →