Comment by nautilus12 | Hacker News Reader

Comment by nautilus12 | Hacker News Reader

Parent Full thread

Both indicators (along with python) that this is an article written by a yet another data scientist that thinks engineering is unecessary. Data science is a field that is plagued by individuals having very specific narrow experiences that are often easy to engineer making them think they are experts in data engineering. Vertical scalability has a hard bound, if your algorithm changes complexity it could become literally impossible to move forward. Horizontal scalability gives you options, this is why we like it, not just because we like to overengineer systems, but because we don't like embracing hard limitations on the systems we engineer, especially when we often don't know yet what problem we are solving. Engineers often don't get the liberty of solving extremely specific instances of problems like data scientists do.

> Data science is a field that is plagued by individuals having very specific narrow experiences that are often easy to engineer making them think they are experts in data engineering.

That's on point, some of the worst and most inefficient code I had to maintain was written by data scientists. They are for sure leagues above any of my data science knowledge in their respective field but the average engineering knowledge and best practices seems pretty low.

Most of the time it's not that data scientists think engineering is "unnecessary," it's that they're expected to deliver analyses and models without the help of engineers, because the engineers are on another team, behind a massive backlog. So they make do with the tools they have.

This is totally fair, why we need to bring the two disciplines closer together rather than have one try to obsolete the other.

Um, yes. From the article:

"The simplest and most common way to implement indexing is by naming files in a directory:"

    mydata/
        2019-Jan.csv
        2019-Feb.csv
        2019-Mar.csv
        2019-Apr.csv

Um.

This is time partitioning, not indexing, but it's commonly done in data lakes for raw telemetry/log data.

    mydata/
        2019/
            11/
               12/
                   2019111200.csv
                   2019111201.csv

Where each one of those CSV files could be 10-100 GB in size.

Usually you want to process it into a columnar format like Parquet from there, though

Reply on news.ycombinator.com