Comment by taeric | Hacker News Reader

Comment by taeric | Hacker News Reader

Parent Full thread

As someone that is completely non visual, I personally don't buy it. Trying to keep things visual will almost certainly choke on all of the extra information that comes with being visual.

I think it is easy to lob complaints at sql. The problem is every successor seems worse. Teams I've been close to lately have taken the python pill. Pandas, they push, is so much easier to understand. This works too an extent. But what we have dodged large sql files, we now have giant python files. Worse, they are trying to engineer things in "logical" ways for code sharing.

What I'm seeing is they moved all of the complexity of most ETL engines into each python file. This feels great at the start, but is like how every Todo list project looks clean in basically any language. Ramp up the variety and size of data sources, and get ready for that clean idea to turn into the mess it thought it prevented.

For my two cents, if your ETL setup doesn't factor in Conway's law in who owns different parts of the data, it will fail in time. Trying to hide split ownership of data is a sure sign that changes will get misunderstood and data will be lost or misused.

> all of the extra information that comes with being visual

I think the key is to not make everything visual. You want to be able to view your system as you would draw it on a whiteboard. I believe it could be done using a restricted syntax of a proglang.

> Teams I've been close to lately have taken the python pill...

Most people have a desired result and want it as fast as possible.

When imperatively using map, groupBy, filter, you can usually get the immediate result you want very quickly. This is the appeal of Pandas, Python notebooks, etc.

But when someone asks: I want to re-compute on change, which pretty much applies to every task...their quick solution is not optimal.

And then if the schema changes, and you're doing all this intermediary stuff...then refactoring is next to impossible.

Any SQL query could also be written in an imperative prog lang and its probably easier to understand and tweak and also trace. An SQL relation does not maintain any info about its source, which then makes incremental updates difficult, and also refactoring.

You can basically ask the question: for every piece of data you see in your company, can you easily see the "data lineage" from its source and all the transformations it was involved in, and how clean and efficient is this pipeline when receiving new data. Then a step further is, how easy is it to change something.

The problem with, "as you would draw it on a whiteboard," is that much of what makes whiteboard drawings useful is his much you omit.

Such that I agree being able to visualize things is nice. But don't confuse the visualization with the system. A step you have to make in many visual tools.

I'm pretty much agreed on most of what you wrote, though.

> But don't confuse the visualization with the system.

I think this is where people have got it wrong in the past.

This tool is an example of something in the right direction: https://www.ilograph.com/

It's just completely disconnected from code.

Agreed that I think that is where folks go wrong.

And the very valid counter claim from folks that do some of this is that there needs to be some mechanism to keep things synchronized between the visualization and the execution.

Reply on news.ycombinator.com