Hacker News Reader

Comment by yakshaving_jgt | Hacker News Reader

You don’t need to understand the internals of a thing to use the thing.

This has not been my experience of using technology effectively. Without an understanding of the implementation details, you inevitably use something inefficiently or for not quite the right purpose.

I cannot imagine anyone using a database effectively on any significant amount of data without understanding indexes, how different joins work, why join order is important, what effect join orders have on performance, etc. Get to a certain scale and it's not enough to know about indexes; you need to understand the structure of b-trees, disk I/O performance, how CPU cache performance affects b-tree navigation even when index is cached in memory, how to use compound indexes effectively to reduce random access through the index, etc.

The constraints of CPU and memory never go away, and if you're trying to scale something, you're going to be limited on either or both of those resources. That in turn forces you to understand execution and memory behaviour of the abstractions you're working with. All abstractions leak when pushed.

> I cannot imagine anyone using a database effectively on any significant amount of data without understanding…

You'd be surprised just what proportion of systems running in the market operate on amounts of data you would not deem "significant".

And as I said in another comment, Haskell doesn't try to pretend that computations run with no hardware constraints.

> All abstractions leak when pushed

Yeah. But we might have wildly different ideas for where that boundary is.

I wouldn't be surprised because I seek employment in areas where my skills are valuable.

> I cannot imagine anyone using a database effectively on any significant amount of data without understanding indexes, how different joins work, why join order is important, what effect join orders have on performance, etc.

But conversely you probably didn't have to understand what filesystem the database runs on, whether it is in a RAID array, whether the network connection was over Ethernet or T1, etc.

All abstractions leak. The question is how leaky they are. In my experience Haskell abstractions are much less leaky than most.

I did and do; when performance at the database level isn't right, you need to look into OS stats, I/O stats, disk stats (the SAN is an area ripe for unexpected contention, RAID5/6 act like a single spindle and inhibit random access in ways that RAID1 or RAID10 don't, stripe length in RAID5/6 bloats the size of small writes, etc.), but I had to stop somewhere :)

> but I had to stop somewhere

Why? You seem unsatisfied with any of the previous layers of abstraction, so where does it end? When can I truly call myself a user of a database?

When you don't have to fix it when it stops servicing requests, or worry about scaling workloads.

When you treat it as a black-to-grey box, not a grey-to-white box.

So I'm confused - how many years did you have to study before you felt confident enough in your grasp of all the underlying concepts to write your first "Hello World"?

Hello World doesn't push much to the limits.

I’m not much of a haskeller, but friends’ war stories indicate that laziness leaks quite a bit.

Aha, yes! You are right. But then so does strictness. See https://augustss.blogspot.com/2011/05/more-points-for-lazy-e... for some examples.

Laziness is a definitely a double-edged sword, with sharp edges.

Thunk leaks happen, but they aren't that scary. They all pretty much have one of a handful of root causes.

It's definitely a wart of laziness, but it's also pretty easy to avoid.

As far as abtraction goes, laziness doesn't leak. In fact, it's a big reason Haskell performance composes. The head . sort example is contrived, but it holds true in more complicated & useful examples as well!

I am in the Database-as-a-Service world.

I have to understand all the intricacies of the system, so people who choose to treat databases as black boxes don't have to. There is only so far this abstraction holds.

Stateless (immutable) code scales horizontally. ACID[1] doesn't.

[1] https://en.wikipedia.org/wiki/ACID

Fun aside, coworker recently had a fun issue where the database server would halt due to running out of disk space... except the disk had over 100 GB left.

Turns out NTFS has a maximum number of file fragments a file can have, and if it exceeds that it will refuse to grow the file (for obvious reasons).

Hardly everyday stuff though.

This interesting, I work as a data engineer but never had to really understand how joins or indexes work because I work 99% of the time with denormalized data. I also do not know much about b-trees. I think you come from the database developer point of you rather than the database user point of view.

There is also other aspect of this, even if I do not understand joins or b-trees I can measure performance so I can figure out which combintion of joins are the faster. The reason that I prefer performance testing over getting to know the theorethical background for certain things is because in many cases your performance also depend on the implementation details, not only on which tree data structure backs your implementation.

It's exactly because the performance depends on the implementation details that when you understand implementation details, it can guide what you test and help find an optimization peak faster - or come up with a theory of another peak more distant, and journey through the performance valley to find it.

Implementation details follow the contours of the fundamental algorithms and data structures. You understand the outline of the implementation details - the bones, if you will - from that. And you can dive into the source (or disassembler - I've done that on Windows) to fine tune knowledge of other specifics, with e.g. gdb stack traces on the database engine when it is abnormally slow to focus attention.

Without having gone to battle together over some performance problems, or me writing significantly longer battle-story posts, we're probably not going to be able to communicate effectively here.

Do you struggle to use a computer due to not having an expertise in semiconductor physics?

<rant> I work with a team that has a 16 TB database, who doesn't understand how it works. They seem really surprised when their queries run slow. </rant>

This is exactly right. I use lenses all the time, but I have absolutely no idea how they're actually implemented, nor do I need to know.

This is abstraction. If there's one thing Haskell does well it's abstraction.

EDIT: It's really bizarre. We see these same responses to all the Haskell-or-Idris-or-whatever threads -- I wonder if there's some imposter syndrome going where "I can't immediately read/write Haskell" somehow morphs into "Haskell is useless". IME it's really rare for people who actually program in Haskell to have serious issues with the language. Yes there are issues from a smaller ecosystem, package management was bad (Stack fixed that), etc. etc. but there are very few fundamental problems with the language. Something so small as just Pattern Matching is a huge increase in productivity. Thankfully, quite a few languages have adopted pattern matching these days (Scala, Rust, TS, maybe even C++23?).

(The really big payoff comes from granular effects, but I'm sure the rest of the world will realize in about 20-30 years' time. The Erlang people already have, albeit in a different way.)

People look at weird syntax and discussions about things that have no resemblance to the problem they are facing, and conclude it must not be useful for anything real. Yes, those problems have no resemblance to any real problem because of abstraction, but most people's experience with abstraction is in a Java-like language where nothing good ever gets out of it.

You don’t need to understand the internals of a thing, until you do. Everything works fine as described in documentation until it doesn't for your use case. You might be lucky and find help from stackoverflow, otherwise you need someone who really grok it.

Sure and at that point you need to learn those internals. That's just the way it is with everything, no?

When things go wrong, you might not have time to do so. When my project started to leak memory at enormous rate, I was able to find the issue quickly enough. But if I didn't know how all those things work, I would spend weeks or months learning all those things. Restarting application every 10 minutes for a week is not a good idea.

How is this different for Haskell than with anything else? If you want to be an expert in something, anything you do need to put in the work and learn it inside-out. There is no royal road. I fail to see how this is specific to Haskell...

It boils down to the steepness of the learning curves.

If I am in the business of system reliability, I will choose the language with shallower rabbit holes.

Abstraction layers are great for builders and terrible for fixers. I am both, so I need to strike a balance.

I agree with you, I don't think that it's different for Haskell.

Haskell's internals are actually pretty easy to inspect!

- Haddock hyperlinked source makes it easy to understand libraries if you need to dig into internals

- ghci makes it easy to interactively learn the language - and a new codebase!

- Haskell can dump the stages of its codegen pipeline (Core, STG, Cmm)

- Profiling and Event logging exist and are easy to use

- You can even write an inspection test to assert certain behavior holds. For instance, that no allocations occur or that some abstraction (e.g. Generic) is truly erased by GHC's optimizations

There was this piece of common knowledge floating around a number of years ago about how you need to know at least 1 level of abstraction beneath you well, and have a working knowledge of the second one below it, to use your tools effectively. I don't recall where the advice floated around or came from but it was something along those lines, and it's pretty true.

I’m not sure I agree with that.

How does this work in the context of CSS? Do people making websites need to understand how WebKit paints the screen?

The word “effectively” seems rather arbitrary here too.

I feel like being able to find an exception doesn't mean the rule is invalid?

How many exceptions should I find to invalidate the rule?

Enough to show it's at least on the same order of magnitude as the number of situations where the rule does hold.

I actually think you do need to understand rendering logic to some extent to use CSS effectively.

For example I have seen many having a hard time understanding why it is trivially easy to align an element to the top of the screen but tricky to align something to the bottom of the screen - something which would be symmetric and equally simple in a typical application GUI framework.

But understanding how layouts are generated makes this clear.

IMO the conceptual level below CSS is in the design sphere - the box model itself. You would not believe the number of people who hack together properties until they get something that looks right, without understanding the box model, who think they're really good frontend developers.

Were you thinking of Joel Spolsky's Law of Leaky Abstractions[0]?

[0] https://www.joelonsoftware.com/2002/11/11/the-law-of-leaky-a...

I don't think so, though it's close.

Only if you have infinite memory. In theory there's no difference between folding left and folding right (if associative), in practice there is a right way and a wrong way.

> In theory there's no difference between folding left and folding right

There's no difference in practice either for a sufficiently small dataset.

> in practice there is a right way and a wrong way

Sure, but that's true of all technologies.

Yes, Haskell can't help you escape the limitations of our world — or indeed our hardware — but it doesn't pretend to either.

> There's no difference in practice either for a sufficiently small dataset.

As a non-Haskell user, just for reference what's "sufficiently small"?

It depends on your needs. This is neither constrained to Haskell specifically nor functional programming more generally.

If you were building a website for your local Italian restaurant, what would your needs be? Do you need an ElasticSearch cluster to handle customer menu item queries? Do you need a database at all?

In Haskell's case it's best to avoid lists entirely, as they're _usually_ not the optimal data structure. But best for whom? Does the beginner care that a set operation would be more effective than a list operation for their given use-case?

Not sure how that's an answer to my question?

In my day job, I frequently generate records returned from a database along with local changes to be posted later, and say compute the sum of one of the columns. That sounded like something I'd use folding for, with my limited knowledge, so I was just curious at which point (order of magnitude) I'd have to worry about doing it this way or that.

But if lists are not to be used, what should I use for the above? And will the data structure you propose be fine with either fold?

Again, it depends. A left fold might be fine (and likely will be). Lists might be fine (and likely are). I don't know anything about the size of your data, or about the business you're writing software to support (who knows, maybe you're writing a FPS game, or some real-time aviation software!).

At this point (taking into account the extent of your knowledge as you yourself described it), my advice would be to just use basic folding and lists. At some point in the future, if you observe some performance issue in exactly that area, you might remember this and think "hmm, memory consumption is bit excessive here; maybe I'll try a foldr instead", or "finding the intersection of these two big lists is a little slow; maybe I'll convert them both to sets instead?"

>There's no difference in practice either for a sufficiently small dataset.

So what happens in practice with unfortunately large datasets?

>Yes, Haskell can't help you escape the limitations of our world — or indeed our hardware — but it doesn't pretend to either

Then what is Haskell's value-proposition when it comes to solving real-world problems?

> So what happens in practice with unfortunately large datasets?

You take a different approach. Haskell provides plenty of options.

> Then what is Haskell's value-proposition when it comes to solving real-world problems?

There are many. You can consult your favourite search engine to learn more.

>You take a different approach. Haskell provides plenty of options.

So do other languages. Why is Haskell special in this regard?

>There are many. You can consult your favourite search engine to learn more.

None that seem to address the problem of diminishing returns.

> So do other languages. Why is Haskell special in this regard?

I never suggested other programming languages don’t also have value. Again, if you want to educate yourself further on a specific technology’s benefits, I invite you to make use of a search engine instead of sea-lioning on a technical forum.

> None that seem to address the problem of diminishing returns.

That’s your opinion. Nobody is forcing you to like Haskell. You are free to just ignore it.

Reply on news.ycombinator.com