Beginner's Guide to Abstraction
jesseduffield.com
jesseduffield.com
The first example in the article is a nice demonstration of this happening well. You are transitioning from a model where geometry is unknown to one where at least simple geometry is known. It may not be the most compelling model change ever, but it at least captures the key point - there is a semantic operation going on: calculate the volume of a sphere. The actual formula for doing so isn't important at the level you want to think about your code, so replacing the formula with a function call simplifies the model in which you're working.
Contrast that with the example of things going wrong in the next part, with the bad "average" function. What unnecessary details are being removed there? Being sure you're calling that function correctly actually takes more work than calculating an average without it. That's not an abstraction, it's an indirection. You still need to track it down and read the code to understand it. That's not something you have to do with sphere_volume in the preceding part.
So how do you know whether duplicated code represents an opportunity for abstraction? You start thinking in terms of the semantics you want to be using. Is the code duplicated because it's doing the same thing, semantically, in your target model? Well then that's an opportunity for abstraction. Or is it just duplicated because it happens to share an implementation? Well then that's just a coincidence. Don't try to share code.
Not to suggest that this is an easy test, of course. It's very possible that two things might be different instances of a common problem that you're unaware of, and you have no idea that they share code because they actually are the same. That's ok, and there's always more to learn. But I think if people put more thought into what abstraction is and why it exists, the questions of when and how to use it fade away.
I disagree with your claim that the 'bad' `average` method is an example of indirection rather than abstraction. Indirection, afaik, is about decoupling two things via an interface so that one isn't directly dependent on the implementation of the other. In this case you are still directly calling the method which will directly execute the code. Maybe the dispute here is around whether we should call these things 'the wrong abstraction' or 'a failed attempt at abstraction' but I think it's a spectrum where you can have a good abstraction but with some parts that don't really belong, and in that case it's still an abstraction.
I'd tend to think that the "chunks of code all performing the same task" aren't the thing you abstract from, rather they are just a poor abstraction of the same domain.
So I don't think the term 'abstraction' should only be reserved for use in reference to the domain because abstraction is already happening at every level anyway.
It's coming from the tradition of Sir Francis Bacon's Empiricism. Some people believe that abstract ideas are mostly acquired thorough empirical observation a.k.a. carefully watching a series of repetitive events. The point here is that "to know" is a synonym of "generalization" for them. So OP naturally puts emphasis on duplicated code as a good opportunity for abstraction.
People from other camps have a very different view on human knowledge, and think that abstract ideas can indeed come from other sources, most importantly, from our reasoning function. For these people, the process of programming is to implement our innate idea into concrete code. For them, an actual piece of code is basically a "shadow" of our abstract concept.
Hell, someone really should write a book titled "The Logic of Programmatic Discovery" ;)
And unfortunately, a lot of people learn the quick reminder "Don't Repeat Yourself" without ever hearing the actually definition of the principle: "Every piece of knowledge must have a single, unambiguous, authoritative representation within a system", which doesn't roll off the tongue as easily. The DRY principle is a bit easy to corrupt and mis-apply, if you don't ever learn what it really means.
Abstraction in my mind is fundamentally about the "higher-orderness" of a thing. These two average methods are just as abstract as the other, since one is not a higher-order operation than the other. I would use the word over-abstract if one was to write a program modeling a dog-walking business (a very specific thing), by writing a system which models actions on entities (a very abstract thing), where walking is an action which may involve one or more entities, and dogs, humans, employees, and customers are all entities. If the core thing you want to do is just a single concretion of the system that you actually built, then you "over-abstracted". I feel like we should not discourage the practice of abstraction, since that's our business. I literally get paid to think about the real world in terms of abstraction and write it up into a computer. Young engineers should not be taught to fear "over-abstraction".
Agreed. This confusion is something Zed Shaw wrote about in blog post called Indirection Is Not Abstraction [0]. Surprisingly it seems it was never discussed properly on HN [1], but it was discussed elsewhere [2][3].
(There's also another blog post by this name by another blogger, by independent reinvention/coincidence [4].)
> Young engineers should not be taught to fear "over-abstraction".
Disagree. As you just showed, unnecessary abstraction is bad. Ineffective abstractions are also bad. It's not easy to get right.
[0] https://web.archive.org/web/20160304022133/http://zedshaw.co...
[1] https://news.ycombinator.com/from?site=zedshaw.com&next=8820...
[2] https://www.reddit.com/r/programming/comments/38hobm/zed_sha...
[3] https://lobste.rs/s/ja5ihv/indirection_is_not_abstraction
I like to think of a program like a tree (main is root, each function is a node, calls are edges). Each sub tree should be a bounded context, in that (ideally) you only have to think about parameters defined in that sub-tree. Leaves implement the “nitty-gritty” (mainly I/O, number-crunching, and complex transformations), and are heavily tested. Each node that isn’t a leaf is either an abstraction composing leaves, or an abstraction composing abstractions. Unit tests for leaves test the nitty gritty, unit tests for non-leaves must only test composition.
I find that human-readable modules have some limits (number of children, height of the tree). You can violate those limits sometimes, but only if you provide some assistance in the form of comments.
Sometimes, a 100-line function is not composing many distinct nitty-gritty ideas. It just really takes 100 lines to express “write this model to the database”.
A 'helper function' might be tightly bound to some other function, i.e. the helper function is sensitive to the internal workings of the function it serves, and is not intended to be called from anywhere else.
And there's still no excuse for source files that are 5000 lines long, of course.
HN discussion of John Carmack's thoughts on how long functions are sometimes preferable: https://news.ycombinator.com/item?id=8374345
I agree with most of your comment, but in my experience (and, I think many may echo the same sentiment) over-abstracting has been always more dangerous than the opposite.
I’d choose a messy, duplicated piece of code over an overly abstracted 10x developer made mess, every single time.
> I literally get paid to think about the real world in terms of abstraction and write it up into a computer.
I love software engineering, but the most satisfying moments in my work involve removing code or not relying on tech to solve problems whatsoever.
With your dog walking example, I'd say each of your listed abstractions would be 'the right abstraction' because you're not bundling up dissimilar things into a single representation as if they were similar. Specifically with dogs/humans, my example about circles/squares is the same: both might conform to the same interface, but you wouldn't want to represent them with the same class. I agree that doing so is perhaps a special kind of mistake for which there may be a better term than 'over-abstraction', though it's not obvious to me what that term would be (over-specified doesn't quite sound right to me).
In this case I'm saying that we're over-abstracting by pulling too much of the dissimilar code between the examples into the one representation (i.e. the method).
But then, they leave. Or they change projects or they just lose their enthusiasm for this major refactor. And you're left with a half baked abstraction in the code. Then another developer comes along, and another, and another and before two long you've got a spaghetti mess of incomprehensible "abstractions".
I was told by a friend once that the iOS app for Facebook has a whole bunch of implementations encapsulating what a "form" is in the app. Many developers came and went with their own ideas of what that abstraction should look like, but none became the one abstraction to rule them all.
I usually found 3-4 ways of styling in a medium codebase on github. I find varying degrees of state management - some built with context and hooks, some imported and traditional mobx/redux. I haven't found many significant opportunities to reuse code without refactoring a lot or doing micro files of 7 lines of code and separating logical components.
I now think of react as a library to build your own frameworks. It aligns perfectly with the javascript ecosystem mentality.
Never try it. A bazillion ways to do basic web service & web things and every google search returns different methods.
Nightmare.
So the only option seems to be to commit to the legacy (in many ways measurably bad) implementation, or start upgrading/rewriting it a little bit at a time.
Of course, the latter is good. Except it leads to the situation you describe with multiple ongoing half-finished abstractions.
There are better ways to do it: not do all the plumbing first, and then the actual business logic, and instead do thin vertical slices at a time. But it still feels... wrong.
Why does this happen? I think a partial explanation is in this Nietzsche essay [1], where he says, "every concept arises from the equation of unequal things" — in other words, the abstractions of the world were built bottom up in our heads, and the Platonic essence of things is just a fairy tale we tell young programmers so they can sleep easily at night.
1. http://nietzsche.holtof.com/Nietzsche_various/on_truth_and_l...
It is typically much more possible to cleanly abstract the behavior for a single interaction/ role, than for the entirety of an entity's state. If you have multiple interactions you might want to use more than one interface.
Once you have clean behavioral APIs for your interactions, you may also be able to use composition, wrapping and delegation to implement/ enhance behavior. This is the 'Strategy pattern'.
The one thing you lose here is object identity -- you can no longer assume a delegate is the entity itself. This is no big loss given the clarity & flexibility that are gained.
Creating the "right" abstraction is _not_ the process of bundling up repeated code and only questioning how "abstract" it should be, it is the process of creating an interface that is familiar and conceptually easy to grasp. This is done through naming, comments, the use of metaphors, etc. From this viewpoint there can be many correct levels of abstraction, some more useful than others. We should aim to create good abstractions at all layers of code, even if finding abstraction bliss is unattainable.
So we make "virtual machines" by creating new "instructions" that extend the physical machine. (Dijkstra called them virtual machines; these days we might say APIs or something.)
We keep doing this: extending level n-2 into level n-1, with the guiding principle that n-1 should be a "virtual machine" that makes it slightly more convenient to express level n. At some level k, the solution to our original problem is trivial.
Other important properties of these abstraction levels is that
- They should do resource allocation and management to the point where a raw resource used by level n should not be visible as such in level n+1.
- Any level should ideally only depend on one level below it.
- No level can ever depend on a level above it -- this creates cycles in the dependency tree and prevents further extension and contraction of functionality.
----
Drawing further from Parnas instead: abstractions should hide design decisions that are likely to change. Typical examples of those are the format of data, layout of data structures, implementation details.
Things that are less likely to change are things that come from the problem domain. Design interfaces based on problem domain concepts, not solution domain concepts. This applies at every level: the abstractions of level n-1 should have interfaces in terms of problem domain concepts of level n.
> The purpose of abstraction is not to be vague, but to create a new semantic level in which one can be absolutely precise.
There is also developing abstractions for south-bound APIs where it goes the opposite way. I declare that I want to be able to do operations X, Y, and Z but I don’t care how it happens. Then an implementer will be the one who knows all of the details specific to their implementation even though I’m the one who “abstracted” by defining the abstract API signatures.
> it is the process of creating an interface that is familiar and conceptually easy to grasp
Sometimes it’s not though. Often times the “familiar” and “easy to grasp” is the API burdened by legacy systems tied too much to one implementation. In this case the process of abstraction is cleanly separating the intent from the legacy crap and defining a new interface that captures the intent and nothing more (not leaking the details of the “how”).
An API that writes bytes to an arbitrary location on a disk is certainly familiar and easy to grasp. But without a filesystem abstraction on top of it, it’s not really useful if you want portability, multiple processing using a disk, etc.
In reality, though, a combination tends to be used, and development is an ongoing process of "I have this set of operations and I would like to construct a component that meets this specification -- how do I join those realities?"
I agree strongly with your second point, though.
When developing a driver framework, you are the consumer and you’re just declaring what you want a provider to do. You don’t know anything about how it will be implemented nor do you care.
They are completely different actors in the system designing the API with completely different goals. It’s not just top-down vs bottom-up unless you’re both the consumer and the implementor, which is just a trivial subset of API work.
IMO fits it better than implementing interfaces.
That is to say, the first step is to find the representation of something which reduces overall complexity, and the next step is to think about naming. In my experience, one of the signs that you've found the right abstraction is that it's easy to find a name for it, because the intrinsic complexity of most things is low (or at least lower than I thought when I started programming), and so it's not hard to find metaphors when your abstraction brings the extrinsic complexity of a system down towards the level of intrinsic complexity of its requirements.
> DON'T BE AFRAID TO DISMANTLE THE WRONG ABSTRACTION
Couldn't agree more with the statement, though I don't completely agree with the author's suggestion to copy paste. Duplicating code _is debt_. It may help us go faster now, but it'll almost inevitably come back to bite. It is manageable if 1/2 people do it 1/2 times - definitely not manageable if 5/6 people do it 5/6 times.
I believe the general hesitation of not touching a piece of code (or, getting by with that optional param) is due to the fear of fucking things up. Having your code test covered gives an amazing amount of confidence to rip apart old abstractions to yield newer ones that serve the purpose of the _current code_. To me, this route is more preferable to duplicating code.
Even with the best of intentions, Hacking an abstraction with that one optional parameter is inevitable. Tests help in our ability to repay that debt faster - on time & in full.
Basically they make all abstractions a lot cheaper - easier to write and easier to throw away. Thereby solving the problem of having a 'wrong abstraction' too early.
This is actually not the main reason you'd want to abstract, and I think the whole article kind of gets it wrong. The main reason to abstract is not to keep code DRY, but to "abstract away" things that are not important in a certain context (or layer). You want to put the formula for the volume of a sphere aside when it's not relevant to what the code at hand is doing and would get in the way of trying to change or understand that code. For example, a really strong case for abstracting that formula is if you're writing code that is calculating the volume of several shapes.
Yes, code duplication is often a sign that you've screwed up your abstractions (ie they correlate), and creating the right abstraction will often make your code more DRY, but it's the means, not the end. The end is code that is easy to read, understand, and change.
My high-level "sniff test" is essentially rubber-ducking (try to explain to yourself, someone else, or a duck) what the code is doing. For instance, you might explain a function called calculate_remaining_space_in_box:
1) we get the volume of all shapes (including spheres) in the box
2) we get the volume of the entire box
3) we calculate the difference between them
In that explanation, you realize there's really no extra benefit to a reader of that code at that level to knowing the exact formula for the volume of a sphere (or any shape for that matter).
There are, of course, other signals to measure whether you've abstracted correctly beyond just code duplication. For instance, and the Single Responsibility Principle is a good example (https://blog.cleancoder.com/uncle-bob/2014/05/08/SingleRepon...). Code that changes for the same reason should usually be grouped together, and code that changes for different reasons (or at different frequencies) should be grouped apart. But again, this is in service of the end goal: making code easy to read and change.
"You want one layer to handle user input and get it into a normalized form. You want one layer to do your actual task. And you want one layer to handle formatting and output to the user. Those are the three layers you always need."
The "do the task" layer can be abstracted again further. But starting it off as a monolithic layer, separated from input and output, is always the right call.
You invariably have to hit hour head sometimes, but I hope clearly written articles with understandable but not completely contrived examples like this one reduce the head knocks for some people
A concrete and easy example is when you're developing a new software and building your common utils/libs. If you've nailed the abstractions, you'll notice that you're able to expand the libs by frequently reusing/leveraging other libs you've built.
Abstractions come to exist from the requirements. IMO the key to a good abstraction is to be able to dissect a requirement into smaller requirements that you are familiar with and have already solved and have a solid understanding of them.
Metaphorically speaking, bad abstractions are ones that converts a requirement into a new polygon shape, and good abstractions are ones that dissect a requirement into one or more shapes that we are all familiar with (circle, square, rectangle, rhombus etc..).
The difference between a polygon and common shapes is that no 2 polygons will look alike unless the requirement is exactly the same and I'd argue that a developer will create a new polygon if asked to solve the requirement twice.
When a new requirement comes in, it's common to start implementing it right away. Create one or more classes with names that maps to the requirement, add few methods etc and voila you have a new polygon shape.
The key point here is to dissect the requirement into sub-requirements that look like familiar shapes (problems you've previously solved). Every once in a while you'll end up creating a polygon here and there for a sub-requirement, which get refactored over time to a known shape.
A good developer can quickly see the familiar shapes that the requirement is hiding behind instead of creating a new polygon shape.
it's easy to write the average function in the article in this way.
def average(arg) when Enum.all(arg, &is_string/1) do
# implementation
end
def average(arg) when Enum.all(arg, &is_integer/1) # for the int
or using pattern matching in the arguments.
def shape("circle", ...) do
# implementation
end
def shape("square", ...) do
# different implementation
end
def is_good(%{ hn: HN }) do
IO.puts "#{HN.someprop} is good"
end
def is_good(%{reddit: Reddit}) do
IO.puts "#{Reddit.someprop} is bad"
end
def multiply_on_two_numbers_otherwise_square(a), do: a * a
def multiply_on_two_numbers_otherwise_square(a, b), do: a * b
It abstracts component development for frontend hiding details about framework.
I applied DRY principle on code that we write. Framework syntaxes are a repetitive hardcode that I tried to abstract.
Pragmatic programmer is the book referred to me by a friend of mine and it definitely works.!
Parametricism is slowly taking over other industries like architecture and industrial design. In essence it's automatic rewrites and programs finding the right abtractions/compositions/organizations based on given parameters, where the designer job is more in the actual problem domain, providing the right paramters to the program and selecting the most promising outcomes of that automated process.
The web moving to site generators and serverless is maybe a glimpse of a future with dynamic site generators, where the generators then get much smarter and responsive to input parameters and surrounding contexts.
https://computersciencewiki.org/index.php/File:Abstract_hear...
https://en.wikipedia.org/wiki/Abstract_data_type
https://www.oreilly.com/library/view/object-oriented-design-...
One of the principles I follow is "the best code goes unwritten" so a good rule of thumb for a good abstraction, for me, is if it reduces (noticeably) the total amount of code. Conversely, if abstracting just increases the total code, I call it an indirection and avoid it.
Meanwhile the cost for getting an abstraction wrong is often far worse. And it’s really easy to get wrong because abstractions by their nature are always built based on yesterday’s data plus good intentions. People are arguing about subjective theories and unmeasured concepts of extensibility, wasting time arguing about Liskov Substitution Principle, SOLID, dependency injection, type system design patterns, etc., but it’s mostly junk that just adds code bloat.
Obviously there are other solutions that don’t require premature abstraction or copy/paste 1000 times (like simple module functions as the core unit of reusability, or a macro system for code injection). But the point is copy/paste gets a bad wrap. It’s simple, straightforward, easy to automate, easy to test, and adds no extra concepts to the code.
“Functional core, imperative shell” is the best advice I’ve found. Ruthlessly avoid object orientation, and when you need it, stick to extremely shallow inheritance. Make everything a module function, and when you write data structures, don’t give them member function logic for their core functions (like search, sort, add items, remove items), rather create functions that accept data structure instances as arguments and perform these operations with no class-like internal state.
I would say this in a slightly different way myself. "Mathematical core, business shell". Code in the mathematical core is defined by it's behaviour. "sort", sorts a list. Functionality in the business shell solves business problems. For instance "add_customer".
In a year's time, sort will do the exact same thing. But "add_customer" may update a different database from what it used to, and in a different way. Code in the mathematical core will never become incorrect because the world has changed. But it can become irrelevant. Sorting may no longer be needed by the application.
heap_sort(my_array)
my_array.heap_sort()
The first interface above just needs the simple top level name “heap_sort” patched or mocked with easy control of the scope. The act of array construction and the knowledge of internal array data is completely isolated away from the act of sorting.
As you start adding parameters to the mix and involve highly stateful internal operations, start mixing with inherited methods, class methods and static methods, and as you chain the fluent interface (eg foo().bar().baz()) all these effects get worse and test complexity becomes unberable.
Basically it boils down to function composition.
f(g(x))
x.g().f()
You don’t need pure functional programming or compiler enforcement to do this either. It is how I’ve structured large C and Python programs in business settings for many years.
https://blog.cleancoder.com/uncle-bob/2019/06/16/ObjectsAndD...
As an example I might want to test code that internally calls heap_sort but replace the heap_sort with a test double (sounds like a silly example but maybe the heap_sort calls out to a heap sorting microservice or something equally dumb.)
With the procedure call and lexical scope, that's not going to happen. With the instance method, that might happen if I pass in an array with a different heap sorting implementation.
----
To be clear: I'm all for functional composition over mutation through interface methods. But it's not, primarily, about syntactic convenience. It's about immutability and referential transparency.
The example you picked is bad for another reason too: heap_sort is not a fundamental method to arrays that should be part of its core definition. So no, clearly it shouldn't be an instance method, but for different reasons!
> and as you chain the fluent interface (eg foo().bar().baz())
Aside: A lone method call isn't a fluent interface (though I suppose it could be part of one), so the naming here is a bit misleading. A fluent interface is a particular type of method chaining, so the second example may or may not be one depending on what foo/bar/baz actually do.
(Take blah then do a then do b then do c then do d then return the result which is if same type as blah).
The functional approach basically moves “blah” all the way to the right.
In terms of mocking builder methods are just as simple (since they return the class rather than modify in place).
(Concretely I am think of something like transformations on dataset, especially if you do different transformations/in different order at different places in code)
The constructors that live in builder classes can just be separate module functions, attached to no class, and use decorators or other non-object-oriented types of metaprogramming to add specializations.
It’s very nice to encapsulate complex creation in a helper function, but builder pattern just takes this idea and adds unnecessary code bloat.
It is so much easier to just get the functionality down first, properly test it and once it is all there, good abstractions within the implementation might just surface from looking at the (duplicated) code.
You can do “test once, reuse everywhere” without any inheritance at all. You could use metaprogramming, sealed types with pattern matching, or macros/code-gen. It’s just tradeoffs to decide between them.
The point isn’t that creative abstraction never wins that tradeoff, rather just that it wins very infrequently compared to how often it is preferred based on parochial “software design” hand wavy reasons.