Practical Railway-Oriented Pipelines in Ruby | Hacker News Reader

Practical Railway-Oriented Pipelines in Ruby

ismaelcelis.com

67 pointsismaelct66 comments

I might be a bit biased (I wrote and maintain a similar gem), but I think this is a great pattern that solves a real problem that is encountered and does so in a very Ruby kinda way. I particularly like the mental model of stringing together multiple service objects into a "pipeline" and the semantics chosen for the API. Kudos to OP for putting this out there!

Thanks. Where can I find your library? I'd love to compare notes!

I use a functional version of this that can chain any "callables" + support extensions on the calling interface https://docs.rubykit.org/kit-organizer/edge/Kit.Organizer.Se...

Nice. Yes the pattern I describe in the article supports any callable too. I should point out that this is not a specific library, just a very bare-bones pattern.

This pattern is effectively how Vagrant (for anyone who remembers that) always worked, also in Ruby! I even gave a talk on it at MountainWest RubyConf back somewhere around 2013, although I compared it moreso to a "middleware" pattern. Even the API/DSL is almost identical.

The middleware pattern had a lot of the same concepts present in this post: we called context "state" and you could use special exceptions to halt or pause a middleware chain in the middle.

This was a really great way for over a decade (to this day!) to represent a long-running series of steps that individually may branch, fail, accumulate values, etc. I don't recall the exact count, but an old `vagrant up` used to execute something like 40 "actions" (as we called each step).

I'm not trying to disregard this blog post in any way, I'm only pointing out this pattern is indeed very useful and has been proven in production environments for a very long time!

Thanks Mitchell (big fan, btw!).

Indeed!https://github.com/hashicorp/vagrant/blob/main/lib/vagrant/a...

Yes, it's not a new pattern by any means, and there's many ways to "halt" the pipeline as you say. For example ActiveRecord stops callback chains if any callback throws ":halt".

Other examples are Redis.rb's pipelining API https://github.com/redis/redis-rb?tab=readme-ov-file#pipelin...

Or more generally any builder-style pattern that composes a set of operations for later execution, including again ActiveRecord's query chaining.

In my article I tried to show a specific implementation using the Railway pattern (where the result must only respond to "#continue?() => Boolean")

I wrote something like this in the past, it doesn't make sense in ruby.

The closest ruby way is:

    result = DoStuff
    return something if result.nil? # error, early exit

    result = OtherStuff(result)
    return something if result.nil? # error, early exit

No additional complexity. Otherwise usually it ends up just rewriting "programming".

With pattern matching there are also some alternatives now.

What do you think of Ruby's built-in function composition? https://ruby-doc.org/3.2.2/Proc.html#method-i-3E-3E

Interesting, never used it. Usually I have methods but not proc, by the time I have to write `.method(something)` a bunch of times I'd it doesn't make much sense anymore.

I love curried functions by default, but if it's not the default, it never works out even in functional languages like Elixir. You need to have every developer (and new hired developer) pretty much on board with that and exclude every library that doesn't do this, or wrap it (huge burden).

Given those, I'd rather stick to what I wrote. Not my favorite, I loved my brief experience with Elm

I understand the sentiment but I think it's highly dependent on context. Where you work at, who you work with, org size, what the problem is, the cost and benefits of each abstraction, etc etc. I think it's our job as developers to put all those things on the scales when deciding what abstractions to use or not use. Whether a pattern is "the default" or familiar is certainly a big factor, but not the only one.

I'm still bitter about having to work on a team with some Clojurists who thought Haskell and monads were bad and stupid, but then forced ROP on the rest of us in a Clojure project.

Odd to see ROP and monads as an either/or problem (see what I did there?). Most ROP implementations I've seen rely on the result monad.

Exactly my point.

I just wanted to make the either/or pun.

I appreciate you.

> see what I did there?

Maybe

Very nice

the code in the blog post has an error that you fixed in the gist:

   def step(callable = nil, &block)

you're missing the nil default in the blog post so it ArgumentError's when passed only a block

Well spotted. Fixed. Thank you!

I’m glad people find value in patterns like these, but I’m so sick of them. Plain Ruby code is so much easier to debug, which is all that matters in the long run. The senior engineer who introduced the thing like this inevitably leaves and nobody cares to learn some bespoke abstraction enough to keep using it.

I’m sure this is great and solves real problems for OP and friends. I just ask that you think twice about it before forcing it on coworkers. Please, this poor soul can’t take it anymore.

Thanks for your feedback. Out of curiosity, when you say "plain Ruby code" what do you mean, exactly? Presumably you're still making use of _some_ patterns that you think are Ok.

If I had to guess, “plain Ruby” is ruby that fails with a sensible stack trace without internal things showing up. That’s always been my trouble with complex abstractions, it can make it very difficult to use with a debugger especially, which is how I always work with Ruby. New flow control models like this are the main culprits to weird debugger issues.

I haven’t gotten a chance to mess around with this specifically, how much of its internals are mixed into execution?

Good observation about stack traces and abstractions. Re. your question, the pattern itself is no different than, say, Rack middleware, so you'd see similar cost and benefits. In essence you're running one callable object after the other.

A pipeline is essentially this

steps = [ ->(r) { r }, ->(r) { r }, ->(r) { r }, ]

Wrap initial data in a common Value object

initial = Result.new(some_data)

Run the Result through the steps, in order

result = steps.reduce(initial) { |r, step| step.call(r) }

That's the pattern, really. A reduce operation.

Re. stack trace, it can add noise because you're iterating over steps instead of calling them procedurally one by one, and you may want to decorate steps (put steps inside steps) for encapsulation, caching, etc. but again no different than Rack.

yes, it is similar to rack. But what rack has this requirement of being pluggable from multiple decoupled codebases (rails or other framework, your app, some gems you're adding to your app). And that also warranted things like "insert_before" etc.

Even with rack, decoupling of code definition (where you've added your middleware) and code execution could be pretty hard to debug, but at least with enough exposure you know where to look, and you usually have only one rack stack in your app.

If all those steps in your pipeline are defined in the same codebase, rack approach becomes much less useful.

> If all those steps in your pipeline are defined in the same codebase

I find that rule a bit arbitrary. For example, I have data-import pipelines where some steps are unique to each task, but some others are shared across tasks. Why does it matter whether the steps are defined in the same codebase or not?

Pretty much this.

I want traces to go to actual code as much as possible.

I also want my IDE to work.

One of the most annoying things in my life right now is aasm. It’s a good idea, but I hate having to know this new dsl, I hate the dynamic methods it defines, I hate having to constantly explain to the confused folks that this random bang method is actually triggering this `event :event_name` over here. I would just so much rather have none of the DSL and meta-programming stuff, even if it’s twice as much “boilerplate”, if that means people who understand normal code can actually trace what the program is doing using the tools they already know how to use.

Certainly don’t mind abstraction like a standardized Result class, it’s more the chaining and custom halt with error patterns that I can tell are going to complicate debugging.

What do you think of the Rack interface?

Or ActiveRecord's query chaining?

ex. User.where(admin: true).joins(:account).order(id: :asc)

It's all versions of the same approach.

And yes all of the above _can_ complicate debugging, for sure. But so can most abstractions. Only use them if the specific problem they solve is bigger than the drawbacks.

To me, Rack and AR are scoped to a specific domain. HTTP request handling and database-backed models are naturally going to involve some internal noise since I don’t want to write a web server or a database interface layer. I want the things I don’t want to write to be called.

Here, this is closer to general purpose flow control. There isn’t really a library I wouldn’t want to write myself here. You show a lot of examples of things that could be accomplished by objects and messaging passing (“plain Ruby”) which might be preferable for some.

All in all though, it’s a really interesting idea! I still really like the Ruby community’s love of bending ruby to their will. (Even if I do think it can cause issues sometimes)

I tried to keep this as idiomatic to Ruby as possible. No meta-programming, no FP machinery like monads, etc. Little more to it than an array of objects and a reduce function. All plain Ruby in my book.

I wonder if just using "throw :halt" (like in AR callbacks, to halt callback chains) or an Enumerator with "raise StopIteration" (which would work) would throw people off a bit less, just on account of those things being more familiar.

Eh, don’t let myself or others influence your library. You will know what’s best, and hacker news will mostly elicit lots of nitpicky reasons as to why you are wrong.

To be clear this is not a library. I'm just describing a pattern and a bare-bones implementation.

I think some abstractions are fine and warranted, even ones you yourself come up with. Rails is a framework consisting of tons of opinionated abstractions. We need mental models to grasp complicated, large codebases.

My point is more to think twice about adding your own, and after thinking twice about it, think about it ten more times. Most people using Rails will already have an understanding or mastery of the abstractions you mentioned, but that understanding guarantee drops off quick with third-party gems, and is a guaranteed no for custom in-house solutions.

Rack and query chaining are great for two different reasons.

Rack because it's simple.

Query chaining in things like AR is fine because it is mostly debuggable up to a reasonable point with things like to_sql, etc. Since the actual query only runs in the end, it is decoupled enough, and the "escape hatch" of Raw SQL is not really a leaky abstraction, is just the of the abstraction itself.

The problem with things like Trailblazer is that it intertwines itself too much with the business logic, so using breakpoints is very difficult. And its internals are also super complicated, not to mention people build even more complex abstractions on top of it (since it's a bit lacking). It works fine for toy examples but anything multi-layer deep ends up making it too difficult to debug and make sense of.

It's always worth reading Against Railway-Oriented Programming[1] before you commit to using it.

Personally I've dealt with enough Trailblazer code now that for business logic I think I'd much rather work with a simple ruby class with a procedural interface.

I like the use of a Result object but not at the expense of exception handling or bloated stacktraces full of library code.

Being lightweight, easy to debug and upgrade is underrated IMHO.

Having said all that, I still think this is a really interesting article and great food for thought depending the usecase.

1. https://fsharpforfunandprofit.com/posts/against-railway-orie...

Very good article, thank you.

I would add that the tagline says "when used thoughtlessly".

For example, I agree that using Result objects to reinvent exceptions is a bad idea. There's a reason they're called "exceptions". Error results should only be used for domain-level errors. ie. things that _your expect_ to go wrong in the domain.

Also note that in my article I use continue/halt, not Ok/Error. At the library level it's just a way to compose functions together with a mechanism to halt processing. Whether something halted execution because it was an error, or any other reason (ex. caching), is up to your app's semantics.

> The senior engineer who introduced the thing like this ...

I actually consider the urge to introduce overengineered patterns like this a sure sign of a "mid-level developer". Senior knows better.

I would have thought that "senior" means assessing things in context instead of falling back to truisms. "over engineering" only makes sense relative to a concrete problem you want to solve.

yes, of course. I usually am all about context. It's just this particular example is over-engineered in most context I've seen (and can imagine) in my 18 years of using ruby. Not much overengineered, I've seen much, much worse, but it is still annoying enough.

I’d say mid-levels will follow along, but newer seniors will confidently throw ideas out for all to observe how gloriously smart and perfect they are.

Amazing.

You both are probably right. I see a lot of difference in how different companies or different dev cultures see seniors.

A lot of what people used to call senior has become "staff" in larger companies.

True true, my org doesn’t even have a staff level, it goes from senior to principal. Not meant as a rebuttal, meant as a reinforcement of role names meaning less.

> Senior knows better.

Sometimes, the cost of keeping really talented senior engineers who have solved a very difficult part of your tech stack is to give them the freedom to over engineer solutions that are probably a bit overcomplicated (maybe like OP). The alternative is they get bored and leave and take a wealth of technical knowledge with them

yes, of course. As I said, it's an urge. I wrote quite a few examples of such stuff myself, ruby is amazing for inventing new control structures and trying things. But then I think about my peers and future self who'll have to change and debug my code later, and spend additional time making stuff simpler. This is actually harder to do.

Programming is about managing complexity. Making stack trace twice as long, hiding control flow in data, most cases of metaprogramming do not reduce complexity, they increase it. Unless your app is all about usage of some specific pattern (a lot of procedural scenarios with sideffects, shortcircuiting on error and no return value) then "railway framework" is not worth it in ruby. Use exceptions (really!). Use throw, maybe.

I'd like to think this is kind of just the cost of discovering that 1 pattern out of many thousands that is actually useful. I am going to guess the Gang of Four didn't just sit down and bang out the entire catalog of design patterns in one extremely productive programming session. They likely battle tested hundreds of different patterns, most of them being thrown out as not useful or a bad abstraction before arriving at the set they published in the book.

I wrote a whole blog post on this because of the amount of harm caused by these types of systems. Interactors, interactions, pipelines or whatever other name, can very easily ruin a codebase in my experience. Already happened multiple times in my career.

Developers don't like to think about abstractions and naming is a known hard problem. People pick this up and now everything is easy. Every name becomes: DoThis, DoThat, DoThisOtherThing and only answers to `call`. No need to think!

Now you have deeply nested procedures that oversimplify their interfaces to match the library and suddenly you no longer have meaningful error types, you have strings. You no longer have abstractions, you only have procedural code with some objects in the middle. The list goes on.

Like most things in development, used in the right places and with careful consideration, this can be very valuable. But this is a very insidious change that turns your code into a functional style with several drawbacks that are rarely considered.

I agree with the drawbacks listed here. I would add that "careful consideration" of any pattern we use is the job description.

I would further add that we should extend that thoughtfulness to the opinionated frameworks many of us rely on. They usually come with hundreds of complicated patterns baked in, and we "oversimplify our interfaces" to match the framework as a matter of course. All I'm saying is that committing to any mental model provided by a design pattern or framework has similar drawbacks. In this case, the mental model is "a big operation is a list of smaller operations in a chain". Use with caution.

> No need to think!

Not sure I agree with fully. If anything, I've found that I need to think harder about what constitutes a step in a workflow, what are the names of each stage, what concept they encapsulate. I can't just chuck everything into a god object or a deeply nested hierarchy tree somewhere.

But again I agree. Using this pattern I've definitely over-complicated or gone down the wrong path at times. I would say though that I found it easier to roll back and change direction when compared deeply nested object graphs, for example.

> Developers don't like to think about abstractions and naming is a known hard problem. People pick this up and now everything is easy. Every name becomes: DoThis, DoThat, DoThisOtherThing and only answers to `call`. No need to think!

This 'framework' doesn't require that though:

    def double_number(r) = r.continue(r.value * 2)
    def add_one(r) = r.continue(r.value + 1)
    def square_number(r) = r.continue(r.value ** 2)

    pipeline = Pipeline.new do |pl|
      pl.step method(:double_number)
      pl.step method(:add_one)
      pl.step method(:square_number)
    end

Just methods, with no need to implement call() because that is what a method/proc/lambda in ruby implements. In other languages like C# you could have an API that can take a Func<Result, Result> so that you can just pass delgates/lambdas/methods to it. And adding a bit more DSL could make the construction of the pipeline even less verbose.

You can use this without going full-Java and creating a dozen (or a hundred) 5-line classes in different files.

Have you ever noticed that everyone who drives faster than you is a maniac, and everyone who drives slower than you is a moron? I think code abstraction fits a similar pattern.

Sortaaaa, it’s definitely harder to read existing code than it is to write new code, which is what you’re getting at. To bend your analogy a little further, we can all identify the person on the road who was clearly wrong, putting us all in danger. It’s the difference between someone slowing down traffic by camping in the fast late and someone else barreling down a residential street 50 mph over the limit. The former is annoying, the latter struck and killed someone.

Not that code is this serious at all. Idk.

I'm so going to steal this

It's stolen from George Carlin.

Attribution would be appropriate!

I rather doubt that George is concerned with attribution at this point, but I freely admit that I was probably misquoting somebody.

Attribution is not only for living people.

You got the quote close enough for a plagiarism complaint. :)

https://www.youtube.com/watch?v=XWPCE2tTLZQ

I wonder if I'm witnessing the birth of yet another Hacker News thought terminating cliche.

There is a place and time for everything. You can drive as fast as you can in a Formula 1 race, but when you're around pedestrians you gotta slow down.

Similarly, complex abstractions can often be fast to write but difficult to maintain. Similar to a lack of them.

But sure, it all has a time and place. Except the attitude of dismissing or never discussing this topic. That's the only thing that is "wrong" here.

I think we're saying the same thing. My intention was to point out the subjective nature of the original comment, not to kill the discussion.

The previous comment seemed more "thought terminating" to me than the Carlin reference response to it.

Are we reading the same comment?

The gist of it was: "This specific thing makes my job difficult, please be mindful of people like me" is just personal experience.

The comment even contains the word "please" in it. And they even acknowledge that it solves other problems.

But it's a bit shitty to dismiss someone's personal experience and plea by mocking with exaggerated quotes from a comedian, just because you happen to disagree with them.

> The senior engineer who introduced the thing like this inevitably leaves and nobody cares to learn some bespoke abstraction enough to keep using it.

They seem to be saying that this inevitably happens and to never use the abstraction. They may have inserted a "please" in there to try to make it superficially appear softer, but they aren't actually being very nuanced at all.

If they drive faster than me, then godspeed. Let them safely pass me and then swiftly disappear into the horizon, hopefully not before I manage to learn a couple badass driver tricks from them.

If they drive slower then yeah they're morons and I need to pass them as soon as humanly possible.

In both cases they become non-factors and I can drive in peace. The fast disappear into the horizon and the slow get left behind. This plays out until I am alone in the road and no longer need to consider the actions of other humans behind the wheel. That is when peace is achieved. Sadly it does not last long.

> hopefully not before I manage to learn a couple badass driver tricks from them.

I mean they're probably tailgating and aggressively changing lanes and don't have any awareness of the dangers of differential velocity between lanes of traffic. I don't think you really need to learn those 'tricks'.

You can still freely let them go and they can give any bored cops ahead of you something to do though.

We use the `interactor` gem at $DAY_JOB and it does 90% of the stuff described here, and the last 10% came naturally and we just kinda engineered it ourselves intuitively.

definitely works better than "fat models" that do everything and ruby on rails callback hell

> If you prefer inheritance over composition

That's not what this boils down to.

These are procedures. OOP is generally about objects sending messages to each other. So this solution is all about executing procedural calls, which has nothing to do with inheritance/composition.

You can very easily have OO code with composition without having to rely on everything having one single interface "call/run/execute".

Perhaps I phrased this badly. I don't think the entire article boils down to inheritance vs composition. But in discussing these patterns elsewhere, some of the pushback has been that many Ruby devs prefer to decompose problems via sub-classing instead of composition of command objects, so I tried to cater to that objection with that line.

Pipeline steps are command objects (a pretty standard OO pattern), so they have a single entry point / public method. But they can still fully leverage any other OO pattern in their implementation. The more complex ones I use may instantiate other objects, pass messages between them, etc.

But the single-method #call API is what makes composition easy. See Rack, or any number of middleware-style designs, for other common uses of this in Ruby.

Yeah, my main gripe is with the phrasing. It gives the impression that if you like composition over inheritance (which people mostly take as true without even thinking about it), you should use this.

I feel like that's the kind of over-simplification that then makes people pick a style like this without fully considering the implications.

Every single place that I've seen use these "command pattern" systems devolved into a complete utter mess of procedures. People forget the basics of OO and write everything into "steps" because that's now the hammer and everything is a nail.

If they stuck to the outer layers as the orchestrator for everything, that'd be great. But that never happens in my experience.

Good observations, thank you. I take the point about properly delimiting the boundaries of the orchestration layer in a system.

I think you're right about how that line comes across, I'll try and improve it.

Reply on news.ycombinator.com