Hacker News Reader

>There is no "sanity" lost, nor are the masses "following fashion". That is unfair to those who put real thought, time and effort in their software.

It's also a forthright description. Developers chase fads, the new shiny, just like everyone else does. Neither are we immune to cargo-culting; I'd go so far as to make the absurd comparison to web apps written in shell script to be an instance of this.

I've been behind the curtain on large Rails apps. Every issue you mentioned is either reasonably mitigated or exists more-or-less equally on other frameworks.

Well, Rails wants to be on a relational database. That's a pretty strong constraint that - say - NextJS or Flask avoid.

I think it's true that it sometimes feels a little crazy that form posts are so much tricker in the SPA world. I sometimes wonder if NextJS has got this right: build a semi-old fashioned web app with a frontend and a backend, so all the data transport in between those is pretty light on code, and then use APIs to data services with nicely defined domain objects.

Still yet to put my money where my mouth is on NextJS though.

Most data is relational. It’s important to embrace the efficiencies and safety guarantees that come along with that.

Yes, a lot of data is relational (although probably most of the world's data is not relational). But not all things need to persist data or are data driven.

If all data belongs to a user or an organization, which is usually the case, and that user does some actions to some objects, is that not relational? It's easy to see how most web apps fall into this pattern.

Building your own API to join data is absolute torture.

> Most data is relational.

No. It's not. It may be in your domain. But it really very much depends on the domain and use-case.

I daresay that purely on information density and amount of gigabytes, most data you use in your business is both document-based and hierarchical: files, docs, directories. But obviously I don't know you or your business, so this is just a wild guess.

This misunderstanding that "data is mostly relational", I believe, comes from the fact that it is easy to make most data relational (edit: to clarify: turning something into X is not the same as something being X to begin with). Much easier than to make most data document-based, indexed, hierarchical or into graphs. For one because the tools (RDBMSes) are mature and omnipresent. Practical: it's easy to put word docs and directories in postgres. But it's hard to have millions of CSVs with profile information accessible on directory structure

And, if what you say were true, that "most data is relational", then the best tool for most of our jobs would be a step-up in relationality from an RDBMS and we should be using graph-databases instead.

I'm talking as a whole. So yes, there are cases where it's not relational but on average most data is relational. A user has many projects. Those projects have many tasks. This is very common to most web-based software used today. Think of all the tools out there that work perfectly with this structure.

Most of our data isn't files, docs, directories. It's data given to us by our users or generated by our own processes. I'm not getting the reference to CSVs and word docs.

Yes, we store uploaded files but the metadata is stored in the database in a row that has some identifier that links to the directory where the file is stored. That works fine. We don't need a document database.

You're just only classifying relational data as data. YouTube's petabytea of data is mostly video and audio. Perhaps some of its data should be stored relationally, but definitely not most of it.

Isn't a hierarchy a relation? Item, item's parent, item's children. It suits itself well to a RDBMS.

> exists more-or-less equally on other frameworks.

You are probably just comparing apples-to-apples here, then. Certainly: replacing Rails with Django or Symfony won't solve the core issues one has with Rails, because Djange and Symfony follow the exact same pattern and architecture.

But architectures (please note that I deliberately don't use the term framework here) like CQRS, Event-sourcing, Event-buses, Actors, Reactive etc. all fit much better for many use-cases than MVC.

All architectures have downsides and upsides. The one's I mention above with which I have experience have severe downsides in other areas. Some of those downsides are solved in Rails, some in other architectures. It's always about weighting trade-offs and choosing what fits best for your use-case.

So please don't just label everyone who made such tradeoffs and chose to forego rails because it doesn't fit them as "chasing fads".

If anything, blindly choosing Rails is just as bad as "following fashion": both is making decisions without any basis.

Django has these same limitations in my experience, but to me the issues are largely caused by the common usage patterns which are encouraged by the official docs and community, and lead to absurd amounts of coupling/ball-of-mud apps.

you can be disciplined about clean architecture/DDD/hexagonal architecture, etc, but you have completely ignore the patterns encouraged by Django, Django Rest Framework, etc etc.

Hide the ORM behind Repository interfaces

Marshall ORM objects into true Domain objects

Do not call to the database in the HTTP adapter/validators

Etc etc

> Hide the ORM behind Repository interfaces

> Marshall ORM objects into true Domain objects

> Do not call to the database in the HTTP adapter/validators

The tradeoff for this is exploding the number of files, directories, and tests, needed for the project. Which has its own cost and overhead.

I'm not experienced in Django.

But doing this in Rails is very hard. It goes against the most strongest of opinions this opinionated framework has. So much, that if you strive for such a design (and you should, IMO) Rails is more in the way than helping you - I know, I tried.

Sure, writing boilerplate is always going to have overhead.

But in my experience the stricter-yet-more-verbose design strongly increases maintainability and velocity in the long term, which should be self-evident due to the massive reduction of coupling.

> An extremely weak side of Rails, is how it is very tightly coupled to the database.

You mean that Rails — being an MVC framework — is tightly coupled to (or defined in terms of, even!) the schema you've chosen for your business domain, as understood by your RDBMS.

If that's a problem for you, it's very likely because you're using your RDBMS wrong. (E.g., not taking advantage of abstractions like views+triggers to provide schema [API] stability atop storage [implementation] mutability.)

An RDBMS schema is an encapsulating API for your data, just like a GraphQL schema is an encapsulating API for your data. It's just not in fashion to use it as one — very likely because the spectre of "non-standard SQL" and inter-RDBMS portability has scared all the framework developers away from actually using RDBMSes for what they're for (instead using them as glorified indexed document stores.)

> it's very likely because you're using your RDBMS wrong.

Sorry. But no.

RDBMSes make poor tools for complex domain logic, business-rules, etc. If your solution to "my database and business-logic are too tightly coupled" is to move more of that business-logic into the database, "you're using your RDBMS wrong".

RDBMSes are near impossible to test. They have no source-control, no collaboration models (pull-reviews, CI/CD). They have no deployment strategies. Lack distribution.

Which is why in e.g. Rails all business logic is in the models, commonly and not in your DB schema.

(and with "no", I do understand that it is possible, just that accessible tools, frameworks or even books and course lack severely).

> just that accessible tools, frameworks or even books and course lack severely

Right. Using an RDBMS "correctly" requires that you live in 1985. The result is actually better operationally (for model evolution, at least) than what living in 2022 gets you, though. If your product or service is "about" the data it captures/models in the DB, living in 1985 usually turns out to be worth the sacrifice.

The deeper problem is that because everyone uses RDBMSes wrong, the RDBMS vendors aren't being incentivized to invest into the developer-UX for the features that differentiate RDBMSes from other DBMSes. These features could give you a 2022 experience instead of a 1985 experience, but the vendors just don't care about the 2022 experience.

The exception to the incentives being analytical data warehouse services, like BigQuery and Snowflake. These companies do have these incentives, but they don't have the bandwidth to actually address them; nor do they operate on an open-core model where anyone outside the company could help them with their roadmap. (I say this in frustration; my own company is waiting on one of these vendors to implement support for arbitrary-precision integer columns that aren't actually just uint128s; and "lack of bandwidth to address features" is the common refrain I hear from them.)

And there's also the fact that the major RDBMS vendors are all stuck in a 1985 mindset, where they their primary users are analysts manually typing SQL. I still have yet to see an RDBMS whose pitch is "this database is for machines to talk to; it [literally] optimizes for the case of predictable — or even pre-registered† — queries, at the expense of efficient execution of arbitrary one-off queries." (Because, who runs arbitrarily-complex reporting queries against their service's production DB, rather than shoving those off to some other data-warehouse system? "Production DB for a service" should be its own product category.)

---

† Picture: your applications would register versioned schemas in advance with the DB, including all the static templated queries the application plans to do; and then, in exchange, the application would get persistent a set of persistent cross-connection-sharable query-handles. Rather than allowing the user to define indices, the DBMS uses the knowledge of what queries the app will perform to build whatever indices best optimize its own query plans. Multiple app-versions can be registered + connected simultaneously, as versioned-API "portals" onto different projections of the same data. Etc. You know, the same 2022 stuff you expect to be handling yourself on the business layer — but exposed as part of the first-class meta-schema of your DB.

This overlooks the actual problem. In Rails, you have a db schema, models are tightly coupled to the db schema, then the community push is to put logic into these models.

Having two models pointing at the same table is disregarded as bad idea.

Then, validation for form data is coupled directly to the db schema, thanks to models, not separated by scenario.

Then, the json rendering part is coupled to the model, because it's "convention over configuration" and the community pushes for DRY everywhere, without concern for the cost. So now, the json api is surfacing any change to the underlying STORAGE mechanism of the data, instead of being stable.

Now, add a react + Typescript interface that so popular nowdays on top of it and the result is that an update to the DB schema causes changes in the model, in the controller, in the json view, potentially breaking typings in the frontend.

This is the most common situation in Rails applications.

Now, let's take this situation 2 years in the future:

The json produced by this app is massive, there are many more new endpoints. Each endpoint has json rendering, but DRY is a hard requirement, without evaluating the risk, so every json rendering for a given model is reused in other json rendering objects.

Now one change of the underlying storage ripples through many, many API endpoints, potentially affecting hundreds of files.

I think you misunderstood the thrust of my argument.

A Rails model that talks directly to an RDBMS table, does indeed couple your business logic directly to your data storage representation. I'm not arguing with that.

There are two ways to fix this, though. And most people only consider one of them.

The "software-oriented" way to fix the coupling — and the one that software developers usually reach for — is to embrace domain-driven Hexagonal architecture on the inward edge: to create a domain layer within the business layer that defines its own non-DB-synchronized domain objects; and then to write an explicit DB gateway that can be Commanded and Queried with those domain objects, where that layer translates those domain objects into relational row-tuples.

(Note that this is separate in concept from embracing Hexagonal architecture on the outward edge. Gateway abstractions + internal-domain/external-view model isolation+translation are well-suited concepts to encapsulating the different rates-of-change needs of your system vs. the external systems it interacts with. That's what the Hexagonal-architecture concept was invented for, and why MVC separates Controllers from Views.)

The "database-oriented" way to fix the coupling, is to continue to model your business domain within the DB, but to model it in a faceted, denormalized way, such that each query-schema gets its own view, and each command-schema gets its own writable view driven by a trigger + stored procedure, and the tables where data actually lives (and what columns/types/indices they have) are an implementation detail of how those views were written. Just like in OOP, you hide the data, encapsulating it behind an API (the views.)

When your Rails models bind to these "schemas", each Rails model then inherently represents a Query-result or a Command-changeset — without having to complexify your business layer. These objects can just be your domain objects. You're getting your domain objects directly from the database, and sending them directly into the database, because the database itself contains the translation layer mapping the domain into the storage representation. Just as with the first approach, you're free to change these mappings at any time — you just add some DB schema migrations which add/update/remove the views backing the models, and re-point your models at the new views. (If you create these views using unique-per-version names, then different versions of your app can even live in peace together, with all their versioned bits-and-pieces of translation layer co-existing and being shared between where relevant.)

Oh, yeah I definitely misunderstood your point.

This makes sense and it's an idea that makes me excited, I wish there was a community drive toward that.

I do think that most Rails developer will dismiss it because some reason like "It's weird to have a Model backed by a view" or something along those lines. I wish the Rails community pushed (way) harder on dramatically increasing SQL knowledge, because it's so important to the whole framework.

Instead most of that SQL knowledge is hidden behind active record, making it incredibly hard to acquire.

There is one thing that this approach doesn't solve, and that is the inherent flaw of activerecord: having Save on the business logic entity makes it so that returning such object back from a business-logic method is dangerous, since now any other code could potentially write to the DB (and this is what usually happens due to lack of software design principles).

I suspect you are going to suggest something along the lines of "set readonly some records and command records are only used in specific spots", which could make sense!

> Or when you have more and more complexity moved into async jobs

How does this cause pain in rails specifically? Any 3 tier architecture is going to have the same headaches.

> How does this cause pain in rails specifically

As long as the jobs are self-contained, cohesive and decoupled, there's no problem.

They hardly ever are decoupled. Often, jobs will reach back into the model-layer (rather than receiving all data with which to work). You are now coupling yet another concern tightly to your Active-records. In practice, e.g. you have 400 jobs waiting, which will do a `deliver_to(Report.find(report_id).recipient.email)` What happens if your domain-model Report changes to have `recepients` instead?

They hardly ever are cohesive. It's hard to orchestrate jobs - the tooling is simply not there in Rails-. So having tiny jobs spawning other jobs and/or sending eachother messages is not the common way: instead we build Giant Balls Of Sequential Code that perform a job. e.g. that ReportJob above will now (1) collect data, (2) prepare and filter it (3) push it into a template, (4) build a PDF from it, then (5) upload that to storage and (6) build an email and attach it. That is violating SRP on so many levels it hurts to write it down.

And they are hardly ever self-contained. As mentioned above, many jobs will reach back into the models rather than having all the data in their job. Worse: jobs in Rails lack own state and management thereof. So they'll often push (intermediate) state through your models into your database. e.g. that ReportJob may trigger a statemachine changes with things like @report.transition_to(:pdf_generated).save!. Causing all sorts of issues with transactional boundaries (what if an HTTP request comes in that updates the same Reports' name, while you are building a PDF from it?) and thus race-conditions, locking and other issues.

ActiveJob and the likes are very good as an aside on a CRUDdy app. To handle simple commands like notifications, emails, uploads, etc. But they lack severely when your domain is mostly command-based. In those cases, architectures such as Event-sourcing or Actors are a much better fit.

> And they are hardly ever self-contained. As mentioned above, many jobs will reach back into the models rather than having all the data in their job

This is usually for a very good reason.

If you passed in a user model into your job directly and then your job executed with that user there's a race condition here waiting to happen where the user might have been altered in between when the user clicked the button to issue the job and then job itself executing.

This could have very bad side effects like the user receiving an email that they opt'd out of or maybe the user doesn't even exist in the system anymore or your job saves the user back to the DB with stale data that the user updated since then.

It's super common and a best practice to send a unique ID of the user as a job argument and do the user lookup in the job at execution time of the job. This is typically something you'd do in any system, not just Rails. You mention "actors" which hints at Elixir but Oban (Elixir's most popular job processing library) also uses this approach of passing IDs because this problem isn't related to your tech stack.

Also, you listed out 6 sequential steps in your ReportJob example. Based on what you wrote it sounds like they need to be done sequentially and can't be further parallelized. Based on no other context and it being a typical web app I'd keep that as 1 job and have 6 functions calls in the job that have the implementation details for each step so the job itself is very readable. I wouldn't make a jumbled mess of jobs-calling-jobs spaghetti there.