90% of software engineering done today is integrating poorly documented APIs

Being intimately familiar with HTTP is fine, reading a lot of RFCs is fine, but there are plenty of well respected, well compensated, influential engineers who do neither, and have no need for either one.

Maybe it's because I live outside the United States, but it seems like it's much harder to build a "pathway" into work that demands that intensive, deep understanding of engineering on computers. At least one that'll pan out in my 9-5.

The supply of problems that require deep knowledge vs framework Kung Fu... my impression from ground level is they're tilting more towards GitHub martial arts every year.

The supply of companies outside the United States with a deep interest in these topics is sparse, to say the least.

You jump from Rails project to Rails project and they all work the same way. You know where everything is immediately. It's fantastic for teams and long term maintenance between different devs because the learning curve of each project is virtually non-existent.

But Rails has gotten bigger and more complicated over the years, which makes it tougher to make the case.

That's why I feel bad for the new generation of developers. When I started developing in 2000s it was enough to know low level abstractions, some design patterns, etc. Along the way I've seen the context in which for eg. MVC was revolutionary, and all the latest frameworks actually solved a lot of real problems (SQL query in PHP garbage all over the place on backend, jQuery/MooTools/ExtJS/Coffescript/AngularJS, supporting IE6, etc.)

A lot of modern tools make sense to me because I know the context they came out of. I feel we've done a shit job of preserving this context and lesions that lead to current frameworks - this will eventually lead people to rediscover all the mistakes that were hit in past, but also saddle us with legacy burden because it's hard to distinguish what's relevant and what's a result of historic constraints.

I for one believe that the best solutions I've produced over the years were things that side-stepped the popular or established frameworks and were more customized and focused solutions. They gave the business an undeniable edge over the cookie cutter competition, and by being well versed in the full stack, my team was able to implement new features at a fraction of the cost.

Anyone who knows the underlying technology can pick up our work in not much more time than learning any off-the-shelf framework simply because our solution has fewer moving parts and the relationships between the business logic and the bedrock are direct and intuitive.

That is only "technically" true, in the way that if you develop any kind of sufficiently complex program then you will have, by necessity, built something which could be called a framework.

Realistically, though, your point is invalid and misses the mark.

To say that programmers shouldn't write pure code without an existing framework is akin to saying that a house builder shouldn't use bricks and lumber to build a house; they should buy a pre-fab kit instead. Which is, of course, nonsense unless you want the shittiest quality most generic house possible.

Because they make it very quick to get a 70% working system that you can sell to people (internally too) before the people with the capacity to get a 100% working system gets their first version.

In “raw" html/js/sql/Java.

"General-Purpose Tool-Building Factory Factory Factory"?

"Why I Hate Frameworks": https://www.gwern.net/docs/cs/2005-09-30-smith-whyihateframe...

It depends.

If you have a nail, and you are using a jackhammer of a framework to pound it in, every developer that comes after you has to learn how to use that jackhammer unless they have previous experience with it. And honestly, most complex frameworks have so many different potential usage patterns that future developers will still have a heck of a learning curve. (I've never seen two uses of redux that remotely resembled each other! After using redux for 3 years I can easily find redux code that I have literally next to no idea what it is doing!)

If you use a hammer, even if it is a bespoke hammer, future developers just have to learn how to pick up that hammer.

Obviously at some point it pays to switch to using an Industry Standard(tm) solution, but for many problems plain old JS or HTML may very well be appropriate.

For example, I was making ad campaign landing pages for my now defunct startup. To ensure speed I wanted them to all be static HTML. I could have used a static site generator of some sort, configured templates and all that jazz, or, instead, I could just write HTML + CSS for 3 landing pages. (Actually I only had to write the CSS once).

Now if I needed to programmatically generate dozens of landing pages, then sure, adopting some static site generating framework may have been worthwhile. But in my case it wasn't. Raw HTML was the correct solution.

That's an extremely open question, and I wouldn't agree that frameworks, in any way, "provide a way for people to collaborate" any more than a whip does. They seem to me to limit the ability to write the code you want for no value other than providing confirmation bias.

The idea that the same thing being outputted on the screen will have two implementations by two uncommunicative engineers that vary in such a way that they become impractical, impossible or even at all harder to debug, is simply a myth. I highlighted the word "engineer" because that's the key point - if you are hired to solve a problem, you are probably going to solve it the easiest and most convenient way. You are trusted to write code, not use an API. I find it hard to buy the idea that 20 year old aspx code is any easier to debug than pure JavaScript written around that time.

As far as why frameworks have become the norm, I would personally put it down to, in no particular order, the following ideas:

- Some are genuinely useful - functionality is limited in the parent environment's API and must be implemented in some way. This is rare, in my (mild) experience.

- The company had learned the lesson that contractors (especially ones that win the bidding wars) write horrific and unmanageable code. Therefore, it decided that a cost-free way to alleviate this issue is to force all engineers, permanent or on-contract, to use a specific framework.

- The above point spread between companies as people (directors of engineering) shifted around, word spread that someone was able to make significant cost savings etc.

- They are in place to allow one to follow idioms better. I recently interviewed for a company that followed, quite closely, the clean architecture book by Martin Fowler, with their own internal framework. They had a good plan in mind - they wanted components to be interchangeable, usually at compile time, and needed to ensure that they followed procedure in order to write code that was manageable at scale in order to achieve this. They made their own framework and have used it for 10+ years. This proper, elegant use of a framework is very rare.

I do not mean to say that frameworks are useless - however, JQuery has always been absolutely worthless to me in my personal endeavours - which often included quite bespoke DOM manipulations - the exact place that JQuery is meant to shine. To me, it is a great example of something that was phased in as a Band-Aid solution to people writing terrible code as they had no ownership of the product.

I will admit that I use little helpers, such as

  function $(usuallyASwearWord){
      // Yes I really use element by id for this when writing my own fun code. Not query selector. 
      return document.getElementById(usuallyASwearWord);
  }

When using a framework, you get most of the benefits right away, and most of the costs happen much later, when abandoning the framework is costly.

Besides pure technical quality, you have to consider more murky issues: who wrote the framework? how long will they be around for? Does their problem match your problem? Will your interests remain aligned in the future? Is there a succession plan? Will new hires be familiar with it? What's the learning curve? Does it integrate with the other N tools you are using?

Ideally, the decision to use the framework or not should be made by someone who has used it before. This is often not the case.

My personal preference is to always use simplest tools that get the job done.

I wish that hiring technical writers to write docs was more common. Most of the new libraries I have to use have documentation that is so poor, I have to use stack overflow + write test cases.

Currently, I am in the middle of taking over a go code hairball from someone who left, and who loved external dependencies. Out of 31 direct dependencies, 11 are not at release 1.0 yet. Many have poor documentation. I believe that when I am done with it, our code will have fewer dependencies, and be half the size.

> it raises the barrier enormously for onboarding new people.

is.. a little ironic. I've never met any of the gazillion frameworks out there that have made it easier to get people up to speed, despite promises to that effect.

Hmm, i'd say that there are frameworks that actually make things easier and better, and some that are much like the incomprehensible Rube Goldberg machines that the previous poster complained about.

For an example of this, you needn't look any further than the Spring framework in Java, perhaps the ultimate example of enterprise bloat of poorly understandable (unless you spend all your time digging into the internals) and overly abstracted non-solutions for problems, and that's even before you pull in 50-100 dependencies for making your dated monolith do whatever the business wants.

There is an absurd amount of XML configuration needed, except that sometimes you need to mess around with annotations in your source code as well, except for those cases when instead you'll want to just overwrite some class with your custom logic inside of the methods, which will be initialized by some Eldritch abomination of class loading and reflection, or maybe you'll need to change a bunch of properties in a config file, with all of this leading to long and arduous error messages that will make you waste hours digging through Stack Overflow.

I was in charge of migrating an older Spring app to Spring Boot, which attempts to alleviate some of these issues (and there was also the fact that the particular Spring version was EOL at that point) and after wasting more time than i'd like to admit of an error after an error after an error, it was decided to just update Spring itself as far as possible and let it chug along in a barely alive state.

The framework wasn't a way of managing risks or "getting things done", it ended up being a risk in of itself and was a massive pain once the system needed to be supported, maintained and also have new functionality developed for 5-10 years or so. That's not to say that there aren't better frameworks out there, but there definitely are some that will be more trouble than they're worth, unless you want to have a person or two full-time just to wrangle them in order and continuously maintain and update them, which for many of the smaller orgs out there is unacceptable.

> If everyone is writing "raw" JS/HTML/SQL/Java/whatever, they are actually building their own individual, unstandardized frameworks.

I've also seem bespoke frameworks be used and those were generally worse - while there was the occasional good idea in them and they could be decent to work with in a limited set of circumstances, getting help when something goes wrong with them was impossible and you were stuck reading the source and mucking about in it. Why is that bad? Because with the larger and more popular frameworks out there you can instead look up solutions for problems or integrations that someone else has written.

I'll admit that pulling in dependencies carries certain risks in of itself, but you'll very quickly realize how nice it is to let someone else step on rakes before and learn from their problems and mistakes, instead of attempt to translate code comments from Lithuanian to your own language when prod is running into issues, because the original devs are long gone and there is no documentation, nor examples of how to use anything in place.

Thus, it is better to use frameworks that have tens of thousands of collective man hours put into them, or even more, as long as they don't suck.

So, in the end, it's probably better to use frameworks that keep you closer to the code that is actually running, like Spring Boot (where at least parts of the config can be done in plain Java), or even look at ones like .NET (the newer ones are a bit like Spring Boot), rather than DSL hell.

You can put a breakpoint in your Java code and see why something isn't initializing properly. You cannot put a breakpoint into your .properties, .xml or .yaml file (or even feasibly find where to put into the over-abstracted configuration read mechanism, or find how it connects to the bit that is actually initializing your data).

There is only one reason, and it's that people don't understand how to use SQL, and that's because it's not taught in computer science degrees, and this also has strong ties to the battle between sun and oracle.

And this makes database developers expensive and hard to hire.

There is no reason to use an ORM if you know SQL and database modelling. The relational model is by far the most powerful tool to solve CRUD use cases, which "web development" is, and that's just simply a fact.

What the ORM will do for you is automate some boilerplate mapper code, which might be a bit boring to write, but is very simple. And the downside is that it will make all of your database modelling, querying, access etc orders of magnitude more obscure and complex compared to just using SQL. And now you will waste enormous amounts of time fighting with the framework to debug the ORM and find out which kind of weird SQL statements it generates, where you would have complete control over this if you just wrote your own.

The obvious solution that would benefit everyone, is to simply train your developers to use SQL, if they don't already know it, and let them take a course if they need to. Not to slap on some stupid framework and make a huge unmanageable mess of the whole system.

Moreover, many ORMs give you the tooling to validate data before hitting the db, which is also another huge plus.

I mainly work with Python on the backend, and I've recently switched from SQLAlchemy + marshmallow + alembic to SQLModel and both have been amazing to work with.

I would never dream of setting up tables through plain raw sql ever again.

If it is the case that I'm moving onto a different language, as Go may be it, I'm willing to learn the ORM, granted that it deals with migrations and validations out of the box.

I remember reading licenses as a teenager and wondering about their legality.

Engineering... I dunno. The digital realm doesn't seem, to me, to have enough of a split between "design" and "actual device," so we can't really split the job into the engineer's task and the manufacturer/builder/etc.

One annoying thing about this the whole "software engineering" discussion is that often casts the engineers as gatekeeping or being snooty and not wanting to include the programmers. It just seems like a different type of job, though, still perfectly respectable and decently prestigious. Nobody gets angry that we can't be called "Software Chefs" or "Software Conductors."

Actually that's one of the things Sussman noted about modern software development: that you have to "perform basic science" on your framework and library code by running experiments, rather than building up things from first principles.

So maybe computer science will become a science, after all!

You hit the nail on the head with mention of contracts.

We need wider adoption of tooling like Pact. (Ironic link to Pact docs follows.)

https://docs.pact.io

If you can build this from common basic components it should turn out ok. I think the trick is discovering what your application needs. What kind of interface it should be and what goes where. The it's mostly the same thing over and over again until it's not. Then this exploratory phase starts all over.

I like to draw experience from game engines. They typically build success around a few key components that have well defined constraints. As long as the right constraints are in place it's possible to make a fun experience. If you have the wrong constraints and you find yourself constantly battling the system then stuff is all wrong and you need to find a new approach.

Is this really suprising for You? I mean I always believed that our deduction skills as humans were really limited and primitive. Just look at conveys game of life or any game as a matter of fact (chess?). The emergent properties of those games with few basic rules are so elusive that we need generations od geniuses to work on them to discover anyhthing beyond simplest things. How can we expect anything better when the rules gets really complex and fuzzy (as in real life)

[1] https://www.youtube.com/watch?v=ZSRHeXYDLko

What I am saying instead is interfaces and composition needs more attention. Shitty APIs are a lack of attention to interfaces. Shitty marshalling ultimately stems from a lack of attention to competitionality, and the cost is that people are incentivized to commit anti-lang-sec sins.

> these are all implementation aspects where the same interface can cover a world ranging from total crap to excellence.

This is narrowly true, but completely misleading. You have to focus on incentivization and emergent behavior.

Good interfaces can be used safely by lazy people. Bad interfaces will never be used will by the most careful people.

Every implementation tests in interfaces below it, and so quality will percolate upwards.