Ask HN: How to get good at designing big and complex applications? | Hacker News Reader

Ask HN: How to get good at designing big and complex applications?

76 pointsrochak46 comments

Hi HN! I am a grad student with a focus on CS, specifically Software Engineering. I like to explore different technologies that are used to build a product and am fascinated by the way people design such big complex systems. For instance, I am taking a class on Operating Systems right now which also requires me to develop my own kernel. I am finding it very exciting, challenging and fun. I would really like to know what all I can do to get better at designing such big complex systems. I try to read open source but it always feels like a challenge. I don't know if I have the aptitude for this but studying CS and the thought processes of people who have spent so much effort in this field gives me a lot of joy. So, if you have any advice, resources, books or tutorials that you think might help me, please do share. I would be really obliged. Thanks!

View on HN

"A complex system that works is invariably found to have evolved from a simple system that worked. The inverse proposition also appears to be true: A complex system designed from scratch never works and cannot be made to work. You have to start over, beginning with a working simple system."

I just made the connection between your statement "A complex system that works is invariably found to have evolved from a simple system that worked." and the common understanding among experienced developers that proficiency in software and the primary goal when designing and building a system is perpetually fighting complexity.

You design and write the cleanest, most simplistic & adequate solutions for problems as they arise and ruthlessly battle introducing complexity as features and scope grow.

Inevitably, after many years you will wind up with a system which by sheer size is complex. But only reason it works is because of the never-ending battle in the fight against complexity.

Yes, have to relentlessly drive down complexity. Starting with simple, pure functions that work and composing them together to build up more complex functionality goes a long way.

Depending on the context this is either simply false or a truism. And thus imho offers no real informational value.

You can't design complex system without some buiding blocks which basically definition of 'complex', but you can design a complex system before it's deployed and it can succeed.

Itreatively designed system can hit a scalabilty wall or plateu where adding features is slower that a redesign. An evolving system design is a subject to a local minima of scalability and it needs careful apporach to maintain it.

This sounds kinda like a rephrased "there's no problem that can't be solved by adding another abstraction layer".

I disagree. The observation that working complex systems evolved from simpler ones is very useful, because from it follows a rule that, if you want to make a complex system, your best chance is to make a simpler system, then iteratively improve it. It practically solves the analysis-paralysis problem that otherwise is inevitable.

Even total redesigns will benefit from the experience and information, that you got from the simpler system(s). These are often hard to derive by other means.

People quote that frequently. Does anyone have counter examples? Eg, did DooM, the witness, Windows “rewrites” (I read certain releases were not really incremental, correct me if im wrong), or the nasa shuttle systems evolve from smaller versions? (Doom has WolfenStein, I forget if it was evolved from it though)

This comes with experience. There is no single answer to get good at the designing complex systems.

The more you work in different applications, the more you discuss with your peers in your organization and other meetup groups, explore different design patterns (Martin Fowler), Follow well known Architects or Company engineering blog [0] posts, Try doing proof of concepts etc. Eventually as you progress with experience, I am sure you will get that knowledge.

I used to follow the below link [1] to understand how the systems work at different companies globally (e.g Facebook, Google, Netflix, PlentyofFish etc). This has very interesting content.

[0] https://github.com/kilimchoi/engineering-blogs

[1] http://highscalability.com/all-time-favorites/

Thanks for the advice and the links! Also, I came across a book called Designing Data Intensive Applications. It was very highly rated on Amazon and on HN as well. Do you think it would help me? It explained a lot of concepts but after reading for a while, I found myself not following it. So, I decided to tackle each topic from scratch.

Do people find Martin Fowler and his peers' work to be good in real world? I'm a system/application architect who spent years coding. I used to read books by Martin Fowler and others religiously. But now, I realize most of the books and talks are really just fluff pieces because the examples worked in perfect condition, where as my experience is from the battle field.

As far as getting good at designing big and complex application, I'll give what I know.

1. Everything complex came from a small system. So make sure your small application works first. Twitter/Google/Facebook didn't get to what they have now on day 1. They failed a lot to get here.

2. Designing monolithic system is a lot easier than micro services.

3. Read a lot of codes and how different component interact with each other. For Python, projects such as Scrapy, Flask and Django are good example to start.

I find the concepts he makes you think about are helpful.

Think through the big problems and make sure you keep them in mind as your are coding the subsystems.

Make sure any code you write could be used by another system in case you need to break it apart later.

Make things pluggable though interfaces.

No global references.

Honestly, I found most books are written more as a basis for people who need something to hold onto and can't do it on their own (yet), and for people to understand the ridiculous enterprise systems. Most of the time, I see good developers take longer to come up with various structures and deciding which one to go for than the time they'd spend fixing these things afterwards. On the other hand, inexperienced devs will trip up no matter what. That's why they are inexperienced, give them a chance and some time. Realistically, good systems are built by good devs who understand you need to continuously maintain and evolve the system according to your needs, and keep an eye on relatively simple things like method structures while doing so. Its easier to refactor when all your methods name themselves, are bite-sized in what they do and you provide a sandbox for other people to work in, or a blackbox for them to work with. And clear errors. Don't be that guy who hides the stack trace and doesn't keep track of the state of the app. Please don't.

Thanks! Iteratively improving and adding features seems the most natural way how a software matures or ‘becomes complex’. I guess I just have to start building something small and learn how to add features that require learning a new concept.

Your first step should be avoid complexity. Reject features (big or small) that don't add value to the system. And if you must add feature, try to make it as specific as possible. Generic features tend to get abused. I personally don't like it, but other do love it.

I agree. Too much generic features end up becoming abstract and make it difficult to find what the original purpose was.

I would add to this that you shouldn't be afraid to refactor when you hit a wall where you can't add the feature you want to add because of the way the other features structured your code. Better to take the time to modify the basic structure a bit to make it work than to try to kludge something together to sort of make it work.

You'll find that doing it right allows other features to be added that you haven't thought of yet. If you somehow force it to sort of work, you'll end up doing that for a lot of other features, too.

Now that I look back, that is how I have approached developing systems and it certainly makes a lot of sense.

I learned about Phoenix servers and some other topics from Fowler and it was valuable for me. I can’t speak to his bigger works

Hi,

There's no single way. In fact, beeing a good software architect is all about having many ways in solving a problem, knowing wich solution is the most adapted depending on the context. All is about culture and cleverness To improve, you can :

- Follow/read the well know architects. For example, I read this blog https://martinfowler.com/ and watch most of the conferencies he gives.

- Learn the design patterns and the architectural ones.

- Read a lot of code, like github repos. It does not care if you're reading repos on a subject close to the one you're interested to. You need to see a lot of architectures to see some clever solutions some peoples have used to solve some specific problems. I don't think there's any shortcut.

- get interested in meta development. Like "why does the languages work the way they work ?", read about how code is produced in real conditions (what are the processes involved), etc.

I’d also add go to excellent meetups with high quality talks. They attract good people, get talking and you’ll have 100 things to google when you get home!

Yes, getting involved not only behind your screen is really important. They attract good peoples that give you tons of subjects to study, but most important, it makes you feel like you're part of a community and you're not doing all of this for nothing, or in a wrong way.

For that part, I think it's true whatever is the field in which you want to improve

This is something I can do! Thanks, I will try to go to some meetups.

What to do when those talks are too complex for my understanding?

They are for me too. Ask questions after the talk. Get the gist if you can even if you don’t grok all the details. Speakers often go too fast through stuff so I don’t take it personal if I don’t understand it all or any of it.

Thanks! For some reason I find it difficult to just approach people and talk about stuff with them.

You're still a studient. It is not possible for you to have a good knowledge of software architecture because it is a really complex domain. If you show interest in it, read about it, go to conferences and ask questions, don't worry. Peoples will act cool with you. Anyway, the only way to progress in dev/software archi/devops/etc is to put your ego aside and accept the fact a lot of peoples knows whay more than you. If you go speak with them and they treat you like garbage, this is their fault, not yours.

If you're really shy/impressed, maybe you can go with some friends and speak about it together. It will help you feel like you understood some things. Architecture is not filled only with youg peoples, and if you're a group of 20+ y/o speaking together, eventually the older ones will come and speak to you. then you can ask them what to read/watch to have deeper understanding

Congrats on your will and enthusiasm, I believe this is already a great first step!

Besides Martin Fowler which was already recommended, I would also recommend the books by Robert C. Martin (Uncle Bob), and his "Clean" series - there are books "Clean Code", "Clean Architecture" - all of them are really good and useful and will give you a solid foundation to build on.

Other than that, the best thing you can do is to get practice & ideally surround yourself with more experienced developers. Either get an internship in a company building something more complex, or participate in e.g. Google Summer of Code.

Doing fun side projects is also a great way - you learn a lot and also have something to show & write about - win-win!

I have got an intern at AWS so I hope I get to see some good design and meet intelligent people there.

I have been looking at clean architecture and trying to apply it to a Unity game. My impression is that industry wisdom favours a lot less OOP and a lot less abstraction in their architectures (eg, Jonathan blows talks, hundreds of posts on Reddit), mostly for performance and velocity reasons.

Do these techniques apply to games, or are they the wrong tool for the jobs?

Have you checked [0]?

[0] https://gameprogrammingpatterns.com/

Check the books of Architecture of Open Source Applications - http://aosabook.org/en/index.html

This is a good find! Thanks!

This book (Designing Data Intensive Applications) was an interesting read: https://dataintensive.net/

Thanks! I knew about this book before but couldn’t go through it a lot because it started to get very tough. So, I decided to read each and every concept from the basics before I jumped into that book again.

It is tough indeed, especially if you don't have a use case for it.

Experience > theory for such a topic. Go ahead if you find it interesting, but don't expect to become an expert on the topic just by reading about it ;)

That is also true!

Study & Experience, that's it.

Also understand the difference between these

     programmer
     application developer
     software engineer
     computer scientists

there are very different roles, it's okay to be one and really good at one.

BTW, you mentioned "design" so I'm going to hinge on that, design not the actual implementation. In that case study existing systems. 90% reading can teach you design, on actual implementation, more like 90% doing.

What if I’m not inclined to just one of these? I like learning things and figuring out why which is why I see myself fitting into every category.

Nothing wrong with that, if you are a generalist you will find yourself intersecting those. Else you will find yourself focused in one. There are very few organizations in the world where you get to do all of those, except for a startup in the initial stage or by you pursing your own side projects. Something like game programming might intersect all (programming - using GPU, app development - the game engine, software engineering - building a scalable game server, computer science - utilizing appropriate algorithms and data structures that most people will tell you they never use in the real world)

if you want to do the programming thing - learn low level programming, buy an arduino etc and learn to program it.

app development - build websites, mobile apps, desktop apps.

software engineering - contribute to a large scale app (kubernetes, postgres, etc)

computer scientist - study latest CS papers, replicate result, write your own paper and come up with your own ideas.

After you define what these terms mean, I think I am more inclined towards programming and software engineering. Thanks!

The first three of these are synonyms. I've had all three job titles, over six different companies for roughly the same work.

No they are not, but that's my opinion.

Let me explain...

A programmer can really program a computer. Think of the person that writes an operating system, a bootprom, all your embedded systems programmers. traffic light, very low level, operating near the hardware level/systems programming, assembly, C, rust, etc. your Linux kernel programs & device driver writers. You give them a device, give them the spec and they can bring it to life.

Application developers - build apps typically using higher level library and frameworks, think of your android, ios developer, microsoft windows app developer, CRUD website developer, they build apps for users. Most app developers call themselves programmers, but IMHO are not. Take away their framework, DB and most of them can't even build much with just a language and the stdlib.

Software engineer - more at the system levels, this is not about programming at the low level or user apps, but engineering software systems. Think of the folks behind AWS cloud, the sort of people that wrote zookeeper, kubernetes, Gnome, Unity, kafka, there's a lot of engineering discipline at this level. Code bases are usually in the millions and never really a single developer.

Computer scientist - new ideas, refinement of old ideas, algorithms & analysis. Their output is mostly papers and with some code to demonstrate the idea. This will be the person that gives you something like the raft algorithm or some new twist on machine learning.

Surely, one can intersect some or all of these, but it's also possible to be just one of these. 80% of the companies out there are hiring app developers and that's what they care about.

The early decision you will have to make is whether you want to be a theoretical specialist who will mostly make money consulting working at a consulting firm or university or whether you will be solving actual business problems down to the metal.

Each has a completely different trajectory at this point in your life.

A great example is to read the great debate between Andrew Tanenbaum and Linus Torvalds over kernel design as its closely related to what you're currently studying.

Linus would have clearly failed Tanenbaum's OS class.

Yet I use Linux every day, as does millions of others, while Tanenbaum's MINIX is on a CDROM that came with the book that I bought for my OS class.

This is not to say Linux is still a monolith, but that would be getting in way too much details.

> Yet I use Linux every day, as does millions of others, while Tanenbaum's MINIX is on a CDROM that came with the book that I bought for my OS class.

You sure about that? Do you use an Intel CPU?

https://www.zdnet.com/article/minix-intels-hidden-in-chip-op...

Andrew, I didnt know that. Thank you.

Now, let me ask you this.

If someone was to ask you either of:

1. "Between MINIX and Linux, which is a great example to study as an OS that's deployed on billions of machines worldwide and used by millions of engineers?"

2. "Between MINIX and Linux, learning administering which, offers me the best job prospects in the general tech industry?"

What would be your answer in each case?

Minix was not voluntarily chosen by Intel's customers...

> How to get good at designing big and complex applications?

Write small and simple programs.

Reply on news.ycombinator.com