HNHacker News
TopNewBestAskShowJobs

fpereiro

431 karma · joined July 27, 2009

federicopereiro.com
submissionscomments
fpereiro··on How I write back ends
I think it does have to have a certain elegance (or better, conceptual integrity), otherwise it won't be able to provide a flexible and understandable interface for handling all the various use cases.
fpereiro··on How I write back ends
Unfortunately this seems to be the case; lockfiles would be unnecessary only if all your dependencies (and their dependencies, recursively all the way down) reference explicit versions, the risk being that a new malicious version would be published. I'll research if there's a workaround.

Thanks everyone for pointing out this issue.

fpereiro··on How I write back ends
I'm curious as to the advantages of using Debian over Ubuntu on the server - is it considered generally more stable or better debugged? This is something I've never explored and am interested in your thoughts about it.

At some point in the future, I'd like to give OpenBSD a try.

fpereiro··on How I write back ends
I also rambled further here: http://federicopereiro.com/why-es5 .
fpereiro··on How I write back ends
The only alternative would be to create an elegant system that is also in itself an open standard. I don't see a way around this double requirement. It sounds like something mighty interesting to work in.
fpereiro··on How I write back ends
Hi there!

- I choose js/node as my server-side language of choice. This is because I love both js the language and node itself, and so far haven't seen the need to try something else. That doesn't mean, however, that it should be the solution for everyone. Particularly if you don't like js - you're probably better off programming in a language you enjoy!

- I have rambled a bit about js style here: https://github.com/fpereiro/ustack#mes5-a-javascript-subset . My approach, however, is very minimalistic and backward-looking (or aspiring to timelessness, if you're being very kind), so I recommend it very cautiously. You might be better off using more language features, particularly those that come in ES6-7-8. For all of this, I'd rather leave this content separately in the ustack repo.

fpereiro··on How I write back ends
I love working with fundamental data structures (sets, lists, hashes) much more than working with relational databases. I feel I'm more productive and my code is shorter. It does, however, require far stricter validations in the code; in general a lot of what a relational database does for you (in terms of type validation, schemas and even data consistency), you must do yourself in code or by using redis carefully.
fpereiro··on How I write back ends
I normally don't work with websockets, just JSON, but perhaps that's because all the applications I work on are basically glorified CRUDs. If your application is essentially a CRUD, however, JSON might be more than enough. I've seen thousands of JSON requests/responses per second being handled by small virtual instances running node.

The first thing I'll do as soon as the product I'm building starts bringing in revenue will be to pay a security expert to perform a security audit of it all. In the meantime, I try as much as possible to eliminate backdoors (use the API for everything), simplify the data flows, drastically reduce the amount of dependencies and of moving parts, and as much as possible use technology that's been around for a few years (nginx, node and nginx all make the cut :).

fpereiro··on How I write back ends
At some point I'll do a public writeup of my planned cost structure. Hetzner's EX62 servers (https://www.hetzner.com/dedicated-rootserver) are about 10 USD/mo/TB of disk and it would be hard to go over their bandwidth limits . Even allowing for RAID and/or multiple locations/servers for a file node, the cost is still low. The main advantage in terms of cost is the possibility of not spending money on outgoing bandwidth.
fpereiro··on How I write back ends
> One second is a significant gap for data loss.

Definitely agree. I would not, for example, create a financial transaction system that is not backed by an ACID database that persists to disk.

If you have time/interest, I'd love to know more about the problems encountered by your team when using redis. In my experience, the main problem is scaling it to multiple instances (because RAM grows linearly with use). If the team also encountered other reliability issues, it would be valuable knowledge to me and perhaps others.

fpereiro··on How I write back ends
Good point. I'm drawn to redis not so much for its performance (which can be comparable to that of a relational database) - but rather for the expressive power of storing data in general purpose data structures (hashes, lists, sets) rather than in relational databases. That expressive power is what I miss when not working with redis.
fpereiro··on How I write back ends
I added a few more notes on when/why to use Redis and when to use something else: https://github.com/fpereiro/backendlore#redis . Redis can take you surprisingly far as a general purpose database. I prefer to use either Redis or a relational database, but I'm in no position to tell you that you should do that too, particularly if MongoDB works well for you.
fpereiro··on How I write back ends
Please see https://github.com/fpereiro/backendlore/issues/5 .
fpereiro··on How I write back ends
I believe that npm no longer allows for republishing existing versions. Wouldn't this restriction remove this vulnerability altogether?
fpereiro··on How I write back ends
I am consdering to move "ulogs" out of redis and into files because of the potential memory usage - perhaps a hybrid approach where only new logs are in redis would work best. I don't have a definitive solution yet. However, this would be a problem when handling tens of thousands of users or more, which makes it a problem I'd love to have!

I'm also considering letting users delete their own history of logins that are older than N days, since it's their data in the end, not mine.

fpereiro··on How I write back ends
Glad to hear! Part of the reason I put all of this out there is to offer a different point of view that has worked for me and others I've worked with, and to stimulate debate and fact-based (or at least experience-based) interactions regarding backend lore, instead of pining for best practices that often are under-scrutinized.
fpereiro··on How I write back ends
Definitely a tradeoff - I'm biased towards working in teams of only 2-3 people. Larger teams would probably benefit much more from more structure and process.
fpereiro··on How I write back ends
> why is all of this still needed ?

That's the million dollar question! It also vexes me that all of this is necessary. I hope like crazy that there will be a radically better approach in 5-10 years. I am not aware of solutions that truly abstract all these problems without creating 1) further, larger problems; and/or 2) severe vendor lock-in.

fpereiro··on How I write back ends
Thanks for your comment!

- I don't use lockfiles - instead I use very very few dependencies, and I always target a specific version of each of them. I know this is not the same as a shrinkwrap, but I'm crazy enough to feel that package-lock.json adds visual bloat to my repo.

- I am biased towards mon because it's probably less susceptible to changes over time than Ubuntu itself, and because it can be installed with a single command. But systemd definitely is a good option too if you're using Ubuntu!

fpereiro··on How I write back ends
You raise a great point.

The web frontends I write always draw views entirely through client-side js, and interact with the server mostly through JSON. The only HTML served is that to bootstrap the page, plus static assets (js & images). If you use server-side rendering, writing an HTTP API might indeed represent more work. And I'm in no position to say that client-rendering is better than server-rendering.

Just added a clarification in this regard in the document. Thanks for bringing this point up.

fpereiro··on How I write back ends
It's perfectly possible and probably won't add much complexity or significant cost. I have no experience, however, in configuring ELB with HTTPS - usually I just go with the known quantity of a local nginx.
fpereiro··on How I write back ends
Thank you for your suggestion! However, I'm a sucker for text diagrams (ever since I saw them in Jonesforth: https://github.com/nornagon/jonesforth/blob/master/jonesfort...), so I strongly prefer to leave them as they stand. Hopefully they're still clear enough and get the point across.
fpereiro··on How I write back ends
I like dealing with the underlying OS - well, not precisely like, but rather the absence of an intermediate layer between the OS and me. Fortunately, dealing with the OS is very straightforward.

However, if Heroku works well for you, by all means use it! Your time is probably better spent working on your app instead of in the infrastructure. Things that work should only be reconsidered when they start to create real world problems - and the changes that you do in response to real challenges tends to be wiser and more lasting.

fpereiro··on How I write back ends
It's actually a fine solution and I often do this with infrastructures that I don't intend to maintain myself. In general, however, I try to minimize complexity, and having an external service feels a tad more complex than running the database inside the instance. But you could probably make a successful case to the contrary - namely, that using a service rather than running your own is actually much simpler and easier to maintain.

I must confess that in this decision, the complexity of the AWS console (which I consider awash in buckets of accidental complexity) usually tilts my balance towards self-hosting.

fpereiro··on How I write back ends
Glad you liked the article!

I don't have recommendations for other articles like these; most of what I wrote came from personal experience or things I learned directly from other people working with me. I also have learned a lot from writeups by people running web services, especially when they encounter issues and/or suffer downtime or data loss. These real-world lessons can be highly enlightening, particularly if they're written in an honest way - I cannot point to any one in particular, but that's what I would look for, especially for services that run a stack similar to the one you're planning to use.

As for Python vs node, I recommend using the language you love the most, as long as the runtime you use is decently performant. node itself is blazing fast, but I would probably use it anyway if it was 10x as slow. I have no experience running Python as a web service (I've only done some local scripting with it) but I'm sure it can work perfectly well and I know of at least a couple projects which use it successfully at big scales.

I recently read a writeup by DHH about Ruby representing only 15% of the infrastructure cost of Basecamp (https://m.signalvnoise.com/only-15-of-the-basecamp-operation...). The takeaway here is that your choice of database and architecture will probably determine your costs and performance, not the programming language.

Regarding alternatives to AWS, I haven't taken the time to consider them, honestly. I use AWS very sparingly (with the exception of S3 - which has a track record that no equivalent service offers, if only for the sheer amount of time they've been at it - and SES, which I use only because I'm already using S3 and I might as well just have one cloud provider). For work with clients, they all seem to prefer embracing the dominant vendor and I don't have any powerful arguments to dissuade them from this.

At the moment I'm only responsible for a couple of services with low traffic (and handling no financial transactions nor pacemakers), so I sleep with my phone off. I've been responsible for largish infrastructures and in those cases, the quality of your sleep drastically declines. Soon and hopefully, however, I'll be running a service with a lot of user data and reliability guarantees, so I won't be able to turn off my phone anymore at night. This time, however, I'll be fully in control of all design decisions and I hope that, by systematic treatment and elimination of sources of failure, outages and emergencies should happen asymptotically less as time goes by - and with any luck, the lessons learnt can also help others that are in the same position.

fpereiro··on How I write back ends
Will take a look at Fabric, thank you!
fpereiro··on How I write back ends
I fully concur with your last statement. I find that the main advantage of having less moving parts is that there's exponentially less things that can go wrong - and that makes reliability possible with very small teams. The setup cost is usually not steep when compared to the overall lifecycle of a project, I fully agree.

The other argument I have against many DevOps tools (and most software tools in general) is that the patterns of the solutions they provide don't generally match the basic patterns of the problems they are solving - the consequence is the sheer complexity of understanding the tool, especially when something goes wrong. In the past, this hasn't only costed me time, but a lot of frustration that I'd rather avoid - even at the price of having to build my own solutions.

fpereiro··on How I write back ends
You raise interesting points!

In general, I don't use Vagrant. I simply use my host OS (Ubuntu) and my local node & redis. There's usually no further moving parts than these, so it is easy to achieve development in an environment that is practically equivalent to the Ubuntu servers/instances in the cloud, the main difference being a different folder path, different files stored and different info on the DB.

When running the app locally, I still use services like S3 and SES - although I might not send emails, to keep things as "real" as possible.

I do use Vagrant when the project requires installation of other tools beyond my basic toolkit (like MongoDB or Elasticsearch) - in this case, I want that software to stay within the VM.

In general, I'd rather avoid virtualization because it represents an extra layer, but if it is necessary for idempotence or having a dev environment that mimics the prod environment, I'd definitely embrace it.

fpereiro··on How I write back ends
The main reason I use nginx is because it is so easy to configure HTTPS with LetsEncrypt & Certbot. Perhaps there's an almost equally easy solution now that only uses node, but last time I checked (2018) it was messy.

In this article (https://medium.com/intrinsic/why-should-i-use-a-reverse-prox...) there's another good reason: by not making node manage the HTTPS certificates, no node library that I use (directly or indirectly through a dependency) can represent an attack surface regarding the certificates. But I must confess this is marginal compared to the ease of installation and maintenance that I already mentioned.

fpereiro··on How I write back ends
Wow, I didn't know that AWS had a managed redis service! I thought Elasticache was only for Elasticsearch. Thank you for pointing this out.

In my experience, setting up redis is very easy and not time-consuming; what is not trivial is to store large amounts of data in it, which requires either getting more RAM or considering a clustering option.

I'll very much have this option in mind for upcoming projects where I'm not the sole owner.

Regarding Ansible vs bash/ssh, I'll probably (hopefully!) come to the point where I'll need something more sophisticated. Ansible is definitely my benchmark for an elegant solution to idempotent provisioning and infrastructure as code. I might not help writing something custom in js, but I definitely will have to go beyond bash and purely imperative approaches. I'll be sure to share what I learn along the way.

← PreviousPage 2 of 3Next →