The unwritten laws of engineering at Stedi
stedi.com
stedi.com
I think they push the bounds of "serverless" for a complex data project. They did discover a bug in IAM (also documented on their blog).
But, really, it's another bunch of opinionated nerds. Isn't that most of HN?
> We are hiring across multiple engineering, product, and design roles right now, so we wanted to post this publicly to give a sense of what it’s like to work here. If this resonates with you, we would love to hear from you.
Serverless architectures require designs that accommodate these constraints upfront, and there's a learning curve to finding the best primitives to adopt. It's akin to building with Lego rather than a 3D printer—you're working within predefined limits. Once you understand the optimal patterns for utilising these constraints, they become guardrails that reduce development and operational costs over time.
In my experience, the bar for requiring "always on" is higher than we might realise. Also, the choice between serverless and traditional elastic computing models isn't always binary. Excellent options like AWS Fargate serve as a practical middle ground.
Of course, this approach is only sometimes applicable. There might be a tipping point where it might make sense to consider renting servers with more traditional hosting solutions. Is optimising operational costs by investing capital in rented or physical infrastructure a strategic decision post-startup?
Do you have some recommendations where to start studying such patterns? I am a bit overwhelmed by all the possibilities and (cost) dangers of cloud computing.
I agree that finding comprehensive, non-sales-oriented best practices can be challenging. One useful resource is Serverless Land from AWS [1], which offers a collection of practical examples and articles.
Mastering serverless involves more than just wiring up functions—it's about adopting Infrastructure as Code (IaC) principles. Over time, you'll develop reusable patterns and tools like Terraform modules that encapsulate best practices such as asynchronous Lambdas with EventBridge subscriptions, SQS Dead Letter Queues (DLQ), and appropriate error handling configurations.
For a deeper dive into the Serverless philosophy explicitly applied to AWS, I recommend checking out Yan Cui's blog [2]. Yan provides insightful articles on serverless architectures and patterns.
I suggest exploring the Reactive Manifesto [3] for principles and best practices in building responsive, scalable systems if you're interested in asynchronous messaging patterns.
[0] https://www.oreilly.com/library/view/serverless-development-... [1] https://serverlessland.com [2] https://theburningmonk.com [3] https://www.reactivemanifesto.org
I would rather stick with EC2s at this point.
But the “what” and the “why” seem to be mostly implied. There are mentions of quality and security, but it is really unclear. I don’t know what this company is trying to actually achieve. As an aside, lists like this are intriguing to me because they seem to value uptime over capability. I get the impression they would be perfectly happy shipping nothing for long periods.
I wouldn't say _happy_, but we were certainly _willing_ to do this for long periods of time to get the right foundation built (for reference, we built for 5 years before launching). This is made explicit in the opening:
> As a result of following the framework, product development may sometimes come to a screeching halt. The work itself may be tedious and frustrating. Progress may take many multiples of the time it would take using other methods. > [...] > That's the difference between principles and preferences: principles are what you do regardless of the cost – everything else is just a preference.
As for this item:
> But the “what” and the “why” seem to be mostly implied.
This is outlined in another linked post: https://www.stedi.com/blog/excerpts-from-the-annual-letter
This is really vague point. What's security? You can spend millenia securing anything without clear target. ssh with root:toor is probably not secure. What about ssh with 12-chars password? Shall we disable passwords altogether? Shall we change port? Shall we implement port knocking? Shall we investigate ssh source code, recompile it, remove anything not needed in this particular server? Shall we rewrite ssh from the scratch using Rust? Shall we build new language with better security focus?
Erk. I get where they're coming from, but it feels like they're choosing lower AWS costs, but in doing so, incurring higher dev costs.
But maybe, that's a trade-off that makes sense for them. Would like to have more background on this balancing act - e.g., what "functionality" means to them.
The other concern I'd have is the risk of unanticipated spend, serverless techs can scale up really well, and your costs can also scale up hand-in-hand.
I guess it depends on how bursty their product is.
Ditto the bit where "As a last resort, we write our own code". That feels like a lot of pain and YAML trying to cobble together other people's expensive code.
It's all the yummy egress and all the other markups you're paying for.
Lots of humans upgraded lots of dependencies without noticing, I doubt whoever is doing it in your org is special enough to be the one who would have caught it. And if they are, they should be working in security research, not bumping dependency versions in package.json.
Also, not sure what JS has to do with the xz attack.
The argument I was responding to is that automating your dependency updates somehow makes you more vulnerable to a supply chain attack.
I could see an argument that waiting X days from a dependency release to when you pull it in gives you a little time for other people to find issues. But that's orthogonal to whether you update dependencies automatically or manually, or whether you do it once/year or every day.
But usually I find that bumping dependencies is a detective work because unlike JS where you can depend on multiple versions of the same package, for .NET you can't. So if you update package A, there is a risk that you also update a transitive dependency C, which package B depends on but on the previous version. So even for what looks like trivial updates it's often a chore. Which is why I'm happy to have just a handful of dependencies.
I don’t think it makes sense for every project, but if recovery options are cheap then I don’t see anything that speaks against it.
Of course you should design and implement secure systems. You owe that to your customers. Still, that's meaningless until you decide what "secure" means. How valuable is your data to your customers and to potential attackers? Who's likely to target you? What level of risk are you willing to accept?
And make no mistake: you do accept a certain level of risk. If your systems are valuable enough, you're going to get the attention of enemy nations who will attack you. How much are you willing to invest to protect against North Korea or Russia? Short of financing a formal proof of every bit of software you run on, from web frontend down to microcode, you're trusting that someone else did their job correctly. And will you run that unprecedentedly secure code on a server farm inside Fort Knox?
Job one in security is deciding how much you need to invest to reach the level of security you need to achieve. If you're storing healthcare data, that's going to be a whole lot of time and money, and even then good luck if a malicious nation wants in. If you're storing grocery store shopping lists for a thousand people, and you're investing $10,000,000 to secure it, please call me because I want in on that action.
Looks like Stedi's in a field where security matters an awful lot and it's great to hear that they take it seriously. However, I'd love to hear what that actually means in practice.
Security is #0. Okay, security against what?
Not the new hotness? Longer startup times, which can hurt if you scale to zero? Requires building a golden image which is more overhead plus you are responsible for OS bugs?