General guidance when working as a cloud engineer
lockedinspace.com
lockedinspace.com
* When choosing internal names and identifiers (e.g. DNS) do not include org hierarchy of the team. Chances are the next reorg is coming faster than the lifetime of the identifier and renaming is often hard.
* The industry leading tools will contain bugs. From Linux kernel to deploy tooling, there are bugs everywhere. Part of your job is to identify and work around them until upstream patches make it to you if ever.
* Maintaining a patched fork is usually more expensive than setting up a workaround
* Your hyperscaler cloud provider has plenty of scalability limitations. Some of which are not documented. If you want to do something out of the ordinary make sure to check with your account rep before wasting engineering time.
* Bought SaaS will break production in the middle of the night. Your own team will have the best context and motivation to fix/workaround them. When choosing a vendor, include the visibility into their internal monitoring as a factor for disaster recovery (exported metrics and logs of their control plane for example)
If only they'd tell you. We had this exact issue on AWS. Seemingly random packet drops. Metrics on both clients and servers were ok, latency specifically was very low when it worked.
Call up support "yeah, you're running into our connection limit". "Oh. What's that limit?" "yeah, I can't tell you that". His solution was that, since this was somehow related to connection tracking in the security group, I could set this to allow all/all, and set up filtering at the NACL level. Turns out I could do it for this particular issue.
This was before there was a possibility to monitor this [0]. Called up our customer manager. "Let me check". A few days later, "yeah, that's not something we divulge".
---
[0] For those who don't know, it's now possible to keep an eye on refused connections (at least on Linux). https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/monitori... -> conntrack_allowance_exceeded
The service rep has been, overall, useless, when he wasn't actually less informed than I was (my knowledge being based on the AWS website).
Naming in general is hard. If you name stuff based on location, use an identifier that won't change, like provider datacenter names, street addresses or customer building codes, not the current tenant or purpose of use.
For products, come up with an internal product/project name and stick to it in everything that is not immediately visible to the customer. At one point you could see the current and three previous names of our product if you popped out an iframe and opened the inspector (logo with name, page title, URL and prefixed log messages).
> Maintaining a patched fork is usually more expensive than setting up a workaround
When your bosses demand additional features a single customer requested, you absolutely have to make them understand that the functionality must be added to the main product.
- anything my company doesn't create or own, is just called whatever it is
- logical components that aren't specific to the org chart can be whatever you want
- anything org chart specific gets a randomly generated code name. an internal website allows you to register and look up any code name across the company. this allows you to find who the hell owns server "ws-prod" in a 6 year old account that nobody seems to maintain. instead you find "peanutcar-ws-prod" and can then look up who registered "peanutcar". (this also prevents the bu-org-group-product-subproduct-env mess that eventually runs over character limits)
- doesn't matter if the rest of the company doesn't do it, I do it for what I manage. later on if it gets adopted, fine, but if not, at least I won't ever have to rename my crap.
Whatever you are maintaining, read the docs completely first. And I mean cover to cover. Not just the one chapter you need to get a PoC up and running. You will wish you had later, and it will come in handy many times over your career. Consider it an investment in your future.
Read books on microservices before you implement them. Whatever two-line quip you read on a blog will not be as good as reading several whole books from experts.
Docker multi-stage builds won't work in some circumstances. Build optimization eventually gets complex, the more you rely on builds to be "advanced".
Don't get me started on how “strong coupling” is shorthand for “coupled in ways I don't like” etc. ... Sometimes I feel like I'm on an episode of “whose line is it anyway?”, where everything is made up and the points don't matter.
> I find that “microservices should only perform a single task” is a really dangerous way to phrase it because we have no idea what the article means by “task.”
First off, it's not dangerous. It's just loosely defined.
Second, it's not important to know their task size to still know that two different (to them!) things shouldn't be shoved together.
If the two tasks fit together so well that it's a mere 5% extra code to do them both, then maybe they are two sides of the same coin. But if "one simple thing" is painful to implement maybe it's really two separate things under one blanket.
> often they settle on “one microservice per bounded context” where “bounded context” means “separated however I want it separated at the time,”
Right, because sometimes 'bounded context' is what Legal tells you about data residency and sometimes it's about optimal latency.
> Don't get me started on how “strong coupling” is shorthand for “coupled in ways I don't like”
Strong coupling is generally easy to define, relative to loose coupling, in any given language/platform. The value of engineering around it depends on the value of doing the thing which requires it multiplied by how often you have to do it.
I'm not sure about this. Some people might take it at face value and end up with dozens if not hundreds of services for a domain that doesn't quite need them. Ergo, the project that they work on might have the development velocity plummet, because it has too much accidental complexity and ops overhead.
That's how you get humorous parody videos like these ("Microservices" by KRAZAM): https://www.youtube.com/watch?v=y8OnoxKotPQ
As anyone who has ever seen SOLID or DRY taken as gospel, many people out there would probably agree with the claim that some general advice can definitely be dangerous.
You're worried that someone junior enough to run with random advice is going to be able to rearchitect their entire codebase into hundreds of microservices and deploy it to production without testing its performance/stability/etc?
I think they should try. It's a great way to learn why, and why not.
> many people out there would probably agree with the claim that some general advice can definitely be dangerous.
It's just mouth noises which aren't dangerous. We need to be able to hear other people's opinions (advice is just an opinion after all) without immediately adopting it.
No, I'm worried that someone senior will do that, because they want microservices on their CV. This has happened in the past and will continue happening in the industry with every new technology or approach - as I'm sure many have seen projects get burned due to being an early adopter of Kubernetes or Kafka, or even microservices in general.
As for how much testing should be done vs how much actually is done, you'd be surprised. While some orgs absolutely do things right, for a large part of them out there it's basically the Wild West - ship stuff and hope it works well enough. The entire e-health system of my country is a good example of this (though I cannot comment on the architecture itself): https://www-lsm-lv.translate.goog/raksts/zinas/zinu-analize/...
Some excerpts:
In December, the availability of e-health was disrupted for seven days (with one exception - on weekdays), but it actually did not work for five more weekdays. For example, on December 7, none of the e-health services were available for more than four hours (4 h 13 min) during the working day. January looks a little better - there are disruptions on six working days, while in February e-health is disrupted for three days, but for four days it can be said that the system does not work.
"If there are three, four interruptions for more than 10-15 minutes during an hour, then of course the user can be lucky to hit during the time when the system is working, but there are users who can only hit during the times when the system is not working, so I also marked red," explains Edgars Goba, deputy director of the National Health Service for information and communication technologies.
One of the users of the system, family doctor Ingars Burlaks says: "I have been working with e-health from the very beginning, since it appeared. Unfortunately, those problems that were there from the beginning, those chronic problems, like the system hanging up or getting kicked out of the system when you get a screen with an exclamation mark, those problems persist."
> I think they should try. It's a great way to learn why, and why not.
This is something I can get behind for internal pilot projects, side projects, or hackathons and such. This is not something that I can get behind for projects with potential contractual penalties.
> It's just mouth noises which aren't dangerous. We need to be able to hear other people's opinions (advice is just an opinion after all) without immediately adopting it.
As long as no adopting advice at face value and no cargo culting follows, sure.
I advise you to take all advice with a grain of salt. :) It's just what someone else thinks, it's not something magic.
> I'm worried that someone senior will do that, because they want microservices on their CV
Heh. I was going to say that the only way this advice is "dangerous" is if a point-haired boss hears about it. Someone padding a resume is the danger though, not the technology they choose.
> This is something I can get behind for internal pilot projects, side projects, or hackathons and such. This is not something that I can get behind for projects with potential contractual penalties.
Just don't merge it into the main branch.
> The entire e-health system of my country is a good example of this [...]
I'll check it out. Thanks.
Agreed, of course. Or maybe even suggestions that have some caution codified into them, such as "innovation tokens" from: https://boringtechnology.club/ Moderation in all things and so on.
> Heh. I was going to say that the only way this advice is "dangerous" is if a point-haired boss hears about it.
This is also true. I would definitely suggest that there are industry wide trends that urge people to rush towards "the next big thing" or sometimes adopt certain technologies or approaches because "others are doing it, so clearly we also have to". Perhaps a lot of it can simply be summed up by hype cycles: https://en.wikipedia.org/wiki/Gartner_hype_cycle
> Just don't merge it into the main branch.
The problem the way I see it, is that people don't just experiment with new architectures within the confines of an existing system. Typically they give in to the "second system effect" in regards to their former tech choices when starting a new project, which can have long term implications and choices that are really hard to change down the line: https://en.wikipedia.org/wiki/Second-system_effect
But hey, at the opposite end of that spectrum, maybe I shout at the cloud too much; exploring new stuff is certainly desirable - there were certainly benefits to being an early adopter of something like Docker or 12 Factor App principles, even though not without seeing what works and what doesn't in non-prod projects first.
Have you tried to advise them otherwise?
> Or maybe even suggestions that have some caution codified into them
But then every suggestion is couched in a ton of warnings. Not only is this a bit useless, but if someone does start to depend on it they're at risk of a suggestion which lost its warning. Like, say the warning runs onto the next page and that page got lost.
There are an infinite number of possible suggestions and all could potentially rabbit-hole a team to death. Why not embrace immutability? Why not go serverless? Why not migrate to on-prem hardware? Why not use a memory-safe language?
It seems far better to build some sort of suggestibility-immunity by listening to others, and their suggestions, until you can understand but not feel compelled to copy their ideas.
Orders are related to Fulfillments are related to Shipments. They are "coupled" in that an Order will trigger Fulfillment, and Fulfillment will trigger Shipment (there's a Payments service in there somewhere too).
It definitely is loosely defined and the rules do get stretched to fit opinions though.
also, with k8s, nothing like deleting the wrong object or making a change and not knowing what it was, N revisions ago.
I learned from using software like Photoshop and Ableton Live, that you shouldn't underestimate the complexity of any software you use.
Take a few days or weeks, if you can, to read docs or do high quality courses on the topic and it will make your life easier in the long run.
Not saying it's the way to go for every situation.
My point: a single dedicated server may be a more reliable, simpler and much cheaper solution than the cloud provides.
I literally do not care about any particular server. If it fails, it kills itself, and a new server comes up and serves traffic.
No alerts, no pages, nothing.
These things are inherent complexities in any situation with backup/DR. You've already got to figure this out. When one machine is degraded and another is trying to recover from db logs, that's a cluster.
> This decision should be made per-project
This is one of the least project-dependent things ever, akin to source control. If you don't have scripted installs you don't know what you're running and you can't reliably test it or upgrade it.
You don't need to run a cluster, or have auto-scaling groups, just because you can.
Depends on who manages the fleet. An EC2 instance provisioned by Elastic Beanstalk is quite easy, the problem is EBS can be an utter PITA of edge cases and weird failure modes.
In what context? Individual instances fail frequently.
I've replaced pets with ASGs that do nothing but sit there unless the EC2 instance fails. Its amazing because you no longer have to baby the pet, it just recovers in a nice clean state, same massive instance..
Pets are a artifact of laziness.
I was with you until this unnecessary point.
An R620 plugged into a switch in a colo, a bash script via cron, or a cloudflare worker are just fine for a lot of use cases. The only time it stops being fine is when you can't afford to do your pet -> cattle migration as you scale up. But I don't think this is a common death for companies.
If you call "cattle" a cloudflare worker or lambda function - fine. But when we are talking about multiple redundant servers with load balancing across them, you really need to justify the cost of that vs the value you squeeze out. Sometimes you're squeezing the juice out of the rind.
That got us the comfort of knowing if the host machine died in the night the task would be rescheduled without a bunch of extra engineering to be able to scale arbitrarily when we only needed a couple of instances running.
My real, pointed question is: what was the opportunity cost of cattelizing that vs any of the other work you could have done in that time? Because given a fresh instance but I'm allowed to use the rest of my infra, setting that machine back up in the morning should just be to instantiating the right docker container on the host and copying the license file in. Depending on what the vendor software does, I probably still wouldn't ASG that until it had actually happened more often than once a month - there are so many more things to stress about! Even if this were the splunk license checker, losing some log files would be quite disappointing but not the absolute end of the world - if you're losing api calls to stripe or the equivalent that's a different matter!
They are not, if they are being configured by hand, using mouse or cli
Doubled up as amazing training / onboarding material for folks that maybe hadn't worked intimately with AWS before (including myself).
> Git should be your only source of truth. Discard any local files or changes, what's not pushed into the repository, does not exist.
Second, don't fool yourself into thinking, just because you wrote a script or configured some provisioning service, that means you've pulled yourself out of the valley of nondeterministic unreproducible production environments. Pets can and do live in git. A majority of the systems I've worked on in both FAANG and non-FAANG in my career couldn't be reproduced from the source repo in the case of a catastrophic disaster (I.e. us-east-1 disappears) or a year of being stable enough to not deploy even though their "source of truth" lives in git. The world moves out from under your repo and many of the steps you take for granted when writing your automation aren't understood by the next engineer running the automation. Then one day you have to go upgrade this one repo and realize it's IaC is no longer compatible with the infra in the real world and you burn a day trying to figure out what's wrong with a 1000LoC YAML file.
> Then one day you have to go upgrade this one repo and realize it's IaC is no longer compatible with the infra in the real world
Thats yet another reason to avoid pets. When I say pets I mean relatively big pieces of SW, usually 3rd party, that need non-trivial configuration. Like Jenkins or Windows or some security gizmos.
While I'm all too familiar with this problem from the NodeJS ecosystem, it's rare to encounter it in Terraform in my experience. The only exception that needs manual intervention in a "cold boot" scenario is AWS Cloudfront using certificates provisioned from AWS ACM - the provider will assume that certificates are already valid when creating the CF distribution and error out if ACM hasn't issued the certificate yet.
Sometimes you can justify using a thing for the wrong reasons.
I recently attached 1x NLB to each of our Swarm clusters to migrate to automatically managed certificates directly attached to the NLB (Digital Ocean).
$COMPANY has maybe ~3 users accessing each production application at a time. So the NLB itself is utterly pointless.
But Engineering no longer have to fix the certificates each quarter after users see an insecure browser warning and email us about it.
100% worth it for $12 pcm (per swarm cluster).
Cattle is the best approach. Practice it and make it your default.
This may make sense if you can only afford a single machine you want to hyperoptimize but it’s not a good workflow for anyone. Not even BSD kernel developers.
Beyond the security surface that’s just asking to brick the machine; if you never make any mistakes it’s because you never tried anything very interesting with it in the first place.
Furthermore, I claim that “you can write a script to edit /etc” is not how an OS should support configuration. Instead of being possible to keep a copy of the config in a script somewhere else, it should be impossible to do anything but that. So basically /etc and UNIX are bad ‘cause they’re not declarative.
The important thing to remember is that improvements should happen gradually in small chunks; you don't have to have a perfect cattle system out of the gate, but you should be working your way there.
More likely than not, if they've got execution permission on your webserver, they can send down a compiler to run. Plus or minus a c compiler doesn't open up much (anything?) they couldn't have compiled elsewhere or done in the scripting language or shell code or ... Unless you're running a static website or something. Plus, it's presumptious that all my servers are webservers. :p Do my other servers (other than build servers) get to have compilers because they're less naughty?
> And why isn’t the official kernel good enough?
Too many drivers, not enough local patches that are worth having but not worth pushing upstream. (Going upstream is a good metaphor, it often takes a lot of effort)
> Beyond the security surface that’s just asking to brick the machine; if you never make any mistakes it’s because you never tried anything very interesting with it in the first place.
That's what the console is for. Serial and IPMI consoles are way more convenient than vga consoles, but I've certainly screwed up bunches of times and had to use a console to sort things out.
> So basically /etc and UNIX are bad ‘cause they’re not declarative.
So you declare. ;) If it helps, I ran the script from Make, which is declarative (although not really the way we used it), and we mostly ran Erlang, which is claimed to be declaritive in Erlang: The Movie, but seems to be useful, so I'm not sure. I've never quite understood how the goal of declaritive system management matches with reality; it feels like a very leaky abstraction to me; the part that changes the system is necessary complexity, and hiding it away adds additional complexity. Maybe if you build a read-only root filesystem declaritively and run that? Which you can if you want? While you're saying I shouldn't do something the way I did it, my team managed a bunch of machines and a complex system with very few people, so I think the method works.
But I think about things differently than a lot of other people; I'm comfortable hot loading code to change uninterupted services, wheras others like to move traffic to new servers (and hope it all moves). You could transfer existing connections to your new servers, and that would be a lot of fun (and probably require a custom kernel), but nobody does that. Instead you either serve old connections with old code, or forceably terminate old connections. If I'm going to hot load code, it makes an immutable system directly opposed to operability. Why would I be interested in making my system less operable?
Adding on: farmera don't kill their livestock at the first sign of injury or illness; they won't do heroic interventions like you might do with a pet, but they'll certainly do simple fixes on the live cattle in the hopes of returning to health and retaining the value.
Just the other day I had to perform some maintenance on a long-running VM hosting some monitoring software. A backup VM is supposedly always running and ready to handle the workload in case of downtime. The switchover seemed to go fine, at first.
Turns out, someone long ago had manually added a cron job to the primary server without adding it to the backup server, without documenting what it does, what permissions it needs, how it works, or why it's needed. This was only discovered after some manager in a different department complained that he stopped receiving the daily report to his e-mail inbox.
If whoever deployed the report generation script took an extra hour to document what the script did, or even better, added it to VCS as part of the provisioning process for the server and re-deployed the server to ensure that the process works as expected, a day's worth of headache could have been averted.
It may only be absolutely necessary there, but it's helpful even for smaller folks.
Over the years, even Debian LTS goes out of support and new features and software should be installed. There's moving systems, doing restores, things breaking and wanting to "reset" to a known working state. Any time you can do something simple with docker or even just (short) step-by-step build scripts, that's a huge win.
I have playbooks for deploying a system, but with npm installs, bower installs, secrets to be hand copied from multiple places, etc, it feels more like pets and it's NOT simple to deploy.
Monitoring/alarming, and knowing what to monitor. Also, properly instrument your services or whatever it is you have. Take time to reflect on what are the signals that tell you operational health. An error metric alone is useless if you don’t know the denominator. Also be careful to avoid adding noisy metrics that cause panic for no reason.
I’m not sure what fault tolerance means in this context. Very handwavy statement. I think if you have dependencies, have a plan and understanding of which ones tipping over will bring down your service or how you can build resiliency. For example, some feature on your page requires talking to a recommendations service. If the service goes down, can you call back to a generic list of hard coded recommendations or some static asset?
As for automation: yeah, have test workflows built into your CI/CD harness. And avoid manual steps there requiring human intervention. Use canaries to test certain functions are up and running as expected, etc
If one of the servers/pods fail, the process behind detects them as "unhealthy" (having a nice monitoring/alarming as you mentioned) and replaces them with a new server with the same software characteristics so, for the end-user, SO, your client. Nothing has changed, the load just moved to a single instance for about 5-10 mins until a new server was deployed.
Absolutely. I've seen so many junior engineers / devs go on about it like this:
Someone higher up: Could you please look at this problem? I need it fixed ASAP.
Jr. Engineer, presented with a problem he's never seen before: No problem, I will look into it!
Someone higher up (the next day): Did you fix the problem?
Jr. Engineer: Sorry, I haven't still gotten around to look at it / I'm still working on it / etc.
Someone higher up: We really need it fixed today, please prioritize it and give me a call when it is fixed.
Jr. Engineer works on the problem all night, feeling stressed out, not wanting to let down his seniors.
(Of course, there are several other ways to get 100 engineers to have the output of 10; but I think microservices makes the engineers a lot less frustrated when it happens compared to more traditional alternatives)
Can you explain this reasoning?
With microservices, you make 10x as much work in such a way that the work seems meaningful and is perhaps even fullfilling. It takes some effort to see that the complexity you are solving day to day is accidental complexity, not essential complexity.
To your original point though, correlation doesn't equal causation, and just because you could abuse a technique to placate your wrong-sized org using this technique doesn't mean that it's the purpose of said technique.
Seems like this falls under "doing work", not creating output. You're doing a lot of work.
> The monolith doesn't work well in multi-team setups..
There are other ways of splitting work between multiple people and teams without introducing a network call.
I could say the same of a monolith -- if it is a big ball of mud then it is not a Proper Monolith ...
Granted, my only experience with microservices was a ball of mud mess microservices. I am sure we will both agree they were NOT doing microservices correctly -- BUT -- they did not know that going in did they?
If microservices is defined by definition to only be about the successsful applications of the technique, then it is not very helpful for those choosing architecture for a new system..
Re shifting services to other teams:
We have a medium-sized-service architecture, not monolith nor microservice. One or two services per team mostly.
By far, the hardest part of onboarding and doing development is learning the business domain and have people understand WHAT to do, why to do it, etc. -- the HOW is usually simple.
Code usually covers only the HOW part. Just shipping parts of what we do to another team and expect them to be on top of things quickly doesn't seem realistic regardless of the size of services..
Teams are homes for business and domain knowledge first. A home for the code is just a byproduct.
It also depends on how much functionality you consider to be “one thing”.
I thought microservices is a solution to scale development teams, not for traffic.
If you have a horizontally scalable monolith, it can scale pretty much as far as you want. If you split services along functional boundaries (i.e., vertical) then a split from 1 to 2 services will in the extreme best case scenario give you 2x scaleup; further splits give you less. So: If load is the issue, work on horizontal scaling, not microservices.
What am I missing?
As you add more functionality to a system, it gains different workloads, which have different needs (e.g. different storage solutions) and need to scale at different rates (e.g. one part of the system is very memory hungry and another is very compute heavy).
As your system gets larger, it's usually more important that its various components are well isolated, both on software and hardware axes, e.g. container isolation vs isolating across different physical zones/regions. Sometimes that isolation happens for security/compliance reasons, e.g. PCI.
As your business grows, you will need to build functional solutions that require different toolchains and platforms. You will acquire other businesses that have built on different platforms. You will not require all of your acquisitions to rewrite all of their software into your monolith.
With extra work, you can solve for many of these problems within a monolithic architecture, but those solutions will typically introduce similar complexities as microservices (e.g. talking to many datastores, composing your monolith of many coordinating processes, configuring independent nodes of your monolith differently, provisioning independent nodes appropriately, etc).
My experience in software is that we draw false dichotomies between approaches and the reality is that they will tend to converge when you put in the effort to consider and address the shortcomings of each approach.
Microservices have their own challenges apart from network communication etc between medium sized services that yiu refer to.
For instance lack of easily available consistency, sometimes needing to basically implement multi-service transactions (see SAGA pattern) and so on... if services are a bit larger, such problems occur much more seldom.
At some point, you have servers, that say, pre compute complex internationalization logic, and therefore are almost databases, and you want to keep them separate. Other times splitting requests that have very different memory vs cpu requirements into different boxes helps. And this is just for performance/costs.
The scariest issues are all operational though: Suddenly my giant fleet is misbehaving, and a rollback means 10K servers get hit, and I am not exactly sure what is going on. Having the functionality separated a little will do a lot of good.
Now, from there to needing microservices is a pretty big gap, but the lone uberservice can get very scary under the wrong circumstances, and the probability approaches 1 as the codebase gets really big.
I am all for dividing services after technical requirements; and certainly any non-O(1) memory requirements for instance one may want to look at a dedicated service.
When we get into the things you say, the definition of "service" gets more important too. Is it separate DB, or only separate k8s pod or VM, or what... If the only constraint is to make computation efficient and robust, the solution space is bigger if one does not add the "should fullfill the microservices ideology" as another constraint.
With microservices on k8s I think it is pretty common for each VM to have many open connections to 15+ databases though, so not sure if I got that particular point.
I ~hate~ dislike this advice. If you can't deploy on a Friday, you need to fix your deployment strategy. By removing Friday from when you can deploy, you're wasting 1/5 of your available days.
Note: deploy != Release[1]. Use flags, canaries etc.
[1]: https://andydote.co.uk/2022/11/02/deploy-doesnt-mean-release...
Edit: hate is far too stronger word for this
Would that still be required if a deployment and a release are decoupled entirely, or is it unavoidable? Genuinely interested!
It's a good interview question as a candidate. If you ask the interviewer when they deploy and they say only Friday (or worse only once a month) then perhaps look elsewhere for your own sanity because it's a sign of serious malfunction either organizationally, technically, or both.
Don't discount a job because of one deployment per month---it really depends upon the service. I joke that a busy year for me involved four deployments to production, but "production" for me wasn't a website, or even a web-based service, but a service involved in the call path for phone calls. Our customer is [1] the Oligarchic Cell Phone company and the SLAs are pretty severe.
I do have to ask---where do you people work where you have multiple deployments per week (or even per day)? To me, that sounds insane!
[1] Still is, even though I left a few months ago, and not because of the lack of deployments, but the shoving of Enterprise Agile [2] on the company by new management.
[2] Which is anything but Agile.
The upshot is that this is fairly rare and we do not have an on-call rotation. If most anything breaks over the weekend, nobody is going to notice or care until Monday morning rolls around.
However, I will admit it is a trade-off; some engineering time does have to be spent to get there, and perhaps that engineering time is better used elsewhere right now.
> Deploy is the act of moving software into an environment and running it.
> Release is the process of making a feature visible to a user or subset of users.
In our case, we deploy often (10s of times per day), and make use of feature flags for releasing. Keeping an eye on data in Honeycomb before and after a deployment and before and after a flag is changed is part of the engineers job.
If something doesn't look like it's working as intended, ideally it is disabled with a flag, rather than needing to figure out how a rollback can be done (often it can't), or rushing a fix and roll forward.
Regardless of how confident you are on your feature-flagging functionality, everyone's going to be unimpressed if suddenly your "minor" deploy has caused a bunch of things to go haywire at 4:30pm on a Friday and now people have to stay back and fix things.
If some of that new code is a flag that controls what customers see, then from a product perspective it’s not a “product release”. Is that what you mean?
The point is, even feature-flagged code can break things unexpectedly.
If you can't afford to give up 1/5 of your available deployment days you have a problem somewhere in your CI/CD system.
The reason we have core hours release only without director approval (aka director approval required outside core hours) is so you don't piss off another team by paging them after hours, and so you aren't trying to shove out a thing on a system that doesn't have good coverage or by turning off the safeties. In a large company I've noticed many engineers assume urgency even where there isn't. As an approver myself, most of the time someone wants to rush is because they've not even had the convo with their manager on if it's worth the risk, they are assuming urgency because that's when the sprint ends or what some TPM added to a jira ticket 4 months ago.
I admit that sounds risky itself (the engineers not having the right risk training) but this is why we have a policy and tooling... most of the times I've dug in they're just very new and worried about perception as a new employee, so my job is to shepherd them through having that convo with their managers which inevitably has the managers saying "yes it can totally wait till monday", and the change is inevitibly a bit more hot than it should be due to accidental deadline pressure.
I do disagree with the absolute statement of not doing it, but I definitely do a risk analysis whenever I ship a change on Friday and avoid anything risky and just push it off until Monday morning.
Not all changes can be put behind flags and canaries don't really fix the issue (unless you are okay in blocking important fixes from being rolled out due to your bad change killing the canary)
If you are a small company, or you do not do extra weekend shifts, I understand your point. Elsewhere, you just want to live an adventure every Friday.
Thank you! So many people running unnecessary things on Kubernetes
There are distinct advantages to that in terms of both development (running a local K8S cluster is relatively easy) and deployment.
ECS has no distinct advantages over K8S (or EKS in AWS land). Particularly now that there are CRDs for K8S that allow you to deploy AWS functionality (eg ALBs, TGs) from K8S.
As far as FAAS goes, I think more people need to go check out Cloud Run as a Knative implementation. Having used it for sometime now it feels like a near-perfect FAAS solution. The only gripe I have is that versioning is a bit dopey. But hey, if I can have autoscaling services with absolute impunity over how my HTTP interface is shaped (looking at you AWS lambda) and without needing to worry about Kubernetes headaches, I’m perfectly happy to embed version names in service domains.
If PaaS, isn’t Gcloud itself the PaaS? For instance cloud run, the product inside Gcloud, is ephemeral and stateless, which wouldn’t be at all good for trying to make a DB.
I believe PaaS predates FaaS, there used to be only IaaS, PaaS and SaaS. In my book App Engine and Cloud Run are PaaS, while Cloud Functions is FaaS. OTOH, there might not be any super clean definition
Can anyone recommend some certifications that are worthwhile? I realize that this is a very broad ask, but the advise is also rather broad.
EKS?? EKS? EC2? EBS..? ELB..? Ah no way it was /data egress/ of course.
There's a lot of conceptual carryover between cloud platform offerings, so getting _any_ of the big 3 (GCP, AWS, Az) is likely to help you out a ton of you're new to the space. Much like how your first programming language took much longer than your second through fifth ones, learning your first platform well enough to get employed is much more challenging than filing the serials number off and learning the new quirks of the other two.
In the absence of further information as to your career goals, I'd lightly recommend AWS. It came into existence years before the others and can offer SLAs that approach "this S3 bucket will outlive you in the event of thermonuclear war".
Azure is where I have lived so far in my career and it seems to be catering more towards enterprise and government needs. I actually imagine finding an Azure shop is harder than an AWS shop if you haven't already worked at one before, but it's a pretty sweet gig otherwise.
GCP goes the other direction from what I've seen - much more startup-oriented, as the newest kid on the block itself. It looked nice from the last time I played around with it.
Kubernetes exists as a useful "stage 2" if you want to go further down the pipeline, as a technology whose business raison d'etre is to commoditize cloud providers. _In theory_ a Kubernetes cluster can be engineered to run unaltered on any of the big 3, since they all offer k8s clusters.
It's also totally cool to say that's okay thanks, I'll stick with simple architectures and a focus on getting MVPs out the door rapidly. For me $DAYJOB is spent between Azure and k8s, but my side projects start the same way every time - SQLite, Django, and _maybe_ Docker Compose to sidecar Litestream if I'm feeling extra infra-inclined that day. Really there's no reason to get dogmatic about anything in a space with so many options.
The number of issues I've seen that turn out to be documented features... (or, more accurately, things just being configured incorrectly)
I feel like this is spectacularly bad advice. "Do not get fooled by shades of grey, things are meant to be either black or white!"
I feel like this could used as one of those "How to 10x career" articles - and be better than all of them.
Completely agree with that.
I like to do short lived credentials using Vault (e.g. vault can create say db access credentials dynamically), but for things like API keys where I can't do that..? Is the Vault KV store the source of truth?
Edit: now that I think of it, for generated short lived passwords we also use SSM but for anything set by a human it’s in version control…
Probably.
Completely agree
> only
Fine, but can substitute "git" as appropriate
> discard any local files or changes
Ok for when deployment is completely and always automated, but for that _one special case_ maybe keep a copy of the old that you can revert to until you're _really_ sure of no unwanted effects. In the meantime, find out how & why that local change got made and what can be done to automate it next time
Have a good logging & rollback strategy well communicated across stakeholders