cbushko

That is an interesting way to do it. Per project would allow you to set give each developer all the permissions they need.

The worry I have with this is that as you grow you will eventually end up with a bunch of dead projects. You need to cleanup that list every so often as employees come and go.

We have a dev project so they could send their logs there.

I really enjoyed this article. I will assume that logs are being sent to a dedicated GCP project.

Can you shorten the retention of logs and traces in Stackdriver? If so, that would be amazing and make this practical for all dev use.

Base64 is your best friend!

You can setup NextDNS at the router level.

I did not see it mentioned anywhere but does this have the same performance problems as Docker Desktop for Mac?

What DB are you running?

They are similar skill sets but the domains are different. I am not expected to understand every javascript framework that is thrown at developers and they are not expected to know the inner workings of our networking, kubernetes or service meshes. I will concede that every company wanting an Ops person to be a full fledged software developer is a little ridiculous.

I have been at this for 25+ years and I remember the days of silos. I remember developers passing off code to QA and it coming back days later with bugs. I also remember a lot of 'not my problem' coming back from developers. Sometimes it wasn't their problem; often it was.

Either way, the person with intimate knowledge of how the code works should be the first person that looks at the problem. In SaaS, that is production and should be the developers (within reason).

As Ops, my goal is to make that as easy for possible for developers. That means automating everything I can so that the right tools are in place for deploying, monitoring and alerting are there. It means that I have to automate spinning up and destroying infrastructure as quick and easy as possible so that we can meet your needs and also keep costs down.

I have also seen companies fail at becoming 'devops' in the most terrible way. They took developers and made them own everything from code to deployment to VMs. The developers had so many pieces to understand that the only guarantee was failure. That was a terrible startup to work at.

I'd also add that Ops is not automating these problems away.

And this only works if your Ops team provides good tooling for deploying, logging, monitoring and alerting.

I have heard of companies deciding to do 'devops' and it turns into a free for all of dev teams having to handle/build things end to end. Everyone loses in that scenario.

I am sincerely curious why you feel this way about DevOps.

I think it comes down to passion and curiosity. Those are two things can happen at any age and education. It is also something that ebs and flows based on energy levels.

I am in Ops and I can understand this point of view. The author is probably overwhelmed with issues that are 'not his problem' and this is his rant on it.

To be fair, Developers are getting slammed with their responsibilities too. At one time it used to be that they could just know one programming language really well, like java, compile their code and hand it off to QA.

Now they have to know a dozen languages, frameworks, do their own testing, deploy the service, monitor it and trouble shoot everything in production in some 'cloud'.

Or they are just being lazy and this guy is sick of it. That is when you do your best to train people up and get them to put in the leg work. Ask pointed questions about if they Googled the error and help them work through the problem. Then add some things to the docs to help others out in the future.

Awesome catch!

Their pricing for GCP is way off. If you were looking for an instance with 32 GiB of RAM, you do not have to buy one with 40 CPU. There are other options such as:

> n2-highmem-4 4 32GB $0.262028

Which turns out to be the cheapest price and that is not even preemptible which is $0.06356

The only datapoint I can give is that moving from AWS to GCP reduced our bill by 2/3rds. That isn't a 1 to 1 comparison though as we did a re-platforming at the same time.

Moving from AWS ECS to GCP Kubernetes was probably the biggest money saver. It reduced the amount of compute needed by a huge amount. Being able to safely use pre-emptive VMs in Kubernetes is also leads to huge savings.

I am sure that if we used Kubernetes in AWS it would have had a similar cost reduction.

So expect the next version of Spiderman to be based on a mRNA spider.

I too had some fun getting a nice neovim setup going but in the end I just wanted to get work done instead.

I settled on https://github.com/LunarVim/LunarVim and then I customize my lv-config.lua with the extra pluggins/settings I want.

I really like 99% of everything with VSCode except, 1) it kills my CPU and 2) the terminal experience is terrible. I spend all day in a shell and I want to move around quickly.

Ageism is a thing but I don't think it is always due to the fact that you are "old". Companies turn old people away because they do not want to pay for "experience".

Why pay twice as much for an experienced developer when you can get a fresh new grad that is "just as productive"? Companies often ignore the fact that there is an experienced Senior Developer sitting in the corner telling that productive Junior Developer what pitfalls to avoid and how to get things done.

The Two-And-Done rule is fantastic and I would take it one step further.

I am correct, A LOT, when it comes to technical decisions, strategies and the Right Thing To Do™ but that does not mean that my decision is Right For The Business™.

I do not have the full view into what my others see.

  - I do not see the CEO getting pressure from investors, board members or customers. 
  - I do not see the Head of Development dealing with not having enough people to get work done, budget issues or other critical issues that I am not involved in.
  - I do not see the Development Manager dealing with people that are having struggles and thus not getting enough work done.

Keep that in mind when someone says no to your perfect idea. Odds are you do not see the entire picture.

I see DevOps as a culture shift that is working towards a culture where developers can easily build, deploy, monitor and alert on their code in production. It is a direct counter-attack to the pattern of developers throwing their code over the wall and hoping that a sys admin will deploy it. You need expertise in tooling and processes in order to achieve the culture shift, that is where these 3 positions come in.

I see each role as an evolution where you go from DevOps -> SRE -> Platform Engineer. Automating the infrastructure and tooling is the catalyst. You can have any of these roles in your company based on how far along the path your company is, how on board management is and how interested your development teams are in growing/learning these practices.

DevOps writes scripts/code to automate CI/CD pipelines, creation of infrastructure, deployments. Once automated you move onto..

SRE writes scripts/code monitoring, alerting, SLOs/SLAs, incident processes. Once automated you move onto..

Platform Engineering where you write the tooling to make all of the above codified. Tooling could be anything from cli tools to kubernetes operators.

I say this as someone that has spent the last 3 years automating my way towards platform engineering.

Manitoba Nurses are on shaky ground too as they haven't had a contract for 4 years.

That is painful for sure.

Would running something like the docker proxy below on an old laptop help?

  TLDR: A caching proxy for Docker; allows centralized management of (multiple) registries and their authentication; caches images from any registry. Caches the potentially huge blob/layer requests (for bandwidth/time savings), and optionally caches manifest requests ("pulls") to avoid rate-limiting.

https://github.com/rpardini/docker-registry-proxy

It seems to me that the concepts of high availability, scalability and resilience are things that are just not taught or emphasized at most companies average at all. As Ops, this is a daily battle of education and prod readiness checklists.

It is even the simplest things that are ignored such as:

  - what happens to your db query when you have 1 million records?
  - what happens when the cache goes away?
  - do you even use a cache?
  - why are you treating your cache as a DB?
  - can we run 2 copies of your service?
  - can your service scale up and down?
  - does your service have state?

Kubernetes has been a boon for us in this department. Since pods can be moved/deleted at a moments notice we have had to make our software resilient and stateless. Yes, kubernetes is complex at first but it pays off hugely when you are no longer getting paged because 'server X hung and needs to be rebooted'.

I definitely agree and I have seen this. I can understand being risk averse to change and unknowns but debt and cognitive load should be a consideration too.

This is why teams creating a billion microservices because it is the pattern of the day gets me a little angry. It is just another unused thing to watch out for.

Sometimes yes, sometimes no.

Is it really junk or are we lacking in time/money/tribal knowledge to understand how it deeply works?

I like to think that I am a Very Good Engineer™ but I am sure that I would produce a similar piece of "junk" if I was under the same lack of requirements, training and time.

  who wrote this garbage??? 
  > git blame
  oh crap, it was me 2 years ago!

I agree with you that most developers do their best work after 6-8 months and that developers get less productive after 24-36 months but for totally different reasons. Developers don't get 'lazy', 'bored' or 'comfortable', they get overburdened by the company.

It is common knowledge that the average developer leaves after the 2 year mark. When that happens, everyone else is left holding the bag and they become the new experts.

Companies are full of self created abandonware.

Another person leaves and it is now two extra things that the vet has to become the expert on. Then three, then four and on and on. Next thing you know, your vet isn't getting any work done because they are holding too many bags. As the expert, they are in too many meetings trying to keep other people productive. They are pulled into too many critical issues and they are trying to mentor new developers in your ever revolving door of new people.

But they are unproductive

No. They are keeping your company afloat and you aren't compensating them for it.

So far it seems to be ok. My stack is 1 customer per namespace and I am running it across around 100+ namespaces, 3 keydb pods each. Some customers are very bursty - I don't have an RPS - but they are holding up for us.

I would still only do redis as a single non-persistent cache if it was up to me.

My company is staying fully remote with 'hub-offices' for those who want to get together. I plan to stay working from home even though the office is an 8 minute commute. I like my co-workers but I just want to work on my craft and build things. Commuting and hanging out at the coffee machine wastes time that I could be doing things.

My company is also experimenting with a 36 hour work week based on the Icelandic Study. As per the study, in order to maintain productivity, organizations need to clean up their processes. Specifically targeted meetings, more documentation and less interruptions. I am sure my coworkers will also be more productive just to keep the shorter work week - I know I will.

8 hours in an office != productivity. Quiet time and the ability to do deep work = productivity. I wish more managers and execs understood this.

For my own productivity, the company that I am looking at for inspiration is Hashicorp. https://works.hashicorp.com/ looks to be one of the best documents on how to do remote work. If anything, it is all well thought out.

I spent months trying to get some clustering solution of redis running in kubernetes. Every single solution was a huge hack such as running a HAProxy in front on it and having that point to the master.

The best solution I found was running keydb in a multi-master, multi-replica mode. All of the pods are masters, any pod can be written to and the keys will be copied over to the other masters/replicas. Performance is decent too.