ruben81adelaide

The things started to go from bad to worse when work_mem parameter has not been set up correctly.

Some queries that requires a large amount of memory to process started to use disk.

Once the autoscaling kicked in, even if we would have realized about it, wouldn't have helped as you are locked out of the system.

The auto-scaling event was triggered earlier than our alerts for low disk available.

What I don't know yet, is that we have been moving services for multiple days, and the service that required the work_mem parameter was in production in that AWS for 48 hours before started to use disk rather than memory to process the SQL Queries.

@fxtentacle

I agree with you. This actually makes a lot of sense. Probably the title reflected the anger and frustration that I was feeling after the poor support.

Actually the company that is being migrated to AWS is quite large, but not large enough to get in spending $1+mio annually. This migration was one system from many others that needs to be moved.

But, still, 10+ years ago the support in AWS was available for free to everybody, you would get in contact with a big nerd that knew the ins and outs of the system and would help you massively on pretty much anything.

Slowly, they allow just paid support, and now not even paying listen to your problems.

At the very beginning, it was the small(-ish) companies (which a few exceptions) which gave name to AWS, and an army of geeks talking all positive about the services.

If now they just work with large corporations, it is a big (and risky) change, which goes against the Amazon Leadership Principle "Customer Obsession".

@evil-olive

Thanks for your feedback. Aurora is something that we would definitely look in the future, but with the levels of workload that we are currently experiencing postgres RDS should work for quite a long time.

> CockroachDB or Cassandra or something similar.

I can't wait for a business use case where I need something like this :) but unfortunately right now this is not the case.

> run out of burstable IOPS but RDS is still partway through a background mirror operation.

This is a very good point. Will check.

> If you have RDS Enhanced Monitoring turned on do look at these metrics.

We don't have it enabled yet, but point taken.

> This is where having an AWS TAM is helpful - you can get them to re-esculate the support issue for a second opinion.

We currently have "Business". I requested support to escalate internally and he said that he could not.

The difference with the next level would be "Enterprise On-Ramp" ( https://aws.amazon.com/premiumsupport/plans/?nc=sn&loc=1 )

By checking the differences, looks like if we move to the next support we gain the following stuff:

- Business-critical system down: < 30 minutes - A pool of Technical Account Managers to provide proactive guidance, and coordinate access to programs and AWS experts - I think that we have already this service via our account manager. - Concierge Support Team - no idea what is this. I will check, but if anybody has any option whether this is worth it from experience, I would like to know.

> Does Cloudwatch confirm that RDS "needed" to scale?

Actually you pointed out to a clue that I missed. I should have checked cloud watch!

The free space graphs shows that at some point something got the 200 GB of space that we originally assigned.

This is a good clue that I am going to dig in.

Thanks for your feedback!

EDIT: Cloudwatch metrics in RDS has been the key to find the source of the issue.

@synicalx thanks for your feedback.

I will see if there is anything visible in Cloud Watch.

> - Other than the (presumably) one change made to that RDS instance at ~4.30AM, were there any other changes made to it, specifically to its storage, prior to the autoscaling event?

At 4.30 the following has been logged: "Storage size 999 GB is approaching the maximum storage threshold 1000 GB. Increase the maximum storage threshold."

However the auto scaling event started at 10.17.

> - Had this service been tested on RDS prior to the migration being performed?

We have performed a dozen migration simulations in our Sandbox Account in multiple weeks. We developed scripts and automation to make the actual migration. The only difference in the Sandbox Account is that the RDS database was smaller in CPU and RAM.

> - Were any other changes made, that may potentially effect your DB? For example a query being changed or something of that nature.

I will double check with the team, but all the migration was fully automated with scripts. I have not been reported any action required outside executing the automation scripts and performing the plan.

@tus666 thanks for the tip. "Collapsing Under it's Own Weight" is more accurate.

There are 3 types of programming languages you can learn: Modular programming, Object oriented, and Functional programming

Python- it has a very simple syntax and it’s easy to learn. It can be used for modular programming and object oriented and it is forgiving if you don’t strictly follow the best practices. Therefore it can help to make thing working while you’re learning

From my working experience in Spain, "error" is a more generic term. The word "bug" is commonly used to specify an error in the software.

Back up with a fix: https://twitter.com/DOStatus

You would expect that a company as big as they are to update their status page as well...Anyhow, at least they communicate via twitter.

Yes, they are down again. They had some downtime a few weeks back as well in the NYC3 region.

Yes, I monitor about 1000 servers in Digital Ocean in New York, London and Singapore. Only New York went down for about 1 hour and a couple of hours later the global DNS system in DO went down as well for about 15 min. They initially didn't made any update in their Status page

They don’t mention other benefits rather than just the pay package. And nowadays, not allowing remote work.

If the containers are truly inmutable, the application has been designed for horizontal scalability and the automation is right, it doesn’t make any difference to manage 5 containers or 1000 containers.

The same applies for virtual machines. The difference is that it is usually easier to accomplish this with containers.

I’ve pretty much been working non-stop since I was 18. I started self-learning when I was 14 and found a job as a programmer a few years after. I’ve only taken a long break when I decided to quit my full time job to create my own company. During that time I pushed my limits learning new technology and finding new ways to improve my code. After all, my business really depended on how good I developed the technology. So I think is worth taking a break to accomplish any objectives you have in mind: studying, travelling, experiencing new things, etc. It’s just one year of your entire life. You deserve breaks once in a while…

In my professional lifetime, I have developed several technologies. Most of them as a hobby, and about 90% become fully functional and production ready.

Some of them got sold; some others went nowhere. Overall, those projects have not made me much direct income - I can develop technology but I am terrible at selling stuff (actually, I am not so sure if I am bad at selling because I have zero interest, and the effort that I made to sell is quite proportional to the interest that I have).

However, all those projects allowed me to have a better understanding of many different facets of software development, and my actual area of expertise is quite deeper and broader compared with other professionals of similar background and years of experience.

It has also allowed me to get to know a lot of people, become a technology speaker, and perform many risky experiments - which I wouldn't dare to do when I am hired in another businesses - which provides me a unique insight on how to actually develop technology (for example, all those experiments confirms that if you are not using data driven approach to make decisions you are being massively inefficient).

Eventually, I decided to go full on with one project related to infrastructure automation (this topic is really cool!!!!). This project evolved to consulting services. The interesting part is that this project is quite interesting and well put together (although sometimes a bit hard to get started) but once you have set it up it is amazing, as I have confirmed with many consulting clients. But when I think about actually selling it, the excitement collapses quickly. I just tell people about this solution when I am requested to solve the problem by the client. (if anybody is interested the project is manageacloud.com, although there is a price structure I am actually not charging anyone).

Did I waste time with this project ? Well, it provided me with a direct income. Allowed me to show expertise to speak in several big technology conferences, and I got around 15 (yes, unfortunately not kidding) interviews with Google where somebody internally got the position(s) Google suggested to me until those interviews were becoming routine. I have managed to work from home for 6 years with consulting services, in very interesting projects, in huge business and well funded start-ups.

Working from home has allowed me to do some crazy stuff such as become black belt and fight in full contact international competitions. This would have been much harder with a 9 to 5 job, without the freedom to work whenever I can.

Are you thinking of making a project ? Throw it all away and go full on for it? Just do it. Why not ?

What will you think of yourself 10 years from now, knowing you had the opportunity and didn’t do it ? Asking yourself what would have happen? I cannot live with this thought.

What is the worst that can happen? Well, I am going to rephrase: How hard would it be for you to find a position similar to the one that you have right now?

Go for it. Well, actually, if you don’t have debt go for it. If you have debt … then it depends.

I agree, Vanguard is one of the best funds with low commissions. Nowadays there's no point putting your money in a bank account, so I looked into ways of investing the money. After carefully considering different options, I decided this was the best way to proceed with the minimum risk and with a good return.

Every month I transfer a bit, without looking if the stock goes up or down. The best goal is to invest intelligently for the long-term. Broad-based index have proven to be a good choice for long-term.

Try the CCNA course of CISCO, “The Complete Network Fundamentals”. I did one of those years ago. It’s a great course to achieve a good base of networking. CCNA is just the first step to a good introduction, if you like it, you can expand your knowledge with more.

It depends on the company culture and its maturity towards software development.

Good PM if they know the product inside out and they have a good insight on how development works provides a lot of value.

However there are some companies that believes that software engineers are “second class citizens” and roles such as Project Managers and Product Owners just gets paid more without any clear justified contribution.

For me this is a red flag that the company has not realised yet the real value of the software engineers.