AWS Is Asleep at the Lambda Wheel | Hacker News Reader

AWS Is Asleep at the Lambda Wheel

lastweekinaws.com

97 pointsmooreds34 comments

This is a great article, and sums up my team's frustrations with AWS Lambda and the AWS engineering leadership in general.

I worked in the past under a slew of former AWS product managers, and I know from them how the AWS roadmap works:

1.) Real, crucial customer feedback is incorporated into roadmap, usually for the core services that are already used extensively like Lambda. 2.) Executives have new, novel, sexy flavor of the month projects like "we need to catch up with that ChatGPT thing my kid was showing the Au Pair the other day!" 3.) Product Management leaders are incentivized to prioritize said new/novel projects, and proceed to do a worker crushing sprint to build what any sane person would call a PoC. 4.) PoC is barely tested and pushed out as a half-baked garbage product. Garbage product languishes. (See AWS Data Science Studio for a great example of giant, useless POS)

Meanwhile, developers are siphoned from real projects like, gee, I don't know, getting Python 3.10 out on Lambda? Or how about when you launch a sagemaker notebook instance you still top out at Python 3.8???? And don't get me started on recent versions of CUDA being available on GPU instances.

My team is relegated to docker containers deployed to lambda due to the antiquated, ridiculous limits on layer sizes for any of our ML oriented lambdas.

I have a buddy who works at AWS and talks about how utterly inefficient they are due to horrific leadership. At this point, you have a choice between a few highly dysfunctional cloud vendors. Google just SUCKS, Azure is a blatant financial rip off, and AWS is always developing new features nobody wants while ignoring the ones we do.

Before my current role, I had to juggle roadmaps and integration partnerships with the Big Clouds. It's insane how much they ship their org chart but also their internal politics?

I am convinced that some of the teams there are really trying to angle for releases that ship for PR rather than the feedback. (Although some AWS PMs really are trying despite everything else.)

What are your issues with Google cloud?

I had a horrific support experience due to a product (it was a predecessor to Colab that they eventually just cancelled) that was (due to a bug that they eventually fixed) triggering the startup of expensive instances randomly that cost hundreds of dollars a week to run. I couldn't get any help, until I eventually out of desperation found a BigQuery evangelist on a Reddit forum who escalated my issue.

Venting rant below:

It's blatantly obvious that Google's culture is completely broken, and they are incapable of hiring people who don't think they're too brilliant and amazing to be customer support. They care about their advertising printing press and everything else is secondary.

Tensorflow was the first out of the gate, quickly got its ass handed to it by PyTorch, and then they just imitated Pytorch, but in a crappy, crowded API kind of way. Angular is hilarious in a sad, pathetic, "wow that guy still thinks he's doing something useful, doesn't he realize nobody cares" kind of way. Dart..... well, I don't even need to say anything more. I bet there are people on HN who deeply regret investing time in learning it. Colab: "We have paid plans! If you pay, it will be only semi-useless!" (I tried the paid version, utterly useless compared to Paperspace)

If anyone from Google is reading this, sorry, but you work at a company that is rotting like early 2000s Microsoft. Even my highly non-technical wife was asking me the other day if I had noticed how shitty the search engine is now.

BTW, am I the only one who bought a Google Home speaker and used to listen to Youtube podcasts on it, only to have a software update make it so that now I can't even use Chromecast to send any Youtube video's audio to my speaker? Really awesome Google. First time in my life I've purchased a product and had it literally lose features a year after I bought it. Also, Google Meet sucks, and has become increasingly bug ridden over the years.

> BTW, am I the only one who bought a Google Home speaker [...] and had it literally lose features a year after I bought it

There are tons of other features that are rotting on Google home/assistant. It used to be that one could set a location based reminder on the assistant ("hey Google remind me when I get home to do the laundry") but they removed that, for example.

Whooowee, pull up a chair and grab a drink

Their IAM is a raging tire fire. Disturbingly stupid for such an advanced organization. I'm on my phone so the rant will have to be left at that for now, but people can bitch about AWS IAM all they like, at least it was designed and thought out versus the GCP "works on my machine, ohh I got promoted now it's someone else's problem". Words fail me how angry GCP IAM makes me

I know this isn't a "cool kid" thing to say but the lack of CloudFormation makes deploying into other people's GCP more painful than necessary, which has an intersection with my IAM comment because Role bindings are authoritative (in both gcloud and terraform) so oh, you want to add a member to `role/editor` but leave the rest of the members alone? Tough! I had high hopes for Deployment Manager but it doesn't support hardly any of the resources and there's no one click URL to roll out the yaml anyway

And congratulations, you just created your first Project in GCP: now you want to run Cloud Functions? Well prepare to spend the next hours of your life turning on the necessary APIs for that project because unlike AWS where everything works to your credit limit, GCP is that blank room in the Matrix. For ever project, every time. It's like GCP has amnesia. What APIs are necessary, you may ask? Who knows, just try something, read the 403 it gives you back, rinse and repeat

From my experiences working there, AWS is a feature mill. They've built their entire sense of self around shipping features. Stuff for Jeff Barr to blog about (that's no diss on Jeff, I think he does an awesome job). They can't get off that horse they've decided to ride because they're concerned if they do they'll be seen as having stopped innovating, but they're also not willing to provide headcount to meet both innovation goals and properly look after what they've already got.

> that's no diss on Jeff, I think he does an awesome job

Honest question: How is Lambda runtime version rot acceptable to an "awesome" top exec like Jeff Barr? Is someone else demanding he siphon resources away from maintaining key services? What data-driven metric says maintaining service quality shouldn't be continually prioritized?

I'm open to any explanation, but the logical conclusion I see if the lack of prioritization only really makes sense if internally, Lambda is considered a gimmicky toy. Where is the "Customer Obsession" mantra here?

A large portion of the most valuable work in software is of the boring, unsexy variety needed to keep key pillars working smoothly.

They’ve provided custom runtimes which removes the need completely for managed runtimes giving you whatever version you like. They also did it in a way with Docker that’s simple and powerful.

If I was an AWS exec I’d say managed runtimes are legacy, with the introduction of containers move to containers for new deploys.

I’ve been critical of Lambda in the past and critical when people use Kubernetes/Docker because it’s Kubernetes/Docker. The container support in Lambda has been a game changer, it’s one of them now they’ve done it you think why didn’t they think of that earlier, it’s a great fit.

> If I was an AWS exec I’d say managed runtimes are legacy, with the introduction of containers move to containers for new deploys.

If they do that they remove one of the key benefits of AWS Lambda and that's that customers don't have to care about the underlying operating system and just have to care about their function code.

And I'm sure if they'd would look at the numbers of how many AWS Lambda Functions use zip packages vs. Docker images they'd see that something like >90% use zip packages. Based on that they'd be insane to deprecate the use of zip packages.

Of course 90% will use zip packages, container support is relatively new in the lifetime of Lambda.

Deprecating zip packages doesn’t mean zip packages no longer work, it means pretty much what they’ve done, ignored the managed runtimes with the view if the current runtime doesn’t do what you want, make your own with exactly what you need using tools you are likely already using with no vendor lock-in.

For things like Python most people don’t care about the underlying operating system from the projects in multiple orgs I’ve worked in. They go to Dockerhub, type Python, pick the image with the highest downloads and run pip install inside it. If you use Go, Rust, anything that creates a static executable you use “FROM scratch” with the only thing inside the container being the executable.

Those who do care about the underlying image are likely large orgs with a set of already blessed base images or people using things like Nix to build containers.

From a large org perspective and developer experience getting a “registry” for free because packaging as a standards compliant OCI image means instead of effectively creating your own registry on top of s3, you get a far more robust one with tagging, asset hashes, layering etc out the box. I never had a issue with zip archives but containers removes a bunch of roll your own boilerplate registry.

This sounds like it would fall under

"Earns trust"

For the Amazon leadership principles.

(From an Amazon cdo engineer looking over at aws, nothing official of course)

Developers gain trust in AWS lambda by having it support the current runtimes, and keeping the old ones running for a long time. Lambda will always run the functions you want it to.

For some tension:

When I've taken free beers to chat with the lambda folks, it seems like they have a linearly growing oe burden for each new runtime version they support, and at least 4 or 5 years ago, I couldn't see a path towards avoiding that

I mean has AWS ever been easy to use outside of S3? If they were simple, companies like Vercel and Heroku would not even exist (both runs on AWS infrastructure).

I feel like SQS, API Gateway, DynamoDB, (largely hidden) SimpleDB, and vanilla Lambda and EC2 are pretty damn easy. (IAM is admittedly semi-baroque layer holding them all together. I struggle to call that easy.)

The complexity and amount of AWS-specific knowledge required to efficiently use it is pretty much absurd.

Even though AWS offers multiple different solutions (AWS SAM, Elastic Beanstalk, AWS CDK, AWS App Runner) that in theory should make consuming AWS services easier, these options still assume you understand and know how to use AWS.

In my previous job, I've been tasked to create an abstraction that will make using these AWS services easier and more developer-friendly.

I failed.

Yet I realized how hard it is, and how many people/companies are struggling with it.

That's why I've founded https://stacktape.com/. It makes using AWS simple to a point where any developer can efficiently use it. Let me know what you think.

> Meanwhile, developers are siphoned from real projects like, gee, I don't know, getting Python 3.10 out on Lambda?

Why isn't competition putting pressure on AWS? A cursory googling around seems to show that Google Cloud Functions is at Python 3.11 and Azure is at 3.8. Which then makes me wonder why people don't use Google Cloud more? And why would anyone use Azure at all?

[0] https://cloud.google.com/functions/docs/create-deploy-gcloud... [1] https://learn.microsoft.com/en-us/azure/azure-functions/crea...

> Google just SUCKS, Azure is a blatant financial rip off

If AWS is the least worst option, and also the most popular, what motivation do they have to improve?

I’m a very happy GCP user. Cloud Functions are ass but Cloud Run is amazing, and as far as Python versions go, if you can make a Docker image of it you can use Cloud Run. Scale to zero. Automatic load balancing. Any Docker image. Yes please.

> We’ve been able to use custom runtimes ourselves for a while, and via the magic of Docker container images we can basically make Lambda run whatever the heck we want it to. Unfortunately, building our own Python runtimes at home isn’t sustainable as a customer

Custom runtimes with Docker images solves so many issues with Lambda. Artifact management for free and the ability to run whatever language/applicaiton you want on Lambda. It works well, it's simple, it's portable. I can run my containers I deploy to Lambda on Lambda, Kubernetes, my local machine, pretty much anywhere if I abstract the AWS services i use.

Building a Python Docker image isn't sustainable as a customer? I don't understand that as most people package apps as containers already as a standard unit of deployment.Most large companies maintain their own base images. It looks like the only in-convenience for Python is doing `pip install awslambdaric`

With custom runtime Docker images I think AWS was smart, removing AWS maintenance as the limiting factor to let users have pretty much full control of the runtime, fairly trivially with tools they already know.

Totally agree. I wouldn't be surprised if Amazon is viewing the premade runtimes as a legacy option, with containers being the default (and eventually only) method in the future.

This is the way. The only drawback of docker on lambda is slower cold start times (which may have changed since I built on lambda)

I feel like AWS is one of those things where you gotta stick to the good parts. A lot of stuff is half baked but there’s a decent slice of it that works quite well

Props to RDS team - they keep those databases updated. Anyone here who can figure out how far behind they are in months cs released db versions - for my workloads I’ve been impressed

Python 3.11 is ~25% faster… Lambda is billed based on compute duration… if you are AWS PM Lead, it’s a challenging eng investment to justify.

Jevons Paradox, though

Disclaimer: I work at AWS in Professional Services.

I can honestly say that I have never been pressured to create a more expensive solution for a customer. Everyone from sales, to account managers, to TAMs are are always trying to figure out how to help customer spend less money on their existing workloads.

Before anyone says I’m drinking the kool aid, I’m 49 years old and this is my 8th job since 1996. I’m the most cynical person you can imagine.

Now there is a difference between wanting customers to spend less on existing workloads and helping them optimize and trying to convince customers to spend more money by bringing in new work.

I've been an AWS customer since the thing before S3 I think was called SQS.

And whether I've been small, or massive, ProServ and TAMs always always did what you're saying.

When we were big enough, you'd even (no joke) get in an actual automobile and drive to us, to yell at us about wasting money.

It was great you did that.

With dogfooding, it's a pretty easy argument

-non-aws Amazon engineer

It definitely feels like Amazon wants everyone to use lambdas for everything in the non-aws world (opinions my own)

It's actually pretty easy:

> Customer Obsession: Leaders start with the customer and work backwards. They work vigorously to earn and keep customer trust. Although leaders pay attention to competitors, they obsess over customers.

>Frugality: Accomplish more with less. Constraints breed resourcefulness, self-sufficiency, and invention. There are no extra points for growing headcount, budget size, or fixed expense.

> Deliver Results: Leaders focus on the key inputs for their business and deliver them with the right quality and in a timely fashion. Despite setbacks, they rise to the occasion and never settle.

https://amazon.jobs/content/en/our-workplace/leadership-prin...

Ran into the same issues with Node. 18 only took 7 months for them to support but 16 took almost a full year. For a paid service this is simply ridiculous. Maybe they can't be ready day 1 (though I sort of think they should have people following node/etc progress and be testing leading up to a release) but 11 months (heck, even 7) is way too long to wait.

This was made much more painful by the fact that M1 macs didn't really play super nice with node14 for whatever reason so my choices were 12 or nothing for a very long time.

Possible explanation: lambda just isn’t profitable enough.

Many workloads that are sporadic / rarely invoked gravitate to lambda.

Id wager that the cost it takes to build and deploy a lambda and manage its logs and security and storage are >> the revenue from invoking it 24 times per day.

+1 ran into the outdated Python Lambda runtime too. Nothing to add, OP said it all.

Seems like a pretty strong case against “serverless.”

There’s always a server. You just have zero control over this one. So you’re stuck defactoring code to run an older version of Python because you can’t upgrade Python.

It would be a case against server less if Lambda were the only option. You’re always free to setup your own container which would still give you control over all of that.

With managed runtimes it used to be you had no control. It was an opaque black box with complete vendor lock-in.

Now it supports Containers it’s a place you can run any compute you can think of as long as it finishes in 15 minutes, uses less than 10gb of memory and you want to trigger it using http, direct lambda invocation or from one of the various aws event sources such as s3, sqs, sns.

I used to avoid Lambda as it was an opaque black box. Now it’s a generic compute platform for short lived jobs using container support.

Reply on news.ycombinator.com