I have ran into scale problems very fast at Google and now am having to use App Engine and add more complication which I don't have personal engineering skill/capacity.
Google support first bumped me up to 12000 max queries per 100 seconds and said that's the limit, but that's not close to enough. We have super bursty traffic (webhooks that aren't concatenated from source).
Google then increased to 100,000 per 100 seconds after they claimed 12k was maximum.
It's still not enough and I think their sampling might be funky because when I look at the invocation/traffic graphs it doesn't align with the stated maximum (still throws quota errors).
Kind of frustrating. The idea of stateless functions without worrying about engineering is a dream! I don't get why Google can't get 'google scale' here. In the past we've used Snowplow on AWS when we had engineering help but that's a huge amount of work/engineering time. The actual accept POST, store to cloud storage, insert to DB is less than 100 rows.
Response time is also pretty bad in the extreme cases. Even though I end the response before processing data I still see 95%/99% spikes when bursting on the far less bursty data source (as in only bursting to 10/second). Just today for this function I got 3764ms 99%!