Very curious how others have fared, though.
Very curious how others have fared, though.
There are 2 major issues that we tend to run into on GKE:
DNS Problems: most likely self inflicted, but they are a pain and can take a while to troubleshoot.
Google Cloud Platform Quota Increases take FOREVER: as far as I am aware Google refuses to allow customers to set organization wide quotas for new projects that get spun up, and there is absolutely no way to automate quota increases requests via an API. Its absolutely infuriating and makes making disposable projects all but impossible, which I was really hoping to set up for my company.
If there is anyone at GCP reading this: Please, please, please allow customers to set organization wide quotas(even for new projects that get spun up) and please give customers an API to manage these things!!!!
Overall though, GKE is pretty great for us.
We've added features to suit our needs. Examples: near real time collaboration on notebooks with user cursors for when a data scientist colleague has a problem implementing something and we need to pair-program.
Automatic logging for parameters, metrics, and model to take the cognitive load off.
Multiple checkpoints for our notebooks.
Ability to turn a notebook into an AppBook: we automatically generate a form for parameters without the author tagging cells so you can train a model with a form or an HTTP request, without changing the notebook. Also useful when we want to allow a domain expert to train models by tweaking parameters that matter to them (no need to have Jupyter). Then click to deploy a model, or click to build its Docker image if they have a developer team that wants to send data to it.
Again, not all scenarios are handled but adding features that we needed in our projects.