The Myth That Machine Learning Engineers Are Not Good General Software Engineers
managingml.substack.com
managingml.substack.com
- Some of our members could go through the whole thing: meeting client, understanding business problem, write code to connect to data sources, get the data, clean it, set up training environments, train models, set up serving environments, deploy models, write the application (web, mobile).
- Some of our members could do a subset of these: feature engineering, training models. XOR build applications and deploy models.
This has put us in a situation where those who cannot "do it all" relied on someone who can do the part they needed. For example, asking someone to deploy their model, or setup their compute environment, etc. The person might be working on something else, or deploying that very person's earlier model version. Blocking.
We wanted to lower the frictions, so we built our "machine learning platform"[0] to allow people to do things that have higher value and impact, and allow people to be independent. For example, our people who are great at training models but cannot deploy them now can with the click of a button. The platform also takes care of setting up environments, scheduling long-running notebooks, automatically tracking experiments (parameters, metrics, models) because we used to rely on people being consistent and remembering to do that, and then we had a divergence of methods to do so (ad-hoc processes).
It also allows people who can "do it all" not to waste time with these tasks that are blocking their colleagues, and work on more valuable things. Just because they can do it all does not mean they ought to be setting up training or serving environments.
- [0]: https://iko.ai
For example, say you have an expert on image processing. That skill set usually involves deep system engineering knowledge because you must write many software components in C/C++, deal with complex compilation of special dependencies, understand low level details of image file formats and compression, as well as core mathematics in Fourier transforms, image processing, linear algebra, as well as modern image models like convolutional neural networks.
Every minute of time you task this person to instead work on devops, kubernetes config, REST API minutiae, etc., you are throwing money away. So structuring it with a platform team to support them is critical even for people who can “do it all.”
Yup. One of the reasons we're building that is that we did consulting and we got paid Total = sum(role[i] * rate[i] * number_of_days_of_role[i])
Say a team member has a role with a rate of 1,500 euros/day invoiced to the client. A lot of these days go to things like:
"Oh, I upgraded Ubuntu and now NVIDIA drivers and Docker engine or Tensorflow isn't working anymore". There you go. Non productive time.
"Oh, the notebook isn't working on my machine because you used a different library version but I can't install it because my environment would break (or I'm afraid it would)".
"By the way, which notebook produced this model? I don't know, let me check my Excel file for which parameters I used. Oops, I forgot to write them down."[0]
"Where are the weights? Is that the last model? You have them saved by date?"
"I produced 15 notebooks but I can only show one or two to the client. I'll have to spin up a VM and "deploy" the model to show it. I'll have to write a Flask application. Meeting is rescheduled. I forgot which VM it was."
How many days per month would you say does anyone involved in machine learning waste doing this? What's the combined loss as is? What's the combined interconnected loss when something happening to you impacts someone else, for example when there's task dependency.
Yes, yes... These would not happen with "good" engineering practices, but they do happen because good practices take time and money to learn. There's very little, if any content, on actual problems in the real world. Most write blog posts with Iris dataset, nice CSVs, and toy projects one can do alone.
Do anything slightly more complex that involves more than one person, and things start to break fast. Even for people who are good developers and follow good practices in software engineering. [good workflow, code reviews, version control, CI/CD, automated tests, etc]. Things break not because they don't know how to do these things.
That's also what's funny when I see tools and platforms created or founded by people who have a background in web dev who'll tell a story of getting into ML and being shocked people don't know how to use Docker or Kubernetes and deciding to fix the ecosystem. Coming from the outside, they think that they'll illuminate people and make them discover Docker. It makes me smile.
Many organizations develop their own solutions to mitigate or solve these problems. So, have you looked at our product and do you have any feedback? I'd very much appreciate it.
- [0]: ~68% of data scientists in a Kaggle survey said they don't track experiments. 68%. https://storage.googleapis.com/kaggle-media/surveys/Kaggle%2...