HNHacker News
TopNewBestAskShowJobs

vtuulos

780 karma · joined February 28, 2011

Helping companies build real-world ML/AI systems.

CEO/co-founder/customer support representative at https://outerbounds.com

I started https://metaflow.org at Netflix where I led ML infra for 3+ years. I wrote a book about my learnings:

https://www.manning.com/books/effective-data-science-infrastructure

Earlier projects:

http://traildb.io

http://tuulos.github.io/sf-python-meetup-sep-2013/

http://discoproject.org

submissionscomments
vtuulos··on Ask HN: Who is hiring? (September 2021)
Outerbounds | Frontend dev | SF / Remote | Full-time

Outerbounds was founded recently to commercialize Metaflow, an open-source Python framework that makes infrastructure easily accessible for machine learning/data science projects. Metaflow was originally started by us at Netflix and it is now used by hundreds of companies across industries.

We care a lot about thoughtful design, overall product experience, and quality of code. We are looking for stunning colleagues who can help us build delightfully usable, technically non-trivial UIs. This is a great greenfield opportunity for a senior frontend engineer who wants to build a product from scratch.

- Metaflow: https://docs.metaflow.org

- More about us here: https://outerbounds.co/workwithus

- Join our Slack to see the project in action: http://slack.outerbounds.co

- Email me: ville@outerbounds.co

vtuulos··on Ask HN: Who is hiring? (August 2021)
Outerbounds | Frontend dev, Systems dev, ML/AI Specialist | SF / Remote | Full-time

Outerbounds was founded recently to commercialize Metaflow, an open-source Python framework that makes infrastructure easily accessible for machine learning/data science projects. Metaflow was originally started by us at Netflix and it is now used by hundreds of companies across industries.

We care a lot about thoughtful design, overall product experience, and quality of code. We are looking for stunning colleagues who can help us at the low-level (interfacing with operating systems, cloud, Kubernetes) as well as with the user interface (building delightfully usable, technically non-trivial UIs from scratch). Background with data science/ML is a bonus but not a must.

- Metaflow: https://docs.metaflow.org

- More about us here: https://outerbounds.co/workwithus

- Join our Slack to see the project in action: http://slack.outerbounds.co

- Email me: ville@outerbounds.co

vtuulos··on Using Argo to Train Predictive Models at FlightAware
This is very cool! If you are interested in using a similar setup for your ML workloads (Argo + K8S), I'd love to chat with you (ville@outerbounds.co).

We are working on integrating Argo & K8S for Metaflow, an open-source ML framework originally developed at Netflix https://github.com/Netflix/metaflow/pull/434

vtuulos··on Launch HN: Journey (YC W21) – a better tool for selling software
I love the idea! The Journey team has a great founder-market fit too :) I am looking forward to using the product by myself.

Congrats for the launch!

vtuulos··on Machine Learning at CNN
Just my experience from Netflix tells that a large company can have tens of use cases for optimization of different kinds, ML included.

Often it is the least hype-worthy cases that are the most valuable for the business (e.g churn prediction, marketing budget optimization etc)

vtuulos··on Machine Learning at CNN
Interesting, thanks for sharing CNN! (Disclaimer: I hack Metaflow for living)

Do other folks find Terraform more appealing than Cloudformation in general?

vtuulos··on Downloading files from S3 with multithreading and Boto3
The topic is close to my heart so maybe one day :)
vtuulos··on Downloading files from S3 with multithreading and Boto3
re: separate process - fault-tolerance is a key requirement. There are a myriad of ways how a highly parallelized, network- and data-intensive code can fail, so isolating it in a separate process is a safer approach than trying to try-except everything and hope it works.

Good catch re: the warning about consistency! The docs were written before the change :)

vtuulos··on Downloading files from S3 with multithreading and Boto3
It is a bit of a hidden gem but Metaflow includes a Boto-based highly parallelized, error-tolerant S3 client that Netflix uses routinely to get 10-20Gbps throughput between EC2 and S3.

Technically it is independent from Metaflow, so you could use it as a stand-alone, high-performance S3 client.

See docs here https://docs.metaflow.org/metaflow/data#store-and-load-objec...

And code here https://github.com/Netflix/metaflow/tree/master/metaflow/dat...

(I wrote it originally - AMA if curious)

vtuulos··on Netflix's Metaflow: Reproducible machine learning pipelines
Coming soon!
vtuulos··on Netflix's Metaflow: Reproducible machine learning pipelines
As of today, Metaflow supports AWS Batch for scaling out and up. While supporting K8S is convenient in the operations point of view (assuming you have a K8S cluster already), it doesn't make the scalability story any better.
vtuulos··on Netflix's Metaflow: Reproducible machine learning pipelines
Reproducibility is a spectrum. A good starting point is to snapshot the exact version of the code that produced a model. Even better, you should snapshot the full dependency graph, including transitive dependencies, of all (compiled) libraries, which Metaflow does with @conda. Together with data snapshots, this gives a pretty good foundation for reproducibility.

Depending on the libraries you use, the exact results may or may not be reproducible on other architectures. If cross-platform reproducibility is important to you, you should choose your libraries accordingly. Metaflow provides the tools for choosing the level of reproducibility that your application requires.

vtuulos··on Netflix's Metaflow: Reproducible machine learning pipelines
No plans but it should be possible technically. We'd need help from the Julia community :)
vtuulos··on Netflix's Metaflow: Reproducible machine learning pipelines
Good question! What you are asking is pretty much the core question for a certain set of ML tasks at Netflix.

Metaflow is rather unopinionated about those types of questions, since they are subject to active research and experimentation. Metaflow aims to make it easy to conduct the research and experiments but it is up to the data scientist to choose the right modeling approach, features etc.

In some cases, individual teams have built a thin layer of tooling on top of Metaflow to support specific problems they care about. I could imagine such a layer for specific instances of transfer learning, for instance.

In general, we are actively thinking if/how Metaflow could support feature sharing in general. It is a tough nut to crack.

vtuulos··on Netflix's Metaflow: Reproducible machine learning pipelines
Metaflow is largely complementary to a job scheduler like Airflow. Technically you could export Metaflow workflows to Airflow, although the specific integration doesn't exist yet. For more details, see this blog article https://netflixtechblog.com/unbundling-data-science-workflow...
vtuulos··on Netflix's Metaflow: Reproducible machine learning pipelines
I'd love to hear more what you have in mind! Feel free to drop by at our chat at https://gitter.im/metaflow_org/community

While it is true that auto-sizing resources is hard and the easiest approach is to oversize @resources, the situation isn't as bad as it sounds:

1) In Metaflow, @resource requests are specific to a function/step, so you end up using resources only for a short while typically. It would be expensive to keep big boxes idling 24/7 but that's not necessary.

2) You can use spot instances to lower costs, sometimes dramatically.

3) It is pretty easy to see the actual resource consumption on any monitoring system, e.g. CloudWatch, so you can adjust manually if needed.

4) A core value proposition of Metaflow is to make both prototyping and production easy. While optimizing resource consumption may be important for large-scale production workloads, it is rarely the first concern when prototyping.

In practice at Netflix, we start with overprovisioning and then focus on optimizing only the workflows that mature to serious production and end up being too expensive if left unoptimized. It turns out that this is a small % of all workflows.

vtuulos··on Netflix's Metaflow: Reproducible machine learning pipelines
If you are curious about how Netflix uses Metaflow to power behind-the-scenes machine learning, take a look at this recent blog article https://netflixtechblog.com/supporting-content-decision-make...

Also I'm happy to answer any questions (I lead the Metaflow team at Netflix).

vtuulos··on Launch HN: Tappity (YC S20) – Keeping kids entertained with science
I just installed it for my daughter. Most educational apps are quite boring but Tappity was immediately engaging, thanks to genuinely funny videos and well designed content. Great job!

Quick user feedback: My daughter tried a course about building an animal. She made choices that produced surprising results ("the eyes are too ugly!") and wanted to go back but she couldn't figure out how - there's no clear back / restart button.

vtuulos··on Why we do machine learning engineering with YAML, not notebooks
We (Netflix) do a ton of prototyping/exploration in notebooks like everyone else. We run many ETL pipelines in production as templated notebooks. When something fails, you can just open a notebook to see the input and the output, which is handy.

We don't deploy or execute ML models in production as notebooks. We have many other solutions for that use case. In particular, check out https://metaflow.org

vtuulos··on Metaflow, Netflix's Python framework for data science, is now open source
Good question. We have many common features covered:

- Container management: See https://docs.metaflow.org/metaflow/dependencies

- Search for artifacts: see https://docs.metaflow.org/metaflow/client

- Automatic publishing of web apps: we have this internally but it is not open-source yet. If it interests you, react to this issue (https://github.com/Netflix/metaflow/issues/3)

Let us know if you notice any other interesting features missing! Feel free to open a GitHub issue or reach out to us on http://chat.metaflow.org

vtuulos··on Metaflow, Netflix's Python framework for data science, is now open source
We are good friends with the DVC folks! If the DVC + MLFlow + Anaconda stack works for you, that's great. Metaflow provides similar features. The cloud integration is really important at Netflix's scale.
vtuulos··on Metaflow, Netflix's Python framework for data science, is now open source
Good question! A typical Metaflow workflow at Netflix starts by reading data from our data warehouse, either by executing a (Spark)SQL query or by fetching Parquet files directly from S3 using the built-in S3 client. We have some additional Python tooling to make this easy (see https://github.com/Netflix/metaflow/issues/4)

After the data is loaded, there are bunch of steps related to data transformations. Training happens with an off-the-shelf ML library like Scikit Learn or Tensorflow for training. Many workflows train a suite of models using the foreach construct.

The results can be pushed to various other systems. Typically they are either pushed to another table or as a microservice (see https://github.com/Netflix/metaflow/issues/3)

vtuulos··on Metaflow, Netflix's Python framework for data science, is now open source
Orchestrating a workflow, which is what Dagster does, is just one part of Metaflow. Other important parts are dependency management, cloud integration, state transfer, inspecting and organizing results - features that are central to data science workflows.

Metaflow helps data scientists build and manage data science workflows, not just execute a DAG.

vtuulos··on Metaflow, Netflix's Python framework for data science, is now open source
the workflow DAG is a core concept of Metaflow but Metaflow is not just that. Other core parts are managing and inspecting state, cloud integration, and organizing results.

At Netflix, we use an internal workflow engine called Meson https://www.youtube.com/watch?v=0R58_tx7azY

vtuulos··on Metaflow, Netflix's Python framework for data science, is now open source
you need more connections than what a single AWS CLI process open to saturate network on a big box. You can achieve the same by doing some "xargs | aws cli" trickery but then error handling becomes harder.

Our S3 client just handles multiple worker processes correctly with error handling.

vtuulos··on Metaflow, Netflix's Python framework for data science, is now open source
Dask is great if you want to distribute your algorithms / data processing at a granular level. Metaflow is a bit more "meta" in a sense that we take your Python function as-is, which may use e.g. Tensorflow and PyTorch, and we execute it as an atomic unit on a container. This is useful, since you can use any existing ML libraries.

I am not familiar with Prefect.

vtuulos··on Metaflow, Netflix's Python framework for data science, is now open source
Here are some key features:

- Metaflow snapshots your code, data, and dependencies automatically in a content-addressed datastore, which is typically backed by S3, although local filesystem is supported too. This allows you to resume workflows, reproduce past results, and inspect anything about the workflow e.g. in a notebook. This is a core feature of Metaflow.

- Metaflow is designed to work well with a cloud backend. We support AWS today but technically other clouds could be supported too. There's quite a bit of engineering that has gone into building this integration. For instance, using the Metaflow's built-in S3 client, you can pull over 10Gbps, which is more than you can get with e.g. aws CLI today easily.

- We have spent time and effort in keeping the API surface area clean and highly usable. YMMV but it has been an appealing feature to many users this far.

Hope this makes sense!

vtuulos··on Metaflow, Netflix's Python framework for data science, is now open source
btw, if you happen to be at AWS Reinvent right now, you can get a stylish, collector's edition Metaflow t-shirt if you drop by at the Netflix booth at the expo hall and/or ping us otherwise!
vtuulos··on Metaflow, Netflix's Python framework for data science, is now open source
yep, Metaflow tracks everything: Your code, dependencies, and the internal state of the workflow automatically.

A big difference between Metaflow and other workflow frameworks for ML is that Metaflow doesn't only execute your DAG, it helps you to design and implement the code that runs inside the DAG. Many frameworks leave these details to the data scientist to decide.

vtuulos··on Metaflow, Netflix's Python framework for data science, is now open source
no relation
← PreviousPage 2 of 6Next →