468 karma · joined July 19, 2015
We have another project to cover the orchestration/pipelines aspect: https://github.com/ploomber/ploomber and we have plans to work on the rest of features. For now, we're focusing on those two.
I think this is a great way to explain what you're doing. I'm working in the same space (ML/DS tooling) and I feel like we, as the ML/DS community, haven't cracked exactly how Rails for data looks like, I actually wrote some ideas on this a while ago (https://ploomber.io/blog/rails4ml/).
Congrats on the launch and best of luck with the product!
We just released debuglater (https://github.com/ploomber/debuglater), an open-source library that serializes a Python traceback object for later debugging.
You can see a quick video demo here: https://github.com/ploomber/debuglater/blob/master/README.md
Countless times, we've scheduled overnight jobs to find out the following day that they failed. While logs are helpful, they are often insufficient for debugging. debuglater allows you to store the traceback object so you can start a debugging session at any moment.
We built this to support our open-source framework for data scientists (https://github.com/ploomber/ploomber), who often execute long-running code in remote environments. However, we realized this could be useful for the Python community, so we created a separate package. This project is a fork of Eli Finer's pydump, so kudos to him for laying the foundations!
The implementation is quite interesting. You can see it here (https://github.com/ploomber/debuglater/blob/master/src/debug...). The serialization step has two parts: it takes the traceback object and wraps it into a new object so it can be serialized; secondly, it stores the source code so you can debug even if you don't have access to the source code!
Please take it for a spin and let us what you think!
We're building tools to help data scientists develop and ship faster. We recently closed our seed round and are looking to assemble a small team of amazing individuals to take our tooling to the next level.
Job board: https://www.ycombinator.com/companies/ploomber/jobs
GitHub: https://github.com/ploomber/ploomber
Website: https://ploomber.io/
I'll ensure we display the "extract_upstream" more prominently in the docs, we've been getting this feedback a couple times now :)
Re: the Jupyter extension injects the cell when the file you're opening is declared in the pipeline.yaml file. You can turn off the extension if you prefer.
Feel free to join our community, this feedback helps us make Ploomber better!
Any feedback for us? What can we do to improve Ploomber?
As an example: one of the reasons why I don't use Kubeflow is because it requires having a Kubernetes cluster up and running, which is an overkill in many cases.
Check out the project I'm working on: https://github.com/ploomber/ploomber
We're working on notebook tooling as well (https://github.com/ploomber/ploomber), but our focus is at the macro level so to speak (how to develop projects that are made up of several notebooks). In recent conversations with data teams, the question "how do you ensure the code quality of each notebook?" has come up a lot, and it's great to see you are tackling that problem. It'll be exciting to see people using both MutableAI and Ploomber! Very cool stuff! I'll give it a try!
One question I have is how to Hydra balances writing postgres scripts vs leveraging system-specific features. For example, I remember going through Snowflake's documentation and found interesting functions for data aggregation. Can I leverage Snowflake-specific features when using Hydra?
The other important difference is that AFAIK, DVC can only run your pipelines locally, and Ploomber can export your pipelines to run in other environments like (Kubernetes, Airflow, AWS, SLURM, Kubeflow), this allows you to run experiments locally but easily move to a distributed environment when you need to train models at a larger scale or want to deploy an ML pipeline.
Having said that, the challenge now is to hit a sweetspot between keeping the Jupyter interactive experience, and providing some features to help data scientists develop modular work. That's where most frameworks fail, so we want to keep our eyes open, and get feedback from both scientists and engineers to develop something that works for everyone.
Finally, since the pipeline is modularized, it's easier to split the work. Some people may work in data cleaning, others in feature engineering, and they can all orchestrate the pipeline with "ploomber build".
You can read more about our approach in this guest blog post we published a few months ago on the Jupyter blog: https://blog.jupyter.org/ploomber-maintainable-and-collabora...
We'd love to hear from your experience! Please send me an email to eduardo@ploomber.io
More on the docs: https://docs.ploomber.io/en/latest/user-guide/jupyter.html