I never thought that ipython notebooks could be used to build production models.
Can you talk about this? How big a machine do you need to run notebooks effectively ? Why a notebook...Why not a Python script,etc
Can you talk about this? How big a machine do you need to run notebooks effectively ? Why a notebook...Why not a Python script,etc
- not too big. I've run tests with performance vs. sampling of data and for the models we use we can sample a significant amount (i.e. use a fraction of the full training data). - If we did need a bigger machine, we're using ensemble models that parallel train nicely.
> Why a notebook...Why not a Python script
I was originally going to use a Python script, but I found it useful to have the notebook output inline performance charts and metrics. It's easier to contain them in the notebook than output a bunch of image artifacts that have to be added to VC. This way I can pull open the notebook and scroll through to check all my visual metrics.
I'm not opposed to ditching the notebook for training entirely, but for now it works just fine.
I really love this quick iterative way of working (atleast in the early days). Could you talk about your production setup of training ? I'm just concerned about performance, etc - is it OK to train manually each time (by opening the notebook, etc)
Not claiming it's a good way of doing this, but just how it is right now.