Comment by pwang | Hacker News Reader

Comment by pwang | Hacker News Reader

Parent Full thread

> Useful for heavy computation on smallish datasets, but not for moderate computation on big datasets. Here at work the datasets are several gigabytes so this would not be a solution.

This is just a closed beta for the very first initial version of the product. We absolutely intend to make it a powerful platform for computation on big datasets.

Many people are storing data in S3, and many datasets are sourced from locations on the web or remote data providers, anyway. With Wakari, you can have extremely efficient access to those datasets via IOPro (http://continuum.io/iopro), which is adding indexing on S3 data in the next version.

We will also be adding sharing, collaboration, and publishing features in the next version. I completely understand that it might not fit every situation (or your particular situation right now), but I hope that eventually you'll find these other features fairly compelling.

> After reading the blog post this looks like an enhanced IPython notebook with some additional packages.

Not the case at all. We provide a fully sandboxed Linux environment, accessible via the browser, with a complete installation of Anaconda Pro. The IPython notebook is an additional feature of Wakari, and not vice versa.

One thing we haven't showcased very much is the fact that you have concurrent access to many different Python environments. Python 2.6 + Numpy 1.5? No problem. Want to try out Numpy 1.7 on Python 3.3? Just change the drop-down, and you get another shell with that. Want to play around with Travis Oliphant's Numba compiler for Python (https://store.continuum.io/cshop/numbapro), but don't want to go through the hassle of installing LLVM, LLVM-Py, and Numba on your own system? You're just one button click away.

Like I said, I can understand that Wakari isn't going to solve everyone's problems right now, but it's fairly cool even in this initial version, and it'll only improve over time. :-)

> We absolutely intend to make it a powerful platform for computation on big datasets.

The datasets I run on tend to be several terabytes and all the clusters that I use utilize various distributed filesystems (eg. Lustre [1] or Dcache [2] or Xrootd [3]) to store these files across local storage on all the nodes of the cluster.

I think for big datasets some kind of support for these distributed filesystems will be necessary as most private (scientific) computing clusters use them. The complication is of course that to use these filesystems, one needs to build support for each access protocol into a python module (off the top of my head I'm not sure if they exist yet; perhaps they do). While most of these filesystems do offer standard posix 'mounting', this is usually not recommended for performance reasons.

Perhaps I'm outside of the intended use case anyway. To a certain extent this problem is solved for my field of research (of course nothing ever works all the time heh), as all LHC/OSG Computing Grid centers run the same Scientific Linux distribution with the theory being that any code that works on one should work on another.

But this sounds great for teaching classes and labs! I look forward to using it.

[1] http://wiki.lustre.org/index.php/Main_Page [2] http://www.dcache.org/ [3] http://xrootd.slac.stanford.edu/

Thanks for the feedback!

You're right that for scientific use cases, the data access story defines mode and methods of computation. Your use case of having a large dataset stored across many files on a distributed FS is one of the cornerstone motivations for building Blaze, next-generation distributed Numpy:

https://speakerdeck.com/sdiehl/blaze-next-generation-numpy

http://vimeopro.com/continuumanalytics/pydata-nyc-2012

Rather than traditional "load all the data into memory" sort of approaches, Blaze is inherently out-of-core, and allows the user to define mappings of an index space onto local or remote files, and then manipulate that structure at the Python prompt as easily as if it were a small matrix stored in memory. There are existing PGAS approaches that are similar in spirit, but they tend to invoke heavyweight MPI machinery or make assumptions about the regularity or structure of the dataset that is being distributed across the cluster. Our guidance in designing and implementing Blaze is to make simple distributed things easy, and hard things possible; we are not trying to solve decades of distributed computing and linear algebra problems as a start-up. :-)

So, hopefully you can see that with Blaze as your data access mechanism, and Wakari as a web-based front-end, you should be able to do large scale compute from within a web browser.

Will you (or do you) offer PyPy support too? It is looking to be the fastest Python implementation around, and even more so as its support for Numpy matures.

Possibly; it is not on our immediate roadmap, but is something we can add if there is enough demand.

If you are interested in accelerating Numpy, have you seen Numba? One decorator can net you a several hundred times speedup over pure python, and in many cases, a significant speedup over Numpy: http://www.slideshare.net/teoliphant/numba-lightning

http://continuum.io/blog/the-python-and-the-complied-python

https://vimeo.com/continuumanalytics/review/53105906/9f5dcbb...

Reply on news.ycombinator.com