Distributed Python for the Erlang Ecosystem | Hacker News Reader

Distributed Python for the Erlang Ecosystem

github.com

66 pointsrcarmo9 comments

For the record in the interest of programmers who don't use Erlang on a daily basis:

It is very common to use Erlang as a "glue" system between other subsystems in larger installations. You will often find the Erlang system controlling code written in other languages, for different reasons: The large Java subsystem which is hard to replace. The C/C++ code you run as a hidden node() in the Erlang distribution cluster. The C-accelerated function you added to the Erlang BEAM VM through dynamic loading of a .so (called a NIF). The OCaml or Go program you communicate with through a port (essentially an Erlang-controlled pipe with a proxy light-weight process inside the Erlang VM). And so on.

Some times, the right tool for the job is another system. The tool 'py' provides yet another such bridge which allows you to interact with Python programs in a very direct way.

This blog post provides some background: http://technicae.cogitat.io/2014/12/improved-python-support-...

What are the differences between this and Disco?

Disco seems to be purely a MapReduce framework?

You can use Disco to execute arbitrary Python tasks or workflows - it is not limited to MapReduce.

Disco was designed for data-intensive workloads. It wouldn't make much sense to use it to supervise Python servers/services, for instance.

Not just that. Disco also provides a distributed storage service for locality of reference when doing Map-Reduce.

This, in turn, might be something you'd use to build a tailored distributed system where you'd call out to Python (and its ecosystem, like Pandas, etc.) to do your own data processing.

My personal interest in it is that it uses LISP Flavored Erlang (lfe.io) and might therefore be nice to use with Hy (hylang.org)

Nice that you mention Pandas ... it's on the list for being supported by lsci (which in turn will be using LFE py).

As for Hy: LFE py comes with custom ErlPort encoders/decoders, and I came this close to writing those in Hy. However, I did want anyone with a Python background to be able to easily read and contribute, not just Lispers ;-)

Regardless, it would be nice to use with Hy (and probably trivial: a requirements.txt update, and then adding some Hy modules in ./python/lfe, at which point you should be able to access them from LFE).

Disco has gone beyond MR with pipelines, too. This allows for more generalized workflows. Great stuff.

The points made below about Disco a great. Similar confusion came up on Twitter with regard to CloudI. As a result of that, I updated the README with clarification on what LFE py is not for, and what you might use in those cases.

Disco is brilliant stuff (I greatly prefer to to Hadoop), and the fact that it has a distributed file system should really clarify its use cases (any why one would choose to use it over something life LFE py which doesn't have one!).

(As you can see, I can't help myself from saying great things about Disco :-))

Reply on news.ycombinator.com