Hacker News Reader

Comment by make3 | Hacker News Reader

There are many cases where the objects are too big to be passed around. Python is used a huge amount in Machine learning and datascience, where being able to do parallel work on stuff already in memory would be great.

View on HN

Can't this already be handled by calling out to a C/C++ or FORTRAN procedure that processes the data in multiple threads? For number crunching, Python is almost exclusively used as glue.

Today's machine learning and data science students don't know how to code in those languages. They know python, and maybe java.

Don't forget R... shudders

What's wrong with R? I know it's not a programmers language but it's great for getting things done.

You CAN handle it, but why should you have to? If it's possible to remove that barrier, then it absolutely should be removed. If the only answer to a problem is "use another language", then the language in question has a limitation that needs to be addressed.

It is not a limitation at all in this case. Python is just a front to Tensorflow and similar libraries/frameworks so GIL doesn't matter there.

So work on data that can not be broken down into smaller chunks? That makes sense, and is something I never come across.

I'm sure they can be broken down into smaller chunks, but is it more efficient if they aren't broken down and instead shared memory is used? If you want parallelism you're obviously already worried about performance.

Something like the web worker primitives might work there (transferables & sharing read only data).

Are those applications often bottlenecked by the CPU, as opposed to GPU or data transfer?

The world of algorithms that run well on a CPU is still much, much bigger than the world of algorithms that run well on a GPU, even in machine learning.

And even if you're fortunate enough that Nvidia designs their GPUs to solve your problem, why should the CPU cores sit idle?

Reply on news.ycombinator.com