Hacker News Reader

Comment by stdbrouw | Hacker News Reader

> But to get going with Julia, I just download Julia and go.

Until you need to do something very straightforward like web scraping or interacting with AWS and find out that nobody's released a package for that yet, so you're going to have to reinvent the wheel before you can get to the scientific question that is of actual interest.

Julia looks very interesting as a language, let's just hope it doesn't end up a ghetto like R (lots of awesome statistical tools, a dearth of libraries and tools for everything else).

View on HN

> Until you need to do something very straightforward like web scraping or interacting with AWS

For web scrapping: https://github.com/porterjamesj/Gumbo.jl For AWS: https://github.com/JuliaParallel/AWS.jl

Definitely, Julia doesn't have as many libraries as Python, but most common needs are covered. And for the rest you can actually call Python through PyCall.jl

This just proves my point, though. Gumbo is an HTML parser, not a scraper. The AWS library "will" support EC2 and S3. Again, this is all perfectly normal and it'd be absurd to expect such a young project to have a rich ecosystem. I think the Moore foundation grant is awesome and hope to use Julia someday. It's just that (1) as a scientist, you really just want to get shit done and (2) because it's marketed solely to scientists, I'm concerned Julia might never develop an ecosystem of general-purpose tools. R certainly hasn't.

Not sure what you mean by "web scrapper" than, but if you just need something to download web pages (in addition to parsing HTML), then Requests.jl covers it pretty well.

IIUC from GitHub page, AWS.jl has most API implemented, just not thoroughly tested. So it's not really mature or robust, but from my experience with packages at the same stage of development it should be pretty usable for a scientist who "just wants to get shit done".

I totally share your concern regarding marketing solely for scientists, though.

Can't you call out to python with pycall for that?

My experience with bridges is that they always suck, and the ways in which they suck are extremely difficult to debug and never obvious until after you've made a significant investment.

Shit I've seen: silent memory corruption, silently dropping the last element of an array if and only if it is the last declared field in the structure, killing the debugger every time you hit the bridge, inaccessibility of critical code because it uses X unsupported language feature, event-loop integration nightmares (IO suppression / null-routing / deadlock), exception incompatibility, unconfigurable signal/interrupt stealing. And that's all without counting the "usual suspects" of documentation, performance, and testing issues.

Maybe the Julia-Python bridge doesn't have any of these problems. But I'm not going to be the one to find out.

The PyCall Julia-Python bridge is shockingly good. It was written by Steven Johnson, also the author of FFTW – the world's state-of-the-art Fast Fourier Transform library for a two decades now. He is one of the best, most thorough programmers I know. When he writes something, it works.

R is for the stats part of the project. It does just that. I'd never implement a full system in it, but it does the stats bit fantastically.

Reply on news.ycombinator.com