Hacker News Reader

Comment by natch | Hacker News Reader

I’m using Anaconda because it was recommended in a step by step tutorial for playing with deep learning.

What would be involved in removing it from my system and moving instead to this set of tools?

Not necessarily looking for s step by step answer, just for general suggestions.

My guess is: find out which python the deep learning tools are using, remove Anaconda, and reinstall the python version needed, using the tools from this post. I’ll need to read up on the tools too. Any pitfalls with this approach?

View on HN

Anaconda does most of the stuff mentioned, and also makes it much easier to install packages based on C/C++ libraries (which most deep learning things are). So you're better off staying with anaconda. It's widely used in commercial data science projects so the idea that noone "takes it seriously" as someone else suggests is a bit silly. I assume they're thinking about a different context to data science projects.

That said anaconda does have a whole variety of extremely annoying quirks, like packages not being backwards compatible with old versions of conda, or conda going crazy and reinstalling itself, or the way the conda-forge repo has far more packages than the official conda repo. It's very far from perfect. But for data science I think it's basically the standard package manager in Python land.

> ... and also makes it much easier to install packages based on C/C++ libraries

I hear this often, though I cannot remember ever running into a pip package where this was an issue. Out of curiosity, could someone point me to a pip package and its conda equivalent where this is the case?

Install scipy on Windows and include the Intel linear matrix libraries.

The numpy+mkl wheel is available on Gohlke's site.

Try pip installing scipy or Numpy and you'll see the value of conda.

Although I mostly use the system's numpy package (usually shipped with the distribution), I just tried installing it via pip and had no problem (obviously just one data point).

As if that still was a problem, now that we have wheels, and especially manylinux wheels.

See Myth #6 here: http://jakevdp.github.io/blog/2016/08/25/conda-myths-and-mis...

Valid points. Still most of them are not issues in my daily work.

And by the way Peter, thank you for the amazing work you do.

Just ran `pipenv install scipy numpy`. Installs like any other package..

I really like conda. It is easy to have multiple enviroments and minimizes the footprint by symbolically linking in what is needed.

This is also a useful read:

http://jakevdp.github.io/blog/2016/08/25/conda-myths-and-mis...

Rasterio, fiona and anything else GDAL based always causes me headaches.

Anaconda was the competition pip needed to become good. When Anaconda was introduced, I (a pip person), was impressed. However, at work we had pip workflows that were working okayish, so I never made the switch.

Today, pip et. al. has so dramatically improved, that I hardly see a reason to use anaconda. I am not using deep learning stuff, so I cannot comment on this, but for most scientific python stuff (scikit learn, pandas, numpy, etc.). Pip and pre-built wheels work very well.

On my new job, I was given a Windows laptop that I happen to use now for my development work (because I am too stupid/lazy to properly configure and maintain a separate virtual linux machine). And I started with anaconda, assuming it was less trouble, but quickly ran into trouble. pip worked like a charm [].

[] Ironically pip is installed via the anaconda base installation if I am not mistaken :)

I could provide a vagrant based repo that lets you spawn your ubuntu VM for dev in minutes. I'll try to get it in github.

Nothing ironic.

Anaconda integrates pip, it doesn't not compete with it.

I find that Anaconda is the best among the virtual python envs I tried; It does as good or better job of separation and tracking installations as any, AND falls back to pip (with complete integration and tracking) when a package is not in the conda repos.

pip inside conda works better than pip outside in my opinion. I don't understand the general sentiment towards (ana)conda.

> like packages not being backwards compatible with old versions of conda

When new capabilities are added to conda, like the ability to support noarch python packages, you're going to need to be working with an up-to-date version of conda to use those packages. Just `conda update conda`.

> conda going crazy and reinstalling itself

Conda will pretty aggressively auto-update itself. Going crazy and reinstalling is a new one though. File an issue with details at https://github.com/conda/conda

> the way the conda-forge repo has far more packages than the official conda repo

Conda-forge is community driven and more "upstream" than the Anaconda, Inc.-provided 'defaults' repositories. Think of conda-forge like Fedora, and 'defaults' like CentOS/RHEL.

I think you could probably use both, side-by-side (but not at the same time/for the same project).

Anaconda might be nicer if the packages you need are C-based and would need compiling on your platform. Depends on your use-case. For experimentation/playing around, stick with what works for you. But if a project required me to install Anaconda, I wouldn’t take it seriously. And pipenv is pretty easy to use, too. So if you plan on distributing it or open-sourcing it, understanding how most other people manage dependencies ouside of conda is going to be useful.

Why not seriously?

Many data science environments are pretty much based on Anaconda installations.

(See e.g. these dockers: https://github.com/jupyter/docker-stacks/tree/master/scipy-n...)

I mean, all packages I know DONT require Anaconda. But if you need the whole environment, sometimes Anaconda is the easiest tool too install all dependencies.

> Anaconda is the easiest tool too install all dependencies.

For hobbyist stuff that’s fine, and I applaud lowering the entry barrier. My issue would be if I needed Anaconda to deploy the project/dependencies into “prod” in some professional capacity, instead of standard Python build tools. Having said that, I’m not terribly familiar with Anaconda, and it seems to leverage virtualenv under the hood, possibly with pre-compiled packages (like wheels?).

"Hobbyist"? It's widely used in research in data science & machine learning.

If you want a setup which works on various systems, and uses Python numeric packages, usually it is the most failsafe way to use with various OS. (Unless you want to put everything in docker.)

Unless by "professional" you mean "building", well - then you have a point.

Yeah. Unless "Hobbyist" now also means my research university's 104 node, 2548 core HPC cluster...

Anaconda does not leverage virtualenv under the hood. Anaconda does not use wheels as part of that. However, you can use pip and virtualenvs within Anaconda, but you'll probably get some inconsistent results.

The reason that Anaconda exists is to make it easier for people to have consistency between their dev environment, the build environment, and the prod environment. If you're seriously going to build all of the scientific Python stack from scratch, correctly and optimally, for your prod environment (and then also do the same for every dev environment you need to support), then you have way too much time and probably haven't actually tried doing it over any real period of time.

The problem isn't that it's impossible. The problem is that there are dozens of different ways to almost succeed, and you don't run into the problems until way too late.

> you can use pip and virtualenvs within Anaconda, but you'll probably get some inconsistent results.

I've never tried virtualenvs inside anaconda (what would the use case be? anaconda already provides a virtual environment)

pip is perfectly integrated within anaconda, in my experience; What inconsistencies are you talking about?

What do you think you'll get from any if these tools that you do not get from {ana,mini}conda?

I must be missing something, but all these use cases (and more) seem to have been covered by miniconda ages ago.

conda works with perfectly well with pip, and creates virtual environments that are at least (in my experience) as good as virtualenv does.

A few years ago Anaconda helped overcome install issues with some packages, but I almost never run into those problems anymore.

One thing to be aware of is that Anaconda modifies various Jupyter configs / installs some of its own kernels. So it can be a hassle to get back to system python + plain jupyter.

https://github.com/jupyter/notebook/issues/1630

Thanks for the replies everybody. One thing that's still confusing to Python tourists (my word for myself since I am usually programming in a different language, but come to Python occasionally to do something) is that everyone talks about pip, when actually it seems pip3 is required to install when using Python 3. Is this no longer the case? Or do people just say "pip" when they mean "pip3"? Or are people actually still living in 2.x land? I'm talking about pip the command line command/binary executable file, not pip the concept / tool name. Similar to the distinction between the capitalized "Python" (name of the language) versus "python" (command entered on the command line/name of the binary on the system).

Any of those could be the case. Guess it's understood that you will put a 2 or 3 on the end of the pip when you want to choose one or the other. And it goes further:

    /usr/local/bin/pip*
    /usr/local/bin/pip2*
    /usr/local/bin/pip2.7*
    /usr/local/bin/pip3*
    /usr/local/bin/pip3.5*
    /usr/local/bin/pip3.6*

Just in case you wish to retain anaconda, you could simply remove anaconda path from your PATH variable in .bashrc file (Assuming you are on Linux). If you need anaconda again, you could add the path to the variable, or run `source ~/anaconda3/bin/activate`

None of these should conflict with Anaconda (or at least Continuum conda, I'm not that familiar with what else is out there) unless you put the conda root bin directory on your path.

Reply on news.ycombinator.com