Yet another hardcore programmy type discovers that usability is infinity more important that how many clock-cycles you save.
Programming languages aren't for computers, they're for people.
Programming languages aren't for computers, they're for people.
"Please don't post shallow dismissals, especially of other people's work. A good critical comment teaches us something."
If someone is wrong and you know more than they do, you can share correct information without putting them down.
The docs for building your own packages are even worse, to the point where you basically are left copying snippets from Conda Forge to build anything nontrivial.
Basically Conda is a tremendous engineering achievement, but it's very much still a "first draft" in a lot of ways, and Continuum/Anaconda made some weird decisions that work against its user-friendliness. Imagine for example if third-party repos on anaconda.org could have a description box, link to a homepage, etc...
I see this issue brought up a lot, but I have yet to see a language that addresses this reliably. By definition setting up an environment for non-programmer is a tall order, what language should they use?
Once the initial Dockerfile is written its very low maintenance, though getting results out of it could probably be made easier for scientific use.
Never needed Conda, but the packaging tool proliferation is an embarrassment, IMO.
As someone who did scientific programming in other languages (Fortran/C++), I can assure you the nonreproducibility was there in those projects as well. Not because of the tech stack but because no one valued reproducibility.
The current situation with notebooks isn't worse. It's more of the same. I think people criticize it more because notebooks are advertised as reproducible research.
Data science has a ton of moving library parts, it is genuinely difficult to distribute precompiled libraries for everyone when you have 2-3 actively maintained CUDA version with 2 cuDNN version for accelerators that change every 2 years. Most team fail to standardize on an environment (say Python 3.8, Ubuntu 20.04, CUDA 11.1, and cuDNN 8) and then get hung up on a dependency not building as if it's Python's fault that it does not have control of your entire OS.
Maybe other languages have fewer libraries, or maybe the habit of grabbing libraries at random evolved concurrently with the rise of Python.
My team has a rule that we don't let a project get past a certain stage without proving that it can be installed and run on a clean machine and archiving all of the necessary repo's with the project. It's easily forgotten that testing your installer is part of testing your program.
It might look worse because many Python projects use tools such as CUDA, which are notoriously dependent on the specific OS, architecture, method of installation etc. But that same issue will exist in most languages - if you're linking against CUDA, you will sometimes have problems with the package installation. Particularly if you try to run the code on a different OS, CPU architecture, using a different GPU, etc.
I don't think it really has anything to do with Python. It just happens that most people doing work that depends on tricky packages such as CUDA also happen to be using Python.
It's not much of a problem for other stacks because they either are fast enough that they have a library written in the same language for problem X (C#/Java/Rust) or they aren't targeting the same type of work (JS, Ruby, etc...). C++ has the exact same problem as Python and I'd argue that it's even worse.
As someone who uses numpy almost daily, I think that numpy is "overextended" beyond its core niche, sure. So - making it work with things outside that niche (e.g. streaming, non-rectangular data, non-uniform data, nonhomogeneous data, etc) is painful. However, 1) there's Pandas for that, and 2) I disagree with "misleading" and "surprising". What makes you think that?
* why distinguish between Series and DataFrame? just give me an interface for m x n matrices or even higher dimensions.
* pure vs. in-place operations. not such a big fan of having multiple versions of the same function, e.g. a more pythonic
df[“my_col”] = series
df.assign({“my_col”: series})
That brings me to another point: if we keep everything purely functional, then python’s syntax is making things a bit awkward. Where in something like JS you could just put every function call with its dot on a new line without the need to assign, in Python this requires putting line break characters or wrapping it in round brackets. This is one place where a language with explicit assignment terminators (semicolons) are a bit cleaner to work with.
All that being said scipy is still a great choice to have both system programming and numerical business logic in one language.
Little things, like some functions want to be called with a tuple of dimensions,
np.zeros((rows, cols))
np.random.randn(n, m)
When you slice a column from a matrix, and get a not an nx1 vector, but a 1d array, it makes me want to shell out $10,000 for matlab (yes, I know I can get a column vector with the slice A[:, [2]], but I shouldn't have to).
This problem leaks out into the ecosystem. For example, when you try to use scipy to integrate an ODE, and pass it an initial condition vector that is nx1, the scipy integrator will silently coerce your vector to a 1d array, pass it to your RHS function, which then either blows up, or more likely, produces silently wrong result because of numpy's insane array broadcasting rules.
This problem further leaks into the ridiculous function hstack. If you just used the function vstack, which made a 2x3 matrix from 2 1d 3 element arrays, you might imagine that hstack would produce a 3 x 2 matrix. But no. It creates a 1d 6 element array. For what you wanted, you actually need np.column_stack.
I think the way Eigen handles this is the most intuitive. You do linear algebra with 2d objects, and cast to arrays for elementwise operations.
There is also a huge inconsistency between what numpy exposes as an object oriented interface vs a "functional" interface. What I mean by this, is that I can call x.sum() on an array, but not x.diff(). For that, I need np.diff(x). There seems to be no pattern to what is exposed as a method vs a function.
The array slicing api is also really inconsistent. For instance, given a 3 element array x,
a = x[5]
a = x[2:5]
Next time I find myself manipulating numerical data, I'll definitely take a look at Pandas!
The tidyverse ironically ends up feeling more Pythonic, with more of a "there is one obvious way to do it" vibe.
jupyter notebooks encourage disorganized, unprincipled programming; chaotic re-running of cells in the face of global mutable state; and prevent budding programmers from learning to use version control because the JSON format was designed without version control in mind.
But even more, working with Jupyter allows you to work out a very detailed explanation of your thought process, describing your ideas, your experiments. Being able to mix code and explanations is really important (and reminiscent of literate programming). You got the same kind of flow with R.
As data scientist, I'm concerned about data, statistics, maths, understanding the problem (instead of the solution). I don't care about code. Once I get my data understanding right then comes the time of turning all of that into a software that can be used. Before that, Jupyter really gives a productivity boost.
For the code part, yep, you need other principles where Jupyter may not be suitable.
I find it easier to read and understand bad code written in Python, than good code written in the C family languages.
In that sense, I'm making it work like Python code in a file. The advantage of code in files is that I can use all of the slick code analysis tools that will warn me about my mistakes. I wish there were something that would let those tools go through the code in a Python notebook from top to bottom.
Ploomber makes it systematic - store notebooks as .py (py:percent files for example), parameterize them with papermill and execute as a batch job. One can view the resulting jupyter notebooks as .ipynb later and produce reports as html if wanted. It's really good already, and better if ploomber gets more development.
The whole reason it works is because it's easy to open the .py notebook and work on it, interactively, in jupyter.
The main idea - jupytext for .py notebooks and papermill for parameters & execution - that's already "stable" and easy for anyone to use for their own purposes.
Any feedback for us? What can we do to improve Ploomber?
As a first impression, I eventually found meta.extract_upstream = False which I think is important. Reason: The code for each step should be a lego piece, a black box with inputs and outputs. That code should not itself hardcode what its predecessor in the pipeline is - you connect the pieces in pipeline.yaml. (extract_upstream = False is not by itself enough to solve this, since you also need to be able to rename inputs/outputs for a notebook to be fully reusable as a lego piece, but it's good enough for now.)
I also for my own sanity need to know more about how the jupyter extension part works, how it decides to load injected-parameters or not. But maybe I could learn that somehow from docs.
In general I want components that are easy to understand and plug together and less magic (but the whole jupyter ecosystem's source code feels this way to me unfortunately, lots of hard to follow abstractions passing things around). But it's developing rapidly and already very useful, thank you so much!
I'll ensure we display the "extract_upstream" more prominently in the docs, we've been getting this feedback a couple times now :)
Re: the Jupyter extension injects the cell when the file you're opening is declared in the pipeline.yaml file. You can turn off the extension if you prefer.
Feel free to join our community, this feedback helps us make Ploomber better!
The one place matplotlib sucks is any kind of interactivity. But other than that, matplotlib has the best, most intuitive interface of all the python plotting libraries I've tried. It's also one of the few libraries that doesn't rely on generating html for a webbrowser, which makes for a miserable workflow.
I still think Matlab's plotting is untouched by open source options.
What python had, was that it was preinstalled on many computers and then had a large cohort of users that are insisting that others use it. And mostly force proclaiming that it is easy and readable.
I'll not claim that it is hard, per se. More that it is not intrinsically easier than any other dynamic language.
For evidence, the main packages that are popular are often clones of packages from other environments that were not widely installed. Jupyter can be seen as free version of many scientific applications. Matlab, Mathematica, etc. Matplotlib is rather direct in it's copy. Pretty sure there are more examples.
Npm, as much as it annoys me, is light-years ahead of anything in python. Quicklisp is rather pleasant, now. Ruby has had gems for a long time.
I grant that it is a hard problem. I am not griping that it is not solved. More that the option community has largely failed to even pick a direction. The most used dependency methods are, in the modern spirit of python, deprecated already.
Hence PEP 582:
> This PEP proposes to add to Python a mechanism to automatically recognize a __pypackages__ directory and prefer importing packages installed in this location over user or global site-packages. This will avoid the steps to create, activate or deactivate “virtual environments”. Python will use the __pypackages__ from the base directory of the script when present.
https://peps.python.org/pep-0582/
I haven’t tried it yet, but there’s already a PEP 582-compatible dependency manager: PDM https://pdm.fming.dev/
I have had many dep fights with Ruby and Node too, and I would disagree that their solutions are in any way better than pip and virtualenv.