Hacker News Reader

You bring up a good point. I'm in the same boat as you in terms of switching from Ruby to Python. Unlike you I've really enjoyed the switch. But also unlike you I've chosen to ignore (i.e. stick my head in the ground) the issues with the dev-tooling. Not that I ever had a strong understanding of how Rubygems worked, other than knowing why I needed to do `bundle exec rake [etc]`. But I've largely delegated the pain of Python package management to Anaconda. Also, I haven't had to deploy any production public-facing apps in Python.

I don't share your experience on the documentation of libraries though. I've generally had no problems working with the major third-party libraries (requests, python-dateutils, Flask).

And I also am reluctant to feel strongly against how the 2v3 chasm has been drawn out, though it's easy to say that as someone who didn't have to make the jump. The main difference from Ruby 1.8->1.9, AFAICT, is that the Ruby community has a near-monolithic devotion to Rails. When Rails left 1.8 behind, not making the jump would basically be career-suicide for a Ruby dev. The downside is that it shows how dependent Ruby is on Rails' momentum. Whereas in Python, a scientist (among several other use-cases) can live on 2.x for a very long time no matter what all moves to 3.x.

python had the opposite problem where so many major libraries didn't take make the jump to Python 3, so even if you really wanted to use Python 3 it was extremely restricting to do so.

I find chained enumerator / block style code much easier to write and understand than list / dict comprehensions too. It's probably the single biggest thing that I don't like about Python; lambdas are simply too awkward, but lambdas + monadic transformations are probably my primary mental abstractions for writing software these days.

Could you give an example of that?

Think: bash shell pipes. Transforming input to output through successive filters and projections, applied in a concatenated fashion and developed interactively in a REPL. Model algorithms as transformations or compositions of the same, where possible, applied to lazy or eager iterators or consumers (the dual of an iterator, that acts as a sink for output) as required. Model data sources and sinks (like databases and network streams) as iterators and consumers. Program by building a data flow graph out of increasingly sophisticated (by composition) stateless widgets, rather than imperative loops etc.

Imagine you have a list of products with prices and you want to calculate the cost of all products that are of a certain type. In ruby you can write this as

products.select { |product| product.type == 'x' }.map(&:price).reduce(:+)

And in Python you can write something like:

    sum([p.price for p in products if p.type == "x"])

Excuse untested code from phone, but I like it better than the ruby version.

I guess it's a personal style choice, the Python code is more performant(less iterations over the data set), but I like the ruby version better.

It's only one iteration if you throw a .lazy in there:

products.lazy.select { |product| product.type == 'x' }.map(&:price).reduce(:+)

I find the order of iteration in Python for comprehensions totally confusing every time. If you want to convert:

    for a in x: pass

into a comprehension, you write:

    [pass for a in x]

Simple, right? So to convert

    for a in x: for b in y: pass

I obviously write:

    [pass for b in y for a in x]

Bzzt. Nope, you have to write it in middle-endian order to get the same behaviour.

FYI, you can leave off the square brackets for a small performance boost. That turns it from a list comprehension into a generator expression, which is lazily evaluated and doesn't create a list container (which would immediately be thrown out anyway). It's like the difference between range(n) and xrange(n).

Nice, thanks for the tip.

With itertools and functools you should be able to program in that style just fine. You'll likely want to give your lambdas names with a "def", but you should be doing that anyway.

As a fellow FPer who does the monadic stuff in other languages, I think it's usually more trouble than it's worth in Python. Instead, trying to use more of the python-specific language constructs can help you get what you want.

Python is pretty imperative, and the language design ethos is pretty against designing around functional programming. But if you try to absorb some of the python-specific stuff like list comprehensions, generators, and context managers, you can write some tooling to help you get a DSL a little closer to what you're used to.

Giving my lambdas names is exactly what I don't want to do - the syntactic overhead is what kills the style, makes it an exercise in boilerplate. It's like Java closures before Java 1.8 - anonymous inner classes implementing interfaces are technically just about enough, but in practice entirely deficient, because the style is entirely unidiomatic.

The way my brain works (at this point in my career, 25 years of programming) is just against the Python grain. I think in terms of data flow graphs. Python wants me to think in terms of procedures and loops.

There are very good reason for naming functions (and ideally dangling them from the class rather than making them inner functions) - this encourages you to separate the 'what' from the 'how' and tends to result in code that is self-documenting. When you inline all your functions the person reading the code is forced to build and maintain multiple abstractions at one time to follow execution flow. Additionally, you are conflating the 'what' and the 'how' which forces the reader to context shift. If you replace the function with a well-thought-out name, code becomes much easier to read.

I prefer Python over Ruby as a language, but you are correct

gem ist superior to pip. Take a look at "pip-tools" if you haven't already, it eases some of the pain.

For web-dev Rails is more stable, more modern and has better maintained library ecosystem than Django. Django is okay-ish but can't live up to Rails. Django still lacks some stuff that i feel it should bring with it out of the box.... dev/test/production setups, asset compilation etc.

On the other hand, if you don't need a full blown web framework and are looking for something more minimalistic, Flask is awesome.

Amazing that Python 2 vs. Python 3 has still not yet been sorted out. I dipped my toe into Python a couple years ago when looking for a general-purpose scripting language and quickly gave up in frustration—Python 3 was by far the preferable flavor, but many libraries simply didn't work. I picked up Ruby and haven't looked back.

Likewise...and when I have to work in Python it's a frustrating pain because the tooling is so much better on the Ruby side.

Heck, the tooling is better on the PHP side even...

A lot has improved since a couple years ago. A majority of major libraries have both Python 2 and 3 support these days. I've since started all new projects in Python 3.

i started with python when 3 dropped in ~2008.917

ignorant of any prior ecosystem i just thought i was getting into the most current

i came for the prebaked arbitrary precision floating point arithmetic and now i use it for prototyping all of my projects, and all the other stuff that i would call general scripting

this 2to3 complaint made sense to me from an employed standpoint:i work as X and use 2.7's Y library; but you said general scripting

i wonder what libraries were you hoping to use but were unable, and for what general scripting purpose?

This was back in late 2013. My general scripting had to do with automating wireless network simulations to explore parameter spaces and slicing & dicing results stored in MySQL or SQLite databases.

I recall several showstoppers in Python v3 including integration into Emacs, accessing a MySQL database, and basic visualization. The library system was also a mess. It didn't seem possible to develop on v3 and expect that the libraries you needed would be available. Or that anyone else would care since they were all using v2.7.

I think a lot of this is due to these problems being solved through standardised processes in python.

For example, with dependencies, production dependencies and dev dependencies are defined through two requirements files: requirements.txt and requirements-dev.txt. requirements-test.txt for testing. No need for tooling, just pass those into pip.

Most python tooling is pretty limited in scope but works well. From my side, I've had more trouble with "turing complete tooling" that ruby has offered me. I have not had to use ruby a great deal though.

I think the difference in philosophy between the two communities is that python stuff will usually value ease of debugging/implementation over "ease of use". Most things in pyhton and its echosystem are straightforward. None of this "Library A adding methods to classes of Library B" stuff (not that it's impossible). Of course, you then end up with more boilerplate.

Py2/Py3 has definitely been a major pain.

Yeah the requirements-{group}.txt convention kind of works, but what happens if you want to save only your frozen test dependencies for example? If you are using a single virtual env you end up freeezing everything in that env not only the test dependencies and its transitive dependencies.

I wish Ruby had the math and scientific library support of Python.

Also, I've been using Jupyter lately to brush up on data structures and algorithms. It's been pretty sweet to write code in a web notebook and render output including digraphs beneath each block of code. Is there anything like that for Ruby?

Not to my knowledge and this is definitely an area where Python is a lot better than Ruby from my experience. My comment pertains more to web development than scientific computing.

You can run a Ruby kernel in Jupyter - https://github.com/SciRuby/iruby

You can use other kernels in jupyter notebooks. I've never tried ruby, but you could try this: https://github.com/SciRuby/iruby

Not only does Jupyter have Ruby support, but you can go back and forth between languages in the same notebook while sharing a context.

> Having used Rubygems and bundler pip feels like taking an enormous step back.

Personally I moved away from relying on language specific tools to do more than just "install library X" and manage my deployment and dependencies using docker. So far it works great for me.

> There seems to be fewer nice libraries

Depends on your domain. Ruby is very good at web development. Python is good at almost everything.

> Python libraries have lacking or hard to find documentation in many cases.

I didn't use Ruby, but JVM languages or Javascript don't come close to Python in the documentation coverage. Especially if you consider SO as a documentation, googling "how do I do X" in python usually gets you on the right track to the high quality solution. Javascript or Java SO post are more like a mine field in comparison.

Did you take a look at pip-tools ? It allows dependency locking

What do you guys mean by dependency locking? What I do with pip is add a requirements.txt with things like:

Django~=1.10.0

And it keeps the env autoupdated with the latest patch revision but not the latest minor revision. pip-tools doesn't seem to do much related to that, no?

The problem with that is that you no longer have reproducible builds. If you checkout a year old commit and built deployment artifacts with it you want the resulting artifact the be the same as when it was created. This is what you get with Bundler and the Gemfile/Gemfile.lock split

You put

gem 'rails', '~> 4.2.7'

in your Gemfile and when you run `bundler install` the exact version of rails you ended up resolving with the above is locked in your Gemfile.lock. Thus if you have a version in Gemfile.lock it is the source of truth and you have reproducible builds. If you want to update rails given the pattern in your gemfile you just issue `bundle update rails` and it will update to the latest conforming version and update your Gemfile.lock

The fact that pip doesn't support this in a good way is a major limitation imo. You can have either reproducible builds or "auto upgrading", but not both. pip-tools solves this, but requires you to use a different tool than pip and by extension you have to teach and explain it to anyone working on your project.

I version my docker images. Inside the Dockerfile I usually install the latest version (i.e. pip install without specifying version). By single version bump of my image version I can trigger update of everything.

If I check out an old commit I can look up docker image version in the source control, download exactly the docker image that was running at that time and start shell in this image. Inside the image I can run "pip freeze" to check exact version of each package that was running at that time. I can then update the old Dockerfile with those specific versions by adding the "==x.y.z" e.g. "pip install oauth2client==3.0.0" in the pip install to reproduce the old build.

Additional benefit is that such scheme works for system packages, or for any other language libraries I can think of.

Yeah that's what I mean by either reproducible builds(e.g locking each dependency to an exact version. Output of pip freeze) or what the GP is talking about which is specifying a version pattern.

> You can have either reproducible builds or "auto upgrading", but not both.

Ah, okay, this crystallized it for me, thank you. Other commenters were talking about one or the other as a limitation, which confused me because pip does do both (just not at the same time).

Yes, this is currently a limitation, and pip-tools seems to solve it. Hopefully pip will get a native solution soon.

I always though the recommended way to handle that split was by using setup.py as the Gemfile equivalent with very loose versioning rules, and for the Gemfile.lock case to not edit requirements.txt but to only generate it from pip freeze (or whatever it was - I haven't used Python for a while).

Yeah I've heard people say this as well, but from my understanding setup.py is more aimed at libraries than end user projects. I¨ve never seen anyone use setup.py for an end user project in any case.

Interestingly enough in Ruby it's the reverse, you use gemspec for gems which doesn't have the concept of locking. Instead you are supposed to specify semver conforming version patterns and then resolution of these happens when the gem is installed. Only end user projects use Gemfile.lock

"There seems to be fewer nice libraries and they seem to be spread across the web whereas Ruby centralises around GitHub. I also find that Python libraries have lacking or hard to find documentation in many cases."

Could you clarify a bit more on this with the real world examples ?

Well for the case of "nicer libraries" I suppose I mean high quality libraries. Also good to note that I am referring to libraries in the web development/backend sphere here. I have a hard time thinking of some real world examples of there being a larger quantity of quality libraries for ruby, it's more a feeling that I get.

Around documentation and usage I think it comes down to the fact that for Python libraries a google search usually takes you to the modules page on the PyPi registry, to a project website or a read the docs instance. I usually find that where ever I end up doesn't provide all the information I want, e.g what's the latest version, what are some typical use cases for this library. As a concrete example let's compare Flask and Sinatra.

For both Flask and Sinatra a google search gives me their respective websites as the top relevant result. In this case I find both websites don't really provide all the information I am after. When I navigate to their GitHub page however the difference is more stark. Sinatra uses GitHub flavoured markdown to render a nice guide with increasingly more complex use cases. Flask just has some non markdown text pointing me in a few different directions. I could probably find all the information I need to setup and use Sinatra from just the GitHub Readme whereas for Flask I'd have to go navigate their read the docs page where all the documentation and usage instructions are spread over multiple pages and sub pages. I guess it comes down to my preference for the more concise format of a long markdown README.md vs the experience with a read the docs like set of pages.

I recon this is a bit rambling, it's kinda hard to put into words what my feelings around this are and why I hold the opinions I do.

Check out conda sometime.

That looks interesting, I've also tried out pip-tools[0] which is really nice. However nice these tools are it seems to a bit at odds with the Zen of Python. Ideally there should be just one single good way to deal with packages. Ironically this is the case for Ruby, but not at all for Python. Ultimately you can always use some alternative method that solves it better, but you then have to take on the burden of convincing your co-contributors that it is a better idea than using pip and you also have to explain it to anyone that joins your project.

0: https://github.com/nvie/pip-tools

It can conflict with any native Python installations. So if you are running any pre built Python libraries which point to any system wide Python, it can be a nightmare. Hue is one example I ran into.