Hacker News Reader

For anything involving data analysis and scientific computing Python really shines - though I'm often surprised the extent to which web-oriented programmers are unaware of this, I read things like "you can write better web services in Go, therefore Python is obsolete".

Python in science is kinda like the One Ring the Binds them - it is the glue that holds together the HPC code at CERN, ESA, NASA, and more, it crunches climate models and delivers your daily weather forcasts. Some really great projects like Cython, Theano, and Numba boost the performance of numeric code to native speeds with a fraction of the hassle of hand-written C. Mountains of machine learning research is done in Python, as well as statistics and analytics. And of course Numpy remains the de-facto standard of N-dimensional array libraries - if you look at numeric computing libraries in many other languages they will be framed as "Numpy-for-X".

Yeah, I don't know

While Numpy/Scipy is great with a capital G, it seems the underlying infrastructure is flaky.

I'm talking about the messy process of building and installing it. And it seems everything is held with duct-tape

Also it seems that "inside Scipy' it is a different world, and that to really "talk to the outside world" there seems to be a data-translation step needed.

> "I'm talking about the messy process of building and installing it."

If you're using Python for scientific computing, you can use the Anaconda distribution of Python, it makes it painless to work with those packages.

https://www.continuum.io

> "Also it seems that "inside Scipy' it is a different world, and that to really "talk to the outside world" there seems to be a data-translation step needed."

Python has excellent ORMs, I'm not sure what you're getting at.

Working in Windows, my biggest issue was "the 64 bit situation", but that has been greatly solved by Microsoft's MSVS for Python (lovely move by Redmond) and the Python Wheels. Now it has truly become `pip install something` or `pip install [wheel file]` for more complex stuff.

That and Cristoph Gohlke[0]'s repository of pre-compiled modules. That guy saved me more than once. Should I ever meet him, I'll sure buy him a beer!

[0] http://www.lfd.uci.edu/~gohlke/pythonlibs/

at least on OS X installing numpy and scipy is very easy - pip installs prebuilt wheels by default, no need to compile anything

The complexity of building NumPy or SciPy is pretty much par for the course in HPC. Most of the libraries it is based on are battle-hardened and ridiculously well tested to a degree not seen in many places, but they date from a different era, including libraries where originally "building" it involved a hole puncher and a bucket of coffee. Getting all those libraries to play nicely together at all is a genuine miracle.

Most any significant scientific package I've ever had to build (which is quite a few) had a complex build system that felt like it was built on top of shaky glass pillars. It's just the nature of what's involved and it's a fundamentally hard problem; it's difficult to replace those libraries because the extreme amount of verification/validation and optimization that have gone into something like LAPACK makes it economically undesirable.

I'd say also that this is part of the appeal of NumPy and SciPy, which is they glue together a huge number of tools that are otherwise disjoint. You don't have to worry (most of the time) about matrix ordering, calling conventions, those mysterious "workspace" variables in LAPACK etc.

It's a really complicated piece of kit, that has also been around since long before modern build systems and conventions existed. That's why you have excellent scientific packages like Anaconda or Canopy, while on linux you can also install your distro's package. Situation is also becoming easier with pre-built python wheels.

Years ago, it used to be harder to install numpy, but this has changed. On MacOS, it's just

% pip install numpy

For more (e.g., other OS'es, build-from-source), see:

http://docs.scipy.org/doc/numpy-1.10.1/user/install.html

Two days ago I took a proof-of-concept project written in Python, using the GMPY2 library (Gnu Multiple Precision library), and ported it to C.

My C code was very complicated in order to be fast. Lots of pointer swapping, bit-shifting, etc.

The code was about 4% faster in C, which was more lines of code, much harder to read (and write), compiled with every imaginable flag that could offer a speedup. Switching operating systems, this code was about 8% faster in C on Linux than Python in Windows 10. C in Linux would take about 9.5 days to complete, Python in Windows 10 would be about 10.5 days.

In addition, Python has the "pickle" module so that I can easily make the algorithm save state to pause and restore it any time I want. What's more, I can probably use mmap to share memory easily without making copies and switch to multiprocessing for an even greater speedup (my first attempt using Pool was a little slower despite using 3 cores, but that's the slowest way to multithread).

All this because you didn't profile your code to see that 95% of the time was spent in GMP routines.

Yeah. I wasn't sure how GMPY was compiled, so I used C with my own compilation of GMP. I bet the flag differences made up the few percentage points.

Plus, with each variable being ~10Mb, I was concerned about memory copying (I studied mechanical engineering, I missed a lot of this stuff in school), and with C I could make memory copying vs passing pointers explicit. All of that was unnecessary.

It was another reinforcement of the idea: "Build it first, then optimize".

Pick Python for the sheer amount of libraries that's available. Not just for web stuff, where Node.js and Ruby do fine, but also for scientific work, for statistics, for embedded work (Raspberry Pi or MicroPython), for systems administration, for command-line tools, for really just about anything you can imagine. And you can do all of those things while not having to switch contexts between different syntaxes and different standard libraries for basic things like regular expressions or HTTP requests.

Picking the best tool for the job often gets in the way of actually getting things done. Python is the true swiss army knife of today's programming languages.

That's a good reason, but note that it used to be the reason you picked Perl.

Still a valid reason for people to chose Perl.

Not advocating for Perl, just on behalf of Perl lovers like my office mate

The difference is that most people can read most other peoples' python code. Perl didn't tend to have that quality.

EDIT Added another "most" to make it clear that not everyone can write readable python.

That depends entirely on who is writing the code. I have seen some truly awful Python as well as truly awful Perl.

This difference is that I've mainly seen decent Python and mainly seen awful Perl.

Although larger network effects might make it harder to turn the ship, this time around.

This is exactly the reason why I use C, C++ and sometimes Fortran. Yet to see a Python library I'd need.

I often dabble in other languages but in the end always return to python (3).

What it offers for me truly is the batteries included and easy, memorable and predictable syntax. (Probably due to experience).

Sure, Go is easier to deploy, Erlang and by extension Elixir are easier to scale.

But python does align very well with my personal though process. I don't think parallel and I don't have race conditions in my mind. If I want to quickly write something that will perform decently, I choose python, because the time I save by not having to learn a new language could be days or perhaps even weeks. I guess it boils down to this: with my current experience, I'm more then 10 times as productive in python as I would be in any other language.

Sure, it would be neat if I had as much experience in Erlang as I have in python, but that will take years, and at the end of the day, I'd rather finish my project this year with python, then next year with any other language.

>Sure, Go is easier to deploy

Go's lack of a package manager making deployment more painful is the thing that made me come back to python.

Go is staticly linked, you don't need to deploy packages, just the binary.

But you have to build those with its dependencies. And the best thing Go came up with is vendorizing.

And good luck keeping everything updated.

(Does go statically link the libc, too? If so, you should be rebuilding all your go applications right now, after the glibc issues reported on Tuesday…)

You can link a Go program completely statically, but it's not the default.

> But python does align very well with my personal though process.

TBH, that's the most rational reason ever for choosing any language. Anything else is much less important.

But, yes, for me the experience is the opposite, Python flies in the face of my thought process in every single aspect of it. I find it limiting, inflexible and dominating. Which means, everyone think differently, and there is no single language that suits all possible combinations of problem domains and thinking styles.

Concur with the 'thought process.' I'm a happier person after a day of writing Python than I am after a day of writing Go. Environments take a long time to mature.

Python finally has module packaging down fairly well (Go still drives me crazy even with the latest vending approach--cannot wait for a major build to break because the module creator switched hosting services).

Python is pretty pleasant to debug. The ability to set breakpoints and then poke around and/or modify values saves me hundreds of hours a year.

Building executables can still be a bit problematic (one reason we used Go for a few projects -- they really have this down). However, latest pyInstaller has been working smoothly on Windows and Linux.

Python is quick enough -- and we know how to scale websites with WSGI and stand-alone apps with asyncio (or Twisted). Rarely does the need for that last ounce of speed outweigh the ability to quickly create and modify apps.

Elixir, of all the new languages, intrigues me the most; not just for the language but being built on top of Erlang's OTP platform. They also seem to be doing a decent job on the packaging and distribution front.

Readability. Ease of scripting. Libs. Multi-paradigm. Simplicity to debug. Good instrospection. Expressivity. Fantastic community. Great documentation (because hey, docstrings). It's a no brainer to teach and to learn or train for and gives you good habits.

And it does the job.

All you said, and also: the REPL and ipython.

Python is still king for really boring and maintainable scripts. I mean boring in the best way. Every Python script o read is clean and elegant. It's easy to follow and well organized. Python 3 doesn't seem to make it any better at this. In fact all Python3 does is annoy me. I never know when I need it or not, it never seems to be clear either. The fact that it's packaged as a separate entity (i.e. /user/local/bin/python3) is even weirder.

Mastering Python-foo is still on my todo list, though. As a Ruby user, I've noticed that Python scripts are much more straight forward and maintable. I still write my scripts in Ruby though because nobody seems to mind. This is perhaps because I try my best write good comments. In general, Ruby is better as Perl replacement than a Python substitute. In fact, one of my coworkers who is an oldschool perl scripters was sold on Ruby over Perl just by reviewing one of my scripts. I think the issue is people don't think of Ruby outside of Rails, where it's basically a readable purely object oriented Perl.

As an aside, I think there is a bigger overall problem of using scripting languages for application development. I wouldn't use Python, Ruby, or Perl for anything that was more than, say, 500 lines. I almost cried when I tried to read the code for homebrew. Not that it's bad code (it's quite good actually), it's just way too much duck typing for me. I could never keep all that type information in my head. I prefer to pass that cognitive load to the compiler.

Fundamentally, python3 fixes some of the serious WATs in python2 and makes it easier to write high quality and maintainable software. For a pretty quick overview take a look at:

https://migrateup.com/main-difference-python-3/

Perhaps if python3 annoys you more than makes you say, "Hey, this prevents lots of latent bugs I might have easily shipped to production before, cool!", then you need to learn python a bit better and re-evaluate what is annoying about it.

I believe the parent was talking about python3 from a users perspective (they said they aren't a Python programmer) For example when a script requires python3 and another requires python. Python3 is also not included in a lot of base systems. That could be annoying.

Sorry, maybe I was unclear. I haven't really started learning python yet. I read it, and could probably write it. I don't know the ins and outs of the language. I don't even think I've written anything more than "hello world" in python, although, I'm pretty sure that program was also valid Ruby -- so it might not even count.

What I was trying to say is that the portability of python3 is bad enough that it effects non-python programmers.

For example, sometimes I want to try a new vim plugin that's written in python. I install it, and it doesn't work. Great. Then I find out it's because OS X's python can't interpret it. So I install python3 (which has a different executable name, also annoying). All good, right? Wrong. Vim needs to be compiled with python3 support because the python extension is exclusive to python. Now I have to recompile macvim with

    brew install macvim --with-python3 && brew install python3

It's not a quick process at all. Especially when I'm just trying something out. This stuff happens with other tools I've wanted to use too.

It doesn't ruin my whole day or anything, but it's a bit irritating to have to manage several versions of an interpreter for a language I'm not even using that language in production. It doesn't help that I use several machines and VMs.

So yeah, I know exactly what is annoying about it.

> or Go.

If you like Go, try Nim - it's like Go but without all the dogma. I wrote a blog post about my little Nim project: https://klibert.pl/output/slock_in_nim.html

> Why should I choose Python 3 for my new projects?

You shouldn't unless it fits all the constraints of your new project. Choosing a language is not, for the most part, a technical decision: sure there are language features and implementation features (those are separate matters!), but in the end it comes down to external factors: your experience with a particular paradigm, your taste in syntax, skills of potential collaborators and so on.

One thing of note, though: there is no rule saying that you can't use more than one language for your project. On the contrary, you can mix-and-match languages with relative ease nowadays (see https://klibert.pl/output/python_interop.html) and, in my experience, it works rather well. For example, in one of my projects I need to scrape a bunch of websites for data and then process this data. I wrote scraper in Elixir, which made dealing with network errors and parallelizing requests a breeze, and used Python with Pandas for the analysis part. It was a very pleasant experience: neither Python nor Elixir were up to the task on their own, but they worked very well together. It's even easier when you use a JVM, .NET or JavaScript-based languages.

In Nim, isn't it the case that concurrency is thread based? Something I really like about Go is that goroutines are cheap green threads, not full OS threads. This lets me spawn as many goroutines as I like without worrying about exhausting a thread pool.

There is some support for coroutines with async/await (http://nim-lang.org/docs/asyncdispatch.html), but I don't know how mature or functional it is, never tested it. As mentioned, for anything involving concurrency I'd rather use Erlang or Elixir; but it may be a good idea to see how well Nim does in this area.

Most of my development experience is in Python, and Clojure's what excites me the most these days. I've had to write a lot of Javascript for work, and it turns out that there's a large community of Javascript developers who wish they were writing Clojure, and they've built libraries like tcomb, Immutable.js and transducers.js to get pretty close.

In the transpilers era, Javascript will never be so bad that choosing another dynamic language is a clear win. I'd go back to Python and Django for a CRUD app, but since Javascript is easier to write in a functional style than Python, I've actually started to prefer it as a first choice.

For personal projects, you can pick whatever language you want, possibly the most fun one that is still practical. Haskell or Elixir look promising in this regard.

For companies, there are more factors to consider, eg. finding devs at a correct price, getting existing workforce to work on a new tech, size of standard/open source library, getting help and bugfixes from community instead of hacking on your own, risk of language/tech losing steam, etc. That's why several comparatively "boring" languages still thrive.

I personally find python in general to align almost perfectly with the way I think about writing software. It's interpreted so I can do things like compiling dynamic python in my code or fall into a python shell for debugging. Majority of time, I can use the standard library to do everything I need. And I love the clean syntax of Python — always a pleasure to look at.

As for Py3, the standard libs are very clean and streamlined, (e.g. io). Oddly enough, the point that I love about Py3 is its handling of unicode vs bytes (which most people find it difficult for some reason).

I'm not sure what you mean by "can't target the browser"? I don't find distributing/deploying python to be difficult or even involved at all. I do get annoyed at the single CPU bound limitation.

If I had to summarize why I'd pick Py3 for new projects, I'd say because it thrives in being minimalist and I hate clutter.

"Better the devil you know". If it's a professional project, I prefer not to have to learn the undocumented potholes of new language while debugging before a deadline.

I never thought I'd say it, but I'm loving programming the new ES2105 version of JavaScript. It really has a great deal in common with Python now. Clearly many programming languages are converging as they implement common good ideas from other programming languages. That's going to mean that whilst Python might have distinguished itself in the past for being streets ahead, in the future it's likely that many languages will also have many of those same good ideas incorporated.

I posted about it:

What language has decorators, generators, comprehensions, modules, but isn't Python?

http://fourlightyears.blogspot.com.au/2015/08/what-language-...

The "all async" model of Node kills it for practical programming compared to Python...

And the worst thing is, Node/v8's JIT would kill Python in performance for a;; the kind of things people write Python scripts for, even if it run all synchronous.

It would be nice if callbacks/promises/async-await were only optional and for network/web things, not for everything.

async/await isn't too bad (es7 via babel)

You can still use the "-Sync" versions of functions in the standard lib.

Yes, but not hundreds of thousands of npm libs.

> I never thought I'd say it, but I'm loving programming the new ES2105 version of JavaScript.

Well, yeah, I'm sure ES2105 is cool, but most of us don't have the choice of using languages from the next century.

You mean yavascript?

  > What language has decorators, generators, comprehensions, modules, but isn't Python?

Clojure, Haskell. :-)

The difference between python an javascript wasn't in features, it was always about predictability. I can pick out errors and expected behavior in python much more easily then javascript, and unless it breaks compatibility at some point, I suspect it will stay that way.

You did not define what that project would be.

I suppose you should first figure out the domain, then figure out the language support and ecosystem around that domain and then pick one. That is, if you are implementing something practical whose aim is to provide value outside of personal learning.

For example, several constraints still lead to a situation where C++ is a prime choice unless one is mostly interestes in hacking in a new domain and not finishing a project.

Note! I'm not dissing trying out languages for the joy of it!

Personally, I would pick Python for experimenting with syntax transforms, data mining and such. And as a basis for any offline tool that needs to chug bits from one place to another. Also, the numerical use cases listed elsewhere.

Basically, for anything that does not need a GUI.

If you are building an application which does a lot of machine learning and statistics, I don't see a better language than Python. It has so many libraries to use for it. Note, I love R, but I can't imagine building an application with it.

You can build apps with R, but the result will be the ugliest code you have ever seen. For me, R for research and prototyping, then Python for productionalizing.

Ugly isn't the problem. It is the poor error catching, erratic memory use, several ways to do one thing due to a lack of features in the standard libraries (regex), etc.

Noob question: What's so difficult about deploying/distributing python?

ATM it takes work to create a deployable artifact.

We package virtualenvs using fpm, others use Docker. But Go just spits out a single binary you can scp anywhere (the build process OTOH is more painful IMHO).

That’s why I’m highly curious about https://us.pycon.org/2016/schedule/presentation/2122/ .

https://glyph.twistedmatrix.com/2015/09/software-you-can-use...

This should be the #1 comment/resource on this thread IMO.

I recently started writing a few Qt applications, and was strongly considering PyQt - I mean, why wouldn't I? The syntax is cleaner and I can get more work done faster using Python than C++.

I spent a few days going back and forth - comparing the performance, design time, etc of both - and in the end I'm just using Qt's JS JIT for dynamic list comprehensions and other things C++ is bad at. In the end, the fact that distributing on everything but Linux would have been reduced to having to bundle CPython in was just a deal breaker.

Linux isn't really bad for deployment, though. You can use the SUSE OBS to build packages for all the major platforms besides Arch, and since my main desktop is Arch, I'm building my software with PKGBUILD's from the get-go.

Python isn't very big (about 10-20 MB or so). What's the problem with bundling CPython?

I've written a production app in Python 3 + QT4 (PySide as opposed to PyQT) - I've bundled with cxfreed, and it's great.

Fully cross-platform.. easy to distribute. Quite nice all around.

That page dismisses py2exe, but I've used it successfully for over a decade. On unix it is as simple as:

  sudo pip install [package/repo]

Except, you often need a full compiler toolchain for all the c dependencies. Then you need to make sure you have the required header files. Then you try and deploy to a small EC2 instance and run out of RAM building lxml or similar.

If your requirements are advanced, make packages on your dev box then deploy those:

https://nylas.com/blog/packaging-deploying-python

It's an interesting idea, thanks.

this +1000

The python users I've encountered fall generally into two camps:

1) The engineer/scientist/researcher/hacker type that has python installed on their snowflake machine, and has a whole bunch of packages installed to make their flavor of snowcones. These users just want python to work, find joy in the syntax, and in general get their work done.

2) the developer that is trying to deploy the same thing to many machines. This is a pain. Often you end up with users from camp 1 trying to do this and you end up with a global python install with batteries and the kitchen sink included that is a dependency for a whole bunch of "server" apps. you update a package (because "my special snowcone needs it") and you break something else that you didn't know to test. This is a nightmare. With pip and venv it is easier to manage this, but I still see that relocatable venv is "experimental" when you read the help. IMHO, the python 3 community needs to make it very, very easy to distribute and deploy my special snowcone maker and all its dependencies to a system without python already installed, and not stomp on anybody elses snowcone maker in the process.

Fair question. I can only speak from my own experience here:

Productivity, productivity, productivity.

Yes, there are faster, safer, and more modern languages out there. However Python for me still hits nails the hell out of getting stuff done, really really quickly. That applies both to the language itself, and from the standpoint of having such an extensive and mature ecosystem of packages to choose from.

> I struggle to find a compelling reason to use Python at all

My issue is I always see Python as the Second Best choice for what I am doing. There is always a better tool for the job. The one thing going for Python is that it is such a generalist BUT when doing work I prefer the Right Tool for the Job approach. I seriously can't think of a job that Python does best.

> I seriously can't think of a job that Python does best.

For me personally it is the best choice for non-throwaway shell scripting once the scripts becomes more complex and/or long lived than a sequence of shell commands.

Python is part of the base install of just about every mainstream Linux distro, so in that regard its only real competitor is Perl.