Hacker News Reader

Comment by pjmlp | Hacker News Reader

> But bottom line, tons of python code out there needs more performance and has no dependency on numpy or any C extensions.

What they need is nice JIT engines like PyPy and not to switch languages.

If no one invested in improving implementations for modern languages and switched to something lower level all the time, we would still be using Fortran for business applications.

View on HN

We still are using FORTRAN for Business Applications and I fail to see the need to switch them to a slower language to modernize them. Unless the switch is solving a program leave them alone. This is why in R there is still plenty of FORTRAN code.

Personally I LOVE Python but I always feel it is the 2nd best choice. I do use Python but it rarely is the best tool to use for the problem you are solving. Maybe Pypy will turn this around, and I hope so.

How can SciPy be fast if it is written in an interpreted language like Python?

Actually, the time-critical loops are usually implemented in C or Fortran. Much of SciPy is a thin layer of code on top of the scientific routines that are freely available at http://www.netlib.org/. Netlib is a huge repository of incredibly valuable and robust scientific algorithms written in C and Fortran. It would be silly to rewrite these algorithms and would take years to debug them. SciPy uses a variety of methods to generate “wrappers” around these algorithms so that they can be used in Python. Some wrappers were generated by hand coding them in C. The rest were generated using either SWIG or f2py. Some of the newer contributions to SciPy are either written entirely or wrapped with Cython.

A second answer is that for difficult problems, a better algorithm can make a tremendous difference in the time it takes to solve a problem. So using scipy’s built-in algorithms may be much faster than a simple algorithm coded in C.

http://www.scipy.org/scipylib/faq.html#how-can-scipy-be-fast...

This is also why the scipy/numpy crowd doesn't care much about PyPy. In this world Python is the equivalent of ducttape. You don't performance optimize ducttape, you want it to be easy to use and to fix any problem you have.

PyPy is taking a similar approach. The difference is Numpy is using CPython's CAPI as the duct tape where as PyPy is using cffi as the duct tape. Both approaches end up using the same underlying libraries.

Too many people just have the wrong impression, thinking that PyPy plans on re-implementing all the libraries that Numpy and scipy use but that's just completely false.

They have re-implemented the Numpy array so that it can take advantage of the JIT and so that parts of an algorithm implemented In Python that uses Numpy can also be optimized. Unlike what occurs when using Numpy under CPython where the Python code does not get optimized unless it is converted to Cython, C Code, or some alternative to Python to have it be optimized.

I like the quote: "Python is generally the second-best language for everything".

I don't understand numerical analysis as business applications.

For me business applications are what you would use Cobol, Clipper, Java, Delphi, C#, Eifel, ....

Of course Fortran makes sense for number crushing, but that is the language domain, not doing CRUD, ETL or distributed computing stuff (not counting MPI here).

Accounting, spreadsheets, transaction processing, business intelligence?

Of course those are business applications as well.

Do you intend to refer to a product done in those areas Fortran in modern days, maybe with Fortran 2008?

Outside scientific computing and heavy number crushing algorithms and libraries, I don't see a use for it.

No, I'm just saying they're business applications. A segment of business applications are numerically focused. Fortran handles them well. So, in that segment, it can be considered a language for business applications with numeric focus.

I'm not saying anyone should use it today lol. Julia and R are the best for that sort of stuff. Way the hell ahead of Fortran. Although I thought about making a nice web application in Fortran just to screw with people who eventually try to extend it and gasp in horror. ;)

http://www.aim.env.uea.ac.uk/aim/info/fortran.pdf

R != Business Application.

Used R as an example of a language that still uses FORTRAN. BUT I think Microsoft and Business Analysis people would not agree with you.

I think you guys are saying the same thing. Ie, because they don't need to switch languages, and PyPy is a JIT engine for Python, these tons of Python code would benefit from if the devs were aware of PyPy.

Devs are already aware of PyPy. If it was as easy as being aware, everyone would switch tomorrow. The problem is that tons of code uses C extensions created with swig or python boost and those take a lot of time to port to PyPy. Another problem is that PyPy is sometimes slower/less memory efficient than CPython when you wrote the code that is optimized for CPython. That is abusing dictionaries and very limited OO use.

PyPy has higher memory overhead for very small units of work but usually once you do more than a couple of seconds of work it usually uses significantly less memory than CPython. As far as I'm concern who cares if a Hello World program uses more memory or some small trivial few lines of code. I'm more interested in real world code which my experience has been that PyPy uses significantly less memory than CPython. If you know of a case where this is not true report it as I'm sure the PyPy devs will consider it a bug.

PyPy also has a slower startup time/overhead so very small amounts of work may end up slower than CPython but in general larger chunks of work will be faster if not its a bug and should be reported.

There is currently some work going on to improve the C extension issue. It's currently in its beginning stages and its goals are to provide both a significant amount of compatibility with existing extensions and a significant reduction in the overhead to call out to C extensions over what is currently implemented in PyPy. On the performance side the new approach was said to remove about 40% of the existing overhead. There are tons of corner cases that still need to be worked on for the new approach to become production worthy and there is no guarantee that there will be no show stoppers that come up or that the tail of corner cases that need to be dealt with becomes a nightmare and the approach is abandoned. If this works succeeds, it will be a game changer as suddenly the majority of libraries would then become compatible with PyPy.

Indeed that's what I meant and perhaps pjmlp thinks that the feature set in numpy could have been / could be implemented in pure python, now that we have PyPy.

What I mean is that I see PyPy very positive, as a way to stay in Python instead of jumping into C, Go or whatever language.

In terms of performance naysayers only get convinced when someone proves them wrong, not because of what we think might be possible.

So it is important that we have research in GC systems, JIT compilers for dynamic languages, optimizers for FP and LP languages and so on.

Otherwise we might as well keep on using just Assembly.

Numpy is way faster than what you would get if you implementing things in pure Python and executed with PyPy. That's why there is numpypy in the first place.

Reply on news.ycombinator.com