Hacker News Reader

Comment by theptip | Hacker News Reader

I think there is scope to replace Excel, but it’s hard. You’re seeing a gradual spread of “data science” tools in finance, as the more technical analysts start to use Python over Excel. But as you say, the management layer still uses Excel to poke and prod a model.

I think that there will be a generational shift here - you are not going to train an SVP to use Python, but the next generation of SVPs might have more exposure and be willing to use Numpy in a Jupyter notebook.

And in the other direction - there is definitely scope to come up with an “Excel isomorphic” Python framework for data science. It’s fairly easy to generate an Excel sheet from Python computations, but maintaining bi-directionality is Hard, and would require restrictions on the Excel side. I think with the right UI, you could do this though.

View on HN

I have a colleague who is a PhD in Applied Math. Pretty bright guy, huge Python/Jupyter lover with many years of experience. Loves to use git, loves to write dozens of unit tests. A true Man of the Future, according to Excel haters.

He wrote some Python to solve some mildly complex business problem. I told him to translate it into Excel for stakeholders. He did, and the answer came out completely different.

It turned out he had made multiple catastrophic errors in the Python. This is not the first, second, or third time this has happened.

Python, and tech-beyond-Excel in general, just isn't the silver bullet software types often seem to think it is. Even experts sometimes seem to do a worse job in it than in Excel.

I love both spreadsheets and Python/Jupyter. But neither is Excel the silver bullet? The reason the error is caught (and the real lesson) is more because he tried to reproduce his work as opposed to a particular technology stack.

I have an equal number of anecdotes where a few lines of easily readable python replace complex Excel calcs that invariably have a mistake somewhere.

The tools aren’t the issue.

That is hardly special to Python->Excel translation, and happens any time someone reimplements a piece of software.

I think Excel has something to do with it. In Excel, you're usually forced to do computations in small steps and look at the intermediate results. In traditional coding, you don't see anything but the final result (unless you ask to see it). It's much easier to assume everything is working as intended when it's not.

Excel has its problems too. Different tools for different jobs. Tech boosters need to understand this and not just cynically assume that spreadsheet lovers are old fogeys who are afraid of their jobs being automated away.

I think you’re just describing poor coding practices

Certainly if you do traditional TDD or something like REPL-driven development, you see the intermediate results and validate the correctness of your code as you go.

Those are great! But they are coding disciplines one must choose to follow, and continue following every step of the way. You aren't inherently following them because that's how the tool works. Sometimes the tool-enforced discipline has advantages.

How would one write tests for a complex, mission-critical Excel spreadsheet? Or use version control?

Spreadsheets often mix the data and code/formulas and the formulas are hidden behind the sheet view and sprinkled across many cells. At least Python scripts separate the code from the data so you can write tests using known good or fuzzing data. And you can use version control to track and review code changes.

I agree that, in the hands of a hypothetical ideal person, Python should be better than Excel for almost everything. But as my story above shows, even people you'd expect to be experts, seemingly following best practice, don't end up being very close to this ideal person.

Honestly, your story just indicates that someone senior needs to have a discussion with the applied mathematician about their coding practices (and they may even need a bit of training in that regard).

His code style is fine. His problem is that he is short on the discipline and focus to validate his results in meaningful ways. An exhortation to follow some vague "coding practices" won't fix that - he'll respond e.g. by writing lots of tests, none of which catch the problem his code actually has.

The two ways I know to get correct work out of him are (1) review it and kick it back to him when problems are found, (2) have him implement what he's trying to do in Excel.

It just sounds like he was never actually trained to program, and letting these bad habits persist isn’t doing him any favours.

I'm not sure I understand the idea that people who are "trained to program" manage to avoid these mistakes.

What specific bad habits do you think can be addressed?

I know it's hard to diagnose anything from some forum posts, but I'll take your speculation as to what he should maybe work on.

So I was mainly a Haskell programmer in grad school before getting into scientific computing, and the main good habit I picked up was working at the REPL and breaking my code up into small, testable functions/components, and thinking very hard about state. The main bad habit I had was expecting the Julia compiler to be as helpful as GHC when refactoring code.

While I was writing my thesis and job hunting, I attended a few workshops aimed at “grad students breaking into industry”. The main thing I noticed from applied mathematics students in particular was they would write out long functions (really hard to debug) or they worked exclusively in Jupyter notebooks (these have super complicated state, so it takes a lot of discipline to be able to translate these into usable code).

You said he "loves to write dozens of unit tests" so why so many errors?

Sometimes it's hard to think of the right tests, especially when you're solving a mathematical model that hasn't been solved before.

Even for things that are intuitive and have been implemented thousands of times before, like web logins and shopping carts, where the tests one should do are not hard to think of... even so, software engineers rarely develop tests that catch all possible bugs on the first try.

Replacing excel is like replacing paper. People always talk about formulas, but most spreadsheets don’t have any. They’re todo lists, or more commonly, static database exports.

Or they have formulas, but the author calculates a few numbers and throws it away. In this case they’re like a scratch pad, or a calculator with a visible memory.

I think it’s certainly possible to make better business modeling tools, and have played with some designs in that space. But they’ll never be Excel. And that’s ok

Reply on news.ycombinator.com