Stop Using Excel, Finance Chiefs Tell Staffs | Hacker News Reader

Stop Using Excel, Finance Chiefs Tell Staffs

wsj.com

627 pointspetethomas417 comments

I would argue that the most commonly used programming language is Excel. But few of the people using it realize they're programming.

It's a brilliant reactive data programming model that makes intuitive sense to non-technical users. They feel empowered to use it to solve problems right now with a computer. They experiment with it, try things, Google how to do more things- just like any programmer does. And they feel capable of doing this because they don't know they're programming.

Within the Amazon warehouse world, I have seen incredible innovation using it. An acquaintance of mine got into development by using it to help save soldiers lives while serving in Afghanistan[0].

I agree that mission critical data needs to get out of it and into a centralized system, but I still feel Excel is an incredible tool in any business.

[0]https://news.ycombinator.com/item?id=7950190

I used to be a Lotus Notes specialist, and one interesting thing I noticed is that from the earliest versions it was designed specifically to be a "step up" from a spreadsheet (Lotus of course also made Lotus 1-2-3, which was the original killer-app spreadsheet before Excel took over).

Notes still let you see data as rows and columns and had very spreadsheet-like functions, many of which could be transferred directly from Lotus 1-2-3 - but it was a full client-server system which supported decentralized replication, built-in messaging and email, and a solid security model (yes yes I know it was also a crazy confused mess and a UI nightmare but that's a different topic).

Back in the day, organizations had teams going around finding the most useful ad-hoc spreadsheets and converting them into simple Lotus Notes databases. Other organizations would sometimes also give staff training in building simple Notes databases - not a whole lot more difficult than doing a spreadsheet - and then have expert teams come in and polish them up as needed. This made business software bottom-up rather than top-down, which turned out to be a whole lot more useful in a lot of cases.

In many ways, even after I left the Notes world for PHP and Rails and JS, my career in business software has still mostly been about converting spreadsheets into more "proper" applications.

Spreadsheets are the 'blue-green algae' of the software world: the simple base level that's everywhere and surprisingly essential.

> my career in business software has still mostly been about converting spreadsheets into more "proper" applications.

Still true for many people in 2017. @patio11 probably said it best:

> Every spreadsheet shared in a business is an angel announcing another SaaS app still needs to be built.

https://twitter.com/patio11/status/655674551615942657

FWIW we ended up in business software after one of our open source libraries to read and write spreadsheets (https://github.com/sheetjs/js-xlsx, demo http://oss.sheetjs.com/) ended up garnering lots of demand from businesses looking to build those "proper" applications

> Every spreadsheet shared in a business is an angel announcing another SaaS app still needs to be built.

It's a pithy quote, you'll get a very different reaction from a finance or small to med-sized business professional.

Excel is the ultimate maker studio that actually lives up to the promise of "build your own mini-app". I'm not saying it can't be improved on. It's just that the core model of flexibility is so compelling, pushing everyone into a SaaS app is invariably going to take away substantial power from the user.

True but there is a threshold. If the problem is small enough, you can't beat an Excel spreadsheet's flexibility and low startup cost.

Having worked with a number of finance teams, the biggest problem is not when Excel is used as an analytics tool but as a financial database. Excel as a source of truth for financial data is truly terrifying. Even financial audits are still often handled with custom data pulls exchanged over Excel spreadsheets.

Shameless plug: that's why we ended up developing one of those SaaS tools (http://modfin.io). It lets finance teams map their source data to a proper, easily auditable accounting ledger so that they don't need to do their "magic" in Excel.

No. An domain-specific SaaS app takes all the power away from the people writing their spreadsheet programs. Only one app is needed, a hosted spreadsheet with backup and versioning an replication. The only domain-specific business is in writing plugins.

A nice quote but getting data between SaaS apps is much harder than between Excel sheets, and modifying a SaaS app with that one little thing you need for your particular case is of course impossible.

> Every spreadsheet shared in a business is an angel announcing another SaaS app still needs to be built.

With new (or prospective) clients that are SMBs, one of the first things I ask is, "so, what are you still using spreadsheets for?"

> Every spreadsheet shared in a business is an angel announcing another SaaS app still needs to be built.

Really? I am working in company which is not handling super-sensitive data (i.e. not a Bank, for example) but I can assure you that if went around proposing to adopt a SAAS for what Excel is used internally I would not get very far.

No matter how good you are as a SAAS provider, our data stay in-house. End of story.

As a Lotus specialist I can see why you'd think Lotus 1-2-3 was the original killer app spreadsheet and as someone who worked retail at the time selling UNIX and Microsoft machines I'd have to say we sold a ton of Lotus 1-2-3.

But we have to give Dan Bricklin and Bob Frankston their dues because Visicalc is the original killer app spreadsheet in the literal sense: people used to say they didn't know if Visicalc sold more Apples or if Apple sold more Visicalc. It truly was the program for which people bought the platform.

You're right - my mistake!

Maybe I did not appreciate it properly, but I remember using Lotus Notes at my work a few years ago, and I can remember it being the most awful piece of software I have ever used. Admittingly I am not a specialist, more an employee trying to use it to do my job.

> I can remember it being the most awful piece of software I have ever used

Many people seem to say the same thing but the only problem I could ever see (in occasional use) was a non-standard UI which non-technical users had trouble grasping. What made it so awful?

It was pretty awesome for enterprise software in the mid-90s, but the world caught up with it soon after.

It is, you're not wrong

What would you use now if you needed the same sort of functionality? I find myself with some unexpected needs to build databases whose specifications and scope I'm still trying to figure out, and for now I'm missing about in Google sheets and figuring Fusion tables will probably be sufficient, but I'd be interested in any alternatives as long as they're secure and shareable.

Some combination of things like Python and SQLite would probably go a long way?

I basically agree - but in my case, instead of Lotus, it was Framework: https://en.wikipedia.org/wiki/Framework_(office_suite) - I think I used versione III and IV, mostly.

This sounds familiar to my, albeit limited, Microsoft Access experience.

Completely agree.

It's common to tell kids (and everyone) that they need to "learn to code." That is what Excel already is! And it's incredibly powerful, especially if you don't have any other coding skills.

But, of course, it's messy. When tens of millions of people can each write their own programs and easily modify them and change them to suit their own needs without needing to recompile or anything, then you get the mess which is Excel.

...and, in fact, that's also what you'd get if you taught everyone how to program Python in high school instead of Excel. You'd get tens of millions of non-interoperable messes of programs (which still get the job done!).

But that's fine. Give humans tools, and they'll find clever ways of using them that you don't intend. And it'll continue to drive people crazy. In some ways, the Excel hate (though obviously not unfounded) seems almost kind of like a bourgeoisie elitism. THIS is what it looks like when you give everyone the ability to code!

> ...and, in fact, that's also what you'd get if you taught everyone how to program Python in high school instead of Excel.

I disagree. Python leaves a readable instruction set, while Excel buries the logic in an opaque two dimensional grid of cells (plus or minus atrocious VBA). I think this gives a better ladder for people to improve their skills to the point of not producing unmaintainable garbage.

The grid of cells is your memory space. Coders like looking at a stack. Spreadsheet users like looking at the registers.

Which you prefer depends to some extent on which end of the problem you're coming from; if you're starting out with a lot of domain knowledge, a spreadsheet is attractive because you can look at the data, see whether the way it's being stored/processed is basically correct by spot checking it, and then automate. I'll still often reach for a spreadsheet if I don't know off the top of my head what the rules or relationships are and I need the equivalent of a whiteboard with logic.

If you have a top-down perspective and you want to automate a process whose fundamentals you know well, code is more comfortable. And you can go back and expand, modify, or adjust it later. But two things I don't like about code for handling large volumes of data are that I have to construct some sort of viewer to format the output (requiring more coding, or slower bug discovery if I press ahead without), and endless typing I like selecting things from menus and tool palettes because while I don't mind holding an instruction set in my head I don't like doing the same for every library I might want to use, especially remembering all the parameters. I hate hate hate writing glue code and want the computer to do that stuff for me.

Fortunately a lot of these problems seem to be receding thanks to tools like Jupyter and some of the promising Flow-based programming tools that let you easily visualize structure, code, and visualization without getting bogged down in details.

I should add that I prefer hacking to software engineering; that is, I'm only interested in programming as a means to an end of having a customizable tool for myself rather than building a product for other people, so I want spend as little of my time as possible on constructing the tool.

> Python leaves a readable instruction set

Not everyone structures programs in the same way, or puts functions in easily found locations, or names things usefully, or even approaches the problem in a way that makes sense to the majority of other people. For a concrete example, see this[1].

> Excel buries the logic in an opaque two dimensional grid of cells

It would be trivial to represent excel grid formulas as a set of predefined functions, or a special data structure that you register functions in, and then it could be in textual form. It wouldn't make it any easier to reason about, but then again control flow isn't always easy to reason about when a program is split into many different objects and functions, until you've internalized a fair amount of the program. Spreadsheets give people a very limited, but structured output and a very limited, but easily reasoned about way or processing. It's different than "normal" programming, but that difference allows for very intuitive use.

> I think this gives a better ladder for people to improve their skills to the point of not producing unmaintainable garbage.

How much work has been put into making spreadsheets easier to share with other people? I'm not sure the same culture has existed for spreadsheets as for programming, where people have needed to share and reason about complex algorithms and data structures. If it had, I'm sure we'd have much more in the way of how to structure your excel well so it can easily be consumed by others, or yourself in the future (if it doesn't already exist, which it may).

1: http://thedailywtf.com/articles/Python-Charmer

That's exactly what happened with Perl when everyone learned to use, modify, and develop CGI scripts in the early days of web-based application development. Dynamically-generated website content worked and evolved into the web we know today, but those scripts were awful and did a lot of harm to Perl's reputation.

Craziest thing I ever found that someone did with excel was implementing a neural network using it. It was meant as a tutorial to teach how neural networks worked, and not meant for production systems, but I do wonder if anyone ever took it further.

Stands to reason that "everyone" is coding wrong. You wouldn't say people are programming just because natural language is turing complete, or because they are while physics is turing complete. One important aspect of information is structure, but programming classes at the low level merely teach syntax. So I'm thinking that therefore spreadsheets are unstructured. Sure that's elitist or at least academic - in comparison to amateurish.

Excel is a great way to code, but a horrible way to decode. Every try to pick apart someone else's spreadsheet.

have you tried to do scientific plots with Excel? I only tried libre calc, which admittedly might lack behind in a couple regards, but it's nigh impossible, no compare to scripting.

When I consulted at a health insurance company (a Blue Cross Blue Shield licensee with 1000 employees, 200 IT staff) about 10 years ago, I was absolutely horrified that the system of record for health care plan data was Excel spreadsheets. They had staff who would update the claims processing system based on thousands of spreadsheets. When you called customer service, the CSR app would auto-open the member's plan information XLS file to the CSR rep. It got the job done - but it was the wrong tool for the job.

We recommended they implement a PDM (product data management) system - it would have saved them millions of dollars (considering they had a staff of 20 whose sole job was to update the claims processing system by hand based on these spreadsheets.) Unfortunately when the 2008 financial crisis happened, management balked at the initial outlay. It's quite possible they're still using Excel to this day.

There are upsides to excel in cases like that…

The health insurance people of various sorts who use this setup are the ones with domain knowledge. With an excel program, they can (1) understand how it works (2) add to it, modify it (3) create a copy and break it (4) debug it (5) have ideas about how it could work differently…

I’m not saying that excel is the tool for the job (sounds like it is definitely not), and those advantages erode as the setup gets more complex and/or has more people using it. But, there are reasons excel gets used this way.

The alternative is often rough. Software projects like this have high failure rates. Getting “requirements” out of users’/clients’ heads and into programmer’s is difficult. It means the standard (natural?) gap between domain knowledge and programming knowledge is restored. That has costs.

I’m going to repeat again that I am not advocating health insurance companies run policies and claims in excel. It is a bad idea, almost certainly. I am saying that we should be curious at why they are doing it this way. I think it’s a “good” question in that it leads to interesting places.

If the excel team was blessed with a plague of inspiration tomorrow morning, Is there a way they could make it so that excel is the right way to manage an insurance business?

The problem is switching would mean transitioning to a generalized system that they would have to work around. The Excel system looks like a nightmare to the HN crowd, but it was built around their business processes. It's basically a boutique app designed specifically for their job, it would be very hard to find a general product that does the job as well.

I've seen this a lot where a consultant will swoop in and tell them that they need to use Peoplesoft or something and the organization spends millions of dollars and years of retraining to get a system that has a lot more friction in their actual workflow and is never quite as good for getting the job done.

Of course there are often side benefits that can't be overlooked, like a much more sane backup solution, interoperability with other providers, etc... But in the end most of the employees will tell you straight up that the old system was better. Too much of the domain knowledge was baked into the old system and doesn't make the transition to the new system.

>When I consulted [...] We recommended they implement a PDM (product data management) system.

>We recommended they get a system that manages their product data

Yep, I believe you are a consultant.

They should be careful though: excel allows a maximum of 1,000,000 rows. I have seen more than one major crisis occur when a spreadsheet ran out of space, yet all of it was needed together to do daily business.

Yeah, updating these sorts of insurance systems was a huge line of work for Rails devs in the late '00s.

You can do crazy shit in excel. After someone opens an excel sheet - download a webpage and save it to disk? no problem. Lets load other excel files in memory and run their scripts too! Don't worry about your computer fan - while we scan your disk for stuff. Just state "On Error Resume Next" and start scanning hardware. Company email accounts are stuffed with one off sketchy excel attachments.

https://stackoverflow.com/questions/27419804/vba-excel-downl...

When you put it like that, ransom1538, I'm not going to click the link!

The idea is that general Excel files do not run macros. They need to be saved with a special extension and you need to manually approve macros, with warnings if the file is in a location where other users can write to the file.

I have no idea how watertight this is though, it’s Microsoft quality (and some of the most ancient code they have) so I wouldn’t bet the farm on it.

I've been working on a spreadsheet which is more amenable to embedding in programs (compiling to an IR which can be translated to high level languages or [eventually] compiler IRs like those of LLVM or GCC). I worked at a company where months were spent recreating models from a spreadsheet produced by subject matter experts, and this was a major cost.

A spreadsheet optimized for integration into software build processes would probably create more than a billion dollars of real value every year with even limited adoption.

While it may be true that Excel is overused, I believe that the spreadsheet is chronically underused as a model for software development.

I'll join in with a bit of show-and-tell of my own, if I may.

I'm working on a plugin for Excel for dev's and other technical people. It lets the user use SQL and C# in Excel, connect to various types of databases and move data in either direction, and provides a runtime for automation.

The idea is basically to let tech users use their existing skills to make life easier. Main areas of use are:

- process (live) data in Excel via SQL and C#/linq

- move data between Excel and databases

- fetch data from various sources using .net (e.g. get data from a AD, file system, REST etc. in a few lines of code)

- replace VBA with C# and .NET for automation

It's also very useful for prototyping. Business logic goes inside C# scripts, and Excel takes care of user input, displaying results, data binding (formulas and graphs) and data storage.

It's commercial but freemium, the url is http://www.querystorm.com.

I posted about it a few times: https://news.ycombinator.com/item?id=15670030 https://news.ycombinator.com/item?id=11583488

Here's an intro video: https://vimeo.com/242216594

Interesting: optimized for integration is what I've been doing too. If you've got a minute to take a look at the links in my profile I'd be keen to hear your feedback.

This idea looks super interesting. Is the source already online, or will be?

Excel is simultaneously flexible enough to solve a wide array of problems and structured enough to enable larger tasks like slicing and filtering data. This article is trying to push for cloud-based silos that solve specific problems, but most of them fall apart in "real-world" scenarios because they aren't flexible enough to support the ad-hoc workflows people have developed over years.

The dystopia that I fear is an endless gauntlet of powerless employees droning 'the computer doesn't have a field for that' or 'the computer won't let me do that'. We already seem perilously close to this, putting the consistency of systems ostensibly built to serve the needs of society over and above the needs of society.

"The dynamic spreadsheet is a good example of such a tissuelike superobject. It is a simulation kit, and it provides a remarkable degree of direct leverage. Spreadsheets at their best combine the genres established in the 1970s (objects, windows, what-you-see-is-what-you-get editing and goal-seeking retrieval) into a “better old thing” that is likely to be one of the “almost new things” for the mainstream designs of the next few years.

"A spreadsheet is an aggregate of concurrently active objects, usually organized into a rectangular array of cells similar to the paper spreadsheet used by an accountant. […] A spreadsheet is a simulated pocket universe that continuously maintains its fabric; it is a kit for a surprising range of applications.”

– Alan Kay, “Computer Software”, Scientific American (1984). pp. 56–57. PDF: https://frameworker.files.wordpress.com/2008/05/alan-kay-com...

Republished as “Computer Software” (1984). pp. 6–7. PDF: http://home.cc.gatech.edu/guzdial/uploads/130/Alan%20Kay%20-...

I definitely agree that spreadsheets (of which Excel is just the latest) have drawn a lot of people into basic programming. But at the same time, it has stunted a lot of people's growth because of its sharp limitations.

Spreadsheets have a lot of similarities to the sort of code bases that as an industry we've been moving away from. E.g., no version control, no tests, opaque code, bugs are hard to find, write-only code, bit rot, random antique scripts used in critical spots, poor reusability, latest version is on somebody's hard drive somewhere, nobody knows which version is the right one.

I also think it's deeply unfortunate that there's no good path from spreadsheet master to real developer. As great as spreadsheets are, they're a cul de sac. I hope this new generation of tools fixes that. If not, somebody sure should.

The issue isn't that you can't program in excel the issue is that excel lacks many of the safeguards that have been built around programming in the last few years.

Version control for excel is terrible at best. Code review is basically non-existent and how could it be when logic is split over several cells and variables are named $A17 or B23.

I think the win with excel is its super simple interface, that said there is plenty of room to improve on the programming language interface.

> variables are named $A17 or B23.

Excel lets you name tables, cell ranges, or individual cells. I have seen spreadsheets put together by people who take the time to do this, and they are remarkably readable!

Way back before they added visual basic into it Excel had what was called a Macro language that was also built in sheets - just slightly different appearance (very wide cells) and altered functionality (to prevent you from doing numerical operations to your macro code). I think this was around version 4?

Anyway, it basically worked like a stack, with the currently selected cell being the instruction pointer. So when debugging you could either look at your data and watch blocks of numbers flashing in and out of existence or changing state, or look at the macro sheet and watch the pointer bounce around as it went to different branches or sat in loops or whatever. There were commands to turn screen updates on and off (which sped things up considerably) so as you got more and more of it working it would go from looking like some confusing nightmare to a smooth list of progress bars and passed sanity checks.

They moved to VBS because programmers were finding the cell model too restrictive/weird, and for integration with Access, but doing it all in cells was remarkably and pleasantly like assembly-level debugging. You'd hack something up quickly, figure out your algorithm, then name the ranges and use and index value, and you could lock ranges and what-all else. It wasn't as easy to read as a page of code (eg no whitespace) but the crudity of it pushed you towards that sort of 1-operation-per-line discipline that assembler demands and really forced you to think about semantic efficiency. I had macros consisting of thousands of cells of code with every single operation commented. You could, in turn, name the ranges that had macro code in them and do code profiling to find out where your performance bottlenecks were, and so on.

If Excel is even a small part of your job, it is worth taking an hour of your time to watch Joel Spolsky's "You Suck at Excel": https://www.youtube.com/watch?v=0nbkaYsR94c

Some HN comments here: https://news.ycombinator.com/item?id=12448545

TIL, thanks for letting me know

Completely agree. I think simply enforcing a version control would be sufficient to solve most Excel woes, however. It doesn't solve manual user error (a la Reinhart-Rogoff), but it's a good next step.

Excel totally totally missed the wonderful opportunity to do so much better than Google Spreadsheets. They could have created a centralized shared way of creating, editing, sharing and storing data. Their limitations as it exists today are ludicrous at best - 10 max users? Give companies a way to keep the data in a centrally managed place with access control and auditing and users the ability to freely share their creations (and data)...

M$ft XL developer here. I attended a conference to meet our Excel guru users and it's fairly impressive what these non-coders achieve. Excel is like an operating system to them, and they even create Excel files to "debug" there other Excel files, e.g. debugging the order of formula calculation and validating the output at each step in their property assets spreadsheets - without using any VBA

MS access is often a way better tool for what people do. The problem is that it requires a bit more work up front, a little more abstraction, and usually some programming. Having said that, most companies have people in IT that can help out something like that together for people but that's not their job.

> it requires a bit more work up front

Because of this "bit more work", it's also much less flexible in the long run.

Excel is not a great database, but it's a highly flexible one.

Does access have a spreadsheet view for bulk editing?

Yes, it’s called the datasheet view.

I think that excel is one of the best applications on the market.

Every single department in any company can benefit from knowing it well.

The problem is getting people comfortable and competent to know it well.

This is where I think google sheets fucking failed: they should have made an effort to make sure every single gmail user had a tutorial to know how to empower their curiosities and interests via google sheets.

> few of the people using it realize they're programming

An idea I've been toying with for a few months now is an automation app (read: window manager) whose UI is somewhere between Excel and GameMaker Studio's drag & drop UI. Like where you just have a giant grid/canvas, and where you can define "routines" and "variables" as groups of cells. If I could figure it out, I think it would open up the idea of an automation app to a broader scope, to people who aren't programmers, and I might be able to make some money off it. Especially considering I made like $500 in donations for Hydra back when I was doing more WM stuff for Mac, so there seems to be some excitement for the idea of programming your Mac, but perhaps it could be a living if I can figure out how to reach out to power users who aren't programmers. And it was your comment that gave me the idea of making it more like a grid instead of more like GameMaker. So, thanks!

I found this just the other day: A User-Centred Approach to Functions in Excel [PDF][2003]: https://www.microsoft.com/en-us/research/wp-content/uploads/...

Honestly the article stinks like an advertorial of cloud vendors trying to push their wares. I agree that Excel isn't a good fit for many things, but blanket rules like 'no excel' seem counterproductive.

Why do we readers take these sorts of things at face value? You know the whole article is a sales pitch.

I suppose you either understand how PR works and simply disregard anything printed or broadcast and do your own research, and accept this system unlikely to change...

Or you get irritated by it, while waiting for the "inevitable" collapse of this promotional/propaganda model...

Honestly I don't know what is happening, but it seemed like the first option, which I've followed for a couple of decades is a good enough cure for disillusionment and depression. (Caused by investing in the second option)

Edit: add quotes to inevitable.

Compare "Improving the world’s most popular functional language: user-defined functions in Excel" https://www.microsoft.com/en-us/research/publication/improvi...

UDFs are the real LPT. Much of gnarly spreadsheeting going around has to do with having to simulate loops with rows or just partition very large formulas (think option pricing formulas) into partial result cells.

Could you explain your first sentence? The acronyms are a bit dense.

Edit: UDF is user defined functions. What's LPT?

Every excel spreadsheet is an error waiting to happen.

Implicit & unexplained business rules, no tests, no version control, unlabeled cell references, hardwired variables, everything is global, no encapsulation, no logs and no documentation.

Even assembly code is easier read and maintain than a complicated excel workbook.

Functional programming before it was fashionable!

This is how I like to explain functional programming to developers today. When you say "Functional Programming", a lot of devs think "scary complicated thing". When you say "Basic excel spreadsheets" they go "anyone can do that!".

Excel is the Swiss Army Knife application - it isn't the best tool, but it's the most versatile and the one you have with you. Good insight about the relation to programming.

In the book "A Small Matter of Programming"[0] (1993), Bonnie A. Nardi makes the case that there are really only two successful end user programming systems - spreadsheets and CAD.

I think that's just as true today, at least as far as business software goes.

0 - https://mitpress.mit.edu/books/small-matter-programming

Many of the larger businesses I have seen(not too many) have a disconnect between the subject matter experts that use Excel to codify their knowledge and the developers. The program(Excel workbook) often isn't big enough to warrant the dev costs when it works "now" but is too important not to in the long run.

I do think this is a tooling issue that could generate code based on Excel workbooks.

I agree that mission critical data needs to get out of it and into a centralized system, but I still feel Excel is an incredible tool in any business.

You would have thought this would be obvious- it seems analogous to perl/python or other such Swiss army knife tools we use for quick one-off solutions that occasionally get pressed into permanent formal use where they shouldn't.

Python (and probably Perl too) work fine for permanent formal use as long as they are engineered as such. I have seen clear, well structured permanent code bases in python. The problem is just that people like coding one off prototypes in python, and those end up in prod; it's just as bad as when Java prototypes end up in prod, but who actually wants to write prototypes in Java, if given the choice?

I agree - it's also really useful as a prototyping language, since its accessibility allows domain experts to really make something that suits their workflow. My current company (Parsley, www.parsleycooks.com) would have been a lot harder to do without looking at the Excel sheets that the more tech-savvy and organized chefs use for logistics work.

> I would argue that the most commonly used programming language is Excel.

Anytime another programmer asks me what language they should learn I tell them Excel and they think I'm joking. Besides being a quick and dirty way to solve a variety of problems it helps you speak in the language of your users (if you work in B2B)

> I would argue that the most commonly used programming language is Excel. But few of the people using it realize they're programming.

Indeed, I'd also argue that the most commonly used database is Excel. Although few people using it realize they're designing schema.

> reactive data programming model

Can we say "one way dataflow constraint system"? They're even called "spreadsheet constraints"...

Agreed! The business critical solutions (!) you can build quickly with Excel that even non-technical people can understand is the real MVP.

"[Excel's non-VBA interface is] a brilliant reactive data programming model.."

Also known as functional programming

https://vimeo.com/42266454

Excel remains a tool.

It can be a really powerful one I agree. It's simple to use, a lot of people knows how to handle it. And it's probably one of the best tools to do some quick data analysis.

However, like every tool, it has its pros & cons. One of the main cons, IMHO, is that data tends to be mixed with processing a lot (but it also simplify usage, so...).

Overall spreadsheets have been really helpful. Retrospectively, there is a good reason why it triggered the personal computers boom in the 80ies. They are good adhoc tools that can help lot. And Excel is the dominant one.

However, there are tons of horror stories around Excel, worst case scenario, it has lead to wrong decisions affecting the world economy.

I've personally a few horror stories (fortunately, not affecting the world economy).

At my last job, we used an internal spreadsheet to evaluate costs and margins when answering contract bids. This spreadsheet was a monstrosity taking ages to compute and it was a nightmare to manage, sync, and share between all the participants.

I've also seen the traditional "on what did you work last week" spreadsheet, quite annoying to update, even more given that my workstation is under Linux...

I've also seen spreadsheets as specification documents, generating an xml file, which in turn, through some xslt, was used to generate C++/C++ headers files describing an interface between two sub-systems. The concept (automating spec to code translation) was really interesting. However, I don't know about you, but I personally hate to have MS Office in the tool chain used to compile a piece of software, specially when it targets Linux and only Linux...

But those are not the most terrifying spreadsheets I've seen. The most terrifying on was a giant spreadsheet basically managing every aspects of a huge project (thousands of piece of equipment, hundreds of sites). This document was so big it took several hours just to open it. It was like a giant CMDB of all the configuration items, and, trust me there were a lot. But it also did more, a lot more. It was generating configuration files for applications, background images for monitoring maps, firewall rules... It was also a tool that helped architect and size the number of equipment required for a given site. I probably don't know the full extent of its functionalities. Just to give you an idea of its complexity, at one point, even the number of cells in a whole excel document was reached (with Office 2003 however). All the auto-generated stuff were always kind of wrong and had to be fixed by hand, which made for a repetitive, annoying and error prone process. Also, there was no clear naming convention, and it was a mess to take the generated stuff and load it in our automation scripts. It was, de-facto, a huge piece of software with no versioning, tests or QA and it showed. There was only one guy behind it, basically managing the beast full time. When he was sick or on PTO, the whole project was impacted. And when he left, it was a huge mess, nobody was able to take over.

Spreadsheets are great, but past an adhoc usage, please, please, please, implement something more serious. And also, keep track of Excel usage to avoid these kinds of monsters. Often there are real needs behind these beasts, but Excel is a really horrible choice in the long run.

Related commitstrip:

http://www.commitstrip.com/en/2014/12/19/the-coder-and-the-b...

> I would argue that the most commonly used programming language is Excel.

I wouldn't say Excel is a programming language - unless you are referring to VBA attached to excel. Excel is a spreadsheet application. It's no more a programming language than Microsoft Word is a programming language.

Since PowerPoint was proven to be Turing complete my bet is on it and not Excel being the most commonly used: https://www.youtube.com/watch?v=uNjxe8ShM-8

PowerPoint, while popular, has no where near the widespread use of Excel. In most businesses, PowerPoint is the sort of thing you use once a month, once a week maybe if you are doing a lot of meetings. Excel gets used daily or even hourly by members of almost every department in businesses large and small.

I would agree with OP that Excel is the most-used programming environment of all time

The phrases "avoid data silos" and "we want single source of truth" have been used repeatedly about Excel misuse/abuse/overuse for 20+ years. The WSJ story talks about the complaint but not about the underlying tension between data redundancy vs data manipulation.

Excel endures because it's the non-programmer's REPL for data. Like a Lisp REPL, using Excel is nimble and has an instant feedback loop. Click "A-Z" to sort ascending, click "Z-A" to sort descending. Move/hide/freeze columns. If one has intermediate-to-advanced skills, one can pivot the data on multiple axis. None of the centralized systems with even the slickest web GUIs can match Excel's capabilities.

That's why the centralized systems like SAP ERP, Microsoft Team Foundation, mainframe sytems, etc offer functionality to export to Excel spreadsheets. (Or they offer export to csv files -- which is 1 step removed from Excel.) With MS TFS, you can even re-import from Excel so in that case, Excel acts as a sort of offline data-entry client for the project management tool.

One middle ground between Excel data silos and the utopia of a centralized system is a shared document store like Sharepoint or Dropbox. The hierarchy would be something like:

1) Excel spreadsheets emailed around. Lots of redundancy; uncertainty of which version is canonical.

2) Excel xls file stored on Sharepoint with automatic version control. Employees must have discipline to only email links to the "one true" spreadsheet and never attach the actual xls file in the emails.

3) The data that was in Excel is migrated to centralized system and all data entry and reporting is done there. However, because the central systems are so cumbersome, this abstraction "leaks" back out to Excel because of the highly desired "export to Excel" functions! (This restarts the dreaded Excel "data silos" complaint cycle all over again.)

There isn't an IT department in the world that can develop centralized IT solutions fast enough to keep up with the ways employees slice & dice data. That's why Excel is "overused".

> There isn't an IT department in the world that can develop centralized IT solutions fast enough to keep up with the ways employees slice & dice data. That's why Excel is "overused".

It wasn't very long ago that I realized that my two decades in IT could be summed up as: making proper applications out of Excel workbooks with feeble macros. I hate Microsoft as much as the next Linux zealot, but dang if they didn't create a world in which I've made a career. Where I'm at now, I'm drowning in opportunities to ameliorate horrific Excel-based workflows, and I'm fairly confident that there's enough work out there to last me until another 20 years to retirement.

>making proper applications out of Excel workbooks

Yes, exactly.

If we're being uncharitable, we can spin Greenspun's 10th rule[1] of programming as:

>"Any sufficiently complicated Excel spreadsheet contains an ad-hoc, informally-specified, bug-ridden, slow implementation of a proper centralized database."

Or, if we're being charitable, we frame it as an internal MVP (Minimum Viable Product):

>"Spreadsheets are the internal 'mvp' that proves the business value before you build the centralized systems. When the spreadsheet becomes unmaintable spaghetti formulas and the xls email workflow crushes under its own weight, that will give the company the evidence and the confidence to spend $1 million and migrate the spreadsheet to a proper centralized database."

[1] https://en.wikipedia.org/wiki/Greenspun%27s_tenth_rule

How did you build that career if you don't mind me asking? I've moved on from that stuff in my early career but would love to get back into it.

None of the centralized systems with even the slickest web GUIs can match Excel's capabilities.

You can very easily use Excel as a client to a central database and get the best of both worlds.

Actually you run into the one of Excel's biggest problems when you do this, namely "Multiple versions of the truth". Here's how that happens: The central database often contains bad data. Users "fix" the bad data in their spreadsheet. Excel doesn't have the ability to write the updated data back to the database. Now everyone has their own versions of "the truth". There are products that attempt to solve this problem by adding the ability to write modified data back to the database. See for example Pebble Reports, http://pebblereports.com/features/ but this is a reporting program and does not have spreadsheet capabilities.

>easily use Excel as a client to a central database and get the best of both worlds.

Yes it seems that way but many spreadsheets exist because the centralized system is an incomplete solution.

For example, a company might have a "human resources tracking" centralized system. (E.g. Ariba, Peoplesoft, etc). It has some standard features like ingesting resumes and extracting "skills" from parsing MS Word docs, etc.

However, the company happens to use a interviewing process where the candidate must pass (1) phone interview and (2) onsite with 5 peer programmers, and (3) hiring manager.

The hiring manager wants to "capture" all of those states (phone_passed=True/False, peer interviews with 5 different numerical scores, manager_pass=True/False). The centralized system doesn't have fields that capture all that so the hiring manager creates a spreadsheet that has those extra columns. Many spreadsheets out in the wild "fill the gap" between incomplete centralized systems and custom IT programming. An adhoc xls file can be created far faster than submitting a change request to the corporate IT department to customize Ariba -- which could take months (or never) to implement.

Or spreadsheets act as the "intersection" of data between multiple systems. Again, one purpose of centralized "data warehouses" was to eliminate Excel "data silos" but again, no IT department can create data warehouse solutions (which also includes all the custom reports) as fast as someone creating spreadsheets and just emailing them around.

Having a hybrid process where Excel acts as a round-trip offline client to a central db doesn't really eliminate the fundamental reason Excel (over)use keeps multiplying like rabbits.

Can you easily integrate with externals systems via REST & SOAP? I've done some quick web searches on this topic and have never seen anything that jumped out at me.

I know Google Sheets gives you a little bit of this with Javascript embedding but I was searching for something more tightly integrated with Excel or LibreOffice.

> There isn't an IT department in the world that can develop centralized IT solutions fast enough to keep up with the ways employees slice & dice data.

That's why the principal way you sell centralized systems to users isn't in the ease of manipulating results but in the ease / automation of data entry. The whole pain point and problem with Excel is in manual data entry, and having a single source of truth, even if you succeed in enforcing its property as a single source of truth, does not automatically solve disagreements between different human sources of data.

That's why tools like Grafana have interesting futures for business users, because they enforce a clear separation between data sources and data visualization, while offering a lot of control to the end user over statistical manipulations and visualizations that they can create, and keeping those visualizations updated in real time, so that they don't have to be re-prepared every time period for some new report.

"One middle ground between Excel data silos and the utopia of a centralized system is a shared document store like Sharepoint or Dropbox." ... 100% agree, and add I find that Google Sheet is the best for this role: 1 source of truth, modification history, read/write permissions and shareable. I am now starting to use spreadsheets for use cases I would have strongly objected in the past. I'm surprised Google sheet isn't more mainstream.

And an API that lets data be used in other 'real' applications

What about Business Intelligence products such as Hyperion? They can provide reports from a defined data model which gets data directly from the database. People can create their pivots and other transformations on the data fairly easily. Is their something that Excel offers that makes it much better than one of these Business Intelligence products in certain situations?

Excel offers a few things in most scenarios, primarily because that any enforced structure always implies lost flexibility, and one of the issues often quoted with Excel is the lack of enforcement of using the right structures and data sources.

A simple one: If I have been asked to do a report, but it turns out I don't have the correct access rights to some data, my manager can simply send it to me in an Excel sheet, or he can log an issue with IT to change the acess rights to a specific key/dimension/hierarchy.

The 'one true data source' move often correspond to tighter access controls when good auditing really would have been the correct solution, hence the above situation is fairly common.

For qualitative data management, but numerical analysis, 1 source of truth collaboratively - I use Atlassian Jira with great success.

Got a complex work stream? Got thousand small problems that needs to be slowly figured out over 50 different main topics? Got a team of 2,50 or 1000? No problem.

1. Just set up a bunch of jira entries while tagging them with the same labels

2. set up filters on these labels.

3. Set up dashboard(s) that give a complete realtime/update to any inquiring user.

Need to report status? No problem, screenshot of the Dashboard and done.

It's really great tbh.

Alternatively, use Google Sheets :) Centralized, single source of truth spreadsheets.

Google Sheets is great and makes sense for a lot of vanilla use cases, but even in its current form, doesn't hold a candle to Excel running on a Windows machine when it comes to doing anything even slightly fancy or non-trivial.

Google Sheets and other similar web apps replicate most if not all of the Excel features you just listed, and it's centralized and more than one person can work on it simultaneously.

> The phrases "avoid data silos" and "we want single source of truth" have been used repeatedly about Excel misuse/abuse/overuse for 20+ years.

Yep. Even if they get centralize data via RDBMs like SQL Server or even god forbid Microsoft Access, office workers are going to want the front-end of that data to be excel.

There is too much knowledge investment in excel for many office workers to want to switch.

> A year ago, Mr. Bell’s team spent hours distributing hundreds of Excel spreadsheets to regional and unit leaders each month for planning and performance tracking of the company’s 415 U.S. restaurants, he said. Now the same process takes minutes.

Then I believe it should say stop abusing Excel.

Excel can be an incredibly useful tool, but it's definitely not universally useful. If you're using it to manually collate data from 400+ sources, you're doing it wrong.

Agree.

Reads to me like, "last year Mr. Bell's team spent hours hammering in nails by smacking them with the wooden part of the hammer's handle"

I agree. Excel files prolly should not be distributed using the same method as a freshly-out-of-the-ditto-machine dictaphone memo from 1967.

And the premise that things have changed enough to warrant reevaluating worn out business processes like Mr. Bell's here is true enough.

But dumping a tool because some out of touch C-level tool says so is a recipe for huge traction loss.

I mean ... people that use nail-guns for a living still have to use hammers.

This is the point that's going to get missed a lot. In the corporate world Excel is the proverbial hammer to nearly every use case involving data manipulation, regardless of how well suited it is for the task.

Edit: I just noticed that I'm the third person to reply to you and we all referenced hammers.

I don't know that Excel was the wrong tool for this task. I'm not sure why the spreadsheets couldn't have been distributed and consumed automatically by an Excel application, or at least by helper applications that were fed by and/or into Excel sheets. An Excel spreadsheet is a good enough form; and I don't think replacing one with a webform or pdf form would be any better.

edit: my favorite thing about using excel is that instead of spitting out an answer, you can spit out an Excel sheet that shows a lot of the work itself, therefore partially explaining the answer to the people that have to consume it. The product of your program can often be another program, giving the person receiving the report the opportunity to tweak or add new parameters to your output. Basically currying accounting calculations.

That is exactly right.

Many moons ago large bank I worked at purchased a company and soon it became apparent that their entire business ran on a Lotus 1-2-3 spreadsheet with 30,000 lines of really dense, uncommented, unindented, spaghetti LotusScript code spread over 2 or 3 functions. The kicker is it would only run in the 16-bit edition of the product for whatever reason. So we had to have a VM to run it in because Windows 2000 had a problem with either Lotus 1-2-3 16 bit edition or the spreadsheet itself.

So yeah, stop abusing Excel - Lotus 1-2-3 in this case.

The core complaint of not having a single source of truth seems more about how the company handles data/filing rather than Excel itself.

Maybe I'm a bit bias having used Excel my entire career and very dependent on it, but everytime I've seen companies steer away from the desktop application with 0365/sheets/tableau its simply not as useful for adhoc data analysis and reporting.

I agree. Those are great products, if you have the staff to support and use them, but this is rarely the case. Every time I see a company implement them, they think it's going to be a silver bullet to reporting and analytics, and most of the time it just causes a mess, because no one really knows how to properly use them, or if someone does, that person doesn't have the bandwidth to keep up with the entire organization's needs. Excel is amazing, it's relatively easy to use, has immense power for dealing with calculations and graphs, and it is cheap. An organization's problem isn't excel, its the fact that they keep 20 copies of the same data and no one is keeping track of which is the source of truth.

Every email reply becomes an independent fork of the data inside the spreadsheet. The issue is storing and sending the file locally, and the solution is storing it in a centralized location. Cloud storage is often used towards this end, but on-premises file shares work very well too.

This is one of those solved problems that companies inexplicably keep working to solve again.

This is a key point. It's easy to branch data, but painful to merge it.

How do you even HAVE a source of truth? You email a spreadsheet to two people, and the both want to work on it... which one is the source of truth? They each have modified the original, and we want to keep both changes.

I mean, this is the problem that distributed version control was designed to solve. Emailing a spreadsheet around and calling one a source of truth doesn’t solve the simultaneous edits problem.

>its simply not as useful for adhoc data analysis and reporting.

<rant>

You wanna know what's even more useful? Code notebooks.

I store everything from my sql queries to my processing pipeline to my models, and my vizualizations. Most notebooks have slide-show modes now so your presentation to business stakeholders is simple to prepare as well, and with the right plotting packages you can draw interactive charts that do all the fancy stuff. And when it's time to update the analysis, you just re-run the notebook.

But business analysts are too freaking lazy to just learn to write some python or R, and they're perfectly happy to get paid to do the same task over and over again when it comes time to making updates.

And don't get me started on source control.

</rant>

Most business analysts who are pulling this kind of data have a finance background. To say that they're "too lazy" because they don't want to do their work using Python seems a bit harsh.

In practice what I've seen is that IT and finance departments don't even let business analysts write their own SQL queries for fear that they'll write bad ones that bring the database server to its knees.

>In practice what I've seen is that IT and finance departments don't even let business analysts write their own SQL queries for fear that they'll write bad ones that bring the database server to its knees

I'll take data warehouses for 600, Alex.

I use jupyter a good amount, but even in a low user environment there's a big worry about version control and code sharing/review

I know there's " automatically check stuff into git" sort of models but would really like an integrated Jupyter + GitHub + Google Drive system so that it's easier to track what's happening and to find older versions of stuff.

Oh yes. I have tried to introduce jupyter notebooks with pandas to data analysts so many times, but they always want to do it in excel.

Do you have any way you'd recommend of getting multiple users working together with code notebooks? I've messed around with Jupyter, but that's about it.

We usually just send .ipynb files around because of the stuff we do, we all usually have the same libraries.

Installing a notebook server is an option though. IBM will be happy to let you pay to have them help, or you can do it the sane way with continuum analytics.

This!

I happen to use and advocate for R/RStudio for ad-hoc work (I spin it as "excel on steroids" when I am trying to convince someone). But it is hard to pull people away from something that has worked for them.

The alternatives to excel in the past have been really, really unpleasant to use. A now EOL product from Oracle comes to mind. It is called "Discoverer"-- that's right, "Discover" + "er". It is a turgid POS requiring extreme suffering and handholding/permission granting from your local friendly DBA-- all this to produce 90's era bar charts that are all but impossible to tweak.

IMO if you really want non-tech users to become more technical and use DBs as single source of truth, use KNIME. It still needs proper reporting addons though, but you can export data, even to tableau.

I saw a presentation about KNIME but was unfamiliar with many of the processes that it allows one to string together to build a workflow, so the notion of building a workflow myself seemed very daunting. Do you think it would allow one to build workflows for basic stuff and gradually grow in their confidence with it?

The presentation mentioned that the company is/was working on reporting add-ons, I think.

Totally agreed. Especially when the alternative is abusing an ERP for ad hoc reports or using a journey map approach that captures the top X use cases, but misses what is actually needed.

Agreed. Excel users should learn more about importing data from other sources and processing that. Excel itself should make it even more obvious.

I'm not sure of the origin, but I was once told a truism that any speccing process, if left long enough, will eventually spec out Excel.

This is NOT about "inefficiencies", "data security" or whatever.

In big biz, the "finance chiefs" (or other persons with purchase authority) usually get pampered by vendors with expensive business trips or whatever... especially at the scales that require SAP or Oracle solutions. So this likely means: the CFOs bought some expensive software that promises to replace Excel without any due diligence, the staff looks at the software and decides it's crap and continues to use Excel => CFO gets mad because no one uses the expensive software.

If you don't believe me, just ask yourselves how Oracle, IBM and Accidenture are still in business despite numerous highly expensive and publicized failures. It's all due to CxO pampering (in addition to a bit of vendor lock in).

This 1000%. The companies you mentioned don’t innovate on tech, they innovate on sucking up to executives and convincing them to buy things that the executive’s own staff recommends against. They have extraordinarily well-oiled sales machinery that deftly bypasses any kind of sanity checking inside the customer’s company and devote all their efforts to keeping it running.

http://bad.solutions is one of my favorite providers of Enterprise-grade software that is not remotely usable.

What's Accidenture?

"Nickname" for Accenture, an IT consulting/bodyshop famous for delivering stuff at waaay less than acceptable quality (or not delivering at all). In addition they're also known for exploiting their employees.

Is it different from other giant IT services firms?

Likely Accenture: https://www.accenture.com

Not sure if the misspelling is "accidental" or on purpose, in which case both it and this sentence are puns.

He probably means Accenture.

https://en.wikipedia.org/wiki/Accenture

A hilarious pun at Accenture's expense.

More like expense of their feed^W clients.

I'm only use spreadsheets here and there., bit this article has the kind of grating stupidity that adds to the conceptual noise we are surrounded with.

To replace Excel with software X, you would have to deliver software that would provide the Same kind of flexibilty. That is a very difficult kind of software to develop!

One very basic and powerful aspect of excel is simply the fluidity of the display. You can cram a lot of information onto a single screen, resize and move rows/columns, use colors.

As apposed to a clunky web app interface that has low information density, inflexible data scheme,and the crippled UX of a typical web app.

That's not to say that the example given in the article doesn't point out a real problem. But I wonder if doing some custom UI and re-factoring while building on the current system the users already understand wouldn't be a much more cost effective strategy.

When I see the clunky interfaces that so many workers are saddled with these days, it makes me sad.

I used to work for a finance department that tried to implement a fancy driver based tool to replace an endless amount of spreadsheets. We spent so many hours and I’m sure a lot of money on the software license and consultants who built it and in the end it just wasn’t flexible enough.

> “I don’t want financial planning people spending their time importing and exporting and manipulating data, I want them to focus on what is the data telling us,” Mr. Garrett said

This!

I know this is reality in so many companies that don't have technically educated staff to tell them about more suitable ways to do it. So the teams get data from one department or system, manually transform data (in the editor via search/replace), and if they are "very sophisticated" they save it in MS Access, join tables, connect the dots, export back to Excel, analyse the data via Pivot Tables etc. And they do these steps repeatedly, every week. Without stopping and asking if there might be better solutions out there. (In German IT curriculums they call this "Medienbruch" = "media discontinuity"?)

I have seen this in so many companies and thought to myself: This is an excellent case for a consulting pitch. And I could literally transform their work.

And I am not talking about selling data warehouse solutions with ETL processes and the like.

Twenty years ago I realized I could do all these preprocessing and transformations with Perl. Now I do these kind of things with Python and Pandas etc. So there are so many solutions that are a huge improvement. Still these departments do it with Excel/VBA/Access.

I have seen many of these companies and potential "build a real database-backed web app" projects during my software consulting career.

This seems like a good idea at first: automate manual processes, have a real database with one source of truth and consistency rules, multiple people can access it at the same time, etc. You can do it if the process is very well known and fixed.

But most of the time a web app would cause more hassle for the companies than having a team of people doing the work "by hand" in Excel would be.

* The process and desired data schema might change every few months - or even weekly when doing something like marketing reports. * Now they need regular maintenance - keeping servers up, doing and testing backups, applying security patches. * But the problem being solved is not big enough to justify hiring a full-time team of software developers to maintain it forever.

Don't forget the all-to-common scenario:

"Executive 1 likes to see the data presented in formats A and B, Executive 2 likes to see it presented in formats C and D, and occasionally E, and Executive 3 will only look at it if you format it as F and cut/paste it to PowerPoint."

So you end up with 20 analysts who all have input, each manually re-formatting their own data in 6 different ways, making sure they all stay in sync. Then someone else collates it all together into 3 different exec-specific presentations, and double checks to make sure they tell approximately the same story. Welcome to life as an office drone!

I used to work as a corporate financial planning analyst and this hit way too close to home.

Then one executive who didn’t like the story would have one of his analysts pull the same data in a slightly different way and adjust it. The executive would then show it to the CFO and tell him that’s the real story. The CFO would turn to you and ask why his number is different than yours. Then you would spend hours with the analyst reconciling the data and building a waterfall chart to send to the CFO to explain the difference.

Then if you’re really unlucky the CFO says he wants to see the data both ways with a waterfall reconciliation every month.

Do you have any suggestions on where to get started with Pandas? I already know Python, but every time I try to get into Pandas I have a really hard time getting any traction. I'm particularly interested in using it to migrate Excel workflows.

I think the problem is they're using Excel as a database as opposed to a spreadsheet. That's fine at small scale, it's easier to manage and even easier to get going, but Adobe scale? They should advise "Stop using Excel for what it's not".

I’ve worked in many large companies, and I have never seen one using SAP or other large tools as single sources. It always has been excel. From my experience due to 3 reasons

- excel is much more flexible, I can add a column, delete those I don’t want etc

- speed: many of those large corporate systems are just damn slow, maybe because of stupid configurations, but still often very slow an annoying

- analysis: It's easy to do some basic analysis without studying CS

I have - worked on getting Hyperion FM implemented across ~30 countries fed by about 40 ERP systems.

All outputs were generated by a .Net application generating read-only Excel spreadsheets (which might have 30 tabs and up to 15,000 values on a tab) - was done this way to meet the very complex reporting requirements. All the ~20 report packs in the company were produced with one click in a simple web app.

Ah-hoc querying of Hyperion was done from Excel using the SmartView plugin.

I agree with those 3 reasons, but I wouldd add a fourth: easy to create your own version of the truth.

I'm not implying this is due to nefarious reasons (though sometimes it is): sometimes the official version of the truth is just not adequate.

Definitely, great point. Tools like Tableau/Power BI make it very easy to get a misleading version of the truth, since it can very easily include misleading data points and/or make calculations in a fashion that leads the user to misinterpret KPIs.

I rather think the real reason is inertia.

I use Excel because my boss wants to use Excel, because all his bosses have wanted to use Excel, because all their previous bosses have wanted to use Excel.

That's how it works in the electronics manufacturing industry, anyway.

Actually, the main reason is that SAP is expensive as hell. The starter package is $1400 per user (one time) or $410 per user per year. And that's just the software. Add consulting services that are always necessary to actually make it fit your organization and you're easily looking at millions of dollars in capital spend.

All of the cases where I have seen Excel sheets get "out of control" are not because someone came along and decided to implement a complex system in Excel, far from it. I tend to see small, well suited, sheets being created in Excel, and then iteratively new features get added

"Wouldn't it be great to automatically get averages each month"

"Perhaps we could pull in exchange rates so we have up to date currency values"

"As we need to compare this to employees here, and because we actually have a list of employees in another sheet for holiday and sickness, why don't we put the employees into another separate sheet and cross reference it from both places"

"When we add a new employee wouldn't it be nice if these sheets automatically update"

"We want to pull in this live data every day at midnight, perhaps we can have an automated job for that"

Then when someone says "hmm, I don't think Excel is the right tool for the job" there is often general agreement, but at that point there are lots of interconnected sheets, formulas, external data being pulled in, cron jobs, etc. and the system manages a lot of different things, none of which you particularly want to be without and despite the complexities, people have been with it over iterations, so they are actually quite familiar with how it all works.

But then a couple of people with the knowledge leave, a new person takes over, and their first job is to remove Excel and come up with a much better system. They try, users complain it doesn't do certain things as well, over time in improves, somehow certain things never seem quite as good, but then new features have been added so people forget and find the next thing to complain about instead!

To be fair, this kind of unobstructed scope creep is how all software gets "out of control," not just Excel sheets.

"Stop using paper", says chief of a company that treats modified copies as if they were originals, "if we all carved our work on to this big stone in the middle of the courtyard then we'd be rid of our paper problems".

"Wall Street is investing into the new pyramids. Here is why."

It is amazing how many simple tasks at a company can be done with Excel that as programmers we often write custom code for. When all you have is a hammer...

At the same time. I have seen banks transfer money between branches by literally FTP transferring an Excel file. At some point there has got to be a better way. At least use SFTP :)

I've thought about this before. Does it mean that our libraries aren't good enough? Do we just not know how to use them?

I often wish there was an intermediate between writing a fully custom piece of software for something and using excel. This about how many times you've worked on a web service (or other technology) and how much of the code is basically identical to every other webservice in that technology minus some unique data models and a few functions of custom logic.

Just something I discovered recently that entirely takes care of the problem if you have ONE PERSON who knows how to code: Google Sheets with the IMPORTHTML() function.

You host a web server that runs Python or PHP or whatever and it queries and pulls data from any number of places, and dumps the information to CSV. Then in the Google Doc, you request the data from your server with the IMPORTHTML function. It populates your spreadsheet automatically, and then you can show graphs and other various things. And it will update on its own!

There you have it! Single source of truth! And you still get to use a spreadsheet!

It's a fucking beautiful, elegant solution to the entire "Excel problem" that I don't see enough people using.

Instead they're going with enterprise 3rd party "solutions", like Anaplan. Who knows how much money they're blowing because the execs don't know tech...

Unless I'm missing something, this only provides a way to display data from a database. But users cannot write to the database via the Google Sheet, and it thus doesn't solve the problem...

Oh god you're right, I just re-read the article and realized what they're doing now.

It didn't even occur to me that anyone would use Excel as the database. Holy hell. Nevermind.

Still, it shouldn't be hard to convert Excel to an SQL database. Then you dump to CSV, and use ImportHTML to have a spreadsheet be your source of information.

You mean the function inside your Doc(ument) will periodically fetch the http://myweb.com/periodically_generated.csv? Can't you just "push" to the target Doc(ument) instead?

Yes, IMPORTHTML fetches your csv, but you can have your CSV populated by your server dynamically on each request. You can't push to a Google Sheet as far as I know.

But polling is just as good for these purposes, your data will be up to a few minutes old.

If you want true live data then you should use something else, but that's not what the finance chiefs are complaining about. They want people to stop manually importing CSV data because it's really slow, and causes lots of errors.

EDIT:

I was going off the assumption that there was actually a database somewhere. Turns out they're using Excel as the database.

I'd still argue that one person who codes can fix it. Take your Excel sheets, put them into an SQL database, and dump to CSV for importing to Google Docs for graph/display purposes.

Just FYI, Excel has that too. Check the 'Get Data' options under the Data tab.

This article seems to typify what PG was talking about in his submarine article [1].

[1] - www.paulgraham.com/submarine.html

It sure does smell like a PR assisted attack on excel in favor of enterprise solutions

This is true, but, I also clicked because of the title. Well baited trap.

Absolutely right. Excel (and spreadsheets in general) are bad social media/groupware/workflow tools... who knew? This just in: word processors make for a lousy DBMS... film at 11!

Points for their SEO/clickbait headline: it hooked me... this time (-1 one added to my personal WSJ lifetime value calculation)

The title is overblown. There's a push to stop using Excel as:

* the single source of truth for a set of primary data

* the mechanism by which regular (i.e. repeated) reports and analysis are conducted

But it overlooks the fact that:

* Excel has, for many years, had real-time integration to pull data from many systems (not just 'Export to Excel', but the ability to pull data via ODBC connectors). I've used this, e.g. to pull trial balances from Sage into Excel for custom, but repeated, financial reporting.

* You can't beat Excel as a prototyping tool, or for ad hoc analysis from disparate data sources (without a huge learning curve for the people we're talking about). Telling finance/accounting professionals not to use Excel is like telling designers to use D3.js instead of Illustrator. D3 might integrate better with data sources, and be easier to version control and share etc., but for a quick sketch you'll use Illustrator.

From the title, you'd assume these 'Finance Chiefs' were expecting none of their staff would even need Excel installed, after some transition period, but this isn't the case. There's always some lag between the needs you have, and the capability of more structured tools.

I've worked with customers to setup support units specifically for Excel, helping their organisations develop and validate Excel spreadsheets and consider migration to more robust solutions when the time is right. As much as I dislike the general use of Excel there is an inescapable fact, Excel is a great source of bottom up innovation and ideas from non programmers which we should endeavour to capture and not just forbid.

The issues raised in this article are far less about failings of Excel but rather typical failures of business to plan long term and put effort/resources into comprehensive solutions.

Don't blame Excel if your processes suck. Excel is a tool, namely a Swiss army knife to edit and present data that is commonly used in business. It can be very powerful if you know what you are doing.

Swiss army knife is a good analogy. Sure, it does lots of things, but if you want to do really big work, you'll probably want a chef's knife, or a saw, or a pair of scissors, not miniature versions of each designed to fit in a small package.

That said, it's a great tool to survive in the woods :)

One reason why Excel should never be used for anything real is because it lacks type safety. For example if you have to sum a columns of numbers and one of them was mistakenly input as text, rather than throwing an error, it will just ignore that entry. There are lots of other problems related to type safety as well.

For example if you have to sum a columns of numbers and one of them was mistakenly input as text, rather than throwing an error, it will just ignore that entry.

There are ways around that. Here's something I just put together as an example:

=EXACT(COUNT(A2:A6),COUNTA(A2:A6))

That will only return true if all the cells in range A2:A6 contain numbers.

There are other approaches that would work, if that doesn't suit you.

As the founder of my own startup, I'm forced to use Excel to do financial projection, etc, essentially all materials for my pitch deck. The only thing that worries me is accidentally doing something wrong without knowing what has changed -- stupid things like accidentally pressing Spacebar and Enter on an important cell. Because of this, I keep looking Undo just to make sure my last step is what I intended to do.

I use IntelliJ and Git, so you know what I'm missing.

What should we use, then?

I desperately need some tool I can tell non-tech people to use where they can input arbitrary structured data.

After the data is there, in any format the user wants, as long as it is structured somehow, millions of programs can be written to use that data and combine it, but somehow the data has to be written FIRST. It is not feasible to come up with the schema, write a software based on that schema and then give it to the non-tech people.

Teach people how to use excel properly:

- treat inputs and outputs separately

- always use simple tables for input (no breaks, first lines are titles, basic vlookups for relationships, etc.)

- use pivot tables to summarize the tables, and build reports on top of the pivot tables.

EDIT: BTW, on a personal level, I've moved to CSV/Jupyter Notebooks combo, but I have no hope that my fellow non-tech workers will do the same...

My current role is at a large company, and a significant portion of my work is getting Excel reports which are broken and fixing them. Most I've never seen working. Some reports have been broken for years but need to be resurrected.

To add a couple more:

- Use actual tables. (Select data+headers, Insert -> Tables -> Table). In most cases the data is tabular but not formatted as a table. Making this one change makes the formulas self-documenting, and rather than selecting rows 2:5000, it will automatically accommodate more data as it's added. Tables can also be named (click in the table, Table Tools [Design] -> Properties -> Table Name). E.g. tbl_sales. This allows nice formulas like

    =SUM(tbl_sales[Sales])

    =SUMIF(tbl_sales[Region],"West",tbl_sales[Sales])

At a certain point, it really needs to be SQL, but between this and pivot tables you can greatly extend the time before a database is needed.

- Add a sheet with instructions. Explain hiccups which may be encountered, and how to fix them.

- Add comments to VBA. There is about 1 person at my facility (other than me) who adds any comments at all. Most use the Macro Recorder then add in the loops and logic as needed. Makes it really tough to figure out.

- Protect cells which hold calculations. I recently saw a spreadsheet which has been giving sub-optimal output for years because the ranges used in a function were messed up and copied down every week.

- Use the built-in styles (Home -> Styles). Same goes for Word (Heading 1, Heading 2, etc.). It makes spreadsheets way more straightforward to read if there are some cells using Explanatory, Input, and Note styles, and it's less abrasive to read. I particularly like the Input style because you can tell users "You'll only need to update the orange cells".

- Use multi-line formulas. Alt+Enter while editing a cell allows multiple lines on a single formula. You can also improve the readability by adding four spaces at the start of the new lines. E.g.

    =SUMIFS(tbl_sales[Total_Sales],
        tbl_sales[Month], "Jan",
        tbl_sales[Region], "West")

- finally, pray that Excel won't corrupt your data (be very wary of dates, texts only composed of digits, and different decimal separators)

I agree. A lot of these tools are coming about because people don't understand how to use excel and frequently Tableau/Power BI is not a better alternative, unless you have no clue about Excel.

> It is not feasible to come up with the schema, write a software based on that schema and then give it to the non-tech people.

Would it be feasible if you only had to write the schema and the software were generated based on the schema?

That would be much better.

I was working on something like that based on https://postgrest.com/, but I'm now reconsidering the approach.

That's a hard question to answer and depends on your size and what you really need.

For small companies, Excel may really be all that you need. But at a certain size, you might want to use something like PowerBI instead since it will also pull directly from the database and you can share the dashboards you create very easily. Large companies like the article are referring to might be able to get by with PowerBI (Microsoft would certainly argue there are still plenty of places where it works) or ERP/HR tools more specifically designed for certain uses.

The same ETL functionality that's in Power BI is also available in Excel 2010+ as "Power Query".

The startup I'm working with right now is building a tool that allows you to create spreadsheet-based business applications.

What I like to call the product is "excel on steroids".

You can check it at www.ninoxdb.de.

This seems like an alternative to Airtable and Fieldbook. Am I wrong?

I know these tools and I like them, but this wasn't what I was talking about in my comment. I meant something more flexible and more arbitrary, like a key-value store non-tech people could edit.

"If I increase the page number, our sales go up. I'm on to something.."

http://dilbert.com/strip/1998-12-12

While we're talking about Excel:

Are there any good tools available for version control and/or diffing of spreadsheets?

If my coworker updates a formula in cell E3 and emails it to be, is there any way for me to know what changed?

Spreadsheet Compare is built into Excel/Office. It's a relatively unknown tool since it came out in 2013.

See: https://support.office.com/en-us/article/Basic-tasks-in-Spre...

We (disclaimer: I'm the co-founder) have developed a Git-based add-on for Excel workbooks. Basically, it starts where Git left off in the Excel world. Here's a short blog post how it works out for one of our open source projects: https://www.xltrail.com/blog/xlwings-is-now-on-xltrail

Excel shared workbook is versioned and comes OTB.

You can see user edits, old cell value and new one.

It gets latest data on save and even does merge

Find me a cloud database with a good GUI for read/write and I'll get off excel.

Databases are meant for tech stacks, not working with lists.

Put simply, Excel is excellent as a terminal node in any data pipeline, but poor as an intermediate node. Open speadsheet -> human plays with data -> human makes decision = good. Open spreadsheet -> human plays with data -> automated process pulls data further along in data pipeline = bad.

Like others said in here people like and use Excel because it is a flexible, visual, programming language. We haven't got alternative solutions for all the people who want to program like that. Instead more and more companies come up with very rigid, workflow based systems who actually haven't got the freedom Excel has. The only proper attempt in solving this problem has been made by people like Bret Victor but nothing commercialized at least as I am aware of. Accounting, finance and even data processing / analytics software companies still live in the Middle Ages in terms of user interactivity and flexibility which is why a lot of us resort to programming languages.

“Excel just wasn’t designed to do some of the heavy lifting that companies need to do in finance,” said Paul Hammerman, a business applications analyst at Forrester Research Inc.

Instead, companies are turning to new, cloud-based technologies from Anaplan Inc., Workiva Inc., Adaptive Insights and their competitors.

The newer software connects with existing accounting and enterprise resource management systems, including those made by Oracle Corp. or SAP SE . This lets accountants aggregate, analyze and report data on one unified platform, often without additional training.

---

reads like a "submarine

http://www.paulgraham.com/submarine.html

Python is my day job and I'm very fluent in the common data science tools like Pandas, but you'll hardly ever catch me using Pandas because Excel is just so much easier for 2d data.

The thing is, I never use Excel because the data I work with every day is almost always multidimensional, so I work with xarray[1]. It drives me crazy to see coworkers using Excel for things that it is not suitable for and having crazy solutions like multiple tables per sheet, multiple sheets, multiple files or other hacks to describe another dimension of data.

[1] http://xarray.pydata.org

I think "don't use Excel" is the wrong solution. The answer is to develop some clear metrics around when to migrate off Excel. We use some pretty big spreadsheets, and they're fine for the intended purpose.

I was toying with this idea in the past, where cells would support, in addition to primitives(numbers, strings, whatever) and functions, a new type which would be identified by a reverse domain name notation path/key (e.g org.hr.employees.cnt), and optionally a comma separated (key-value) pairs as arguments.

Whenever a value for such a unique (key, arguments) cell would be required, the application would collect all such that need updates -- or, you could click a button to update them all --, create a message that would contain all paths and arguments, and auth info (each user would require to authenticate with some centralized system, so that the auth key would be used for that RPC, and that the service could check for authentication and authorization ), call out to some service and get values for each of those cells (value could depend on the authenticated user, or could even be denied depending on privileges ).

Furthermore, the application would cache each such cell value (auth key, resource path, arguments digest, value) and it would periodically update them, or do so on demand (so that you can use it say on a plane, and refresh them when you land). That’s the gist of it, but there’s more to it and there are some optimization opportunities I am ignoring here, but it could work.

Then again, for all I know, there are already such services, or apps that kind of do this sort of thing.

Huh, I thought only TechCrunch was the advertising "news" secret arm of Silicon Valley investors? I didn't realize the WSJ did advertising "news" pieces also.

After building a couple of years of financial systems, I have a hypothesis - if you could build a Jupyter Notebook with an editable spreadsheet view that is backed by a database and has something like Atom editor Teletype's CRDT (conflict free replicated data types) to enable collaboration...that would be an Excel killer.

Incidentally, there are a lot of people who move from Excel to Salesforce ! Because of the ability to program it like an Excel sheet.

Pretty much every CRUD app I've ever written was to replace something that was already being done in Excel, and the reason was always to get more of one or more things a RDBMS gives you. So I can say this from a knowing place: good freaking luck replacing Excel's other capabilities. 30 years of work by thousands of Microsoft devs is not easily replaced. It's like a moat around their business that they built by brute force.

Accountants invented spreadsheets. Programmers made electronic spreadsheet programs. The reason why spreadsheets are so popular among otherwise not technically people is because it was developed by business people over the course of centuries.

Unfortunately, if you're not technically inclined and someone gives you a big hammer, at a certain point you're gonna have to stop hammering everything you encounter and learn how to use a screwdriver.

When I worked at a large investment bank we spent a ton of time and effort on a system that allowed equity analysts to use excel and sync their data back and forth to the firm's centralized system. it was a great approach because it kept the analysts happy, who were all masterful excel users and thus very efficient and happy with it.

Joel Spolsky's essay "Controlling Your Environment Makes You Happy" comes to mind.

Something that almost always gets left out of these discussions is how quickly you can spin a new dataset into a highly polished, ready-to-print-for-executive-review report IF YOU HAVE MASTERED EXCEL HOTKEYS. For context, I have meaningful experience doing data analysis using various combinations of SQL, python, ruby, R/tidyverse, and excel with lightning-fast fingers. In a previous role, I regularly needed to summarize new datasets to executives, and I found that nothing was quicker and more flexible than the following workflow: copying and pasting raw data into an unformatted "data" sheet, creating a new sheet from a well-formatted template, and quickly writing whatever "SUMIFS" prototypes I needed then filling out the rest of the sheet by copying those formulas across cells.

Often excel users are stereotyped as having error-prone, sporadic workflows where they are often mistakenly writing over raw data, etc. And that is often a valid stereotype. However, I believe that there exist scenarios where excel is part of the optimal workflow.

I'm on the market for another startup idea and one of the most ubiquitous advice I'm hearing nowadays is that "there's a startup in every Excel sheet".

From that perspective WSJ is doing a great job in making the bed for these ideas: startups will eventually want to sell to those "Finance Chiefs" which is never an easy job.

I'm in London too, doing a bootstrapped Excel startup. See my profile for contact details.

"If you want to go fast, go alone. If you want to go far, go together." -African Proverb

Desktop apps like Excel are fabulous at letting you work alone with data in front of you.

But they are a shitstorm of horror when you try to work with other people, collecting data from multiple places. And sure, Office 365 is a step in the right direction, but it's not nearly enough. Google Sheets is slightly better, but it's still not enough.

To be frank, the same way that "anyone can do SQL," we've fallen into a trap of thinking "anyone can do Excel," and it's just not true. And no amount of improving the commercial product Excel will make it work for businesses like they need, in my opinion.

At some point you need to hire engineers, mathematicians, statisticians, process engineers, business analysts, etc., to help you manage your data.

Asking people to "Learn Excel" in order to replace a real team of qualified domain experts is folly.

Excel is remarkable, yes, but I'd argue that given its nature pretty much every reasonably complex workbook can be expected to have at least one bug. Having worked with them a lot in many different contexts, I've seen just how easily errors are introduced and I think they should never be used for research or finance where the cost of errors is high. At it's core the issue is that it's really hard to create variations and tests which can tell you if something is wrong... Much harder than it is to do this in a normal programming paradigm. Yes, the barrier to entry is lower, but the trade-off is much increased error rate and ongoing brittleness. Even if it's "right" at one point there's no guarantee it will be right in the future. What if someone copies/pastes data into a crucial worksheet and misses a column? It's so incredibly easy to screw up!

This really sounds like another case of whats old is new again. In a way, I think it speaks a bit to the obsolecence of what used to be 'systems professionals' that things have swung so far into the end user domain.

If the finance team (and I have been part of one early in my career at a major conglomerate) is doing essentially the same repetitive analysis on periodic data sets, it is their bad for not reaching out to their systems people for a better way. Those types of reporting can be canned at a much earlier level and output generated/delivered in the desired format directly to the end users. And nothing precludes still making data available for new types of analysis.

Excel is great for small operations, ad hoc stuff and just quick and dirty work (a REPL of sorts as others have said). But not as a primary/crucial reporting tool for large operations.

> “You’re still going to use Excel for the things you’re not using a tailored solution for,” he said.

I was shocked to hear about my old company still using Excel heavily for key business operations. I think those are the ones that should be replaced by bespoke software.

On the other hand what likely happens is you need a way to quickly perform some analytical task and your company doesn't have the process or speed to whip up some software quickly. After seeing how a lot of Fortune 500 companies work I'm not sure they ever will with compliance and other regulations in the way.

Excel is very accessible and will probably be the way businesses whip up analyses in the face of business fire-drills unless they have software engineers embedded in their teams or they get their current staff to learn how to code.

This is everywhere. This is the reason I say that every company and organization, whether they realize it or not, needs software engineers. The things that grow (and they do grow rather than being built) in the absence of access to software engineers are overly complex and fragile.

Do software engineers want to work on these problems though? IME, we all love to work on problems one step higher than our current level.

Have seen Excel derivatives pricers that ran long Monte Carlo simulations (yes, RAND() is often good enough) that took several minutes -- press F9 for a new estimate of a price, as well as simplified stripped-down pricers that got within 0.001% of our real internal pricing system but was nicely laid out and explainable to clients.

The only thing that Excel is awkward at is unlimited recursion. For everything else, if you can't turn your computation into a nicely laid-out spreadsheet, then you probably don't understand your problem well enough to explain to others.

Had a fun time doing many of the Euler problems in Excel, even though they theoretically many require recursion. Oftentimes I got the answer more quickly and easily than in Python.

I used to be in the camp “oh you know Excel and PowerPoint, ahah!”

But working full time and having to create Excel spreadhseets on a regular basis as an engineer (mainly for budgeting and capacity plannig, but also do some schedule/timeline display), I have to say once you know how to cross reference things in Excel, life saver...

Automating spreadsheet creation has helped save hours of manual recompiling every quarter (do this Python is pretty pleasant). Yes. The scrolling and wrapping really sucks, but nonetheless everyone understands excels. I have been given tours and free trials from SaaS companies to try their AWS budgeting interface. They are great for some (like some prettt graph), but just not easy and flexible at all.

"Keep checking plurals readers tell headline writers" springs to mind as a response!

But, joking aside, Excel is the most amazingly useful and adaptable tool but it's incredible how many obvious improvements have been left out. Parts of it are stuck in the stone age. As others have mentioned, version control and information flow are hugely limited with it and there's a real feeling that it went backwards for things like charting (I suspect it's partly that other methods are now more intuitive). And why on earth don't they do something (anything) to make pressing F1 not be such an annoying experience...

Sounds more like they should get a proper ERP and/or HR system to solve this.

Having read the article, I feel as if the headline is a bit misleading and clickbaity.

A more accurate headline in terms of the article's contents would be something like, "Stop Using Excel For ERP, CRM, And Workflow-Based Use Cases, Finance Chiefs Tell Staffs". Or to be extra snarky, "Follow Known Best Practices, Finance Chiefs Tell Staffs".

Not that attention should not be paid to the article's subject matter -- there is SO MUCH wood to chop in terms of helping enterprise customers run their companies better and/or get rid of inefficiency and general jankiness.

Shudder

This reminds me of my 2 months working for a Big 4 accounting firm in, basically, SOX auditing.

We mapped out controls on our little 14" laptop screens using excel spreadsheets with tens of thousands of rows and hundreds if not thousands of columns.

I'd say we were writing novels in Excel but, to be honest, there were probably far more words than the average novel contained.

Talk about using the wrong tool for the job -- and using low-paid monkeys on inadequate equipment rather than developing a proper application for the task (or paying someone else to do the same).

I'll never look at excel the same way again.

As someone in a systems / ops engineering role it frustrates me endlessly when someone sends me information / data in the form of an excel spreadsheet or a word document.

My only are the applications slow and buggy (unless I’m guessing you’re running Windows?) the file format is very hard to work with / from if you want to do anything with it outside the vendors application, data quickly becomes stale and it’s not easy to populate automatically using standard tooling let alone share the information generated there from in an accessible, digestible format.

How much does it cost to get an advertisement like this in the WSJ?

Article mentions companies by name: "This is just an advertisement! Shill journalism!"

Article doesn't mention companies by name: "The reporter didn't include any specifics! Lazy journalism!"

I work for one of the companies mentioned, and we don't / did not pay for the name drop. Maybe one of the others did, but we did not.

Is there the usual kind of database on the backend for your system?

Likely not that much. HuffPost, Business Insider, and others articles can be bought for a few hundred bucks.

Exactly! I read it as such as well.

Advertising usually costs 80k.

what are they advertising?

"Instead, companies are turning to new, cloud-based technologies from Anaplan Inc., Workiva Inc., Adaptive Insights and their competitors.

The newer software connects with existing accounting and enterprise resource management systems, including those made by Oracle Corp. or SAP SE. This lets accountants aggregate, analyze and report data on one unified platform, often without additional training.

Adobe switched to Anaplan early last year and many of the tasks previously performed in spreadsheets are now done in the system, maintaining “one source of truth,” Mr. Garrett said."

Easy:

> Instead, companies are turning to new, cloud-based technologies from Anaplan Inc., Workiva Inc., Adaptive Insights and their competitors.

Indeed Very fast into the article I classified it as advertisment.

Probably referring to the alternatives mentioned

Instead, companies are turning to new, cloud-based technologies from Anaplan Inc., Workiva Inc., Adaptive Insights and their competitors.

Could also be the updates in Excel.

Some time ago Telco carriers used to keep their plant assets data in Excel, PDF, Autocad, images, paper etc. I work for a company that does software to centralize all that info in a database. Once they start using the software the big issue is that they want to keep different versions of the same data all centralized. Sort of a version control system, the complexity of maintaining data integrity in such environment is massive. It is like they would like to return to a decentralized system where they can do whatever they want with their data.

That's not happening.

There is currently simply no viable substitute. (And don't tell me LibreOffice). Excel's mindshare is off the charts in the finance world.

Even if LibreOffice was anywhere close - it wouldn't matter...the finance staff wants what they already know & since they approve the $$$ they usually get what they want.

For certain tasks other applications may be more suitable, but as a general swiss army knife it's here to stay

We use Google sheets at work too - almost exclusively for shared trackers though which don't require any serious functionality.

This article appears to be about how senior leadership wants to pay people that are not financial analysts to create reports that can then somehow be used by financial analysts (probably after exporting to CSV so they can work in Excel?)

This sounds great if you know exactly what reports you need and have the resources to build them, but I'm not really sure how this kind of initiative means "stop using excel". This article is advocating for a solid API / data management / data catalog program from what I'm reading.

In my enterprise, PowerBI, Flow, and Powerapps are the microsoft products helping to move data out of excel, or at least helping to make it visual and shareable.

If any of those products grows a more user friendly model for _thinking_ about how to do things (Flow is arguably the closest) they could revolutionise how things are done here. PowerBI has enormous power to express data as stories, but the technical hurdles are still high. It's not _harder_ than excel, but there's a competency cliff.

One step forward to manage spreadsheets would be if there was a way to enforce relational database table looking sheets. I mean, you should be able to lock a sheet in a way that anything you enter to a column must be of same type, you can't addd random formulas wherever, but you need to add data one row at a time, if you miss primary key, no other changes are allowed, if you have a defined formula for a column, it is guaranteed to be the same in all data rows etc.

>Kayla Davis, who runs financial planning at ABM Industries, relied on Excel to pull data from a motley of disparate accounting systems

"Motley of disparate accounting systems" is your problem, not Excel. Heavy Excel use/misuse by end analysts is a symptom of poor data practices, not a cause. If Excel is being used to compile and distribute complicated/big/important reports, it's extremely likely they lack better and more coherent tools and data sources.

In our 150-people team as part of a 1,500 people company, we use a ton of Excel, but a significant part of our structured data is in https://airtable.com, which I can't recommend high enough.

It combines the structured-ness of a database with the ease of use of a spreadsheet. And it lets you slowly evolve your datamodel (we adjust fields/tables every few months and it usually only takes 15 minutes).

Sounds like their workflow is the problem, not their tool. I'm no shill for Excel, but it is without a doubt powerful and can be quite programmatic. Maybe Excel could get a plugin system like Atom or VS Code or Sublime that could handle integration with centralized platforms / databases.

Just wondering: can you make an API call / ORM query to a database from within Excel macros? How much I/O can you do with macros?

You can do basically anything I/O within an Excel macro (VBA script):

You can talk to a database: https://stackoverflow.com/questions/18313899/vba-new-databas...

Interact with the filesystem: https://stackoverflow.com/questions/11528694/read-parse-text...

Open arbitrary sockets, with some caveats: https://stackoverflow.com/questions/125921/mswinsock-winsock...

And more!

Doesn’t excel allow using any data source for the data? So you can e.g populate a table from a web service/database etc?

The thing with excel is that it’s repl-like and lets you sketch up a solution. If users could build more data driven apps (I don’t doubt it’s possible but it should be easy) like those Access popularized then you could start off using sketch data and finish with using a “single source of truth”.

There are things that Excel is good for and some that it's not. This article seems like it's blaming the tool instead of the people who are using it incorrectly.

Also I laughed when one of the complaints was "Older versions of Excel don't allow multiple users to collaborate on the same document". Well, is that Excel's fault, or should they just go and get the latest version?

IKR. And "On the other hand, older versions of Google Sheets, well, didn't exist"

I used to joke with my colleagues that you never have to write "Excel" in your CV :)

Now that I am grown up a bit, I recognize it can be useful. The real life problem, for me, is recognizing when comes the moment to stop sending spreadsheets by mail and write software, instead.

Or buy it. Lots of people buy sofware, they tell me. Why don't they feel the pleasure of crafting every single bit by themselves?

In my experience the problem with Excel is it gets used as a database often because there are more Excel gurus than DB gurus. The Excel guru in my department had to update his version of Office to 64-bit and add RAM because our spreadsheets were getting too large. Even he admits that Excel isn't the best choice for this but we laid off all the Access/SQL people years ago.

I don't think the article is suggesting stop using spreadsheets - it says some companies are adopting "cloud" based solutions that offer collaboration features and other tools.

Not sure if these cloud offerings have the same breadth of features that Excel has as I've never used them, but Workviva seems to be some sort of cloud based spreadsheet and analytics tool.

I have produced thousands of reports in excel format in the financial services industry. If there were an easier api to just have an excel sheet pull/push data from/to a web service, this would be the best of both worlds.

The interface is very familiar for many professionals, its just a matter of coordinating and working off a the same data.

My side project does this by serverizing Excel sheets. Take a look at my profile for links.

Shouldn't you be able to do the same thing R does with excel?

I mean at some point if you cannot teach everyone a programming language, you can at least give them some tool that can do a similar job.

Teaching programming is so expensive, the money has always been into making programming just easier and easier.

They had me until the part where they complained about using old versions excel. Then upgrade it, and while you are at it upgrade your windows XP machines too. Why are these companies complaining about using old software when they are the ones who are always last to upgrade?

The problem is very simple. Everybody understands how to sum cell A1 with cell B1 and have a result in cell C1. Summing arrays, though, like in an R dataframe, is much more difficult to understand. You don't see it and you need to train your brain to think in SQL terms.

There are lots of great tools to solve the single source of truth problem. foe example, I've used Looker (which is connected to our data warehouse) to spit out JSON feeds or CSVs which excell and google sheets can live-import.

Doing this gives us the best of both worlds

Deploying Excel in production is like deploying a beadboard in a piece of consumer electronics. You just give away too much control to keep your deployment robust. Having said that, Excel is an excellent tool - just not for everything under the sun!

Paid advertising? I use LibreOffice and it fullfills all my needs.

I understand, there may ve different needs in big corporations, of course. But that doesn't mean "everyone should stop using Excel and buy our cloud SW", sigh.

In the article, and in these comments, are people distinguishing between Excel and Google Sheets, or are they lumping them both together as “excel”? Where I work I don’t think anyone uses Excel, but Google Sheets is common.

I hate Excel, can't quite put my finger on why, but I'm a big Google sheets fan when it's used with their apps scripts. You can do a lot using the sheet as a data store combined with functional JavaScript.

Look, I hear you... I love Google Docs over Microsoft Word... and for simple things I like Google Sheets.

But given how much shit is in Excel, and how many edges cases have just been worked out already in Excel... man sometimes it's just nice to have Excel.

Want to de-dupe a list? There's a button for that in Excel. (I know there are tools... but look my point is just that it's so much easier in Excel.)

Want to convert some list of dates that were saved based on a 1904 calendar over? Easy, Excel even offers to do it when you paste. Google docs? Yeah not so much. You're left doing this by hand... 4 years and 1 day off... ugh.

Want to see all the formulas and know how they work? You get only so much visibility into the apps for Google Sheets... I don't really want to run code / grant access to things unless I know what they are doing. Not keen on adding 3rd party functionality that may be sending my data back to the mother ship.

I love me some Google Sheets... but just saying when I need to get shit done, Excel is usually a slightly more robust experience.

“I don’t want financial planning people spending their time importing and exporting and manipulating data.“

That is literally all I did when I was a financial analyst and is how I became a programmer. This should be automatic, I thought.

"[Microsoft] wouldn’t comment on what share of [Office 365] customers are using the program for financial analysis and accounting.

Good, I would be concerned if they knew exactly what each customer was using their software for.

I remember one IronPython book that made a spreadsheet application that allowed the user to use python itself for the spreadsheet functions. I think it was a great idea.

The software itself was available for download for a time.

There was a startup called Resolver Sytems pursuing this. They meanwhile abandoned it in favor of a different project but did open-source it: https://github.com/pythonanywhere/dirigible-spreadsheet

We can use Excel as long as we are aware of its limits/loopholes e.g. http://0.30000000000000004.com/

Sounds like misuse of excel rather than don't use it.. no mention of powerpivot , query tool or connections. also there is much more to consider when switching to a cloud solution.

I worked at nPower, in MIS, they used to link about 16 spreadsheets together because they "didn't like" SQL. These are one of the UKs big six energy companies.

One killer missing feature in Excel for me is proper UTF-8 support. Connect Excel to your db, save a csv file to import to your website and say goodbye to your umlauts!

Why does this feel so much like sponsored content for Workiva?

I think all the features he listed as reasons to switch away from excel are in excel. IMHO the issue there may be more training, process and good data management.

It seems like using Google Sheets or the online Excel is the answer. Single source of truth, collaboration, and keep the same fundemental tools.

Workiva's stock price has jump to an all time high after the wall street journal article. This is great news for them ;)

This CFO should talk to CIO - before making such a blunt statement. So one silver bullet to solve all the problems.

Feels like an ad

it's an add in deed.

This is just an ad for workiva, right?

It's weird how no one can adopt office365 and online Excel like they can with Google sheets.

No one has yet mentioned the shortcut keys, so I'll add this dimension.

When you show up to work on wall street, your new buddies will yank the mouse out of your machine and remove the F1 key. Then you can begin your apprenticeship with a fresh perspective.

The point is, you can be incredibly productive in excel against a very large problem space. It can't handle GB datasets, but once you have filtered down to a manageable size you can answer questions via ad-hoc analysis, automate financial reporting documents, solve multivariable regressions, run machine learning algorithms (see 'Data Smart') etc. with aplomb. I understand the objections to the data model and its inappropriate uses, but excel can take raw data from disparate sources to finished documentation for regulatory requirements and helps you automate most of that -- I don't think the finance industry is going to move away from it. I certainly don't perceive an understanding of the costs involved with building a different way.

>“I don’t want financial planning people spending their time importing and exporting and manipulating data, I want them to focus on what is the data telling us,” Mr. Garrett said

I hear this complaint all the time from business leadership, Mr. Garrett doesn't seem to understand that extracting/munging/preprocessing and then analyzing data is inextricable from the end product which is visibility and understanding at the business leader level. There isn't a follow-up to say Mr. Garret is hiring data engineers and software developers to serve the now-liberated 'data focusers'. I'm very skeptical that these sentiments are different than the past, or are a signal of changing times ahead.

Lastly, I'd to share the following funny snippet from the reprehensible though satirical leveraged sellout blog:

>I turned and tapped Josh on the shoulder to get his attention. I didn’t say a word, but my face said “Yo…yeah you, McKposer.” We locked eyes. I slowly and dramatically reached for my mouse cable. Staring him down, I methodically unplugged it from my computer’s USB port. The cable came out slowly but smoothly and with it’s final, climactic release, I heard Josh gulp with fear.

>I continued to glare at him as I started to “fly around.” My eyeballs burned through his unweathered, 60-hour-a-week-when-he-should-have-done-90 face as I hid and unhid sheets and conditionally formatted and applied validation to cells with triple nested conditional and indirect functions and then culminated by on-the-fly writing a macro that took my data and made it into a Marimekko (and we don’t even use that shit!). I never turned to look at the screen, and my eyes never left his.

This story has less to do with Excel than it does having staff capable of using technology

This article could have been written 20 years ago, and it probably was in some other form.

This feels a lot like a submarine ad.

Also if this guy thinks he's going to escape silos by putting everything in the cloud, he's quite mistaken. The cloud is just code running somewhere else. It does not change fundamental computing paradigms.

This sounds like an ad.

Excel shouldn't be used for much other than trivial things:

http://www.sciencemag.org/news/2016/08/one-five-genetics-pap...

https://www.economist.com/blogs/graphicdetail/2016/09/daily-...

he authors found that Microsoft Excel would often interpret “SEPT2”, which corresponds to the gene Septin 2, as “September 2nd”. The programme also tended to mistake identification codes like “2310009E13” for numbers in scientific notation—in this particular instance, the code would be read as 2.310009 times 101

Excel is fine for the home user. But the implicit conversions of input data can play havoc with any complex analysis.

Plus, it's limited precision can cause errors such as subtractive cancellation, etc. In order to correctly calculate complex formulae, the calculations must be done with an understanding of the limitations of the computers.

e.g. you don't calculate (a^2-b^2) for large 'a' and 'b'. Instead, you calculate (a+b)*(a-b). That has the same mathematical result, but is not affected by subtractive cancellation.

    > Excel shouldn't be used for much other than trivial things

Good luck with that, but it ain't gonna happen anytime soon.

Many "non-trivial" applications got their start at someone's desk in the form of a shitty excel spreadsheet with horrific macros. Is it optimal for "complex analysis"? No, but it doesn't matter because most things are computationally simple. The hard part is the intricate business logic and the timeliness of getting results and these things are not "trivial" at all.

Instead of complaining about how terrible excel is, the community here should be providing alternatives. And no, engaging a software team/consultant for a million dollars to develop bespoke applications or interfaces to enterprise systems for every little project isn't a viable solution for people who need to get stuff done pronto.

A real alternative to Excel is no small feat.

I agree with that. Modern software development has version control, unit tests, package managers, etc. Ideally we get support for some of those concepts in something that feels like a spreadsheet to the end user. (I'm not sure if it could ever be retrofitted to Excel)

> Instead of complaining about how terrible excel is, the community here should be providing alternatives.

Yeah... like I'm going to compete with MS by writing my own Excel replacement. That just isn't realistic.

What would be realistic is for MS to acknowledge that a large proportion of their customers use Excel for a particular purpose. And then tailor their software to the needs of their customer base.

But why would MS care? The researchers already bought Excel. So why "fix" it?

On top of that, no one is aware the the published papers are crap.

The real solution would be for journals to deny publication of papers based on shitty Excel analysis.

This is what screwed me over as a herbarium curator (a herbarium is a type of natural history museum for plants):

Excel completely corrupts dates before the year 1900. It just boggles the mind -- if your spreadsheet has dates for 10,000 years from now that works fine, but if there are any dates from the 19th century or earlier, it can cause big problems.

>Excel is fine for the home user. But the implicit conversions of input data can play havoc with any complex analysis.

Can you not turn all of that off (and only need to turn it off once per sheet/file)?

Not really. With care, you can usually get the format right temporarily, but it often changes for non-obvious reasons and there's no way to see that the format is wrong until it mangles your data.

In general, one of the worst things about Excel is there's no obvious way to lock anything to keep it from getting changed by accident.

Tell that to tens of thousands of researchers.

Heck, even integration has been re-discovered and published:

https://fliptomato.wordpress.com/2007/03/19/medical-research...

If the tool doesn't work correctly by default. Then it's likely the wrong tool for the job.

The default conversions are fine for 99% of use-cases. But no one should mistake Excel for a robust data analysis tool.

Can anybody think of other examples of products or tools that fall into the category of: “many people use them, even though management tells them not to, because it helps them get stuff done.”

I’m not a mechanic, but is WD40 an example? Or maybe some tool like a hand held circular saw, in home construction?

And, other examples from the tech world?

Trello. In the last few jobs I've had there's almost always been a developer who has their own personal to do list on a trello board, even if we had robust tracking systems like jira and rally.

Is that such a bad thing though?

Having tickets in JIRA is for the team to track and plan work that needs to be done, not for storing personal garbage.

I thought the spreadsheet revolutionized accounting in the 80’s, not specifically Excel.

My dad brought home an IBM PC in 82, using Multiplan for chemical mixing calculations. He thought it was the cat’s meow. Though Multiplan was the predecessor to Excel, the spreadsheet had been out since late 70’s.

This is just an ad for the cloud services they mentioned. Suits whining about not having absolute control over every scrap of information is not news.

Why does a powerful Excel alternative like https://calculist.io/ (that is not a spreadsheet) and others don't get much attention?

"Tell Staffs" really I am a dyslexic and even I wouldn't make such a trivial mistake with plural /singular.

The CFO of a company tells the staff of the company to stop using Excel; the CFOs of several companies tell the staffs of those companies to stop using Excel.

no its not

Several CEO's (plural) tell their Staff (singular ) a CEO only has one set of employees.

The headline implies that all the CEO's have multiple sets of staff

Reply on news.ycombinator.com