Comment by submeta | Hacker News Reader

Comment by submeta | Hacker News Reader

Parent Full thread

> “I don’t want financial planning people spending their time importing and exporting and manipulating data, I want them to focus on what is the data telling us,” Mr. Garrett said

This!

I know this is reality in so many companies that don't have technically educated staff to tell them about more suitable ways to do it. So the teams get data from one department or system, manually transform data (in the editor via search/replace), and if they are "very sophisticated" they save it in MS Access, join tables, connect the dots, export back to Excel, analyse the data via Pivot Tables etc. And they do these steps repeatedly, every week. Without stopping and asking if there might be better solutions out there. (In German IT curriculums they call this "Medienbruch" = "media discontinuity"?)

I have seen this in so many companies and thought to myself: This is an excellent case for a consulting pitch. And I could literally transform their work.

And I am not talking about selling data warehouse solutions with ETL processes and the like.

Twenty years ago I realized I could do all these preprocessing and transformations with Perl. Now I do these kind of things with Python and Pandas etc. So there are so many solutions that are a huge improvement. Still these departments do it with Excel/VBA/Access.

I have seen many of these companies and potential "build a real database-backed web app" projects during my software consulting career.

This seems like a good idea at first: automate manual processes, have a real database with one source of truth and consistency rules, multiple people can access it at the same time, etc. You can do it if the process is very well known and fixed.

But most of the time a web app would cause more hassle for the companies than having a team of people doing the work "by hand" in Excel would be.

* The process and desired data schema might change every few months - or even weekly when doing something like marketing reports. * Now they need regular maintenance - keeping servers up, doing and testing backups, applying security patches. * But the problem being solved is not big enough to justify hiring a full-time team of software developers to maintain it forever.

Don't forget the all-to-common scenario:

"Executive 1 likes to see the data presented in formats A and B, Executive 2 likes to see it presented in formats C and D, and occasionally E, and Executive 3 will only look at it if you format it as F and cut/paste it to PowerPoint."

So you end up with 20 analysts who all have input, each manually re-formatting their own data in 6 different ways, making sure they all stay in sync. Then someone else collates it all together into 3 different exec-specific presentations, and double checks to make sure they tell approximately the same story. Welcome to life as an office drone!

I used to work as a corporate financial planning analyst and this hit way too close to home.

Then one executive who didn’t like the story would have one of his analysts pull the same data in a slightly different way and adjust it. The executive would then show it to the CFO and tell him that’s the real story. The CFO would turn to you and ask why his number is different than yours. Then you would spend hours with the analyst reconciling the data and building a waterfall chart to send to the CFO to explain the difference.

Then if you’re really unlucky the CFO says he wants to see the data both ways with a waterfall reconciliation every month.

Do you have any suggestions on where to get started with Pandas? I already know Python, but every time I try to get into Pandas I have a really hard time getting any traction. I'm particularly interested in using it to migrate Excel workflows.

Reply on news.ycombinator.com