With a new tool, spreadsheet users can construct custom database interfaces
news.mit.edu
news.mit.edu
Then you have ancient tools like HFM or BPC that try to "enhance" spreadsheets and spreadsheets consolidations. They don't work. They are barely reliable and they cost a lot of money for no reason (they are simple SQL databases, they can't even be compared to much more complex software like salesforce or similar).
Then Corporations are now being sold this all "big data" thing, which is old by now for tech audiences, but it's just new for most of the big companies nowadays. Unfortunately, while big data has a lot of potential, it further moves away investments on good old small data where probably there is much more ROI to grab simply because solving this problem is not that difficult and not that expensive if you want to actually fix it.
Then now everybody wants to do Predictive, but Predictive won't beat saving hundreds of employees thousands of working hours by improving the efficiency of how we handle data and data insights. You can literally create a new workforce out of the many hours you would save with better efficiency in this area. Without even considering that on top of efficiency, you get much better, more fact based, decisions driven by the overall increase in transparency which you lose when insights are spread around thousands and thousands of files.
There is some light at the end of the tunnel. Software solutions like Tableau go in the right direction, but they do not provide the much (more) needed tools to properly manipulate and manage data and, especially, consolidations and integration. The only way to get out of this mess is to control the data flow but especially to have 1 and only 1 data flow. That means once your insights are approved and locked, every other view should read this data, there should be no manual intervention any more. If the locked data is wrong, it will be wrong for everybody, which is a good thing compared to having multiple copies of the same and then people fighting on which one is the right one.
1. Everyone is willing to sell you something that is going to manage all of your disparate legacy data, except that somehow nothing ever seems to die in the Enterprise and instead you end up with users of the unmaintainable legacy system AND a second system on top (and third, and fourth) that now also all need to be maintained.
> Analysis and reporting are the easy part.
I don't agree on this point. They're just as hard as ever. Of course you have Excel, Reporting Services, and Power BI. But everyone wants you to formulate the QUESTIONS as well as the ANSWERS. As soon as they discover all of the issues you've found, now it's bury-your-head-in-the-sand-time, because they just want to whittle it down to that top 1% of 1% of problems so that their stats look good; whack it into a daily email, which goes onto a daily KPI sheet, and the underlying issues they originally wanted to know about of course never (or rarely) get fixed. Progress!
2. Let me tell you about Big Data! It's a codeword for:
* We know you've never archived or thrown anything away before, that you have data coming out of your ass.
* We also know you'll never use it. You don't have the funds, inclination, or staff to use it. But one day we're going to build something that will magically do it all for you. So you better keep holding onto it.
* Oh yeah and we have something to sell you to help you keep holding onto it all until that magical day arrives! Please send us a purchase order. Thanks.
I love data. I am sure that some companies like energy companies and telecoms do actually have to deal with massive amounts of data. But Big Data has become a management buzzword that doesn't mean much except retaining everything and spending money.
If you retain everything you never have to make a decision about what to destroy. Management loathe having to make a decision to destroy anything that they might be held liable for, but also love spending money on something that they can use to justify that they're solving a problem.
Thus the success of Big Data.
3. Predictive. I can't comment. Everyone talks about it but I don't see the solutions in the real world. Most of us are stuck in Excel / SQL and just starting to dip our toes into R. When everything else in life is hard, predictive becomes just another harder thing to avoid.
> There is some light at the end of the tunnel.
What you're talking about is Master Data. Of course the well for this has already been poisoned long ago and woe to all who sip from it.
Everyone loves starting an MD project that siphons data in. But the second people have to manage the data inside and start cleansing it and making decisions - it falls in a heap. It rarely gets to the stage where data is then written back to source systems it's game over (LOLOLOL many a vendor I have left in the dust just asking them about this - "That's phase 2" they say - and they've never had any single customer reach it).
--
Back on point about the article though, it looks like an interesting way to structure data and it could probably do as an Excel add-on of some kind. I expected something that would create a database but unless there's more than the video this is just writing queries?
This is all well and fine, I guess, if it adds usability. But nobody's jobs are going anywhere. The article says this tool can create anything SQL-92 can. Good luck... I deal with shit daily that you couldn't even begin to imagine.
I found things to do... but it is not the core of the problem. I am lucky my boss is quite high on the hierarchy and may be convinced to attack that core problem. But it will take a looooong time.
And no, most of them would not take long time. They just need a political decision, but could be solved in a couple days.
As much as I hate Sharepoint, it's actually, as of this year, come close to solving this problem. Full fledged Excel (and Word) documents synced between many authors working simultaneously (section-wise edit locks until next explicit save/sync), no one ever emails anything around. It's not all the way to a Git-like solution, and (at least for the moment) tied to IE on Windows, but compared to the old ways, it's a winner.
I haven't used this in production, but messed around with it for a while. Seems to do what this article is suggesting.
Link to the project page: http://people.csail.mit.edu/ebakke/sieuferd/
Source code or it doesn't exist ;)
Dude I can build a house. I've only used a drill once in my life, I've never mixed concrete, or bricks, but I can. I will lay stuff on top of each other and use superglue and whatever it takes.
But nobody will ever want to live in and it would not be safe. That's why you get a professional to do it.
A user might somehow be able to magically skip gathering requirements and just dive gung ho into creating a structure for what they want. Great. And you end up with the legacy Excel monstrosity turds that we've been trying to eradicate from the Enterprise for the past two dozen years.
It is apparently more useable than Access, which is damning by faint praise.
And it is not just reporting. In some cases inputting large quantities of tabular data is just done by writing Excel macros that create CSV files to upload to the system (and sometimes, horrifingly, the data are just retyped...).
Without the ability to react quickly that smaller businesses have, I noticed that some form of resistance from IT is usually a big factor: lack of resources, divergent priorities or visions, unclear benefits from hypothetical feature, limited architecture, end-of-life maintenance states that end up lasting years, etc.
Reality changes, systems fall behind, users adapt and spreadsheets are born.
One root cause is that it's comparatively hard to add new data in a RDBMS - it usually requires a full development life cycle. On the other hand, spreadsheets are free, provide immediate benefits, allow users to define and refine requirements as they go, and can be abandoned at the drop of a hat (low risk). Quite the conundrum!
From the article:
> At present, Bakke’s tool enables query construction on an existing database, but it doesn’t enable the direct entry or modification of data. He expects to begin adding that functionality over the next six months
Querying data is a problem with many good solutions nowadays, the bigger question is where and how to write that new data the users want to play with (other than in a spreadsheet).
As far as "add new schema" goes, I'll claim that the database is just another software component, and if you pursuing an agile strategy, that you should be modifying that schema with every sprint. My client, for whom I do data warehouse consulting, updates the schema every two weeks. Because we are responsive to the business, they never revert to spreadsheet solutions. Of course they are free to create there own custom reports in spreadsheets - we'd never try to take that away. In fact we make it as easy as possible to run business curated queries and dump the data to Excel. Our newest feature for these power users is to allow them to give back to us the Excel reports they built and we will "push and deliver" - put the current data into the raw data sheets and deliver the report - usually to Sharepoint.
Getting good business software solutions in place merely requires that the business hire some good developers. Why more don't do so remains a mystery. My best guess, like all things "corporate" is fear and laziness.
Not that I do not believe you but... How hard have you looked? In my experience the largest the corporation the more chances there are that IT just does not realize what the situation is. "Shadow governance by spreadsheet" is something you notice only if you actually work in the same office withe business users for a few days, in my experience (or when the whole thing has grown so much that they cannot really manage it anymore then ask you to fold it back into the main system).
Of course, YMMV.
A newer system which replaced some of the functionality and cost many millions of dollars will only export as .xls and only import from .csv (no API offered). Better than typing it in by hand, but it still pains me every day :-/