Ask HN: How to go about the goal of democratising data at our organisation
We(@ squadvoice.co) wanted to take your help/suggestion in something we're trying and figuring out if to build or not. Below is the problem statement that we're focusing on majorly. Become a data-driven organization where anyone can make decisions & leverage data in a self-service model without spending too much time on data retrieval or data query formation. Why we're thinking we should build a solution for the same?
1) Ad-hoc analysis requires engineering bandwidth at a lot of time. And due to that product features get delayed.
2) New queries require engineering support at a lot of times.
3) Operational queries become stale (inconsistent) with changing business logic.
4) Queries timeout, run slow, etc.
5) We take lots of decision basis the data, and if data fetching and gathering is slow and time taking it is gonna slow whole org down.
We do have SQL dashboarding and things in place, but new requirements keep coming up(I guess we're trying to find new data points every day to see what we can improve which is great honestly) and application+business logic change so often that it is hard to keep it updated.
Now, we wanted to understand:-
1) How people in other organizations like yours are solving the same?
2) What are the things you tried what works and what does not?
3) How do you validate whether it is something that you should solve for or not?
4) What're technologies and gotchas that you think should we be aware of and consider for this kind of things?
5) How do you make sure, whatever you build is scalable and alterable such that if you change your application logic in future things still remains sane.
6) How do you make sure, people are able to get data they need as fast as they can.
I am already sorry if this post looks long and some redundant elements, but I wanted to communicate problem and what we're looking for help in as clear as possible.