1. Pay a ton of money to Microsoft for Azure Data Lake, Power BI, etc.
2. Spend 12 months building ETLs from all your microservices to feed a torrent of raw data to your lake.
3. Start to think about what KPIs you want to measure.
4. Sign up for a free Google Analytics account and use that instead.
Okay, sounds reasonable enough for a complex enterprise.
> to feed a torrent of raw data to your lake
Well, there's the problem. Why is it taking a year to export data in its raw, natural state? The entire point of a data lake is that there is no transformation of the data. There's no need to verify the data is accurate. There's no need to make sure it's performant. It's just data exported from one system to another. If the file sizes, or record counts match, you're in good shape.
If it's taking a year to simply copy raw data from one system to another, the enterprise has deeper problems than architecture.
Yes, it makes life harder for the data engineers in the future, but it might turn out that analysts only ever need 5% of the data in the lake, and dealing with these schema changes for 5% of the data is easier than carefully planning a public schema for 100% of it.
It can be helpful to include some small amount of metadata in the export though, with things like the source system name, date & time, # of records, and a schema version. Schema version could easily be the latest migration revision, or something like that.
But if I haven't spent the effort to extract it, do I really own it?
Then you just don't keep red data for longer than 30 days.
Consider:
Source Data -> Data Lake -> ETL Process -> Reporting DataWarehouse(s)/DataMart(s) -> User Queries
vs
Source Data -> Data Lake -> User Queries
vs
MonolithDB -> User queries
vs
MonolithDB -> ETL Process -> Reporting DataWarehouse(s)/DataMart(s) -> User Queries
A schema change in the source data should be easily updated in the ETL process in example 1. Most changes are minimal (adding, removing, renaming columns). And for a complete schema redesign in the source data, a new entry in the data lake should be created and the owners of the ETL process should decide if the new schema should be mangled to fit their existing reporting tables or to build new ones. Across the four models I outlined above, the first is by far the easiest to update and maintain, IMO.
Another strategy is that the service has an explicit API or report specification; that way, the team that owns the services also owns the problem of continuing to support that while changing their internal implementation.
Of course, whether the benefits are worth the cost is probably organization specific, just like microservices in general.
https://en.wikipedia.org/wiki/Data_lake
The reasons for it are hard to explain succinctly in HN comment but if you look up data lake there will be a lot of explanation. But it basically comes down to "is it better for the data integrators or the data consumers to massage the data?" And data lakes was the insight it's really great when the consumers decide how to massage the data.
To me this sounds great. And honestly you should do the same thing with a monolith. Nothing worse than "oh you can't make that schema change because a customer with a read-only view will have broken reports".
The main benefit of a dumb copy is that the production service is not impacted by reporting, only a copy is. This relates to performance (large queries) but also implementation time.
On the other hand, the notion that “microservices == completely independent of everything else” is an unrealistic one to hold.
But yeah, it's funny how these projects get complicated in larger organizations. Personally I would have rolled something even simpler on gnu/posix tools and scripts, in rather less than a month.
Use off the shelf stuff but be prepared to have to move in a (relative) hurry.
“Data lake” may not be the right answer, but GA certainly isn’t.