At a recent place of employment I created and was responsible for a database that had about that many records and in actuality was a single 2tb postgres db and completely unremarkable.
I never claimed to have worked with big data.
At a recent place of employment I created and was responsible for a database that had about that many records and in actuality was a single 2tb postgres db and completely unremarkable.
I never claimed to have worked with big data.
It's what is and isn't in the data - often a lot of junk in my experience if the source system is a legacy system that has evolved
what meaning that data has within a completely different system
what the demands are on the completeness of that data is in the new system
how to deal with exceptions
and whether that data can ever be frozen, or whether it is still online (as in the case of banking transactions)
This is unlikely to be simply a technical problem of ETLing tables, changing date ranges from inclusive to exclusive and mapping some address fields.
Of course the size of the data after a certain point does make a big difference to risk planning and business continuity planning. It's not possible to rollback and try again within the migration window should a catastrophic issue occur, and it's not possible to simply run some bulk updates to fix issues during the go-live validation.
It is noted though in this project that the data migration itself was not found to contribute to the failure.
If you have a simple db structure with a few tables and very clear data/index rules then billions and billions of records is pretty easy. Your indexes cut out 99% of the work and everything runs smooth and efficient.
But then you can have eldritch horrors where your stored procedures look like seedy detective novels where you chase join after join and have scary high memory requirements on execution.
Personally I would say it is a lot. No other number mentioned in the article even approaches a billion. Billions are big. It might be not be true big data big, but it is still a lot of customer records for a migration project (depending on what exactly they were trying to do within the migration) and it does illustrate why they had so many issues, because there was a lot to deal with.
(Of course this is just speculation. I have no insider knowledge.)
As a sibling comment states a screw up of this magnitude is never simply a technology issue — it requires bad management at many levels.
Still need to write all the procedures, test it, then do it on live system again.
It might make prototyping easier (...or harder) but that's about it