Do you teach people about "hygiene", e.g. data provenance, versioning, and how to design schemas? I work on "big data" stuff at one of these major companies, and IMO the state of things is pretty sad. A typical pipeline involves a bunch of files strewn about a distributed file system, or a pretty messy database, especially when multiple teams are involved.

The tools (I use) don't encourage good practices or have good defaults. You have to put in extra effort and write proper metadata, etc.

I think things are just new so this kind of issue doesn't get much attention yet. Curious to see if anyone has written anything about it. I guess academics and government and people who have to keep data around for a long period of time will have thought more about this.