The variation of complexity of transforms across Enterprises and teams within Enterprises is quite large.

Spark and Hive (that I was product manager for, and used by hundreds of Enterprises) are used quite heavily in data engineering. Many transforms fit well into relational/set based model.

In your particular case, perhaps due to the nature of your transforms, it might not have been the best fit.

Also, if your data is small, it's not always a great fit. However, in many large companies, if they have 500 transforms and 30 are small - there is simplicity in using a single technology.