The trend has been to shift as much work possible to the current generation of Data Warehouses, that abstract the programming model that Spark on columnar storage provided with only a SQL interface, reducing the space where you'd use Spark.
It makes it very accessible to write data pipelines then using dbt (which outcompeted Dataform, though the latter is still kicking), but then you don't have the richer programming facilities, stricter type systems, tooling and the practices of Python or Scala programming, you're in the world of SQL, set back decade or two in testing, checking, and a culture of using them, and with little tools to organize your code.
That, if the team has rebuked the siren songs of a myriad of cloud, low-code platforms for this or the other, with even fewer facilities to keep the data pipelines under control, be it that we call control any of: access, versioning, monitoring, data quality, anything really.