i pity the developer who has to maintain tagless final plumbing code after the “functional programming enthusiast” moves on… in a Go first org no less.
I’ve been doing this too long to shoehorn in my pet languages if I didn’t think they’re a good fit. And I think that scala/FP + Flink _is_ a good fit for this use case.
We did also explore the go ecosystem fwiw - the options there are limited (especially around the data tooling like iceberg) and go is simply not a language that’s popular enough in the data world.
Python’s typing system (or lack thereof) is a huge hinderance in this space in general (imo), and Java didn’t cause many happy faces on the Eng team either, but it’s certainly an option. I just find FP semantics a better fit for data / streaming work (lots of map and flat map anyways), and Scala makes that easy.
Also no cats/zio - just some tangles final _inspired_ composition and type classes. Not too difficult to reason about, not using any obscure patterns. I even mutate references sometimes. :-)
If integrating with java tools was important then personally I’d ask “why not Clojure”.
:)
The BEAM Golang SDK work came from Googlers working on Beam that were Golang fans, and internally there were Golang-oriented tools for batch data processing that needed a migration path forward.
Historical Note: Apache Beam also originated from Google as "Dataflow"
In most well-resourced companies, you’re probably not going to have to ask your Go engineers to fix data pipelines in Scala.
As for well resourced I guess it depends, that variable usually doesn't mean much, as we can see by companies firing whole departments, while swimming in profits.