Ask HN: Best tools for image processing pipeline at scale?

Hi HN,

I am a junior python dev. I have been looking for some sort of tool or framework that would support a large scale image processing pipeline. Currently, I built out a pipeline in pure Python and use Pillow, numpy, and other packages to download, manipulate, and upload images.

I have been looking into apache airflow, spark, kubeflow, and some other tools. None seem to really fit the bill of large scale image processing, but I would love any suggestions for tools or frameworks I should consider or reconsider.