Heron Donated to Apache Software Foundation | Hacker News Reader

Heron Donated to Apache Software Foundation

blog.twitter.com

28 pointstylertreat10 comments

> Today, we are proud to donate Heron to the Apache Incubator where the community will continue to grow and thrive under the guidance of the Apache Software Foundation

Why do I keep reading this as "Today, we are proud to donate Mittens the cat to the local animal shelter where the kitty will continue to grow and thrive under the guidance of the SPCA."

When I was doing a lot of ASF administration (Board, VP Incubator, VP Legal), I was always keenly aware that the core personnel on the projects coming to the ASF were sensationally brilliant engineers and charismatic thinkers. The idea of "guiding" such people like cats... is just way weird.

Given the perception (fair or not) that the ASF is the place that large projects go to die, I'd like to know why this donation is a good fit for the ASF and what resources are in place to help it flourish there.

If potential users are running on Mesos, streaming data from Kafka and writing results to HDFS, they're presumably not that worried about relying on Apache Foundation managed software.

Not to say that this project won't die - most projects do, but I think the "where projects go do die" stereotype is unhelpful. ASF is probably the best chance a project has to move beyond the control of a single company.

My company is considering "upgrading" our Storm topologies to Heron. Given that Storm is Heron's predecessor, and Storm itself is maintained by the ASF, I wonder what this donation means for Heron.

p.s. does anybody have experience with Dhalion, Heron's autoscaling thing (is it a module? or a plugin, idk). All I could find on it was a research paper and a barren github repository; how do you use this thing?

Great! Thanks for the questions. The Dhalion repo (https://github.com/Microsoft/Dhalion) contains only the general Dhalion APIs. These are not Heron specific, that's why they are not part of Heron. In Heron, the healthmgr module essentially integrates these Dhalion APIs with Heron. Currently, Heron contains two Dhalion policies. The first one automatically restarts instances that exhibit backpressure. The second one scales up or restarts instances depending on some topology characteristics. Please join slack and we can show you how to use/modify the appropriate Heron yaml files so that you can use Dhalion. Dhalion does not require a separate installation -- it is already integrated with Heron and just needs to be configured depending on your use case. We are currently working on more aggressive autoscaling policies which should be part of the code base soon. Please let us know if you have other questions.

Thanks for the reply!

For us, the ability to self-tune is one of Heron's most attractive features. Tuning components/executors in Storm was a big pain point. Have you guys ever tried using the Dhalion autoscaling combined with hardware autoscaling from a cloud provider (i.e. Azure, AWS EC2)?

No, we have not used auto-scaling with automatic hardware provisioning. We assume that the hardware is there and we just request more containers (e.g, YARN containers). Theoretically, it should be possible to do that if the autoscaling policy is extended to request for additional hardware before changing the parallelism of a stage. But this functionality is not currently there.

Great! If you could join the heron slack - we can help you with upgrading and how to use Dhalion -

http://heronstreaming.herokuapp.com/

RIP Heron (;,;)

Reply on news.ycombinator.com