Ask HN: Best way to build a data collection pipeline?

Bistro Streams is an open source light-weight stream analytics engine: https://github.com/asavinov/bistro

It is a general-purpose and highly configurable data processing engine which can be applied to many workloads and scenarios including data integration, data migration, ETL, big data processing etc.

Snowplow open source or managed solution ($$$) - give it a look. http://www.snowplowanalytics.com

Kafka is probably the best option in this case.