627 karma · joined June 30, 2011
This is quite interesting. Founder of RudderStack here (we are a CDI or simply an open-source Segment equivalent). I have seen a similar pain point across some of our customers. They use RudderStack to get data into S3 (or equivalent) and then run some kind of post-processing Spark jobs for analytics/machine-learning use cases. Managing two setups (RudderStack on Kubernetes + Spark) is a pain.
A singly managed solution with Spark on Kubernetes makes so much sense. Would love to figure out how to integrate with you guys.
We have thought about this quite a bit. The way to address this is to make offering your OSS as-a-service from day-1. Initially, you are small and AWS won't care. Once you become big, you should be able to compete with AWS on the service offering - afterall you know your code best.
The problem arises when the OSS vendors had a different business model (open-source, on-prem support only) and AWS is able to completely own the as-a-service market.
I myself have used two of these projects heavily (Airflow, SuperSet) and am thankful to the teams for building such great products.
But did this make any business sense for AirBnB to spend on these? In this post COVID world, very unlikely this would continue.
Segment (or RudderStack) is primarily for routing event-data from your apps to multiple destinations (cloud SaaS, warehouse etc). On the other hand, AppFlow is for syncing data that lives on cloud applications with AWS products like RDS, RedShift etc. So, both the nature of data is very different (event-streams vs database update batches) and also the destinations where they are sent (multiple cloud applications + data-warehouse VS only datawarehouses/databases).
Segment also has a product similar to AppFlow called Segment Sources but that's not what Segment is known for. There are a bunch of other products in that space like Blendo & FiveTran.
Wondering if there are any legal issues around this (can cable networks go after this saying it's their content)? What is being played on my TV is entirely up to me but then this argument breaks down in some cases (e.g. car odometer - I can't replace the reading). Might be worth nailing down.
There were some high profile cases of people taking millions of dollars of loans and running away (search for Neerav Modi, Mehul Choksi) so I am not making this up.
Would love to understand your use case a bit more.
We wanted to give out an update from our last HN post so we didn't get into the depth of what the product is. Kind of assumed everyone knows segment but yes a fair point.
We have something called Transformations (user defined functions) by which you can modify the event structure from the Rudder BE. You write the transformation function (currently javascript) on our UI and that gets executed in the backend on your event stream. Using the transformation, you can also control where the event goes to. This is helpful when say you want to change the destination without pushing an update to mobile app
Peter (segment's CEO) had a great answer to this in the thread https://www.quora.com/What-is-the-advantage-of-using-Segment...
TO summarize and highlight the main reasons
1) With GTM you still have to write code. Specifically, you need to figure out how to send events to a destination following their API and JS library. With Segment and Rudder, you just call a couple of generic functions identify, track, page etc.
2) You often want a copy of the events in your own S3 Or Redshift or Snowflake. GTM doesn't help you there but Rudder/Segment can.
3) We have some features like Replay etc which lets you send historic events to a new destination. Say you signed up for a new Analytics tool and want to send all your historic events - Replay helps there
4) Finally, Segment has this product called personas which lets you create audiences (e.g give me all people who have done X but not Y and then send that to MailChimp for emailing). We too are working on it.
We don't post pricing because we're still figuring it out (always a tricky subject for OS project). We charge a platform fee and a small per node charge (beyond 1 node) but there are many dimensions there - node size, number of nodes, etc and we haven't figured out all the combinations. The platform fee generally starts at $2k/month but we offer discounts for start ups/open source projects/non profits, etc.
Right now we more focused on growing our open-source adoptions & customers and we want to talk to our prospects to better understand this.
As a benchmark 1 node can handle ~1K events/sec.
Let's say you have a website or mobile app and you want to collect all the user interaction events (clicks, searches, impressions) in a data-warehouse like Snowflake.
RudderStack can do it for you. We have SDKs (Android, iOS, JS, Python etc) which you can use to send events. We have a corresponding backend (you can run it or we can host it) which will collect these events and dump into your warehouse.
You often want to send (subset of these) events to other 3rd party websites too like Amplitude for analytics, Braze for marketing automation and so on. RudderStack can forward events to those too so you don't have to embed multiple 3rd party SDKs and understand their library etc.
The second part of this is to create a unified view of the customer as you mentioned and take action on that (e.g. find all churning users and send them email). That's where Segment has a product called Personas and we are working on something similar but it's not launched yet.
Though even before, you can build that customer view on top of your event data in your data warehouse by using SQL or a tool like Looker. For example, you find all churning users by writing a simple SQL query. And then send that result by something like Looker Action Hub.
Happy to discuss more here or offline (email: soumyadeb@rudderstack.com)
We have 3 separate pods - one for the RUdder core BE, one for the transformation (javascript which map events from Rudder format to destination format as well as user defined transformations) and one running Postgres.
Please give it a shot. Email me soumyadeb@rudderlabs.com if you run into any issues.
The README says AGPLv3. Or am I missing something?
We had long discussions on and off HN about this topic. The biggest problem with SSPL is it is not an open-source license and we got a bunch of pushback (rightly so) for calling ourselves open-source.
We initially went with SSPL to help prevent cloud providers from offering this as a service. However, there was a viewpoint that AGPLv3 already provides this and there is no reason for SSPL. Also, neither has been held up in court.
In the end, we prioritized something which our community felt strongly about. The business risk was questionable anyway.
Quick question: I tried setting it up but its asking for Write access to the pull requests. I am a bit wary about giving write access - is this required?