HNHacker News
TopNewBestAskShowJobs

soumyadeb

627 karma · joined June 30, 2011

Founder of RudderStack, an opensource alternative to segment https://github.com/rudderlabs/rudder-server/
submissionscomments
soumyadeb··on Launch HN: Data Mechanics (YC S19) – The Simplest Way to Run Apache Spark
Thanks a lot. Will follow up with you.
soumyadeb··on Launch HN: Data Mechanics (YC S19) – The Simplest Way to Run Apache Spark
>Many of our customers use us for their ETL pipelines, they appreciate the ease of use of the platform and the performance boost from automated tuning.

This is quite interesting. Founder of RudderStack here (we are a CDI or simply an open-source Segment equivalent). I have seen a similar pain point across some of our customers. They use RudderStack to get data into S3 (or equivalent) and then run some kind of post-processing Spark jobs for analytics/machine-learning use cases. Managing two setups (RudderStack on Kubernetes + Spark) is a pain.

A singly managed solution with Spark on Kubernetes makes so much sense. Would love to figure out how to integrate with you guys.

soumyadeb··on The third wave of open source migration
Some open-source projects have companies behind them - e.g. I am the founder of RudderStack, an open-source segment.

We have thought about this quite a bit. The way to address this is to make offering your OSS as-a-service from day-1. Initially, you are small and AWS won't care. Once you become big, you should be able to compete with AWS on the service offering - afterall you know your code best.

The problem arises when the OSS vendors had a different business model (open-source, on-prem support only) and AWS is able to completely own the as-a-service market.

soumyadeb··on The third wave of open source migration
Great point!! A perfect example is AirBnB - look at the number of open-source projects that came from there.

https://airbnb.io/projects/

I myself have used two of these projects heavily (Airflow, SuperSet) and am thankful to the teams for building such great products.

But did this make any business sense for AirBnB to spend on these? In this post COVID world, very unlikely this would continue.

soumyadeb··on Amazon AppFlow
Founder of RudderStack here (we are an open-source Segment alternative)

Segment (or RudderStack) is primarily for routing event-data from your apps to multiple destinations (cloud SaaS, warehouse etc). On the other hand, AppFlow is for syncing data that lives on cloud applications with AWS products like RDS, RedShift etc. So, both the nature of data is very different (event-streams vs database update batches) and also the destinations where they are sent (multiple cloud applications + data-warehouse VS only datawarehouses/databases).

Segment also has a product similar to AppFlow called Segment Sources but that's not what Segment is known for. There are a bunch of other products in that space like Blendo & FiveTran.

soumyadeb··on Show HN: Panther v1.0 – Open Source, Cloud-Native SIEM
Great to see an open-source alternative to Splunk SIEM!! Thanks for making this and all the best.
soumyadeb··on Launch HN: Taiv (YC W20) – Replace TV commercials with content people care about
Love the idea!! I see a use case even as an individual consumer - when I am watching something live (News, SuperBowl), I would rather watch something interesting than watch the stupid ads.

Wondering if there are any legal issues around this (can cable networks go after this saying it's their content)? What is being played on my TV is entirely up to me but then this argument breaks down in some cases (e.g. car odometer - I can't replace the reading). Might be worth nailing down.

soumyadeb··on Ask HN: What skills will be useful during an upcoming recession/depression?
Or get a loan from an Indian bank. The interest rate doesn't matter since you don't have to pay the loan back :)

There were some high profile cases of people taking millions of dollars of loans and running away (search for Neerav Modi, Mehul Choksi) so I am not making this up.

soumyadeb··on Ask HN: What projects are you working on now?
Working on RudderStack (https://github.com/rudderlabs/rudder-server), an open-source segment alternative.
soumyadeb··on Show HN: RudderStack, open-source CDI (a.k.a. open-source Segment)
One of our users has setup logstash to load data from S3 to ElasticSearch. Would that setup work for you?
soumyadeb··on Show HN: RudderStack, open-source CDI (a.k.a. open-source Segment)
No, the cloud-hosted version is under a TOS which has an enterprise license.
soumyadeb··on Show HN: RudderStack, open-source CDI (a.k.a. open-source Segment)
Thanks team Mattermost. Super excited to be working with you guys!!
soumyadeb··on Show HN: RudderStack, open-source CDI (a.k.a. open-source Segment)
Yes, that is the goal of the user transformation. You can add an element_id field while in the event before it is forwarded to destinations.

Would love to understand your use case a bit more.

soumyadeb··on Show HN: RudderStack, open-source CDI (a.k.a. open-source Segment)
If you are open to it, would love to work with you (e.g understand your requirements) as we develop the personas product. Can you please email soumyadeb@rudderstack.com
soumyadeb··on Show HN: RudderStack, open-source CDI (a.k.a. open-source Segment)
Sorry, Customer Data Infrastructure and segment.com. A platform which makes it simple to collect event data from your apps and send it to 3rd party destinations (including dumping into your warehouse).

We wanted to give out an update from our last HN post so we didn't get into the depth of what the product is. Kind of assumed everyone knows segment but yes a fair point.

soumyadeb··on Show HN: RudderStack, open-source CDI (a.k.a. open-source Segment)
That's right. You would have to specify at a per-event level (by specifying a flat to Rudder JSON) where to forward that event.

We have something called Transformations (user defined functions) by which you can modify the event structure from the Rudder BE. You write the transformation function (currently javascript) on our UI and that gets executed in the backend on your event stream. Using the transformation, you can also control where the event goes to. This is helpful when say you want to change the destination without pushing an update to mobile app

soumyadeb··on Show HN: RudderStack, open-source CDI (a.k.a. open-source Segment)
Not at all, this is a great question and we get it all the time :)

Peter (segment's CEO) had a great answer to this in the thread https://www.quora.com/What-is-the-advantage-of-using-Segment...

TO summarize and highlight the main reasons

1) With GTM you still have to write code. Specifically, you need to figure out how to send events to a destination following their API and JS library. With Segment and Rudder, you just call a couple of generic functions identify, track, page etc.

2) You often want a copy of the events in your own S3 Or Redshift or Snowflake. GTM doesn't help you there but Rudder/Segment can.

3) We have some features like Replay etc which lets you send historic events to a new destination. Say you signed up for a new Analytics tool and want to send all your historic events - Replay helps there

4) Finally, Segment has this product called personas which lets you create audiences (e.g give me all people who have done X but not Y and then send that to MailChimp for emailing). We too are working on it.

soumyadeb··on Show HN: RudderStack, open-source CDI (a.k.a. open-source Segment)
Fair point.

We don't post pricing because we're still figuring it out (always a tricky subject for OS project). We charge a platform fee and a small per node charge (beyond 1 node) but there are many dimensions there - node size, number of nodes, etc and we haven't figured out all the combinations. The platform fee generally starts at $2k/month but we offer discounts for start ups/open source projects/non profits, etc.

Right now we more focused on growing our open-source adoptions & customers and we want to talk to our prospects to better understand this.

As a benchmark 1 node can handle ~1K events/sec.

soumyadeb··on Show HN: RudderStack, open-source CDI (a.k.a. open-source Segment)
That's right. What we have today is more of a bus but for customer event data.

Let's say you have a website or mobile app and you want to collect all the user interaction events (clicks, searches, impressions) in a data-warehouse like Snowflake.

RudderStack can do it for you. We have SDKs (Android, iOS, JS, Python etc) which you can use to send events. We have a corresponding backend (you can run it or we can host it) which will collect these events and dump into your warehouse.

You often want to send (subset of these) events to other 3rd party websites too like Amplitude for analytics, Braze for marketing automation and so on. RudderStack can forward events to those too so you don't have to embed multiple 3rd party SDKs and understand their library etc.

The second part of this is to create a unified view of the customer as you mentioned and take action on that (e.g. find all churning users and send them email). That's where Segment has a product called Personas and we are working on something similar but it's not launched yet.

Though even before, you can build that customer view on top of your event data in your data warehouse by using SQL or a tool like Looker. For example, you find all churning users by writing a simple SQL query. And then send that result by something like Looker Action Hub.

Happy to discuss more here or offline (email: soumyadeb@rudderstack.com)

soumyadeb··on Show HN: RudderStack, open-source CDI (a.k.a. open-source Segment)
We use helm charts to deploy Rudder. Repo here https://github.com/rudderlabs/rudderstack-helm

We have 3 separate pods - one for the RUdder core BE, one for the transformation (javascript which map events from Rudder format to destination format as well as user defined transformations) and one running Postgres.

soumyadeb··on Show HN: RudderStack, open-source CDI (a.k.a. open-source Segment)
Yes, this is a very standard use case. A single node RudderStack (m4.xlarge) should be able to process more than 1K/sec. And Redshift should receive the data in 30 mins (configurable parameter).

Please give it a shot. Email me soumyadeb@rudderlabs.com if you run into any issues.

soumyadeb··on Show HN: RudderStack, open-source CDI (a.k.a. open-source Segment)
Thanks for the pull request :) Merged...
soumyadeb··on Show HN: RudderStack, open-source CDI (a.k.a. open-source Segment)
Thanks for pointing it out. The LICENSE file was old, removed it now.

The README says AGPLv3. Or am I missing something?

soumyadeb··on Show HN: RudderStack, open-source CDI (a.k.a. open-source Segment)
Yes, very recently did we switch from SSPL to AGPLv3 (as noted in updates above)
soumyadeb··on Show HN: RudderStack, open-source CDI (a.k.a. open-source Segment)
Oops, that was old.

We had long discussions on and off HN about this topic. The biggest problem with SSPL is it is not an open-source license and we got a bunch of pushback (rightly so) for calling ourselves open-source.

We initially went with SSPL to help prevent cloud providers from offering this as a service. However, there was a viewpoint that AGPLv3 already provides this and there is no reason for SSPL. Also, neither has been held up in court.

In the end, we prioritized something which our community felt strongly about. The business risk was questionable anyway.

soumyadeb··on Show HN: RudderStack, open-source CDI (a.k.a. open-source Segment)
Thanks team @Grofers for the kind words and for helping us build this out as our early patrons.
soumyadeb··on Show HN: RudderStack, open-source CDI (a.k.a. open-source Segment)
Founder here - happy to answer any questions
soumyadeb··on Show HN: RudderStack, open-source CDI (open-source Segment alternative)
Founder here. Happy to answer any questions
soumyadeb··on Launch HN: DeepSource (YC W20) – Find and fix issues during code reviews
This looks awesome - congrats on the launch.

Quick question: I tried setting it up but its asking for Write access to the pull requests. I am a bit wary about giving write access - is this required?

soumyadeb··on Show HN: Opensource.builders, open-source alternatives to commercial software
Haha, this is funny.
← PreviousPage 5 of 8Next →