aruntdharan

Thanks! Yes, for sure - pivoting at the right time is important.

Hi HN! We are excited to share something we have been building for the last year.

Castled is a marketing automation platform built on data warehouses like Snowflake, BigQuery, Redshift, and Postgres. Castled allows you to directly use the user data in Snowflake and target them across channels like Email, Sms, WhatsApp, push, and In-app - without having to copy the data to another tool.

We would love you to try it out and give us your feedback. We have created a demo database and put some dummy data there so you can try us out without any hassles. If you have a google account, directly log in using https://app.castled.io/auth/login, or else use the signup link to log in.

Hi Thom - Yes Clickhouse is definitely there on our roadmap. We already have multiple customers asking for it.

Hi HN, We're Arun, Frank and Abhilash from Castled Data(https://castled.io). Castled is a marketing automation platform built directly on top of modern data warehouses like Snowflake, BigQuery, Redshift, and Postgres. Here is a quick demo:

https://www.loom.com/share/671cc9fb11c648cfb00ea5d4fe1d8ec4

Hi HN, We're Frank, Abhilash, and Arun from Castled Data(https://castled.io). Castled is a marketing automation platform built directly on top of modern data warehouses like Snowflake, BigQuery, Redshift, and Postgres. Here is a quick demo:

https://www.loom.com/share/671cc9fb11c648cfb00ea5d4fe1d8ec4

We started our journey by building an open-source Reverse ETL solution to make the warehouse data actionable to marketers. However, after talking to 100s of marketers, we realised that modern B2C marketers needed to use billions of customer data points from the data warehouse to run personalised marketing campaigns. While Reverse ETL could sync all this data to marketing tools like Braze, Iterable, etc., the traditional marketing solutions were fundamentally not designed to store and process so much data.

The restrictions surfaced in the form of illogical data volume-based pricing and restrictive data retention - usually just a few months. Moreover, copying the data to multiple external tools created more complications w.r.t security audits(e.g., SOC2, GDPR, etc.) and other data privacy protocols. The data engineers also faced numerous issues maintaining the Reverse ETL pipelines, which often failed due to API timeouts, rate limits, etc.

That's when we changed course to build a warehouse-native marketing solution. Since we do not store any customer data with us, marketers could finally use all the data from the data warehouse to engage their customers without any tradeoffs. The headless approach also enabled us to offer marketers a more rational pricing model.

Marketing automation solutions have always been built on top of databases, which allowed transactional queries and point lookups. So solving the same use cases on top of a data warehouse that only supports analytical workloads would typically be a significant challenge. But luckily for us, 95% of the B2C marketing use cases required bulk access patterns, which data warehouses were uniquely designed to solve and even enabled us to offer capabilities that were not possible with traditional solutions.

One major challenge was supporting real-time campaigns, where we had to react to product events in sub-second latency using data from Snowflake. This required point lookups in Snowflake on each event, which was not scalable. We had to solve it using "contextual" data caches on our side. But we expect the advent of technologies like Unistore to make it easier to support use cases that require low-latency lookups on top of the data warehouse.

We have been building Castled for a year. Until now, we have enabled many businesses to solve their marketing data problems by replacing the combination of Reverse ETL + B2C marketing solutions with Castled. So, if you have a data problem in marketing, feel free to sign up and try our product for free - no credit card required. We have our solution hosted at https://castled.io.

Sorry about that! But its actually the registration for signup and not for updates. Thats why we had to block personal emails.

Thanks for letting us know!

Thats a great question! We dont use updated timestamps to compute deltas, as thats unreliable and can cause data loss depending on your transaction window.

We keep snapshot tables on your data warehouse(in our own custom schema, so that you dont have to provide Castled write access to any of your production schemas). The snapshot tables are then used as the baseline to compare the query results everytime the pipeline runs. Frankly, we have not really seen a use case of transferring 5 billion rows in a Reverse ETL pipeline. This is mostly because of the fact that our destination apps are mostly transactional systems and cannot really store so much of data. For example, salesforce destination can store max 10GB of data. Because of this, we are storing the actual tuple values in the snapshot. We have easily scaled our pipelines to compute deltas from queries which returns up to 100 million records. To optimize this further, we are also considering to keep the hashes of the tuple values instead of the actual values.

Yes, we need to know the primary key of your query results. This is required to handle failures and to remove the failed records from the snapshot table, so that those can be retried on the next pipeline run.

Thanks for the input!

We can definitely consider that. But I feel its a lot of config and can be error prone. For instance, source-destination field mapping configs might be complex and have various issues like data type mismatches, typos in field names etc and a user interface is better suited to guide you through the entire process.

But I see value in exporting the config to a github repo after the pipeline is created and thereafter future edits can be done via the github repo. Does that make sense?

You mean the warehouse/app credentials, when you say sync configurations? If so, yes, that seems like a great idea. Infact I think your warehouse credentials are already there in your dbt repo in a specific format. Castled can directly read those credentials from there.

Fixed :)

Thanks!

A year back when we started to built Castled, this technology which syncs data from cloud warehouses to your operational tools did not have a name. The term "Reverse ETL" became popular somewhere in the beginning of 2021. We used this term, since we know that the data community knows this technology by this name now.

But my personal take is that "Reverse ETL" is still a new technology in the sense that it completes the modern data stack, which is built around cloud data warehouses.

Just checked it out. Looks interesting. Thanks

The founding team of Castled comprises mostly founding engineers from HevoData, which is an ETL solution. Having built both ETL and Reverse ETL solutions from scratch, we have realized that the architecture required to support both ETL and Reverse ETL needs to be drastically different. Your cloud data warehouse is powerful enough to change the entire architecture of the product depending on which side of the pipeline the warehouse is on. So we believe you need different products to support both ETL and Reverse ETL. But agreed the same tool can provide both products.

I will have to check how airbyte supports both. Regarding Castled, Regular ETL is there in our mid term roadmap.

Noted. We will definitely consider adding support for python. Thanks for the suggestion.

Thanks for the tip. We believe it's still the data engineers who will have to write these reverse connectors as well. We understand that python is probably the preferred choice of language for them. We have also seen that a majority of them understand java as well. But we are willing to support multi-language functionality in future, if the community demands it. Do you think majority of the data engineers do not understand Java?

Thanks for the suggestion. Yes, a devkit would work if its just about building new connectors. But we believe that the community would want control over the entire project rather than just the connectors module. We also wanted to provide a usable version of the product for free to the community, which you can self-host and maintain yourselves as well.

Thanks! We also have a subscription based hosted solution hosted at https://castled.io