Alanhlwang

Definitely, I can jump into what we meant by brittle—we mainly meant that SQL scripts are hard to debug/undescriptive, you can't parametrize and customize error messages that you receive from transforms, and you can only execute one complete statement at a time that are often chained together with CTEs (which is a nightmare if its a statement of 400 lines of SQL). Python makes it easier to debug since we turn the approach from a declarative to a procedural one, and that's even the case with breakpoints when you write your actual transformers in Python.

This is a great point—in terms of dynamic tables/materialized views, the software engineering best practices ie modularization, testing, version control are not as intuitive/straight-forward to apply in comparison to dbt and Serra. We can also add a direct SnowflakeDynamicTableWriter into our framework to work with the best of both worlds!

We see this working more alongside Airflow—we see Airflow mainly centered as an orchestrator/scheduler to chain together your ETL steps after you've written your data transformations and connections. With Serra, we're a flexible dev tool to write these transforms and connectors. I think you could accomplish something similar but the implementation would be unwieldy (ie breaking up 4 SQL script tasks into 40 modular transform blocks), whereas we see ourselves falling into the camp of being able to integrate with Airflow and have great error logs in those 4 SQL tasks that you have simplified into Serra configs.

Thanks for these links! We consider ourselves an ELT and ETL tool—if you run a Serra job in your own warehouse (ie Databricks), you can easily specify extracting from AWS, loading the parquets into your warehouse, then transforming them with our config block approach (ELT).

The same is true for ETL. If you have a spark cluster separate from your warehouse, you can define your config file to run in the order E T L: you can extract from your data source, run the transformations on a separate cluster, then load it to your warehouse.

These are great links, we'll take a look

This is super insightful thanks a ton for this gold mine.

On the python vs yaml part—definitely could've made that way more clear in the demo. Right now are framework lets you call these python objects in your yaml file, but we are working on just a python-centric implementation as well for those that do not want to interact with yamls.

On the loom and licensing choice—that's a great point. One of the main issues we ran into is getting adoption as we originally just tried licensing out the framework (mega fail ofc)—found out the hard way that no dev wants to buy something to try it out. We're definitely flexible on our license and will take all this feedback into account.

On the barrier of entry—also super insightful. We're working on a local UI offering that will be a 'config' block builder that will be free for all installs. We're implementing a DAG view similar to Airflow on the transform level. We also want to make it super easy to see your code and preview how it changes with this local UI (and have a list of all the params you need for your spark objects without having to go through the docs). We also want to flesh out more features especially on the translate side, as well as host on the cloud.

With the complexity issue that's something I ran into Disney as well! As the product grows we definitely want to flesh out our transformers based on the scripts we see. For now, the developer can make one-off transformers—we actually have a catch all "SQL transformer" for cases where you want to just pass in your sql (similar to a dbt model) and run it that way. That way it's a fail safe for if you have one specific transform that you feel is super hard to break down, you can fall back on dbt's way of just modularizing the SQL into a transform, and reference it however many times you want as an input block later on.

Thanks so much for the congrats, will definitely reach out and would love to have further discussions in the thread as will.

Thank you we appreciate the support!

Yep we only showed our configuration file which instantiates the said Python objects. Definitely would've made the demo clearer, and if you want to look at the actual objects that our config is referencing, definitely check out our repo!

These are all great points, and no, Disney didn't use dbt. We wanted the demo to show how you can modularize SQL into reusable objects that you can fully customize error logs for, while also adding the value of handling all steps of ETL in-house. You could definitely write this script modularly using dbt, but we feel like that value add of having connectors that easily integrate with your transforms (e2e), as well as taking the software engineering best practices that dbt applies a step further by turning each transform and connect into objects that you can test, modularize, and customize error logs for is our main differentiator.

Definitely a great point, adding one today

This is awesome, the ability to assess image composition according to photography 'rules' especially is super intriguing. Could you provide some examples of the specific rules your algorithm considers and how they contribute to the overall image quality evaluation? Do you guys see this feature being able to replace studio photographers?