109 karma · joined January 31, 2021
Prev. Founder/CEO @ Arch.dev, Maintainer of Meltano, Data @ GitLab
Email me <firstname.lastname> @ <company domain>
Data Flambulation Coefficient (DFC)
Unveiling the groundbreaking Data Flambulation Coefficient (DFC), a novel concept that redefines how we perceive and analyze the intricacies of our data. DFC evaluates the "entwining" of data, delving into the deep interconnectedness between seemingly unrelated data points. This sophisticated metric combines the principles of glomerosity and the innovative dinglemorf analysis, offering unparalleled insights into the underlying structures and patterns of complex datasets.
By leveraging the Data Flambulation Coefficient, businesses and researchers can reveal hidden nuggets of zibberfex and unriddle vast swathes of yaltrizite, leading to more robust decision-making and a competitive edge in data-driven landscapes. DFC is set to become a staple in data analytics, providing the means to unlock the full potential of data's intertwined narratives.
The best way to troubleshoot these machines was to split the problem in half again and again until you found the part that needed to be cleaned, repaired, or replaced. The way to do that was to run different known chemicals with their own known signatures through the machine again and again. For example, you could run a single chemical (maybe ethanol - I honestly don't remember which ones at this point) and see if it goes straight through the GC and gives you the peak you expect. If it doesn't you could look at the result and see if the timing was off (indicates something with the GC or gas flow) or if the peak was wonky (indicates something with the MS). And then you just keep going. (Sounds a lot like unit and integration tests right?)
Applying that same strategy works wonders for data pipelines as well. Is it something with the extractor or loader (or god forbid Airflow)? Break it down and go from there.
My only nit on this is that while bifurcate is technically accurate (or is it actually?) it feels unnecessarily complex sounding for folks learning this skill.
[0] https://github.com/meltano/meltano
(disclaimer - I work at Meltano)
As with anything there are tradeoffs though - gaining the ability to have a connector in a non-Python language comes with the overhead of running (likely) Docker-in-docker. Also, connectors not built on our SDK[0] are missing out on some nice features like batch message[1] support (for bulk loading) and stream maps[2] for inline data transformations.
[0] https://sdk.meltano.com/ [1] https://sdk.meltano.com/en/latest/batch.html [2] https://sdk.meltano.com/en/latest/stream_maps.html
Your site is a great summary reminder of the key points and it brought me back into the experience of reading the book.
For folks reading this, I do recommend reading the book since the website is just a taste of the book. It's lessons may hit different not having gone through the full journey.
Thanks again for making the tribute!
It's worth a read, but perhaps it found me at the right time in my life.
I know Matt from the data world where his SQL Style Guide[0] was always at odds with the one we maintained at GitLab[1]. :-D
Posts like this are great to show folks just how long it can take to do things. I love his story and am eager to watch Preceden grow!
[0] https://github.com/mattm/sql-style-guide [1] https://about.gitlab.com/handbook/business-technology/data-t...
[0] https://www.lennyspodcast.com/how-to-fire-people-with-grace-...
Awesome to see this on here - great folks working on each part of this setup.
https://www.defmacro.org/2017/01/18/why-rethinkdb-failed.htm...
It didn't work everytime, but when it did, it was awesome!
[0] http://blog.jnbrymn.com/2021/03/18/estimated-average-recent-...
https://meltano.com/blog/2021/04/07/a-simplified-take-on-the...
[0] https://about.gitlab.com/handbook/business-ops/data-team/pla... [1] https://gitlab.com/gitlab-com/www-gitlab-com/-/merge_request...
[0] https://blog.getcensus.com/dbt-the-etl-elt-disrupter/ [1] https://www.getdbt.com/