I definitely had a feeling of "is this it"? I was comparing it to Rails's "batteries included" approach for webdev, and that is such a sprawling ecosystem. I was a little underwhelmed when I first saw dbt. Ok, the incremental builds also require a little more SQL (three more lines in my case) to handle "fresh build" vs "incremental build", but this is so smooth compared to something custom. Switch a transform from a VIEW to a TABLE to an incrementally built TABLE by changing one configuration value.Sure, this is equivalent to "just" keeping a list of sql files in order, but now I don't even need that list. Reference other transforms, or data sources like the csv above, and dbt figures out the order to run the transforms and/or load the csvs.csv file to the project and dbt loads it as a table.
So easy to be able to just add - unique or - not null to a yaml doc and have test run on upstream tables. Declaring existing tables as sources makes it easy to add declarative tests to them.I only use it for about a dozen transforms, but already find some of the more minor seeming features to be useful: I see dbt's strengths all in the testing, and the configuration that enables 100% declarative transforms. dbt doesn't touch loading except for static fixtures like. In my experience so far, dbt doesn't overlap at all with other CI/CD tools because it doesn't address at all when your transformations are run.ĭbt overlaps somewhat with Airflow, since Airflow does have some capability to do transforms as I understand it, but I think Airflow's strength is also sequencing and scheduling? And perhaps better at Loading than Transforming. I have significant experience in CI/CD for delivering and testing applications, but little experience with Airflow. I've only used the standalone CLI based part of dbt.
0 kommentar(er)
