Most SQL-based teams should not consider themselves as engineers...
I recently encountered a team with over 350 DBT models, 0 unit tests (although DBT has a dedicated package for this purpose), 0 documentation, and 0 diagrams built to visualize relationships between models (the only reference available to understand their models was the models themselves)... Additionally, more than half of the DBT models contained over 350 lines of code, with nested queries upon nested queries and CTEs stacked on top of each other.
With Snowflake only needs files, theres no need for anything else (orchestrators python, etc..). People should start learning more product features (Snowflake), instead of creating 1000s lines of external code replicating what SQL does much much orders of degree faster.
Sad most junior developers think creating all this mess is still necessary in modern cloud times where SaaS like snowflake only need files created via CDC snd that's it. Python should only be used AFTER it lands inside a CDW like Snowflake. I have experienced production level mess of +150k lines of python doing aggregations, converting/merging files with files lol.. Please stop doing that!! load whatever files you have directly from source snd let SQL do what it is good at.
Most SQL-based teams should not consider themselves as engineers...
I recently encountered a team with over 350 DBT models, 0 unit tests (although DBT has a dedicated package for this purpose), 0 documentation, and 0 diagrams built to visualize relationships between models (the only reference available to understand their models was the models themselves)... Additionally, more than half of the DBT models contained over 350 lines of code, with nested queries upon nested queries and CTEs stacked on top of each other.
I rejected that project...
With Snowflake only needs files, theres no need for anything else (orchestrators python, etc..). People should start learning more product features (Snowflake), instead of creating 1000s lines of external code replicating what SQL does much much orders of degree faster.
Sad most junior developers think creating all this mess is still necessary in modern cloud times where SaaS like snowflake only need files created via CDC snd that's it. Python should only be used AFTER it lands inside a CDW like Snowflake. I have experienced production level mess of +150k lines of python doing aggregations, converting/merging files with files lol.. Please stop doing that!! load whatever files you have directly from source snd let SQL do what it is good at.