I agree with the sentiment. Medallion in my mind has always been about the “quality” of each particular dataset. The terminology created a lot of frustration within Nike as well. This is due in part to Data Mesh coming along around the same time as Medallion was popularized. Now you have “golden” data products that are fit for purpose (which can be argued as gold-layer “data mart” with data contracts).
My big pet peeve is that most of the design patterns skip “accountability” and when people read about the various layers (be it with traditional ETL -> raw/cleansed/normalized->X) then they can go so far in the wrong direction as hiring teams to manage “layers” - teamB does “silver”….
It adds a lot of communication overhead to daisy-chain the needed business context from one developer to the next to the next to the next etc.
It's much easier for the same data engineer to own the ingest and initial transformation of a dataset brcause they often depend on each other. How you ingest affects how you transform. Making two teams align with each other to change something comes at a corporate crawl pace compared to the same developer just doing it.
Pragmatic dev covered the sentiment. The less number of people between data ingestion and the end result (report, ai model, etc) the better since you can retain context. Context is lost without lots of communication.
Apot on now if we can make Data Vault the default model approach for Silver/confirmed, I will be happy.
I agree with the sentiment. Medallion in my mind has always been about the “quality” of each particular dataset. The terminology created a lot of frustration within Nike as well. This is due in part to Data Mesh coming along around the same time as Medallion was popularized. Now you have “golden” data products that are fit for purpose (which can be argued as gold-layer “data mart” with data contracts).
My big pet peeve is that most of the design patterns skip “accountability” and when people read about the various layers (be it with traditional ETL -> raw/cleansed/normalized->X) then they can go so far in the wrong direction as hiring teams to manage “layers” - teamB does “silver”….
Cheers
Could you please elaborate on why having teams managing layers is the wrong direction?
It adds a lot of communication overhead to daisy-chain the needed business context from one developer to the next to the next to the next etc.
It's much easier for the same data engineer to own the ingest and initial transformation of a dataset brcause they often depend on each other. How you ingest affects how you transform. Making two teams align with each other to change something comes at a corporate crawl pace compared to the same developer just doing it.
Pragmatic dev covered the sentiment. The less number of people between data ingestion and the end result (report, ai model, etc) the better since you can retain context. Context is lost without lots of communication.