... better be good ...
Although Databricks community driven, is it possible to import this library natively in Microsoft Fabric notebooks or other notebooks?
If you can pip install a Python package, you can use it
No you can’t. Read the License file in the repo. You can use it with Databricks only
Superbly written, as always. Can 100% relate on the first part 🤣
Great article 👏
Have you tried cuallee?
Near zero dependency and data frame agnostic.
Looking at it now, looks very basic, lack of checks, no filters, no reporting, no quarantine table..
I only see single column checks. Is it possible to check if combination of col 1 col2 is unique?
Thanks for the article, whats different from Soda or GE?
One thing i have noticed is that you can prohibit bad data being ingested in to your storage and store them in a separate loacation.
Correct me if I'm wrong, Soda and GE quality checks data once the are ingested in your storage.
I see, you mean the quarantine tables.
Yeah I use GE but we have written custom on top of it to handle that case. Seems this tools already provides that.
Yes correct. You will get one df that passes the dq checks and then another df for quarantine.
Thanks
Although Databricks community driven, is it possible to import this library natively in Microsoft Fabric notebooks or other notebooks?
If you can pip install a Python package, you can use it
No you can’t. Read the License file in the repo. You can use it with Databricks only
Superbly written, as always. Can 100% relate on the first part 🤣
Great article 👏
Have you tried cuallee?
Near zero dependency and data frame agnostic.
Looking at it now, looks very basic, lack of checks, no filters, no reporting, no quarantine table..
I only see single column checks. Is it possible to check if combination of col 1 col2 is unique?
Thanks for the article, whats different from Soda or GE?
One thing i have noticed is that you can prohibit bad data being ingested in to your storage and store them in a separate loacation.
Correct me if I'm wrong, Soda and GE quality checks data once the are ingested in your storage.
I see, you mean the quarantine tables.
Yeah I use GE but we have written custom on top of it to handle that case. Seems this tools already provides that.
Yes correct. You will get one df that passes the dq checks and then another df for quarantine.
Thanks