The final screenshot that indicates reading and looking at the table in code. Is that also via Daft? What have you found the best way to manage querying and rolling back to versions of delta tables?
Yes via Daft. I manage and use Delta Lake tables that are in the 300TB+ range, multiple; I've never had to roll back versions, or go hardcore up front on the data quality, and reliable pipelines to avoid having to do such things. Otherwise use MERGE statements to to create idempotent pipelines.
I have limited experience with Delta Lake tables, certainly not working with that scale of 300TB , but compared to raw storage, isn't one of the advertised benefits the ability to have history? What would be the use of history if not to query or restore to an earlier version ?
Good stuff…nice and clean and “real time baby!”
The final screenshot that indicates reading and looking at the table in code. Is that also via Daft? What have you found the best way to manage querying and rolling back to versions of delta tables?
Yes via Daft. I manage and use Delta Lake tables that are in the 300TB+ range, multiple; I've never had to roll back versions, or go hardcore up front on the data quality, and reliable pipelines to avoid having to do such things. Otherwise use MERGE statements to to create idempotent pipelines.
I have limited experience with Delta Lake tables, certainly not working with that scale of 300TB , but compared to raw storage, isn't one of the advertised benefits the ability to have history? What would be the use of history if not to query or restore to an earlier version ?
Awesome article! I could better understand how to work with a image through your article!