This is an enjoyable article to read and gives fresh ideas on how to build a data stack that is flexible and emphasizing on great dev experience. Thanks for sharing!
Really well done walkthrough of the ephemeral compute pattern. The SQLMesh virtual environments pointing dev models to prod to save costs is clver, and honestly something Id seen described before but never actually implemeted. Seeing the full stack run on GitHub Actions without needing Airflow or Dagster is kinda refreshing too, keeps the operational overhead way lower for smaller teams.
Great deep dive into building lightweight analytics infrastructure. The emphasis on SQLMesh's virtual data environments avoiding unnecesary compute is exactly what teams miss when they over-engineer from day one. I've been curious about DuckLake for a while and this walkthru makes the R2 integration way less intimidating than I expected.
🙌
This is an enjoyable article to read and gives fresh ideas on how to build a data stack that is flexible and emphasizing on great dev experience. Thanks for sharing!
Glad you enjoyed it! It’s so fun you can build data stacks just stitching awesome, local first tools. We live in an interesting era
Excellent article, thanks!
As an alternative for cloud storage, there is Apache Ozone: https://community.cloudera.com/t5/Developer-Blogs/Building-an-Open-Lakehouse-with-Apache-Iceberg-and-Apache/ba-p/413422
Good to know! I’ll have to check it out
Neat. This would be even cooler if the object store was local also!
From the config standpoint, that’s even easier to set up honestly. Both dlt and sqlmesh can read from and write to ducklake in local
Very nice article, thanks!
Really well done walkthrough of the ephemeral compute pattern. The SQLMesh virtual environments pointing dev models to prod to save costs is clver, and honestly something Id seen described before but never actually implemeted. Seeing the full stack run on GitHub Actions without needing Airflow or Dagster is kinda refreshing too, keeps the operational overhead way lower for smaller teams.
Great deep dive into building lightweight analytics infrastructure. The emphasis on SQLMesh's virtual data environments avoiding unnecesary compute is exactly what teams miss when they over-engineer from day one. I've been curious about DuckLake for a while and this walkthru makes the R2 integration way less intimidating than I expected.