Thanks for the article, Daniel! I find event-driven pipelines are to some degree the sweet spot. They are timely, simple (just a python lambda for example, no need for spark), easy to monitor and trace, often represent the actual data generation process, and even extended their realm with delta tables/iceberg/hudi. What's your take on them?
Thanks for the article, Daniel! I find event-driven pipelines are to some degree the sweet spot. They are timely, simple (just a python lambda for example, no need for spark), easy to monitor and trace, often represent the actual data generation process, and even extended their realm with delta tables/iceberg/hudi. What's your take on them?
I do enjoy putting Lambdas to use, although they need some work to increase memory and time limits to be effective for big data.