Great article! Funny how "external orchestration" always boils down to someone writing a notebook in python to handle auth, errors, triggering, polling, gathering metadata, alerts.... it is a wonder we don't see more python orientated services also claiming to be external orchestrators!
Lakeflow is still missing many essential features. It lacks basic orchestration capabilities such as global workflow timeouts, task prioritization, and the ability to control or assign job pools. In addition, monitoring is very limited—there is no reliable way to track real-time workflow execution status because Databricks does not provide real-time logging for Lakeflow workflows.
Great article! Funny how "external orchestration" always boils down to someone writing a notebook in python to handle auth, errors, triggering, polling, gathering metadata, alerts.... it is a wonder we don't see more python orientated services also claiming to be external orchestrators!
Truth
Lakeflow is still missing many essential features. It lacks basic orchestration capabilities such as global workflow timeouts, task prioritization, and the ability to control or assign job pools. In addition, monitoring is very limited—there is no reliable way to track real-time workflow execution status because Databricks does not provide real-time logging for Lakeflow workflows.