I don't think data teams will move in that direction because of the skills and mental models they bring. As you said most are not software engineers and embrace managed services which get better by the day. Python will remain the high level interface. Most DEngs I know want to get things done and not think too deeply about implementation details. Obviously it's still great to have the flexibility to choose a different model. Maybe when they get older and bored but by then they may be managers or accepted their fate.
Very interesting, But how about Observability / Monotoring?
In the case of you example adding the change in the existing Batch infra allow easy monitoring without changing much.
How do you think we can still have centralized observability & monitoring with Microservices in DE contexts.
I don't think data teams will move in that direction because of the skills and mental models they bring. As you said most are not software engineers and embrace managed services which get better by the day. Python will remain the high level interface. Most DEngs I know want to get things done and not think too deeply about implementation details. Obviously it's still great to have the flexibility to choose a different model. Maybe when they get older and bored but by then they may be managers or accepted their fate.
Daniel, what do you think about a comparison of implementing microservices with Lambda vs native Spark or Orchestrator.
This could reduce cost drastically..