1 Comment

These are solid tips.

A few more:

Use single node clusters where spark is not being used or for smaller workloads.

Also, use Databricks jobs for multistage jobs and share the cluster across the tasks/notebooks. Job will finish faster and you won’t pay for the startup time.

In Azure, VM reservations can also reduce cost significantly for regular workloads.

Expand full comment