Start by challenging the service
Databricks, Synapse and Fabric are powerful tools, and expensive ones. Before tuning their settings, we check with the team that the service fits the need: volumes, frequency, expected latency, team skills. For simple processing on a few gigabytes, a SQL database, an Azure Function or a Data Factory activity may be enough. It is the build vs buy logic, applied to data.
For a new project on Databricks, we recommend starting with job clusters, and opening all-purpose clusters only for specific needs.
Databricks
- Job clusters rather than all-purpose for scheduled workloads: the DBU rate is significantly lower, and the cluster stops with the job.
- Short auto-termination on interactive clusters: 10 to 30 minutes of inactivity rather than the default.
- Autoscaling with a sensible maximum number of workers, and an alert when it is reached.
- Cluster policies to enforce these rules for everyone, rather than relying on each developer.
- Spot instances for workers in non-prod and on workloads that tolerate interruption.
- Well-maintained Delta Lake (OPTIMIZE, data clustering, incremental processing): less data read, so less compute.
Azure Data Factory
Its pricing model is confusing: activity runs, DIU-hours for copies, vCore-hours for data flows, integration runtime. The most effective levers:
- Review frequency and volumes with the product owner and the business: a refresh every five minutes is rarely needed.
- Delete inactive pipelines, which are still billed.
- Check the cost of a managed VNet integration runtime before migrating: it can double a factory's bill.
- Set a cost alert on every factory: a stuck pipeline costs money every day (see our real cases).
Azure Synapse
- Short auto-pause on Spark pools, and the smallest node size that handles the load.
- Lower maximum node count for autoscaling: an overly generous upper limit is quickly reached.
- Shared Spark pools across teams and projects.
- Delta mode loading rather than full reloads.
- Stop schedules in DEV and UAT, delete unused node pools and SQL databases.
- Pause dedicated SQL pools when unused, and reserve them when they run continuously.
Microsoft Fabric and Power BI
- Capacity at the right SKU, measured with the capacity metrics app, and autoscale rather than a SKU sized for the peak.
- Pause non-prod capacities outside working hours.
- Reserve the production capacity, which runs continuously.
- Clean up unused Power BI datasets, and set refresh frequency to the real need.
Measure, then track
We track data spend separately in the dashboard, with the prod and non-prod split, and we measure the effect of each action on the bill. On one data platform, combining cleanup, Synapse and Fabric reservations, autoscaling and fewer refreshes cut monthly spend by about a quarter in three months. Every size or frequency change is made while monitoring performance.