When I’m faced with the task of asking myself, or being asked, “What size cluster do I need for this?”, my answer never changes: it depends on what you’re going to do with the data, not on how much of it you have. “Just pick the biggest one you can” is almost never a good idea — it costs more, and a lot of the time it doesn’t even fix the real problem. It just papers over it for a while.

I’ve been thinking about this for a bit, so I finally sat down and wrote it out. I’m mixing what Databricks’ own docs say with things I’ve picked up from forums, tutorials, and YouTube videos over time. This isn’t meant to be a rigid recipe. It’s more the mental checklist I run through whenever someone hands me a cluster that’s “running slow” or “keeps crashing” and nobody’s quite sure why.

Concepts worth having straight before you touch any config

Every Databricks cluster runs on a single driver (the one that coordinates everything and executes the Spark plan) plus a bunch of workers/executors, which are the ones actually processing data in parallel. Before you go picking a size, it’s worth having these terms fresh in your head — some are config knobs, others are things that just happen to your job and that you’ll eventually end up staring at in the Spark UI:

Concept	What it is
DBU	The unit Databricks bills compute in, per hour. How fast you burn through DBUs depends on the node type and the serverless tier you’re on.
Cluster policy	A template that limits what someone can configure (node type, autoscaling, spot yes/no…). Exists so people don’t each invent their own cluster size by eyeballing it.
Serverless compute	You tell Databricks which policy to use and it picks the hardware. You don’t choose node type or worker count.
Photon	The C++ execution engine that speeds up SQL and ETL without you having to touch a line of code.
Autoscaling	The cluster scales workers up and down on its own, based on the load at any given moment.
Spot instances	Cheaper machines that can get interrupted at any time. Fine for dev, risky for anything with an SLA.
Cluster pools	A set of instances kept warm and ready, so spinning up or scaling a cluster doesn’t take the usual few minutes of provisioning. You feel this the most in ML, when you’re iterating on a model over and over.
Shuffle	Moving data between executors over the network — happens with a groupBy, a non-broadcast join, or a repartition. It’s the most expensive thing Spark does; almost any weird slowdown traces back to this.
Shuffle partition	Each chunk that results from that data movement. Its size is controlled by spark.sql.shuffle.partitions (200 by default), not by the number of input partitions — which is why it’s worth tuning separately as your cluster grows.
Data skew	One partition (input or shuffle) ends up with way more volume than the rest — say, a groupBy key that’s overrepresented, or a join with a dominant key. The executor stuck with that partition takes much longer than the others, and the whole job sits there waiting on it even though everything else finished ages ago.
Spill (to disk)	An executor runs out of memory for a task and Spark starts writing intermediate results to disk so it doesn’t crash. It works, but it’s a lot slower than keeping everything in memory — it’s the clearest sign you’re short on memory per executor, not on workers.
Broadcast join	Instead of moving both tables over the network, Spark ships a full copy of the small table to every executor. It avoids the shuffle, but if that “small” table turns out not to be so small, it can blow up memory on every executor, or on the driver.

All of this comes from Databricks’ compute configuration docs and its sizing considerations, in case you want to go straight to the source.

The sequence I follow to size anything

There’s no magic formula, but there is an order that works for me:

1. Estimate the input volume (not the output): number of records times average row size.
2. From that, figure out how many Spark partitions you want. Databricks’ rule of thumb is that each partition should weigh between 128 and 200 MB in Parquet/Delta.
3. Convert partitions to cores, aiming for somewhere between 2 and 4 tasks per core under normal conditions. Neither 1-to-1 (you waste parallelism) nor 20-to-1 (the scheduler becomes the bottleneck).
4. Estimate memory per executor, leaving room for deserialization, shuffle buffers, and any spill if it comes to that.
5. Start small, with autoscaling on, and check the Spark UI before declaring that size “good.” If you see disk spill, huge shuffles, or executors sitting at 30% CPU, throwing more machines at it won’t fix anything.
6. Use cluster policies so this doesn’t have to get reinvented every time someone opens a new notebook.

And if you want a quick back-of-the-envelope calculation, here’s the one I use:

data_size_gb     = num_records × bytes_per_record / 1e9
partitions       = data_size_gb × 1024 / 150       # ~150 MB per partition
cores_needed     = partitions / 3                   # 3 tasks per core (midpoint of the 2-4 range)
workers          = ceil(cores_needed / cores_per_node)
total_memory_gb  = data_size_gb × overhead_factor   # 4x for simple ETL with no shuffle, 8x+ for heavy joins, groupBy with many keys, or skew risk

Heads up: this is only meant to give you a first number to start testing with, nothing more. Not even Databricks promises a closed-form formula — the right ratio depends on the type of operation (a filter isn’t the same as a join, and a join isn’t the same as a window), not just on data volume. And I keep the memory range (4x-8x) wide on purpose: with no heavy shuffles or joins, 4x is usually plenty; with groupBy on many keys, big joins, or skew risk, better to lean toward the high end, since that’s exactly where spill tends to show up.

Why input size and output size aren’t the same thing

This is the part most people skip, and the one that causes the most headaches once you’re in production.

When Spark reads a dataset, it splits it into partitions and each worker keeps its own slice. If you’ve got 8 GB spread across 4 partitions of 2 GB, no node needs more than 2 GB of RAM to read its part — it’s distributed, everyone handles their own piece and that’s it.

The problem shows up when you do something like .collect(), .toPandas(), or a broadcast join. There, things flip completely: all those workers send their results back to a single node, the driver. The same 8 GB that were comfortably split into 2 GB per worker now have to be pulled together in one place, and if the driver doesn’t have the memory to hold the full 8 GB, it blows up.

That’s why the driver needs to be sized around the largest result you’re going to bring back — not the full input dataset — and around whether there are operations that don’t run inside Spark and therefore execute entirely on it. Workers, on the other hand, get sized based on the heaviest partition a single executor is going to have to hold in memory.

And this same problem can happen between executors too, with the driver having nothing to do with it: if a partition ends up unbalanced — data skew, a groupBy key that concentrates way more volume than the rest, a badly partitioned join — a single executor can end up with a lot more load than the others and fail on memory, even if the cluster as a whole has plenty to spare. Adding more nodes doesn’t fix that: the node that gets the heavy partition is still the same one, whether you’ve got 4 workers or 40.

To sum it up:

	What drives the sizing
Driver	The size of the largest result you’re going to bring back (.collect(), .toPandas(), broadcast) — plus any operation that doesn’t run in Spark
Workers	The size of the heaviest partition an executor has to hold, the concurrency you need, and how many workers depending on the type of workload (more on this below)

Something that stuck with me from those tutorials: not all of a node’s RAM is available to Spark. Databricks keeps a chunk for the operating system and its own internal processes. In the Executors tab of the Spark UI, the “Storage Memory” column tells you what’s actually left, and it’s always less than the total RAM the instance type advertises. A “16 GB” node might really only be giving you 4-5 GB of actual storage memory. If you calculate assuming you have all the RAM free, you’re going to come up short.

Now, by type of workload

Light ETL (incremental loads, a few GB, simple transformations)

No need to overthink this one: general purpose works fine, though if the bottleneck is purely read/write and transformation with little in-memory state, going straight to compute-optimized also works. This kind of workload tends to be CPU/IO-bound, not memory-bound.

A small cluster with autoscaling, especially if the input volume changes from one run to the next. Here it’s better to have several small workers than a few big ones, since the tasks are light and benefit from parallelism. A reasonable starting point: 1-4 workers, min=1, max=4.

Serverless is pretty much the ideal case here — you only pay while it’s running, and this usually runs for a short while. Spot instances, sure, unless the SLA is very demanding (a preemption can drag the run out right when it shouldn’t). If shuffle times start weighing heavily against the total, or the job grows faster than the input volume, that’s when you know you undersized it.

Heavy ETL (big joins, aggregations, shuffles, hundreds of GB and up)

Compute-optimized if the bottleneck is pure CPU; memory-optimized if there are big joins, groupBy with many keys, or windows, since those operations hold quite a bit of state in memory.

Autoscaling with a generous ceiling: size the cap for the worst case, not the average, and let autoscaling scale down on its own during normal loads. Now, when the bottleneck is the shuffle itself, sometimes the fix runs counter to what you’d expect: instead of adding more small workers, reducing the number of workers and using bigger nodes lowers the number of shuffle partitions and the network traffic between nodes. There’s no fixed rule here — you have to try both directions and compare in the Spark UI.

It’s worth tweaking spark.sql.shuffle.partitions (200 by default) on large clusters, aiming for each shuffle partition to weigh between 128 and 200 MB after the shuffle. And turning on Photon, which Databricks documents as giving 2x to 8x improvements on heavy joins and aggregations, with no code changes.

Spot instances need more care here: a long backfill that gets cut off by preemption can end up costing more in retries than what you saved using spot in the first place. The usual move is mixing on-demand (driver and part of the workers) with spot for the rest. And if you see “disk spill” in the Spark UI (Stages → Spill), that means insufficient memory per executor — not necessarily a shortage of workers.

Machine Learning

If training fits in a single machine’s memory (scikit-learn, pandas, XGBoost without distribution), what actually matters is having a big driver — memory-optimized — not adding workers. A .fit() running in a single process gets absolutely nothing out of having twenty tiny workers sitting around it. As a quick reference: if the dataset doesn’t exceed about 70% of the driver’s RAM, you’re fine going single-node.

When training is actually distributed (Spark MLlib, Horovod, distributed XGBoost/LightGBM), the pattern changes: few nodes but big ones, with more memory and CPU per node — not lots of small nodes like in light ETL. And GPU only if the algorithm actually takes advantage of it (deep learning, some boosting methods); for the classics — regression, trees, most tabular cases — it doesn’t help and just drives up cost.

If you’re iterating on a model several times a day, a cluster pool saves you the provisioning time on every run. You really notice it when the “spin up cluster, train, tear down cluster” cycle repeats often.

Feature engineering on raw data gets sized like any ETL job, based on volume, regardless of how big the training afterward is going to be — they’re different workloads and don’t need to share a cluster. For inference: batch gets treated as ETL; real-time goes through serverless Model Serving, not a cluster you size by hand. And always use the Databricks Runtime ML instead of the standard one — it already ships with MLflow and the GPU/distribution libraries sorted out.

Analytics, BI, ad-hoc queries

For this, SQL Warehouses (serverless or pro) are the right call, not a general-purpose cluster. They’re built exactly for handling lots of short queries at once, with Photon on by default.

What matters here isn’t “how many workers” but how many concurrent queries you need to support, and for that, multi-cluster autoscaling — several SQL clusters running in parallel — outperforms one bigger cluster. Makes sense if you think about it: the typical analytics pattern is high concurrency with low volume per query, the exact opposite of a heavy ETL job.

Databricks uses t-shirt sizes for warehouses (2XS through 4XL). Start at S/M and only go bigger if individual queries are slow, not if the problem is that there are just a lot of them at once — in that case, what you scale up is the number of clusters, not the size. If instead you’re using a classic Spark cluster for ad-hoc notebooks, the same logic applies: few workers with autoscaling, prioritizing concurrency over moving a lot of data.

Always turn on auto-stop (10-15 minutes of inactivity is reasonable) — analytics is the case where you pay the most for a cluster sitting on with nobody using it. And if the dashboard keeps hitting the same queries over and over, Delta Cache saves you compute before you need to bump up the warehouse size.

A quick reference table by volume

Don’t take this as a recipe, it’s just a starting point for your first test:

Input volume	Light ETL	Heavy ETL	ML (training)	Analytics (SQL Warehouse)
< 1 GB / < ~1M records	Serverless, 1 small worker or single-node	“Heavy” doesn’t really apply at this volume	Single-node, medium driver	2XS-XS, auto-stop 10 min
1-50 GB / ~1M-50M records	1-2 workers, autoscale 1-4	2-4 medium workers, autoscale up to 8	Single-node with a large driver, or distributed if iteration is heavy	S-M
50-500 GB / ~50M-500M records	2-4 workers, autoscale 2-8	Autoscale 4-16, memory-optimized, Photon	Distributed, 4-8 workers	M-L, evaluate multi-cluster if there’s concurrency
> 500 GB / > 500M records	Check whether “light” is still the right category	Wide autoscale (8-32+), Photon, on-demand/spot mix	Distributed with GPU if applicable, 8+ workers	L-XL + multi-cluster autoscaling

Again: this table is a starting point for your first test, not a size you copy and call it done. A 100 GB dataset with no skew and another 100 GB dataset with a key that concentrates 40% of the volume land in the same row — and they’re going to need pretty different clusters. The row tells you where to start; the Spark UI tells you whether you came up short, went too big, or whether size was never the problem in the first place.

Loose things worth keeping in mind

Autoscaling on almost always, except for low-latency structured streaming, where rescaling introduces pauses you don’t want — there, a fixed size aimed at the 90th percentile of load works better.

Don’t oversize “just in case”: a huge cluster with a badly distributed partition won’t run any faster, because the bottleneck is still that same partition. Before bumping the size up, check for skew by comparing max time against the median in the Stages tab.

If the source generates lots of tiny files — typical of frequent incremental ingests — compact with OPTIMIZE in Delta Lake before thinking about more compute. That problem is usually about I/O, not CPU or RAM.

Use cluster policies so nobody ends up with a giant cluster on a notebook that’s “just for testing something.” Spot for dev and workloads that tolerate retries; on-demand, at least for the driver, for anything with an SLA. And whenever the workload allows it, serverless by default: it takes the manual sizing problem off your plate and matches spend to actual usage.

And above all: measure before resizing. The Spark UI (Stages, Storage, Executors) and cluster metrics are the source of truth. “It feels slow” is not a diagnosis.

Cheat sheet: a quick summary to keep handy

Topic	Rule of thumb
Partitioning	< 1 TB: Liquid Clustering + auto-compaction. > 1 TB: Z-Ordering on heavily filtered columns. Avoid over-partitioning.
Shuffle partitions	Aim for 128-200 MB per partition after the shuffle. Tune with spark.sql.shuffle.partitions (200 by default) — raise or lower it based on volume and skew, don’t leave it at default for large clusters.
Tasks per core	2-4 tasks per core under normal conditions. 1:1 wastes parallelism; go much higher and the scheduler becomes the bottleneck.
Memory	Not a fixed number per core, depends on overhead: ~4x the partition size for simple ETL with no heavy shuffle, ~8x or more with big joins, groupBy on many keys, or skew risk.
Driver	Size it for the largest result you’re bringing back (.collect(), .toPandas(), broadcast), not for the full input size.
Node type	Compute-optimized if the bottleneck is pure CPU/IO. Memory-optimized if there are big joins or a lot of in-memory state (groupBy, windows).
Batch vs. streaming jobs	Batch: autoscaling + spot wherever the SLA allows. Low-latency streaming: fixed size aimed at the 90th percentile, no autoscaling.
Photon	Turn it on nearly whenever you can — documented improvements of 2-8x on heavy joins and aggregations, no code changes.
Skew	Spotted in Stages by comparing max time vs. median. More nodes won’t fix it, the heavy partition is still the same one. Mitigate with salting, broadcast join, or AQE (adaptive query execution).

Here’s a quick example, using the back-of-the-envelope formula from earlier: a 25 GB dataset, ETL with some joins in it, overhead factor of 6x (the midpoint of the 4-8x range).

partitions          = 25 × 1024 / 150 ≈ 171
cores_needed        = 171 / 3          ≈ 57      # 3 tasks per core
workers             = ceil(57 / 4)     = 15      # 4-core nodes
total_memory_gb     = 25 × 6           = 150     # 6x factor for the join
memory_per_executor = 150 / 15         = 10 GB

With that, you’d start out with roughly 15 workers of 4 cores and 10 GB each — pretty far from the 50 machines at 2 GB each that the classic “1 partition = 1 core” math with fixed memory per core spits out. That shortcut leaves no room for shuffle or for real JVM overhead, and for 25 GB it’s also a considerable over-provision compared to what you’d actually need.

To go deeper, besides the Databricks docs I linked above, I also leaned on a handful of tutorials and YouTube videos that helped me nail down the driver vs. workers part and see real examples right inside the Databricks screen.

Sources

• Compute configuration reference
• Cluster sizing considerations
• SQL warehouse sizing and scaling
• Photon
• Cluster policies
• Databricks Runtime for Machine Learning