“Sometimes the best engineering lessons don’t begin with architecture diagrams. They begin with someone asking a very innocent question.”
Apache Spark & Azure Synapse — The Story I Wish Someone Had Told Me
The Girl, The Grin, and the 18-Hour Pipeline
If you’ve worked in a software company long enough, you know there’s always that friend.
Mine was Sam.
One morning, while I was pretending to be productive and staring at Azure Synapse jobs, Sam walked over, leaned against my desk with the most suspicious grin imaginable and whispered,
“Bro…”
I didn’t even look up.
“Hmm?”
“There’s a new joinee.”
“Okay?”
“You should definitely mentor her.”
I laughed.
“That’s your reason?”
He looked around dramatically before lowering his voice.
“No… she’s beautiful.”
I rolled my eyes.
“You’ve literally never recommended someone because they write good code.”
He shrugged.
“Exactly.”
Then he walked away, still laughing to himself.
About twenty minutes later, someone stopped beside my desk. “Hi…”
I looked up. She smiled. “I’m Jenny.”
“Oh…”
She was the new mentee. And for a second my brain completely forgot what programming language I used. She pulled a chair next to mine.
“So… Sam said you’re Charlie.”
I silently made a mental note to murder Sam later. She opened her laptop, rotated it toward me, and pointed at an Azure Synapse pipeline.
“So…”
“…is it supposed to take this long?”
The pipeline had been running for 18 hours and 17 minutes.
I blinked. “No.”
She looked confused. “No?”
“It should’ve finished in under an hour.”
Just then I noticed Sam walking past our aisle. He looked straight at me… raised both eyebrows… gave me the most wicked grin I’d ever seen… and disappeared around the corner before Jenny could notice. Almost as if to ask,
“Well? I told you.”
She turned around. “What happened?”
I looked away from the corridor as fast as humanly possible. “Nothing.”
She looked back at me… smiled…
and returned to the screen.
Men will be men.
Let’s Fix It Together
I told Jenny something I’ve told almost every new engineer I’ve mentored.
“Forget Spark for ten minutes.”
She looked surprised. “Seriously?”
“Yeah.”
“Because if you understand how Spark thinks, every optimization afterwards becomes obvious.”
She nodded. “So…”
“Imagine our data isn’t ten thousand rows.”
“It’s a billion.”
She nodded again.
“Actually…”
“Imagine it’s twenty billion.”
“And every single row has maybe thirty or forty columns.”
“Now imagine that dataset keeps growing every single hour.”
She slowly closed her laptop. “I think my laptop just resigned.”
I laughed.
“Exactly.”
On your own laptop, complexity usually looks like this:
O(n × m)
where
- n = rows
- m = columns
Your CPU does everything. Your RAM stores everything. Life is simple. Corporate engineering is different. Now there’s another variable.
k = number of machines
Your complexity suddenly starts looking more like O((n × m) / k)
Not mathematically perfect… but it’s a fantastic mental model. Instead of one computer processing one billion rows… imagine 100 machines each processing ten million rows simultaneously. That’s why companies use Spark. Not because Spark is magically faster.
Because parallelism changes the economics.
With 100 parallel machines it’s almost 100× faster
Jenny looked impressed. “So Spark just splits the work?”
“Mostly.”
“But that’s only half the story.”
I drew three boxes.
External Analytics DB
│
▼
Apache Spark Cluster
│
▼
Save Data back to External DB“This,” I said, “is almost every modern data engineering pipeline.”
We read data from somewhere. Transform it. Write it somewhere else. Simple. Except…
Spark almost never executes your code immediately.
Jenny frowned. “So when does it execute?”
I smiled. “Perfect question.”
“Let’s build the pipeline first.”
One Production Pipeline
Imagine this is our real production notebook inside Azure Synapse. Every hour… a Tumbling Window Trigger wakes up. It tells our notebook:
“Process only the data between these two timestamps.”
That’s important. Because we never want to scan years of data if only one hour changed.
from pyspark.sql import SparkSession
from pyspark.sql.functions import (
col,
sum,
avg,
count,
broadcast
)
spark = (
SparkSession.builder
.appName("SalesAnalytics")
.getOrCreate()
)
windowStart = mssparkutils.env.getJobParameter("windowStart")
windowEnd = mssparkutils.env.getJobParameter("windowEnd")
sales = (
spark.read
.format("com.microsoft.kusto.spark.datasource")
.option("kustoCluster", cluster)
.option("kustoDatabase", database)
.option(
"kustoQuery",
f"""
Sales
| where Timestamp >= datetime('{windowStart}')
| where Timestamp < datetime('{windowEnd}')
| project
CustomerId,
RegionId,
ProductId,
Amount,
Timestamp
"""
)
.load()
)
regions = spark.read.parquet("/lakehouse/dim/regions")
products = spark.read.parquet("/lakehouse/dim/products")
result = (
sales
.join(
broadcast(regions),
"RegionId"
)
.join(
products,
"ProductId"
)
.groupBy("RegionName")
.agg(
sum("Amount").alias("Revenue"),
avg("Amount").alias("AverageOrder"),
count("*").alias("Orders")
)
)
(
result.write
.mode("overwrite")
.partitionBy("RegionName")
.parquet("/lakehouse/gold/hourly_sales")
)Jenny looked at the notebook. “It doesn’t look that complicated.”
“It isn’t.”
“The difficult part is understanding what Spark is secretly doing.”
What Spark Is Secretly Thinking
The very first line that surprises everyone is this one.
sales = spark.read...Jenny pointed at it. “So…”
“It loads the data?”
“No.”
“It describes how to load the data.”
She blinked. “What?”
I smiled. “This is Spark’s biggest trick.”
Lazy Evaluation
Spark is unbelievably lazy. Every line before an action is simply a promise. When you write
sales = spark.read...Spark says
“Okay.”
“When you eventually need this data…”
“I’ll remember how to fetch it.”
Then we join.
.join(...)Spark says
“Cool.”
“I’ll remember that too.”
Then we group.
.groupBy(...)Again… Spark doesn’t execute. Another promise. Another note. Another instruction. Nothing actually happens. Jenny looked horrified. “So…”
“Nothing we’ve written has actually run?”
“Exactly.”
The cluster hasn’t touched the database. No network traffic. No CPUs. No memory. Nothing. Spark is simply collecting instructions.
The DAG
“Imagine planning a road trip.”
“You don’t start driving after deciding your first stop.”
“You plan the whole route.”
Then… once everything is decided… you start driving. Spark works exactly the same way. Internally it builds something called a
Directed Acyclic Graph
or simply… the DAG.
Instead of executing line by line… Spark quietly builds a graph that looks something like this.
Read Sales
│
▼
Join Regions
│
▼
Join Products
│
▼
Group By Region
│
▼
Aggregate Revenue
│
▼
Write ParquetNothing moves. Nothing executes. Spark is simply planning. Why? Because planning allows optimization. And that’s where Spark becomes brilliant.
Catalyst Optimizer
I asked Jenny a question. “If I told you to move houses…”
“…would you carry one chair at a time?”
She laughed. “Obviously not.”
“You’d probably group things together.”
“Exactly.”
Spark does the same thing. Once it sees the entire DAG…
its optimizer called Catalyst, starts rewriting your code. Not your source code. Its own execution plan. It begins asking questions like
- Can I move this filter earlier?
- Can I remove unused columns?
- Can I merge these operations?
- Can I reorder these joins?
- Can I reduce network traffic?
- Can I avoid reading unnecessary files?
Without you changing a single line… Catalyst frequently produces a dramatically faster execution plan. It’s like having a senior engineer quietly reviewing every query before it runs.
Tungsten
Jenny leaned back. “So Catalyst makes smarter decisions.”
“Exactly.”
“But once those decisions are made…”
“…someone still has to execute them.”
That’s Tungsten’s job.
Catalyst decides what should happen. Tungsten decides how to execute it as efficiently as possible. Instead of creating millions of tiny Java objects…
Tungsten uses
- off-heap memory
- binary data formats
- whole-stage code generation
- cache-friendly layouts
to keep CPUs busy instead of garbage collectors. The result?
Less memory overhead. Less JVM garbage collection. Better cache utilization. Much higher throughput.
Most Spark users never directly interact with Tungsten. They simply enjoy pipelines that finish dramatically faster because it’s working quietly underneath everything they write.
Partitioning — The Real Superpower
I drew one last picture.
Without Spark:
Machine
██████████████████████
1 Billion RowsWith Spark:
Machine 1
██
Machine 2
██
Machine 3
██
...
Machine 100
██Every machine receives a partition. Each partition is simply a slice of the dataset. Every executor processes its own partition independently. No machine waits for another. No single computer becomes the bottleneck. This is why Spark scales. Not because one CPU becomes faster. Because one hundred CPUs work simultaneously.
I looked at Jenny. “So remember this.”
“In college… you optimized algorithms.”
“In production… you optimize movement.”
Moving less data. Reading fewer files. Reducing shuffles. Keeping partitions balanced. Avoiding unnecessary network calls. Because once your datasets reach billions of rows… the CPU usually isn’t your enemy anymore. The network is. And Spark spends the rest of its life trying to avoid it.
Joins, Shuffles, Caching, Fault Tolerance & Why Everyone Loves Parquet
Jenny stared at the notebook for a few seconds. “So…”
“If Spark is already splitting work across hundreds of machines…”
“…why do people still complain about Spark jobs being slow?”
I smiled. “Because splitting the work is the easy part.”
“The expensive part… is getting the machines to talk to each other.”
Broadcast Join — The Fastest Join You’ll Ever Get
I highlighted this line.
.join(
broadcast(regions),
"RegionId"
)Jenny looked confused. “What exactly is broadcast() doing?”
I grabbed a marker. “Suppose you’re giving tomorrow’s meeting agenda to a hundred employees.”
“You have two choices.”
Option 1: Every employee walks to your desk… asks for a copy… walks back. One hundred trips.
Or…
Option 2: You simply email everyone beforehand. Zero walking.
Broadcast Join works exactly like the second option. Instead of moving billions of sales rows around the cluster… Spark copies the tiny lookup table, our regions table, to every executor.
Executor 1
───────────────
Sales Partition A
Regions Table
Executor 2
───────────────
Sales Partition B
Regions Table
Executor 3
───────────────
Sales Partition C
Regions TableEvery executor already has the lookup table locally. No network movement. No shuffle. Just a very fast in-memory hash lookup.
Jenny nodded. “So Spark duplicated the smaller table… instead of moving the huge one.”
“Exactly.”
She looked impressed. “So the rule is always broadcast?”
I laughed. “If only life were that simple.”
When Broadcast Is a Terrible Idea
Imagine your lookup table isn’t 500 rows.
It’s 500 million rows.
Now broadcasting means copying hundreds of gigabytes… to every executor.
Your executors immediately run out of memory. Jobs fail. Executors die. The cluster starts crying. Broadcast joins are incredible… only when one side is genuinely small.
Usually
- lookup tables
- dimensions
- configuration tables
- country lists
- product categories
Anything comfortably fitting into executor memory.
Shuffle Join — The Network Tax
“Now imagine,” I said, “both tables are huge.”
Maybe - Sales and Customers. Both contain billions of rows. Broadcasting either one would be ridiculous.
“So what now?”
Spark performs something called a shuffle.
Imagine every employee in a company has thousands of papers… but suddenly everyone needs papers relating only to their own department.
Sales papers. HR papers. Finance papers. Engineering papers.
Everybody starts exchanging papers.
Machine A ─────► Machine C
Machine B ─────► Machine A
Machine D ─────► Machine B
Machine C ─────► Machine DThat’s a shuffle. It is to simply move records across machines until matching keys live together, thus the join or aggregate queries become independent and simple to handle. And… maybe a key is big enough to be put on two machines.
The algorithm itself isn’t difficult. The movement is. And movement is expensive.
I told Jenny something I wish someone had told me years ago.
CPUs are cheap.
RAM is cheap.
Network traffic is expensive.
If Spark jobs feel slow… nine times out of ten… you’re paying the network tax.
Sort Merge Join
She pointed at the second join.
.join(
products,
"ProductId"
)“So Spark didn’t broadcast this one.”
“No.”
“Why?”
“Because it decided something else was better.”
Spark’s default strategy for two very large datasets is usually the Sort Merge Join.
The process is surprisingly simple. First… both datasets are shuffled. Then… each partition is sorted. Finally… Spark walks through both sorted datasets together. Exactly like merging two alphabetically sorted phone books.
A C D G H K M
A B D E H M ZWalk together. Match keys. Produce rows. Repeat.
Sort Merge Join scales beautifully. But… it pays for
- one shuffle
and
- one sort
which are usually the two most expensive operations in Spark.
Jenny leaned forward. “So how do experienced engineers optimize Spark?”
I answered almost immediately. “They spend their lives trying to avoid shuffles.”
Because every shuffle means
- network traffic
- serialization
- disk spill from memory
- waiting
- garbage collection
- more waiting
Avoiding one shuffle often saves more time than rewriting an entire algorithm.
Then Came The Most Important Line
I highlighted one line.
sales.cache()Jenny laughed. “That’s the famous line?”
“The one that saved seventeen hours?”
“Yep.”
“It doesn’t even look important.”
“I know.”
“That’s why everyone misses it.”
Caching
I asked her a question. “If I asked you to photocopy the same document five hundred times…”
“…would you walk back to my house…”
“…every single time…”
“…to collect the original?”
She laughed. “Obviously not.”
“I’d keep one copy here.”
“Exactly.”
That’s caching. Spark normally behaves like this.
Database
↓
Read
↓
Transform
↓
ActionEvery action starts from the database again.
Again. Again. Again. Again.
Jenny looked surprised.
“Even if nothing changed?”
“Even then.”
Spark simply follows its DAG. It doesn’t assume you wanted yesterday’s data.
Now imagine we cache.
Database
↓
Read Once
↓
RAM
↓
Action
↓
Action
↓
Action
↓
ActionOne network read. Hundreds of in-memory operations. Suddenly everything becomes dramatically faster.
I told Jenny something every Spark engineer eventually learns.
If your DataFrame comes from somewhere expensive… and you’re going to reuse it… cache it!
It’s one of those rules that’s almost never wrong.
But Cache Isn’t Magic
Jenny immediately asked, “So why doesn’t everyone cache everything?”
Good question. Because memory isn’t free. Executors only have so much RAM. Cache too much… Spark starts evicting partitions. Eventually… it spills to disk. Then performance collapses again.
Good engineers cache
-
expensive reads
-
reused datasets
-
intermediate results used multiple times
They don’t cache everything.
Wait…
“Then why do people call count() immediately afterwards?”
She pointed at this.
sales.cache()
sales.count()“Isn’t that pointless?”
“Actually… it’s brilliant.”
Remember Lazy Evaluation? .cache() itself doesn’t execute anything. Spark simply writes another note.
Remember... when you eventually compute this... keep it in memory.Nothing happens yet. The cache is still empty.
count() is the first action. Spark finally executes the DAG. Reads the data. Stores it in RAM. Returns the count. Now… every future operation reuses that cached copy. Without the action… nothing gets cached. That’s one of Spark’s sneakiest behaviours.
Fault Tolerance — Losing Machines Without Losing Sleep
Jenny suddenly asked, “What happens if one machine crashes halfway through?”
I smiled. “This is where Spark becomes genius.”
Traditional distributed systems often replicate data everywhere. Spark usually doesn’t. Instead… it remembers how every partition was created.
Imagine someone deletes this partition.
Partition 17Spark doesn’t panic. It simply walks backwards through the DAG.
Read
↓
Filter
↓
Join
↓
Group
↓
Partition 17“Oh…”
“So it just recreates it?”
Exactly. Spark stores recipes. Not the food itself. Every partition knows who its parents were and which transformations created it. This concept is called Lineage.
Because of lineage… losing one executor usually isn’t a disaster. Spark simply rebuilds the missing partition somewhere else. No manual recovery. No restoring backups. No engineer waking up at 3 AM.
The Last Piece — Why Everyone Writes Parquet
Jenny looked at the final line.
.parquet(...)“Couldn’t we just write CSV?”
“We could.”
“But we’d regret it.”
I drew two pictures.
A CSV file.
Alice
Sales
100
Bob
HR
500
Carol
Sales
300Rows. Everything together. Now imagine your manager asks
“What’s the average salary?”
CSV has to read - Names. Departments. Cities. Addresses.
Everything. Just to calculate one column.
Now Parquet.
Names
Alice
Bob
Carol
──────────────
Departments
Sales
HR
Sales
──────────────
Salary
100
500
300Columns live together. Need only salary? Spark reads only salary. Nothing else.
Now imagine, one trillion rows. Two hundred columns. Your dashboard needs three columns.
CSV reads 200 columns.
Parquet reads 3.
That’s the difference between minutes and seconds.
Parquet gives us much more than that. It also supports
- compression
- predicate pushdown
- column pruning
- efficient encoding
- vectorized reads
Catalyst loves Parquet. Spark loves Parquet. Databricks loves Parquet. Synapse loves Parquet. Basically… everyone loves Parquet!
I leaned back. “So let’s recap.”
Spark’s performance isn’t magic. It’s simply thousands of tiny engineering decisions working together.
Broadcast the small tables. Avoid unnecessary shuffles. Cache expensive reads. Let Catalyst rewrite your plan. Let Tungsten execute efficiently. Recover failures through lineage. Store analytics data as Parquet.
Jenny smiled. “So… Spark isn’t really about writing code.”
I smiled back. “No.”
“It’s about moving as little data as possible.”
She nodded slowly. “I think I finally understand why that missing .cache() mattered.”
I laughed. “You understand the theory.”
“Now let me show you what actually happened to our pipeline…”
“And why one innocent Azure trigger nearly brought the whole thing to its knees.”
The Azure Trigger That Nearly Melted Our Cluster
Jenny leaned back in her chair. “I think I’ve got Spark now.”
I smiled. “You understand Spark.”
“But… you still haven’t seen why our pipeline ran for eighteen hours.”
She looked back at the notebook. “It was because of the missing .cache(), right?”
I shook my head. “That was only half the story.”
“The other half… was Azure Synapse.”
It All Started With a Tumbling Window Trigger
Every hour, Azure Synapse automatically started our notebook. Not using a Schedule Trigger. Using a Tumbling Window Trigger.
There’s a huge difference. A normal schedule trigger simply says
Run every hour.
It doesn’t care what happened previously. A Tumbling Window Trigger is smarter. It divides time into fixed windows.
00:00 ─────────► 01:00
01:00 ─────────► 02:00
02:00 ─────────► 03:00
03:00 ─────────► 04:00Each window represents one independent execution. Every run receives
windowStartTime
windowEndTimeand processes only that slice.
Jenny nodded. “So if the 1 PM run fails… the 2 PM run still knows exactly which data belongs to it?”
“Exactly.”
That’s why enterprises love Tumbling Window Triggers. They’re stateful. They support retries. They support dependencies. They’re perfect for ETL pipelines.
Then Production Happened…
Everything worked beautifully. For months. Until one Friday. Because of course… it’s always Friday. One hourly run failed. Then another. Then another. Soon we had nearly a day’s worth of failed windows waiting. Azure did exactly what it was designed to do. It remembered every failed window.
Backfill
Jenny frowned. “What happens after the issue gets fixed?”
I drew another timeline.
09:00 ❌
10:00 ❌
11:00 ❌
12:00 ❌
13:00 ❌
14:00 ✅Azure says
Great.
Let’s process everything we missed. That’s called a backfill.
Instead of processing only the newest hour… it starts replaying every failed window. One after another. Or… sometimes… several simultaneously depending on the trigger configuration.
Normally… that’s exactly what you want. If you’re processing financial transactions bank transfers medical records or orders… you absolutely cannot lose data. Every missed window must eventually execute.
Our project… was different.
The One Detail Nobody Questioned
Our analytics database already kept track of processed rows. Every query looked something like this.
Sales
| where Processed == falseNotice what’s missing? There was no timestamp filter.
windowStart
windowEndInstead… every execution simply asked
Give me every row that hasn’t been processed yet.
Whether that row arrived, five minutes ago or three hours ago, didn’t matter. It would eventually be picked up. The pipeline was naturally idempotent.
Jenny stared at me. “So… the trigger windows weren’t actually necessary?”
“Bingo.”
We weren’t processing hourly snapshots. We were processing everything still pending.
The Domino Effect
Imagine twelve failed windows. Azure suddenly launches them. Each notebook executes exactly the same query.
Processed == falseEvery notebook receives almost exactly the same massive dataset. Twelve notebooks. Reading the same billions of rows. From the same external database. At the same time.
Our Spark pool suddenly looked like this.
Notebook 1
↓
Read 2 TB
Notebook 2
↓
Read 2 TB
Notebook 3
↓
Read 2 TB
Notebook 4
↓
Read 2 TB
...Nobody was doing anything wrong. Every component behaved exactly as designed. Azure retried failed windows. Spark executed notebooks. The database answered queries. The problem… was the architecture.
The Database Started Fighting Back
External analytics databases are incredibly fast. They’re also incredibly protective. Suddenly receiving twelve enormous scans simultaneously isn’t exactly polite. Connections started queueing. Memory pressure increased. Requests slowed down. Then… intermittently… we saw this.
HTTP 500
Internal Server ErrorNot every run. Just enough to become annoying. Sometimes one notebook succeeded. Sometimes another failed. Sometimes all of them failed. Sometimes they all succeeded. The worst kind of bug.
The intermittent one.
Jenny sighed.
“I hate intermittent bugs.”
“So does everyone.”
The Spark UI Told the Real Story
Most engineers immediately look at logs. I almost never do. I open the Spark UI first. Because Spark tells you exactly where it’s spending time. The executors weren’t busy. CPU utilization was low. Memory wasn’t exhausted. Most stages looked like this.
Waiting...
Waiting...
Waiting...
Network Read...
Waiting...
Waiting...Spark wasn’t computing. It was waiting. Waiting for data. Waiting for the external database. Waiting for the network. That’s when everything clicked.
Then I Saw Something Else…
Inside another notebook… I noticed something strange.
for region in regions:
sales = (
spark.read...
.load()
)
sales.filter(...)
sales.write(...)Jenny frowned. “That doesn’t look terrible.”
“It doesn’t.”
“Until you remember Lazy Evaluation.”
She suddenly froze. “Oh no…”
I smiled. “You see it now.”
What Spark Actually Did
Developers often think like this.
Read Once
↓
Loop
↓
WriteSpark didn’t. Spark saw something completely different.
Read Database
↓
Filter
↓
Write
────────────
Read Database
↓
Filter
↓
Write
────────────
Read Database
↓
Filter
↓
WriteEvery iteration. A brand-new DAG. Every write… started from the database again.
Imagine - 500 regions. 500 writes. 500 complete database scans.
Over the network. Repeated. Again. And again. And again.
The notebook wasn’t slow because Spark was inefficient. It was slow because, we accidentally asked Spark to perform the most expensive operation possible, hundreds of times.
The Missing Line
The fix looked almost embarrassing.
sales = (
spark.read
.format(...)
.load()
)
sales.cache()
sales.count()Jenny laughed. “That’s it?”
“That’s literally it.”
Remember.
.cache() only tells Spark
Keep this DataFrame after computing it.
It still doesn’t execute. That’s why we immediately call
sales.count()That first action forces Spark to
- read the database once
- populate executor memory
- build the cache
Every operation afterwards comes directly from RAM. No additional network reads.
Then we rewrote the loop entirely. Instead of
for region in regions:we simply let Spark do what Spark does best.
(
sales.write
.partitionBy("RegionId")
.parquet(...)
)No manual loop. No repeated DAG. No repeated reads. Spark distributed everything automatically. Exactly what it was built for.
The Result
Before:
- Hundreds of database scans
- Massive network traffic
- Multiple retries
- Executor idle time
- Intermittent HTTP 500 errors
- More than 18 hours
After:
- One database scan
- Cached in executor memory
- Zero repeated reads
- Automatic partitioned writes
- Stable execution
- 45 minutes
Same hardware. Same Spark pool. Same cluster. Same data. One line. One architectural decision. One lesson.
Jenny sat silently for a few seconds. Then she opened another repository. Scrolled for maybe thirty seconds. Looked back at me.
“…Charlie?”
“Yeah?”
“I think this pipeline has the same bug.”
I walked over. She was right. Another DataFrame. Another expensive remote read. Another missing .cache().
I laughed. “Congratulations.”
“You’ve officially stopped writing Spark…”
“…and started thinking like Spark.”
And that’s the difference. The best Spark engineers don’t memorize APIs. They memorize where the data moves.
Because once your datasets grow from gigabytes… to terabytes… to petabytes… the bottleneck is almost never the code you write. It’s the data you accidentally move. And Spark will faithfully move every single byte… unless you teach it not to.
When Spark Is the Wrong Tool, Synapse vs Databricks, and One Last Smile
The notebook finally completed. I looked at the clock.
44 minutes and 52 seconds.
Jenny compared it with the execution history.
“Eighteen hours…”
“…to forty-five minutes.”
She looked back at me. “I still can’t believe one missing line caused all that.”
I smiled. “It wasn’t one missing line.”
“It was one missing understanding.”
When You Shouldn’t Use Spark
Jenny suddenly asked, “So…”
“Should we just use Spark for everything?”
I laughed. “Absolutely not.”
One of the biggest mistakes new engineers make is assuming distributed systems are automatically faster. They’re not. Distributed systems have fixed costs. Before Spark even starts working… it has already spent time
- network communication
- starting executors
- allocating memory
- creating JVMs
- scheduling tasks
- building the DAG
- planning stages
That’s a lot of overhead.
Suppose someone hands you a CSV containing 20,000 rows.
Should you spin up a Spark cluster?
Definitely not.
Pandas would finish before Spark even finished warming up. Maybe even Polars. Maybe plain SQL. Spark shines when your laptop starts crying. Not before.
As a rough guideline… Use Spark when
✅ You’re processing tens or hundreds of gigabytes.
✅ You have billions of records.
✅ You’re performing massive joins.
✅ You’re building scheduled ETL pipelines.
✅ Multiple machines actually help.
Don’t use Spark when
❌ The data fits comfortably in memory.
❌ You’re cleaning a small CSV.
❌ You’re building a quick prototype.
❌ You’re doing single-row lookups.
❌ The startup time costs more than the computation.
Good engineers don’t use the biggest hammer. They use the right one.
A Small Moment I Didn’t Expect
We wrapped up for the day. Jenny packed her laptop. Closed her notebook. Stood up. “Thanks, Charlie.”
“No problem.”
She smiled. “I think Spark finally makes sense.”
“You’ll hate it again tomorrow.”
She laughed. “I probably will.”
She started walking back toward her desk. For some reason…
my eyes followed her. Not intentionally.
Just…
the way your eyes quietly follow someone after a good conversation. Maybe because she wasn’t just another new engineer anymore. She’d gone from asking
“Is it supposed to take this long?”
to explaining broadcast joins back to me in a single afternoon. There was something satisfying about watching someone genuinely understand something difficult.
She reached the end of the aisle. Almost disappeared behind the partition. Then… for the briefest moment… she turned around. Our eyes met. Again. She smiled.
a small, genuine smile. Almost like she was silently saying,
“That was actually fun.”
Then she disappeared around the corner. I stood there for another second. Long enough for Sam to suddenly appear from absolutely nowhere. He folded his arms. “So…”
I sighed. “Don’t.”
He looked toward the corridor where Jenny had just disappeared. Then looked back at me. That same wicked grin from the morning slowly returned.
“I didn’t say anything.”
“You didn’t have to.”
He burst into laughter.
“You know…”
“I’ve never seen you explain Spark with this much enthusiasm.”
“It wasn’t because of her.”
“Mhmm.”
“It wasn’t.”
“Sure.”
I threw a pen at him.
He dodged it effortlessly.
“See you tomorrow, professor.”
And just like that…
he walked away laughing.
Some people debug production systems.
Sam debugs people.
