Distributed computing is a computing approach where a problem is divided across multiple machines that work together. In the early 2000s, the rise of frameworks like Hadoop and MapReduce provided a way to distribute computation across “clusters” of computers, allowing companies to process massive datasets beyond the capabilities of a single machine. While powerful, MapReduce introduced new limitations which inspired researchers at the UC Berkeley to develop Apache Spark in 2009. Spark was a faster and more flexible way to distribute data processing by keeping intermediate results in memory. It also provided high-level APIs for data scientists and engineers. This article will explore why Spark was created, how it works, and how common data analysis workflows can be performed in both PySpark and sparklyr. Let’s get after the fact!
Let’s start at the beginning.
Imagine you start a small company that sells products online. You likely record quantities, prices, dates, customer data, and other relevenat pieces. Perhaps every once in a while you open a csv or-if you are really fancy, query a database- to analyze your data. You calculate profits, identify trends, and learn more about your customers. Awesome! I love using data to make decisions.
Now let’s look at your company 10 years later. Your business has become a massive success. Hundreds or thousands of orders are coming in every hour. This is fantastic, but introduces a new problem: you wouldn’t dream of opening this dataset as a csv. The data is waaay too large. Just calculating an average could be a serious investment in computing time. And something like training a neural network to predict which customers should be given which coupon codes… forget about it. Datasets that are only a few gigabytes can quickly become impractical to analyze using traditional tools. When data grows beyond the limits of a single machine, we need a new approach. Enter: Distributed Computing.
The idea behind distributed computing is simple: instead of asking one computer to do all the work, divide the problem across multiple computers and let them work together. This group of computers is called a cluster.
Think about a simple task: calculating the average order total for every transaction your company has ever made. Right now, your computer can easily load the data, add up every order total, and divide by the number of purchases. But what happens when your company has collected millions- or even billions!- of transactions over the next 10 years? Computers have limits. They have a finite amount of memory, processing power, and storage. Eventually, the dataset becomes too large to fit comfortably on one machine.
Distributed computing solves this by breaking a large problem into smaller pieces.
Imagine your company’s sales data is divided into ten smaller datasets. Instead of one computer calculating the average order total for all transactions, ten computers can each calculate the average for their assigned portion of the data. Once each computer finishes, the results can be combined to produce the final answer.
The basic idea looks something like this:
Traditional Computing:
+---------------+ +--------------+ +-------------+
| Large Dataset | ----> | One Computer | ----> | Final Result|
+---------------+ +--------------+ +-------------+
Distributed Computing:
+---------------+
| Large Dataset |
+---------------+
|
+--------------+--------------+
| | |
v v v
+-----------+ +-----------+ +-----------+
| Split 1 | | Split 2 | | Split 3 |
+-----------+ +-----------+ +-----------+
| | |
v v v
+-----------+ +-----------+ +-----------+
| Computer 1| | Computer 2| | Computer 3|
+-----------+ +-----------+ +-----------+
| | |
v v v
+-----------+ +-----------+ +-----------+
| Result 1 | | Result 2 | | Result 3 |
+-----------+ +-----------+ +-----------+
\ | /
\ | /
\ | /
v v v
+--------------------------+
| Combine Results |
+--------------------------+
|
v
+-------------+
| Final Answer |
+-------------+Simple as that. Distributed Computing: the computation is distributed. The problem is divided into smaller tasks, those tasks are solved simultaneously, and the results are combined at the end.
But before you run off to buy a fleet of computers for your business, distributed computing introduces new challenges. How do we decide how to split the data? How do computers communicate with each other? What happens if one computer fails halfway through the process? How do we make sure the final result is correct?
These questions are exactly why distributed computing systems like Hadoop and Spark exist. They provide the framework needed to coordinate many computers working together while hiding much of the complexity from the person analyzing the data.
If you only remember one thing from this article, make it this: Spark does not make any single computer more powerful. Instead, it allows many computers to work together as if they were one.
But this raises an important question: how do we actually coordinate hundreds or thousands of computers working together?
The solution was a programming model called MapReduce, introduced by Google in 2004. MapReduce broke distributed computation into two main steps: Map and Reduce.
The Map step takes a large dataset and breaks it into smaller pieces. Each computer processes its assigned portion of the data and produces intermediate results. The Reduce step takes those intermediate results and combines (or reduces) them into a final answer. More specifically, MapReduce accomplishes this by converting each record into a key-value pair.
Your sales data might look like this:
Order 1: Coffee - $5
Order 2: Shoes - $80
Order 3: Coffee - $7
Order 4: Book - $20
Order 5: Shoes - $60
The map step converts each record into key-value pairs:
(Coffee, 5)
(Shoes, 80)
(Coffee, 7)
(Book, 20)
(Shoes, 60)
Before the reduce step, MapReduce groups values with the same key:
Coffee -> [5, 7]
Shoes -> [80, 60]
Book -> [20]
The reduce step combines the grouped values:
Coffee -> Total: 12, Count: 2
Shoes -> Total: 140, Count: 2
Book -> Total: 20, Count: 1
Finally, the average order total can be calculated:
Total Sales = 172
Total Orders = 5
Average Order Total = 172 / 5 = $34.40
Or, to really drive this point home:
MapReduce:
Large Sales Dataset
|
v
-------------------------------------------------
| | |
v v v
+---------------+ +---------------+ +---------------+
| Computer 1 | | Computer 2 | | Computer 3 |
| | | | | |
| MAP | | MAP | | MAP |
| | | | | |
| Coffee, 5 | | Coffee, 7 | | Book, 20 |
| Shoes, 80 | | Shoes, 60 | | |
+---------------+ +---------------+ +---------------+
\ | /
\ | /
\ | /
SHUFFLE AND SORT
(Group matching keys together)
|
v
+--------------------------------+
| Coffee -> [5, 7] |
| Shoes -> [80, 60] |
| Book -> [20] |
+--------------------------------+
|
v
+---------------+
| REDUCE |
| |
| Coffee: 12 |
| Shoes: 140 |
| Book: 20 |
+---------------+
|
v
+---------------+
| FINAL RESULT |
| |
| Total Sales |
| = 172 |
| Orders = 5 |
| Avg = $34.40 |
+---------------+This example looks absurdly silly with an example dataset this small. You’ll just have to imagine this is a big dataset.
MapReduce was revolutionary because it made large-scale distributed processing accessible and reliable. However, it also had limitations. Every step in a MapReduce workflow required writing intermediate results back to disk, which made repeated calculations slow and inefficient.
These limitations created the need for a faster and more flexible approach. Enter: Apache Spark.
The problem is that many data science workflows are iterative. Consider a machine learning model that predicts which customers are most likely to respond to a coupon. Training that model requires repeatedly analyzing the same dataset, adjusting the model, and evaluating the results (think backpropagation and gradient descent). In a traditional MapReduce workflow, each iteration would require intermediate results to be written back to disk before the next step could begin. This process works, but it introduces significant overhead and can become slow. Apache Spark was created to address these limitations.
1. Spark reduces unnecessary disk operations Instead of requiring intermediate results to be written back to disk after every operation, Spark can keep data in memory and reuse it across multiple computations. This is especially valuable for iterative workflows like machine learning, where the same dataset may be analyzed hundreds or thousands of times. (The advantage of memory over disk is speed: accessing data in Random Access Memory (RAM) is orders of magnitude faster than reading and writing data from storage.)
2. Spark expanded the MapReduce programming model Rather than forcing developers to think only in terms of “map” and “reduce” steps, Spark introduced a broader set of transformations that describe how data should be modified. Filtering rows, creating new variables, grouping data, and joining tables are types of transformations.
Under the hood, Spark still follows the same fundamental idea as MapReduce: divide a large problem into smaller pieces, process those pieces in parallel, and combine the results. The difference is that Spark can build an optimized execution plan for a series of transformations and determine the most efficient way to execute them across a cluster.
In other words: MapReduce gave us a way to distribute individual computations. Spark gave us a way to build entire data workflows that can be distributed and optimized.
Before working with Spark, there are a few important terms to understand.
Cluster: A group of computers, called nodes, that work together as a single system to process data.
Driver: The coordinator of a Spark application. It creates the execution plan and assigns tasks to executors in the cluster.
Executors: Processes that run on worker nodes to execute tasks assigned by the driver and perform the actual data processing. A single node can run one or more executors, meaning the executor is the software doing the work, not the computer itself.
Partitions: The smaller pieces of a dataset that Spark processes independently.
Lazy evaluation: Spark does not immediately execute operations when they are written. Instead, it builds an optimized plan and executes the work only when a result is needed.
With these concepts in mind, we can now look at how to set up Spark and perform data analysis using both PySpark and sparklyr.
Spark can be accessed through several programming languages. In this article, we will demonstrate Spark using two popular APIs:
dplyr syntax.Both interfaces communicate with the same underlying Spark engine. The difference is only the language used to write the code.
First, we load the required packages:
library(sparklyr)
Attaching package: 'sparklyr'The following object is masked from 'package:stats':
filterlibrary(dplyr)
Attaching package: 'dplyr'The following objects are masked from 'package:stats':
filter, lagThe following objects are masked from 'package:base':
intersect, setdiff, setequal, unionNext, we connect to Spark:
sc <- spark_connect(
master = "local",
version = "3.5.1"
)In a production environment, this connection could point to a cluster containing hundreds of computers. For this demonstration, we are running Spark locally, meaning our computer is acting as the cluster.
Remember the Spark architecture:
Driver
|
v
Executors
|
v
Partitions of DataThe driver coordinates the work, while executors perform the computations on different partitions of the data. The workflow is the same whether Spark is running on one computer or thousands of computers.
To demonstrate Spark, we will use the Superstore dataset, a fictional retail dataset containing customer orders, products, sales, and profit information.
Our goal is simple:
What is the average order total for our store?
The dataset contains individual line items, meaning one order may appear across multiple rows. Before calculating the average, we need to group the items belonging to the same order and calculate the total value of each order.
First, we load the data directly into Spark:
orders <- spark_read_csv(
sc,
name = "orders",
path = "why-spark_files/Superstore.csv",
header = TRUE,
infer_schema = TRUE
)Unlike traditional R workflows, this does not load the entire dataset into R’s memory. Instead, R sends instructions to Spark, and Spark manages the data processing.
We can view the first few rows:
orders %>%
head()# A query: ?? x 21
# Database: spark_connection
Row_ID Order_ID Order_Date Ship_Date Ship_Mode Customer_ID Customer_Name
<int> <chr> <chr> <chr> <chr> <chr> <chr>
1 1 CA-2016-152156 11/8/2016 11/11/20… Second C… CG-12520 Claire Gute
2 2 CA-2016-152156 11/8/2016 11/11/20… Second C… CG-12520 Claire Gute
3 3 CA-2016-138688 6/12/2016 6/16/2016 Second C… DV-13045 Darrin Van H…
4 4 US-2015-108966 10/11/2015 10/18/20… Standard… SO-20335 Sean O'Donne…
5 5 US-2015-108966 10/11/2015 10/18/20… Standard… SO-20335 Sean O'Donne…
6 6 CA-2014-115812 6/9/2014 6/14/2014 Standard… BH-11710 Brosina Hoff…
# ℹ 14 more variables: Segment <chr>, Country <chr>, City <chr>, State <chr>,
# Postal_Code <int>, Region <chr>, Product_ID <chr>, Category <chr>,
# SubCategory <chr>, Product_Name <chr>, Sales <chr>, Quantity <chr>,
# Discount <chr>, Profit <dbl>At this point, Spark knows: - where the data is located - what columns exist - how the data should be processed
However, Spark has not necessarily performed every computation yet. This connects to one of Spark’s most important ideas: lazy evaluation. Spark builds a plan for the work and executes it when a result is actually needed.
First, we create a new column representing the value of each item:
orders <- orders %>%
mutate(
Order_Total = as.numeric(Sales)
)Now we group together all items belonging to the same order and calculate the total order value:
order_totals <- orders %>%
group_by(Order_ID) %>%
summarize(
Total_Order_Value = sum(Order_Total)
)Conceptually, Spark is performing the same divide-and-conquer process introduced earlier:
Many Individual Rows
|
v
Map:
Calculate order totals for each order
|
v
Reduce:
Combine matching Order_ID values
|
v
Average Order TotalFinally, we calculate the average order value:
order_totals %>%
summarize(
Average_Order_Total = mean(Total_Order_Value)
)Warning: Missing values are always removed in SQL aggregation functions.
Use `na.rm = TRUE` to silence this warning
This warning is displayed once every 8 hours.# A query: ?? x 1
# Database: spark_connection
Average_Order_Total
<dbl>
1 461.The result is a simple business metric, but the process demonstrates the core idea behind Spark. Instead of requiring one computer to process every row, Spark can divide the work into partitions, process those partitions in parallel, and combine the results into a final answer.
This is the same fundamental idea behind MapReduce, but Spark provides a more flexible way to build and optimize entire data workflows.
Spark provides APIs for several programming languages, including Python. PySpark is the Python API for Apache Spark and allows data scientists and engineers to interact with Spark using Python syntax.
The underlying Spark engine is the same as the one used by sparklyr. The driver, executors, partitions, and execution plans all work the same way. Only the interface used to communicate with Spark changes.
Note: This demonstration was developed using
sparklyrlocally. PySpark requires a compatible Java/Spark environment, which was not available on this machine. The following code shows the equivalent PySpark workflow.
In PySpark, we begin by creating a Spark session:
from pyspark.sql import SparkSession
spark = (
SparkSession.builder
.master("local")
.appName("Why Spark Demo")
.getOrCreate()
)This creates our connection to Spark. Just like spark_connect() in sparklyr, the Spark session allows our code to communicate with the Spark engine.
We can load the same Superstore dataset directly into Spark:
orders = (
spark.read
.option("header", True)
.option("inferSchema", True)
.csv("why-spark_files/Superstore.csv")
)At this point, Spark has created a DataFrame representing our data. The data is managed by Spark rather than loaded directly into Python memory.
We can preview the data:
orders.show()Just like in sparklyr, our goal is to calculate the average order value.
First, we create a column representing the value of each item:
from pyspark.sql.functions import col
orders = orders.withColumn(
"Order_Total",
col("Sales").cast("double")
)Next, we group individual rows by order and calculate the total value of each order:
order_totals = (
orders
.groupBy("Order_ID")
.sum("Order_Total")
)This is the same divide-and-conquer idea discussed earlier:
Individual Order Rows
|
v
Map:
Calculate values for individual records
|
v
Reduce:
Combine rows belonging to the same Order_ID
|
v
Average Order TotalFinally, we calculate the average order value:
from pyspark.sql.functions import avg
order_totals.select(
avg("sum(Order_Total)").alias("Average_Order_Total")
).show()The important takeaway is that the Spark engine is doing the same work regardless of whether we use R or Python. The syntax changes, but the concepts remain the same:
The language is different. The distributed computing model is the same.
Spark was created to solve a simple but important problem: sometimes our data becomes too large for a single computer to process efficiently. By distributing computation across multiple computers, Spark allows organizations to analyze massive datasets while hiding much of the complexity of managing a cluster.
Throughout this article, we saw that the core idea behind Spark is not new. Spark builds upon the same divide-and-conquer principles introduced by MapReduce: break a large problem into smaller pieces, process those pieces in parallel, and combine the results. The difference is that Spark provides a more flexible framework for building entire data workflows through transformations, optimized execution plans, and in-memory processing.
Using the Superstore dataset, we demonstrated how a simple business question—calculating the average order value—can be transformed into a distributed computing problem. Whether using R through sparklyr or Python through PySpark, the underlying concepts remain the same: the driver coordinates the work, executors process partitions of data, and Spark determines the most efficient way to execute the computation.
While most analysts will not need Spark for every dataset they encounter, understanding Spark becomes increasingly important as organizations collect larger and more complex datasets. The goal is not to replace traditional data analysis tools, but to recognize when a problem has grown beyond the limits of a single computer.
The most important idea to remember is this:
Spark does not make one computer more powerful. Spark allows many computers to work together as if they were one.
Let’s get after the fact!