Top 50 PySpark Interview Questions and Answers (2026)
Top 50 PySpark Interview Questions and Answers (2026)
PySpark interviews are notoriously tough. Whether you are preparing for a role at a top tech company or a data-driven startup, knowing PySpark deeply is non-negotiable.
This guide covers the most frequently asked PySpark interview questions from basic to complex with clear concise answers.
Basic PySpark Interview Questions
1. What is PySpark?
PySpark is the Python API for Apache Spark. It allows you to write Spark applications using Python, combining Python simplicity with Spark distributed computing power.
2. What is a SparkSession?
SparkSession is the entry point to programming with Spark. It combines SQLContext, HiveContext, and SparkContext into a single unified interface.
from pyspark.sql import SparkSession
spark = SparkSession.builder.appName("MyApp").getOrCreate()
3. What is a DataFrame in PySpark?
A DataFrame is a distributed collection of data organized into named columns. It is conceptually equivalent to a table in a relational database but with richer optimizations.
4. What is the difference between RDD and DataFrame?
| Feature | RDD | DataFrame |
|---|---|---|
| Optimization | Manual | Catalyst optimizer |
| Type safety | Yes | No |
| Ease of use | Low | High |
| Performance | Lower | Higher |
5. How do you read a CSV file in PySpark?
df = spark.read.csv("data.csv", header=True, inferSchema=True)
df.show()
6. What is a transformation in PySpark?
Transformations are lazy operations that return a new DataFrame. They are not executed until an action is called. Examples: filter(), select(), groupBy().
7. What is an action in PySpark?
Actions trigger the execution of transformations. Examples: show(), count(), collect(), write().
8. What is lazy evaluation?
PySpark does not execute transformations immediately. It builds a DAG of operations and only executes when an action is called.
9. How do you filter rows in PySpark?
df.filter(df.age > 25).show()
10. How do you select specific columns?
from pyspark.sql.functions import col
df.select(col("name"), col("age")).show()
Intermediate PySpark Interview Questions
11. What are the types of joins in PySpark?
PySpark supports: inner, left, right, full, cross, semi, and anti joins.
df1.join(df2, df1.id == df2.id, "inner").show()
12. What is a broadcast join?
A broadcast join sends the smaller DataFrame to all worker nodes, avoiding expensive shuffles.
from pyspark.sql.functions import broadcast
df1.join(broadcast(df2), "id").show()
13. What is a shuffle in Spark?
A shuffle is the process of redistributing data across partitions. It is expensive because it involves disk I/O and network transfer.
14. How do you handle null values in PySpark?
df.dropna().show()
df.fillna({"age": 0, "name": "Unknown"}).show()
df.filter(df.age.isNull()).show()
15. What is the difference between cache() and persist()?
cache() uses default storage level MEMORY_AND_DISK. persist() allows you to specify the storage level explicitly.
16. What is partitioning in PySpark?
Partitioning divides data into chunks distributed across the cluster. Use repartition() to increase and coalesce() to decrease partitions.
17. What is the difference between repartition() and coalesce()?
| repartition() | coalesce() | |
|---|---|---|
| Shuffles data | Yes | No |
| Use case | Increase partitions | Decrease partitions |
| Performance | Slower | Faster |
18. What are window functions in PySpark?
from pyspark.sql.window import Window
from pyspark.sql.functions import rank
window = Window.partitionBy("dept").orderBy("salary")
df.withColumn("rank", rank().over(window)).show()
19. How do you write a DataFrame to a file?
df.write.csv("output/", header=True)
df.write.parquet("output/")
df.write.json("output/")
20. What is schema inference?
When inferSchema=True, Spark automatically detects column data types by scanning the data.
Advanced PySpark Interview Questions
21. What is the Catalyst Optimizer?
Catalyst is Spark SQLs query optimization engine. It converts your code into an optimized execution plan through four phases: analysis, logical optimization, physical planning, and code generation.
22. What is Tungsten execution engine?
Tungsten optimizes CPU and memory usage using binary encoding instead of Java objects, reducing memory overhead and GC pressure.
23. What is data skew and how do you handle it?
Data skew happens when some partitions have significantly more data than others. Solutions: salting technique, broadcast joins, repartitioning, Adaptive Query Execution.
24. What is Adaptive Query Execution?
spark.conf.set("spark.sql.adaptive.enabled", "true")
25. What is Delta Lake?
Delta Lake is an open-source storage layer that brings ACID transactions to Apache Spark. It supports time travel, schema enforcement, and upserts.
Practice on PySparkLab
Practice all these questions interactively at PySparkLab — no setup required.