Joins in PySpark dataframe

 In PySpark, joining DataFrames is very similar to SQL joins, but you use the join() method on a DataFrame. Here’s a clear breakdown:

🔑 Types of Joins in PySpark

• Inner Join: Returns rows with matching keys in both DataFrames.

• Left Outer Join: Returns all rows from the left DataFrame, and matching rows from the right.

• Right Outer Join: Returns all rows from the right DataFrame, and matching rows from the left.

• Full Outer Join: Returns all rows when there is a match in either DataFrame.

• Cross Join: Cartesian product of both DataFrames.

• Semi Join: Returns rows from the left DataFrame where keys exist in the right DataFrame.

• Anti Join: Returns rows from the left DataFrame where keys do not exist in the right DataFrame.

⚙️ Syntax Example

python

# Assume df1 and df2 are DataFrames
# Join on column "id"

# Inner Join
result = df1.join(df2, df1.id == df2.id, "inner")

# Left Outer Join
result = df1.join(df2, df1.id == df2.id, "left")

# Right Outer Join
result = df1.join(df2, df1.id == df2.id, "right")

# Full Outer Join
result = df1.join(df2, df1.id == df2.id, "outer")

📌 Notes

• You can join on multiple columns:

  python

  result = df1.join(df2, (df1.id == df2.id) & (df1.name == df2.name), "inner")
  

• If the column names are the same in both DataFrames, you can pass a string instead of an expression:

  python

  result = df1.join(df2, "id", "inner")
  

• After joining, you may want to drop duplicate columns using .drop() or rename them with .withColumnRenamed().

👉 Do you want me to show you a practical example with sample data (like two small DataFrames and their join results), or a real-world use case (e.g., joining customer and order tables)?