This project is a training ground for learning and experimenting with Apache Spark, using the PySpark API. It is set up to be run locally, making it easy to test Spark applications and data processing logic.
These instructions will get you a copy of the project up and running on your local machine for development and testing purposes.
Before you begin, ensure you have the following installed on your system:
- Python 3.12+: This project is configured to use Python 3.12 or newer.
- Poetry: This project uses Poetry for dependency management.
- Java Development Kit (JDK): Apache Spark runs on the Java Virtual Machine (JVM), so a JDK (version 8, 11, or 17 is recommended) is required.
-
Clone the repository:
git clone <your-repository-url> cd spark-training
-
Install dependencies: Use Poetry to install the Python dependencies locked in
poetry.lock. This creates a virtual environment and installs packages likepysparkandjupyter.poetry install
The primary way to interact with this project is through Jupyter Lab, which provides an interactive environment for running Spark code.
-
Launch Jupyter Lab: Run the following command from the project's root directory:
poetry run jupyter lab
-
Open a Notebook: Once Jupyter Lab opens in your browser, you can create a new notebook or open an existing one to start writing and executing PySpark code.
When creating a SparkSession, you can specify various configuration properties to tune its behavior. A common example seen in this project is setting the number of shuffle partitions, which is crucial for performance, especially in a local setup.
from pyspark.sql import SparkSession
spark = SparkSession.builder \
.appName("SparkTraining") \
.config("spark.sql.shuffle.partitions", "8") \
.getOrCreate()This value is a good starting point for local development but should be tuned for larger datasets or production environments based on available CPU cores and memory.