Run All Environments using cocker-compose

Prerequisites

Install Docker

Getting Started

1. Download this repo, open a terminal, and navigate to its location.

2. Build and run the containers

docker-compose up -d

3. Check everything is ready:

docker ps # should show hadoop-playground and spark-playground up and running

Access Hadoop

docker exec -it hadoop-playground /bin/bash

Access Spark

docker exec -it spark-playground /bin/bash

Access Jupyter Notebook

1. Connect to the container:

docker exec -it jupyter-playground /bin/bash

2. Copy the server token:

jupyter server list
# Prints something like:
# http://a0542f50464f:8888/?token=d48af4be9f9c9ccc9226d2b69f689374097f1f26244b8f71

3. Copy the token and load in your browser localhost:8888/?token=d48af...

Cleaning Up

docker-compose down # Stop and remove containers

Just run Jupyter Notebook with a few command:

1. Install Docker

2. Start Docker Daemon (Or just open the Docker app)

3. Create a workbook location (e.g. on your desktop):

mkdir ~/Desktop/pyspark-notebook

4. Pull the Jupyter docker image

docker pull jupyter/pyspark-notebook

5. Runt the image, mapping ports and work folders:

docker run -p 8888:8888 -v ~/Desktop/pyspark-notebook:/home/jovyan/work -d --name notebook  jupyter/pyspark-notebook

6. Connect to the container and find the token

docker exec -it jupyter-playground /bin/bash
jupyter server list
# Prints something like:
# http://a0542f50464f:8888/?token=d48af4be9f9c9ccc9226d2b69f689374097f1f26244b8f71

7. Copy the token and load in your browser localhost:8888/?token=d48af...