Skip to content

Commit 400c041

Browse files
jruauxjruaux
committed
docs: Updated notebook
1 parent e390dda commit 400c041

File tree

2 files changed

+276
-77
lines changed

2 files changed

+276
-77
lines changed

‎redis-spark-notebook.ipynb

Lines changed: 276 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,276 @@
1+
{
2+
"cells": [
3+
{
4+
"cell_type": "markdown",
5+
"metadata": {
6+
"application/vnd.databricks.v1+cell": {
7+
"cellMetadata": {},
8+
"inputWidgets": {},
9+
"nuid": "08b3c927-0b10-42d3-823b-b04dd2def3ac",
10+
"showTitle": false,
11+
"tableResultSettingsMap": {},
12+
"title": ""
13+
}
14+
},
15+
"source": [
16+
"# Connect Spark with Redis Cloud\n",
17+
"Spark and Redis together unlock powerful capabilities for data professionals. This guide demonstrates how to integrate these technologies for enhanced analytics, real-time processing, and machine learning applications.\n",
18+
"\n",
19+
"In this hands-on notebook, you'll learn how to make efficient use of Redis data structures alongside Spark's distributed computing framework. You'll see firsthand how to extract data from Redis, process it in Spark, and write results back to Redis for application use. Key topics include:\n",
20+
"1. Setting up the Spark-Redis connector in Databricks\n",
21+
"2. Writing data to Redis from Spark\n",
22+
"3. Reading data from Redis for application access\n",
23+
"\n",
24+
"## Databricks Cluster Setup with Redis Connector\n",
25+
"\n",
26+
"1. Set up a new Databricks cluster\n",
27+
"1. Go to the cluster's **Libraries** section\n",
28+
"1. Select **Install New**\n",
29+
"1. Choose **Maven** as your source and click **Search Packages**\n",
30+
"1. Enter `redis-spark-connector` and select `com.redis:redis-spark-connector:x.y.z`\n",
31+
"1. Finalize by clicking **Install** <br/>\n",
32+
"Want to explore the connector's full capabilities? Check the [detailed documentation](https://redis-field-engineering.github.io/redis-spark)\n"
33+
]
34+
},
35+
{
36+
"cell_type": "markdown",
37+
"metadata": {
38+
"application/vnd.databricks.v1+cell": {
39+
"cellMetadata": {},
40+
"inputWidgets": {},
41+
"nuid": "aafb8639-2a2f-4c0c-911d-aa4eec118608",
42+
"showTitle": false,
43+
"tableResultSettingsMap": {},
44+
"title": ""
45+
}
46+
},
47+
"source": [
48+
"## Setting Up Redis Cloud Environment\n",
49+
"\n",
50+
"Redis Cloud offers a fully-managed Redis service ideal for this integration. Follow these steps:\n",
51+
"\n",
52+
"1. Register for a [Redis Cloud account](https://redis.io/cloud/)\n",
53+
"1. Follow the [quickstart guide](https://redis.io/docs/latest/operate/rc/rc-quickstart/) to create a free tier database\n",
54+
"1. From your Redis Cloud database dashboard in the Configuration tab click on Import. In the dialog window select Google Cloud Storage as the source type, paste the following in the source path: [gs://jrx/nobels.rdb](gs://jrx/nobels.rdb), and finally click Import.\n",
55+
"1. Examine the keys you've imported using **Redis Insight**, Redis' visual data browser. From your Redis Cloud database dashboard, click on **Redis Insight** and explore the data we imported in the previous step. \n",
56+
"\n",
57+
"## Configuring Spark with Redis Connection Details\n",
58+
"\n",
59+
"1. From your Redis Cloud database dashboard, find your connection endpoint under **Connect**. The string follows this pattern: `redis://<user>:<pass>@<host>:<port>`\n",
60+
"1. In Databricks, open your cluster settings and locate **Advanced Options**. Under **Spark** in the **Spark config** text area, add your Redis connection string as both `spark.redis.read.connection.uri redis://...` and `spark.redis.write.connection.uri redis://...` parameters. This configuration applies to all notebooks using this cluster. Note that it is recommended to use secrets to store sensitive Redis URIs. Refer to the [Redis Spark documentation](https://redis-field-engineering.github.io/redis-spark/#_databricks) for more details.\n",
61+
"\n",
62+
"\n"
63+
]
64+
},
65+
{
66+
"cell_type": "markdown",
67+
"metadata": {
68+
"application/vnd.databricks.v1+cell": {
69+
"cellMetadata": {},
70+
"inputWidgets": {},
71+
"nuid": "6a73287f-3746-4126-be49-24931838a670",
72+
"showTitle": false,
73+
"tableResultSettingsMap": {},
74+
"title": ""
75+
}
76+
},
77+
"source": [
78+
"## Reading from Redis\n",
79+
"\n",
80+
"To read data from Redis run the following code:\n"
81+
]
82+
},
83+
{
84+
"cell_type": "code",
85+
"execution_count": 0,
86+
"metadata": {
87+
"application/vnd.databricks.v1+cell": {
88+
"cellMetadata": {
89+
"byteLimit": 2048000,
90+
"rowLimit": 10000
91+
},
92+
"inputWidgets": {},
93+
"nuid": "574c93b6-8c31-4f4e-b5f8-e9414845c71c",
94+
"showTitle": false,
95+
"tableResultSettingsMap": {},
96+
"title": ""
97+
}
98+
},
99+
"outputs": [],
100+
"source": [
101+
"df = spark.read.format(\"redis\").option(\"keyType\", \"hash\").load()\n",
102+
"display(df)"
103+
]
104+
},
105+
{
106+
"cell_type": "markdown",
107+
"metadata": {
108+
"application/vnd.databricks.v1+cell": {
109+
"cellMetadata": {},
110+
"inputWidgets": {},
111+
"nuid": "caf94ef3-5337-401f-b2d2-a43ff58b8e22",
112+
"showTitle": false,
113+
"tableResultSettingsMap": {},
114+
"title": ""
115+
}
116+
},
117+
"source": [
118+
"## Reading from Redis in Streaming Mode\n",
119+
"\n",
120+
"The following code reads data from a Redis stream, appending data to a streaming in-memory dataframe."
121+
]
122+
},
123+
{
124+
"cell_type": "code",
125+
"execution_count": 0,
126+
"metadata": {
127+
"application/vnd.databricks.v1+cell": {
128+
"cellMetadata": {
129+
"byteLimit": 2048000,
130+
"rowLimit": 10000
131+
},
132+
"inputWidgets": {},
133+
"nuid": "b72cc53e-9c7d-4e10-a828-83a98b1a9021",
134+
"showTitle": false,
135+
"tableResultSettingsMap": {},
136+
"title": ""
137+
}
138+
},
139+
"outputs": [],
140+
"source": [
141+
"import time\n",
142+
"streamDf = spark.readStream.format(\"redis\").option(\"type\", \"stream\").option(\"streamKey\", \"stream:nobels\").load()\n",
143+
"query = streamDf.writeStream.format(\"memory\").trigger(continuous=\"1 second\").queryName(\"nobels\").start()\n",
144+
"time.sleep(3)\n",
145+
"streamDs = spark.sql(\"select * from nobels\")\n",
146+
"display(streamDs)"
147+
]
148+
},
149+
{
150+
"cell_type": "markdown",
151+
"metadata": {
152+
"application/vnd.databricks.v1+cell": {
153+
"cellMetadata": {},
154+
"inputWidgets": {},
155+
"nuid": "63658b45-2684-440f-9d2d-b0d84bb4af8f",
156+
"showTitle": false,
157+
"tableResultSettingsMap": {},
158+
"title": ""
159+
}
160+
},
161+
"source": [
162+
"## Writing to Redis\n",
163+
"\n",
164+
"1. Let's use the `df` data we imported earlier and write it back to Redis as JSON.\n",
165+
"1. Refresh **Redis Insight** and notice the new JSON keys prefixed with `spark:write`."
166+
]
167+
},
168+
{
169+
"cell_type": "code",
170+
"execution_count": 0,
171+
"metadata": {
172+
"application/vnd.databricks.v1+cell": {
173+
"cellMetadata": {
174+
"byteLimit": 2048000,
175+
"rowLimit": 10000
176+
},
177+
"inputWidgets": {},
178+
"nuid": "37786ad3-b0ec-49a9-8839-75ff907e4cae",
179+
"showTitle": false,
180+
"tableResultSettingsMap": {},
181+
"title": ""
182+
}
183+
},
184+
"outputs": [],
185+
"source": [
186+
"df.write.format(\"redis\").option(\"type\", \"json\").option(\"keyspace\", \"spark:write:nobel\").option(\"key\", \"id\").mode(\"append\").save()"
187+
]
188+
},
189+
{
190+
"cell_type": "markdown",
191+
"metadata": {
192+
"application/vnd.databricks.v1+cell": {
193+
"cellMetadata": {},
194+
"inputWidgets": {},
195+
"nuid": "f6ba51ee-705b-4e92-89e6-8530978ba987",
196+
"showTitle": false,
197+
"tableResultSettingsMap": {},
198+
"title": ""
199+
}
200+
},
201+
"source": [
202+
"## Writing to Redis in Streaming Mode\n",
203+
"\n",
204+
"We can also write to Redis in streaming mode.\n",
205+
"1. Replace `<catalog>`, `<schema>`, and `<volume>` with names for a Unity Catalog volume and run the code below.\n",
206+
"2. In **Redis Insight** refresh the database and notice the new hash keys prefixed with `spark:writeStream`\n"
207+
]
208+
},
209+
{
210+
"cell_type": "code",
211+
"execution_count": 0,
212+
"metadata": {
213+
"application/vnd.databricks.v1+cell": {
214+
"cellMetadata": {
215+
"byteLimit": 2048000,
216+
"rowLimit": 10000
217+
},
218+
"inputWidgets": {},
219+
"nuid": "51a55f3b-8912-417d-8f78-57aad0cf86ca",
220+
"showTitle": false,
221+
"tableResultSettingsMap": {},
222+
"title": ""
223+
}
224+
},
225+
"outputs": [
226+
{
227+
"data": {
228+
"text/plain": [
229+
"<pyspark.sql.streaming.query.StreamingQuery at 0x7fd64dc7a4e0>"
230+
]
231+
},
232+
"execution_count": 78,
233+
"metadata": {},
234+
"output_type": "execute_result"
235+
}
236+
],
237+
"source": [
238+
"catalog = \"<catalog>\"\n",
239+
"schema = \"<schema>\"\n",
240+
"volume = \"<volume>\"\n",
241+
"\n",
242+
"path_volume = f\"/Volumes/{catalog}/{schema}/{volume}\"\n",
243+
"dbutils.fs.cp(\"http://storage.googleapis.com/jrx/nobels.jsonl\", f\"{path_volume}/nobels.json\")\n",
244+
"\n",
245+
"checkpoint_dir = f\"{path_volume}/checkpoint\"\n",
246+
"dbutils.fs.mkdirs(checkpoint_dir)\n",
247+
"\n",
248+
"spark.readStream.schema(\"id INT, year INT, category STRING, share INT, firstName STRING, lastName STRING, motivation STRING\").json(f\"{path_volume}/*.json\") \\\n",
249+
" .writeStream.format(\"redis\").outputMode(\"append\") \\\n",
250+
" .option(\"type\", \"hash\") \\\n",
251+
" .option(\"keyspace\", \"spark:writeStream:nobel\") \\\n",
252+
" .option(\"key\", \"id\") \\\n",
253+
" .option(\"checkpointLocation\", checkpoint_dir) \\\n",
254+
" .start()\n"
255+
]
256+
}
257+
],
258+
"metadata": {
259+
"application/vnd.databricks.v1+notebook": {
260+
"computePreferences": null,
261+
"dashboards": [],
262+
"environmentMetadata": null,
263+
"language": "python",
264+
"notebookMetadata": {
265+
"pythonIndentUnit": 4
266+
},
267+
"notebookName": "redis-spark-notebook",
268+
"widgets": {}
269+
},
270+
"language_info": {
271+
"name": "python"
272+
}
273+
},
274+
"nbformat": 4,
275+
"nbformat_minor": 0
276+
}

‎redis-spark-notebook.py

Lines changed: 0 additions & 77 deletions
This file was deleted.

0 commit comments

Comments
 (0)