Add instructlab-knowledge pipeline

notebooks/.gitignore

@@ -0,0 +1,18 @@
+# Created by https://www.toptal.com/developers/gitignore/api/jupyternotebooks
+# Edit at https://www.toptal.com/developers/gitignore?templates=jupyternotebooks
+
+### JupyterNotebooks ###
+# gitignore template for Jupyter Notebooks
+# website: http://jupyter.org/
+
+.ipynb_checkpoints
+*/.ipynb_checkpoints/*
+
+# IPython
+profile_default/
+ipython_config.py
+
+# Remove previous ipynb_checkpoints
+#   git rm -r .ipynb_checkpoints/
+
+# End of https://www.toptal.com/developers/gitignore/api/jupyternotebooks

notebooks/doc-preprocessing-to-sdg/chunking.ipynb → notebooks/instructlab-knowledge/chunking.ipynb

@@ -5,12 +5,20 @@
    "id": "7b33678f-67d2-48a1-801f-302622e43e0f",
    "metadata": {},
    "source": [
-    "## Goal\n",
+    "## Chunking\n",
     "The goal of chunking for InstructLab SDG is to provide the teacher model small and logical pieces of the source document to generate data off of.\n",
     "\n",
     "In this notebook we are doing chunking with Docling[https://docling-project.github.io/docling/examples/hybrid_chunking/#hybrid-chunking].\n",
     "\n",
-    "First let's ensure docling is installed."
+    "The input to this notebook is a docling JSON file created after a docling conversion, or a directory of docling JSON files."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "d9f268fd-35d2-4c7a-8cfa-47630de00837",
+   "metadata": {},
+   "source": [
+    "### Dependencies"
    ]
   },
   {
@@ -272,8 +280,7 @@
     "            c = dict(chunk=chunk, file=file.stem)\n",
     "            all_chunks.append(c)\n",
     "    except ConversionError as e:\n",
-    "        print(f\"Skipping file {file}\")\n",
-    "# print(all_chunks)"
+    "        print(f\"Skipping file {file}\")"
    ]
   },
   {
@@ -286,14 +293,26 @@
     "To view the chunks, run through the following cell. As you can see the document is broken into small pieces with metadata about the chunk based on the document's format"
    ]
   },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "ff88cf5c-1315-4eca-afcd-25706eaf7d6b",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# print(all_chunks)"
+   ]
+  },
   {
    "cell_type": "markdown",
    "id": "84826055-a7f1-4334-a12b-bbc07a523199",
    "metadata": {
     "tags": []
    },
    "source": [
-    "## Save the chunks to a text file each"
+    "## Save the chunks to a text file for each chunk\n",
+    "\n",
+    "Each chunk is saved to an individual text file in the format: `{docling-json-file-name}-{chunk #}.txt`. Having chunking in this format is important as an input to create-sdg-seed-data notebook."
    ]
   },
   {