Skip to content

Commit

Permalink
Update notebook
Browse files Browse the repository at this point in the history
  • Loading branch information
lmartinfuertes committed Apr 9, 2024
1 parent f3a00b6 commit 1dca955
Showing 1 changed file with 85 additions and 72 deletions.
157 changes: 85 additions & 72 deletions docs/source/notebooks/01_promptmeteo_usage.ipynb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,13 @@
"id": "b559000b-5af7-49ed-8e0a-c9c4765167f0",
"metadata": {},
"source": [
"# Promptmeteo Usage - Save and Load Model"
"# Promptmeteo Usage - Save and Load Model\n",
"\n",
"LLMs are truly revolutionizing the world, enabling humans to do things we couldn't do before, or making them so much easier and faster to do.\n",
"\n",
"Promptmeteo leverages the power of LLMs to democratize the data science process. This means you can easily train a model and make predictions with it in a few, comfortable steps.\n",
"\n",
"In this tutorial, you are going to create a model to perform document classification (a classic NLP task), train it with a manageable amount of data, save it, load it and use it to classify new documents."
]
},
{
Expand All @@ -22,12 +28,17 @@
"metadata": {},
"source": [
"### 1.1 Instance a DocumentClassifier\n",
"In this example, we are going to create a model that performs a classic NLP task: document classification. It's one of the tasks inPromptmeteo has a specific task to do that"
"Luckily, Promptmeteo has a specific task to do document classification. You only have to instance the class DocumentClassifier, and it's very easy to do so. You have to set:\n",
"- `language`: the language in which the task is going to be performed (currently, only English or Spanish are supported)\n",
"- `model_name` and `model_provider_name`: the model and provider you want to use (currently, [these](https://paradigmadigital.github.io/promptmeteo-docs/definition.html#available-models) are supported)\n",
"- `prompt_labels`: the categories in which you need to classify the documents (free choice)\n",
"\n",
"Let's use `google/flan-t5-small`, which is free, in English, and go for the classic sentiment classification, that is, using the `positive` and `negative` labels to classify opinions."
]
},
{
"cell_type": "code",
"execution_count": 1,
"execution_count": 21,
"id": "4106fe4b-4ee3-4763-a54a-7dcfe9febc89",
"metadata": {},
"outputs": [],
Expand All @@ -48,13 +59,15 @@
"id": "fd0d8729-635c-4ee9-b59f-d3a9646cf253",
"metadata": {},
"source": [
"### 1.2 Prompt Templates\n",
"First, we need to write a helper function to print YAML files in a proper way."
"### 1.2 Prompts\n",
"The first thing you need to know about Promptmeteo is that we think **prompts should be treated with the same respect as code**. Therefore, they shouldn't be hard-coded or taken lightly; instead, they should be carefully designed and properly versioned. In the end, they are the way we communicate with the LLMs. That means a mistake in a prompt can result in major unwanted outputs, while a really good prompt can unleash all the power from the LLM.\n",
"\n",
"That said, prompt templates in Promptmeteo are written in YAML and saved in files. So first, we need to write a helper function to print YAML files in a proper way."
]
},
{
"cell_type": "code",
"execution_count": 2,
"execution_count": 22,
"id": "5c6baaf4",
"metadata": {},
"outputs": [],
Expand All @@ -74,20 +87,22 @@
"id": "c7bed89f-1fea-4d23-a56a-1076990147c4",
"metadata": {},
"source": [
"#### 1.2.1 Predefined prompt templates"
"#### 1.2.1 Prompt templates"
]
},
{
"cell_type": "markdown",
"id": "e533b297-de29-4e55-8404-1d58c424db5f",
"metadata": {},
"source": [
"Promptmeteo has predefined prompt templates for each of the availables languages, models and tasks. This allows the user not to wonder about defining the prefect prompt but rather, to parametrize the prompt with the parameters of the use case. For the document classification task in english with OpenAI, the prompt template is:"
"Promptmeteo has predefined **prompt templates** for each of the available languages, models and tasks. This allows the user not to wonder about defining the perfect prompt, but rather to parametrize the prompt template with the parameters of the use case, without neglecting the prompts.\n",
"\n",
"Let's have a look into the prompt template for the document classification task in English with FlanTL:"
]
},
{
"cell_type": "code",
"execution_count": 3,
"execution_count": 23,
"id": "29a6f79c-95ca-4fe9-a945-6c8379888d64",
"metadata": {},
"outputs": [
Expand All @@ -97,13 +112,22 @@
"text": [
"TEMPLATE \n",
" I need you to help me with a text classification task. {__PROMPT_DOMAIN__} {__PROMPT_LABELS__}\n",
"{__CHAIN_THOUGHT__} {__ANSWER_FORMAT__}\n",
"{__CHAIN_THOUGHT__} {__ANSWER_FORMAT__} {__SHOT_EXAMPLES__} {__PROMPT_SAMPLE__}\n",
"PROMPT_DOMAIN \n",
" The texts you will be processing are from the {__DOMAIN__} domain.\n",
"PROMPT_LABELS \n",
" I want you to classify the texts into one of the following categories: {__LABELS__}.\n",
"PROMPT_DETAIL \n",
" \n",
"SHOT_EXAMPLES \n",
" Examples:\n",
"\n",
"{__EXAMPLES__}\n",
"PROMPT_SAMPLE \n",
" \n",
"\n",
"{__SAMPLE__}\n",
"\n",
"CHAIN_THOUGHT \n",
" \n",
"ANSWER_FORMAT \n",
Expand All @@ -128,67 +152,47 @@
"id": "744b81ee-2bac-4e20-9860-9670fddf1b55",
"metadata": {},
"source": [
"#### 1.2.2 Prompt text - Prompt injection"
"#### 1.2.2 Prompt texts"
]
},
{
"cell_type": "markdown",
"id": "a45d81af-4601-4366-94f5-09bc9039890b",
"metadata": {},
"source": [
"This template is used to build a prompt text by adding the stament bellow into the template. This technique is called **prompt injection**"
"The prompt template is used to build a **prompt text**, which is the final text to pass to the LLM. We haven't provided any example yet, only the labels, so it looks like this:"
]
},
{
"cell_type": "code",
"execution_count": 4,
"execution_count": 50,
"id": "a89220b2-de0d-48a3-9f80-d624f5d877d9",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"I need you to help me with a text classification task. I want you to classify the texts into one of the following categories \n",
" positive, negative. In your response, include only the name of the class predicted.\n"
"I need you to help me with a text classification task. The texts you will be processing are from the domain. I want you to classify the texts into one of the following categories: positive, negative.\n",
" In your response, include only the name of the class predicted. Examples:\n",
"\n",
"{__EXAMPLES__} \n",
"\n",
"{__SAMPLE__}\n",
"\n"
]
}
],
"source": [
"prompt_print(clf.task.prompt.template)"
]
},
{
"cell_type": "markdown",
"id": "3d9870f4-f36d-490c-86cc-e7fa7bba1165",
"metadata": {},
"source": [
" "
]
},
{
"cell_type": "markdown",
"id": "17cf474a-1003-477b-8f92-2cc898a3b7b3",
"metadata": {},
"source": [
"#### 1.2.3 Injecting labels - Zero-shot Prompting"
"print(clf.task.prompt.template)"
]
},
{
"cell_type": "markdown",
"id": "81a5bf76-0eb0-442e-9f45-77472a7acb70",
"metadata": {},
"source": [
"We can see tha not only the prompt has beenn build correctly, but also the labels `['positive', 'negative']` from the class inicialitations has been injected. Given the labels without more examples is called **Zero-shot prompting**"
]
},
{
"cell_type": "markdown",
"id": "00571ba9-d404-4342-bb7d-138823f1216f",
"metadata": {},
"source": [
"I need you to help me with a text classification task. I want you to classify the texts into one of the following categories: **positive, negative.**\n",
"Please provide a step-by-step argument for your answer, explain why you believe your final choice is justified, and make sure to conclude your explanation with the name of the class you have selected as the correct one, in lowercase and without punctuation. In your response, include only the name of the class as a single word, in lowercase, without punctuation, and without adding any other statements or words.\n"
"We could use the above text as the prompt as is, it's a valid approach called **zero-shot prompting**. But usually we will need to show the LLM some examples of what we want it to do; and that means we need to train it."
]
},
{
Expand Down Expand Up @@ -228,20 +232,20 @@
"id": "65ba3d55-1061-43b7-acfd-13ff8d83370f",
"metadata": {},
"source": [
"### 2.1 Train function"
"### 2.1 `train()` function"
]
},
{
"cell_type": "markdown",
"id": "0707a324-78a1-4b58-b318-5f2808eb3620",
"metadata": {},
"source": [
"The model can be trained by giving examples to the function `train()`"
"As in any data science pipeline, you can train models in Promptmeteo. You simply have to give examples to it using the `train()` function, providing the texts (`examples`) and their expected classification (`annotations`):"
]
},
{
"cell_type": "code",
"execution_count": 5,
"execution_count": 25,
"id": "dc3141d3-8c0b-4dac-a472-cccd4c3d4540",
"metadata": {},
"outputs": [],
Expand All @@ -265,46 +269,49 @@
"id": "c2df6a6d-4d38-4e42-a045-22d7e463a224",
"metadata": {},
"source": [
"### 2.2 Examples Injection - Few-shot prompting"
"### 2.2 Examples injection"
]
},
{
"cell_type": "markdown",
"id": "1591ccbb-25ab-4610-9f9d-9e4a6313cac2",
"metadata": {},
"source": [
"When a new sample cames, the examples are added to the prompt to help the model to improve the answers. This technique is called **Few-show prompting**. This examples are choosed rom those that are more related to the new sample passes for makein inference. We can see that now, the prompt with the examples has the following aspect"
"Each example is added to the prompt to help the model improve the answers. When the number of examples is low, this technique is called **few-shot prompting**.\n",
"\n",
"These examples should be chosen from those that are more related to the new sample passed for making inference. We can see that now the prompt with the examples has the following aspect:"
]
},
{
"cell_type": "code",
"execution_count": 6,
"execution_count": 52,
"id": "092ffb73-2bc8-42db-afb1-ca885c75962c",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
" I need you to help me with a text classification task. I want you to classify the texts into one of the following categories: positive, negative.\n",
" In your response, include only the name of the class predicted.\n",
"I need you to help me with a text classification task. The texts you will be processing are from the domain. I want you to classify the texts into one of the following categories: positive, negative.\n",
" In your response, include only the name of the class predicted. Examples:\n",
"\n",
"INPUT: I like it\n",
"OUTPUT: positive\n",
"I like it\n",
"positive\n",
"\n",
"INPUT: I hate it\n",
"OUTPUT: negative\n",
"I hate it\n",
"negative\n",
"\n",
"INPUT: i am happy\n",
"OUTPUT: positive\n",
"i am happy\n",
"positive \n",
"\n",
"INPUT: NEW EXAMPLE\n",
"OUTPUT: \n"
"I love it\n",
"\n"
]
}
],
"source": [
"print(clf.task._get_prompt('NEW EXAMPLE'))"
"example_for_inference = 'I love it'\n",
"print(clf.task._get_prompt(example_for_inference))"
]
},
{
Expand All @@ -320,30 +327,30 @@
"id": "f711ac62-309a-441c-adfb-aa009d0cd334",
"metadata": {},
"source": [
"### 2.3 Save Model - FAISS"
"### 2.3 Save model"
]
},
{
"cell_type": "markdown",
"id": "37d27b6a-3696-44d0-a234-7a36615ce524",
"metadata": {},
"source": [
"The examples given to the function train are saved in a vectorstore in local (with FAISS) and this vectorstore can be serialized to disk. Saving this examples in disk allow us to easily reuse it from new use cases without having to recover the original data again"
"The examples passed to the `train()` function are saved in a vectorstore in local (with FAISS) and this vectorstore can be serialized to disk. Saving these examples in disk allows us to easily reuse them for new use cases, without having to retrieve the original data again."
]
},
{
"cell_type": "code",
"execution_count": 7,
"execution_count": 27,
"id": "c748ded2-28a3-4e03-a813-f62e454f16c6",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"<promptmeteo.document_classifier.DocumentClassifier at 0x7f32142f4d50>"
"<promptmeteo.document_classifier.DocumentClassifier at 0x7ff4221a8ee0>"
]
},
"execution_count": 7,
"execution_count": 27,
"metadata": {},
"output_type": "execute_result"
}
Expand Down Expand Up @@ -397,12 +404,12 @@
"id": "f8e2c604-67c7-4834-8552-d69206762eb1",
"metadata": {},
"source": [
"For reloading a model we have to intanciate a `DocumentClassifier` and use the function `load_model()`"
"Now that we have saved a model, we can load it. To load a model we have to intantiate a `DocumentClassifier` as we did before and use the function `load_model()`:"
]
},
{
"cell_type": "code",
"execution_count": 14,
"execution_count": 28,
"id": "cae37b63-2780-4581-8dac-89a1919c0c64",
"metadata": {},
"outputs": [],
Expand Down Expand Up @@ -438,12 +445,12 @@
"id": "69571f3e-9e19-4956-bc41-2ae2aff2717e",
"metadata": {},
"source": [
"By calling the function `predict()` we can use the prompt created with the examples loaded again to predict the classification over new data"
"And now we are ready to predict labels for new data! By calling the function `predict()` we can use the prompt created with the examples to predict the classification over new data:"
]
},
{
"cell_type": "code",
"execution_count": 15,
"execution_count": 53,
"id": "298b74ce-20a8-44b3-b946-65a7000f35ee",
"metadata": {},
"outputs": [
Expand All @@ -453,7 +460,7 @@
"[['positive']]"
]
},
"execution_count": 15,
"execution_count": 53,
"metadata": {},
"output_type": "execute_result"
}
Expand Down Expand Up @@ -499,10 +506,16 @@
"id": "89aa2277-0785-4143-9a88-d7cd7be528ea",
"metadata": {},
"source": [
"* In this example we have shown how Promptmeteo can be used as a machine learning frameworks such as Scikit-Learn o Pytorch. It has a similar interface with allows to save the results from the training in a binary file and reuse it. This eases the integration of LLM solution in ML pipeline tools such as Sagemaker or Vertex.\n",
"* In this example we have shown how Promptmeteo can be used as a machine learning framework such as Scikit-Learn or Pytorch. It has a similar interface which allows to save the results from the training in a binary file and reuse it. This eases the integration of the LLM solution in ML pipeline tools such as Sagemaker or Vertex.\n",
"\n",
"* Promptmeteo does not only include code data to simplify the integration to LLM model and services. It also include predefined Prompt Engineer logic for different models and tasks. It allows to focus on developing a solution rather than writting prompts, and ensures that the prompt has been tested correctly by Promptmeteo, which make this kind os solution less error-prone."
"* Promptmeteo does not only include code data to simplify the integration of LLM model and services. It also includes predefined prompt engineering logic for different models and tasks. It allows to focus on developing a solution rather than writting prompts, and ensures that the prompt has been tested correctly by Promptmeteo, which makes this kind of solution less error-prone."
]
},
{
"cell_type": "markdown",
"id": "dccfd4e3",
"metadata": {},
"source": []
}
],
"metadata": {
Expand All @@ -521,7 +534,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.6"
"version": "3.9.5"
}
},
"nbformat": 4,
Expand Down

0 comments on commit 1dca955

Please sign in to comment.