diff --git a/notebook/problems.es.ipynb b/notebook/problems.es.ipynb index 1d259142..31e997a9 100644 --- a/notebook/problems.es.ipynb +++ b/notebook/problems.es.ipynb @@ -1 +1,130 @@ -{"cells":[{"cell_type":"markdown","id":"d9d036c6","metadata":{},"source":["# Problemas de prueba de hipótesis"]},{"cell_type":"markdown","id":"d157c4db","metadata":{},"source":["## Ejercicio 1\n","\n","Usted es un nutricionista que investiga dos tipos diferentes de dietas para ver si existe una diferencia significativa en la pérdida de peso después de un mes. Eliges dos grupos aleatorios de personas; un grupo sigue la primera dieta y el otro sigue la segunda. Al final del mes, se registra la pérdida de peso (en kg) de cada persona:\n","\n","| Diet 1 | Diet 2 |\n","|:-------|:-------|\n","| 2.0 | 3.0 |\n","| 2.5 | 3.2 |\n","| 3.0 | 3.1 |\n","| 2.8 | 2.9 |\n","| 2.3 | 2.8 |\n","| 2.7 | 3.0 |\n","| 2.5 | 3.2 |\n","\n","Con estos datos se busca responder a la siguiente pregunta: ¿Existe una diferencia significativa en la pérdida de peso promedio entre las personas que siguieron la primera dieta y las que siguieron la segunda dieta?\n","\n","Para sacar conclusiones, siga los siguientes puntos:\n","\n","-Enunciar la hipótesis: hipótesis nula y alternativa.\n","-Realizar la prueba para comprobar la hipótesis. Puede utilizar la prueba t de Student.\n","-Analizar las conclusiones."]},{"cell_type":"code","execution_count":1,"id":"8e0f3cf0","metadata":{},"outputs":[],"source":["# TODO"]},{"cell_type":"markdown","id":"f432e18e","metadata":{},"source":["## ANOVA\n","\n","**ANOVA** (*Analysis of Variance - Análisis de variación*) Es una técnica estadística utilizada para comparar las medidas de dos o más grupos. La idea detrás de ANOVA es descomponer la variabilidad total de los datos en dos componentes: variabilidad entre grupos y variabilidad dentro de los grupos:\n","\n","- **Variabilidad entre grupos**: Esta variabilidad se refiere a las diferencias entre las medias del grupo. Si esta variabilidad es considerablemente mayor que la variabilidad dentro del grupo, podría ser una indicación de que al menos una de las medias del grupo es diferente.\n","- **Variabilidad dentro del grupo**: Esta variabilidad se refiere a la dispersión de los datos dentro de cada grupo. Si todos los grupos tienen una variabilidad similar, entonces cualquier diferencia notable en las medias de los grupos podría considerarse significativa.\n","\n","Las hipótesis en ANOVA suelen incluir:\n","\n","- **Hipótesis nula** ($H₀$): Las medias de todos los grupos son iguales.\n","- **Hipótesis alternativa** ($H₁$): Al menos una de las medias del grupo es diferente.\n","\n","Si el resultado de la prueba ANOVA es significativo (por ejemplo, un valor p inferior a un umbral como 0,05), esto sugiere que al menos la media de un grupo es diferente."]},{"cell_type":"markdown","id":"fc5f3721","metadata":{},"source":["## Ejercicio 2\n","\n","Un agricultor decide probar tres tipos diferentes de fertilizantes para determinar si alguno es superior en términos de producción de maíz. El agricultor planta maíz en 15 parcelas idénticas y utiliza los tres fertilizantes (5 parcelas para cada tipo). Al final de la temporada mide el rendimiento de maíz (en kg) de cada parcela, con el siguiente resultado:\n","\n","| Fertilizante 1 | Fertilizante 2 | Fertilizante 3 |\n","|:-------------|:-------------|:-------------|\n","| 20 | 22 | 24 |\n","| 21 | 21 | 23 |\n","| 20 | 23 | 22 |\n","| 19 | 22 | 23 |\n","| 20 | 21 | 24 |\n","\n","Con estos datos busca responder la siguiente pregunta: ¿Existe una diferencia significativa en el rendimiento promedio del maíz entre los tres tipos de fertilizantes?\n","\n","Para ayudarte, sigue los siguientes puntos:\n","\n","-Enunciar la hipótesis: hipótesis nula y alternativa.\n","-Realizar la prueba ANOVA.\n","-Analizar las conclusiones.\n","-Si un fertilizante es mejor que otro, ¿cómo podemos saberlo?"]},{"cell_type":"code","execution_count":2,"id":"eecad0ca","metadata":{},"outputs":[],"source":["# TODO"]}],"metadata":{"interpreter":{"hash":"9248718ffe6ce6938b217e69dbcc175ea21f4c6b28a317e96c05334edae734bb"},"kernelspec":{"display_name":"Python 3.9.12 ('ML-BOOTCAMP')","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.11.4"}},"nbformat":4,"nbformat_minor":5} +{ + "cells": [ + { + "cell_type": "markdown", + "id": "d9d036c6", + "metadata": {}, + "source": [ + "# Problemas de prueba de hipótesis" + ] + }, + { + "cell_type": "markdown", + "id": "d157c4db", + "metadata": {}, + "source": [ + "## Ejercicio 1\n", + "\n", + "Usted es un nutricionista que investiga dos tipos diferentes de dietas para ver si existe una diferencia significativa en la pérdida de peso después de un mes. Eliges dos grupos aleatorios de personas; un grupo sigue la primera dieta y el otro sigue la segunda. Al final del mes, se registra la pérdida de peso (en kg) de cada persona:\n", + "\n", + "| Diet 1 | Diet 2 |\n", + "|:-------|:-------|\n", + "| 2.0 | 3.0 |\n", + "| 2.5 | 3.2 |\n", + "| 3.0 | 3.1 |\n", + "| 2.8 | 2.9 |\n", + "| 2.3 | 2.8 |\n", + "| 2.7 | 3.0 |\n", + "| 2.5 | 3.2 |\n", + "\n", + "Con estos datos se busca responder a la siguiente pregunta: ¿Existe una diferencia significativa en la pérdida de peso promedio entre las personas que siguieron la primera dieta y las que siguieron la segunda dieta?\n", + "\n", + "Para sacar conclusiones, siga los siguientes puntos:\n", + "\n", + "- Enunciar la hipótesis: hipótesis nula ($H₀$) y alternativa($H₁$).\n", + "- Verifica los supuestos: Antes de realizar la prueba, menciona los supuestos que deben cumplirse para poder aplicar la prueba t de Student de forma válida (independencia, normalidad y homogeneidad de varianzas).\n", + "- Realizar la prueba para comprobar la hipótesis. Puede utilizar la prueba t de Student.\n", + "- Analizar las conclusiones: Basándote en el p-valor, decide si rechazas o no la hipótesis nula y explica lo que esto significa en el contexto del problema." + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "id": "8e0f3cf0", + "metadata": {}, + "outputs": [], + "source": [ + "# TODO" + ] + }, + { + "cell_type": "markdown", + "id": "f432e18e", + "metadata": {}, + "source": [ + "## ANOVA\n", + "\n", + "**ANOVA** (*Analysis of Variance - Análisis de variación*) Es una técnica estadística utilizada para comparar las medidas de dos o más grupos. La idea detrás de ANOVA es descomponer la variabilidad total de los datos en dos componentes: variabilidad entre grupos y variabilidad dentro de los grupos:\n", + "\n", + "- **Variabilidad entre grupos**: Esta variabilidad se refiere a las diferencias entre las medias del grupo. Si esta variabilidad es considerablemente mayor que la variabilidad dentro del grupo, podría ser una indicación de que al menos una de las medias del grupo es diferente.\n", + "- **Variabilidad dentro del grupo**: Esta variabilidad se refiere a la dispersión de los datos dentro de cada grupo. Si todos los grupos tienen una variabilidad similar, entonces cualquier diferencia notable en las medias de los grupos podría considerarse significativa.\n", + "\n", + "Las hipótesis en ANOVA suelen incluir:\n", + "\n", + "- **Hipótesis nula** ($H₀$): Las medias de todos los grupos son iguales.\n", + "- **Hipótesis alternativa** ($H₁$): Al menos una de las medias del grupo es diferente.\n", + "\n", + "Si el resultado de la prueba ANOVA es significativo (por ejemplo, un valor p inferior a un umbral como 0,05), esto sugiere que al menos la media de un grupo es diferente." + ] + }, + { + "cell_type": "markdown", + "id": "fc5f3721", + "metadata": {}, + "source": [ + "## Ejercicio 2\n", + "\n", + "Un agricultor decide probar tres tipos diferentes de fertilizantes para determinar si alguno es superior en términos de producción de maíz. El agricultor planta maíz en 15 parcelas idénticas y utiliza los tres fertilizantes (5 parcelas para cada tipo). Al final de la temporada mide el rendimiento de maíz (en kg) de cada parcela, con el siguiente resultado:\n", + "\n", + "| Fertilizante 1 | Fertilizante 2 | Fertilizante 3 |\n", + "|:-------------|:-------------|:-------------|\n", + "| 20 | 22 | 24 |\n", + "| 21 | 21 | 23 |\n", + "| 20 | 23 | 22 |\n", + "| 19 | 22 | 23 |\n", + "| 20 | 21 | 24 |\n", + "\n", + "Con estos datos busca responder la siguiente pregunta: ¿Existe una diferencia significativa en el rendimiento promedio del maíz entre los tres tipos de fertilizantes?\n", + "\n", + "Para ayudarte, sigue los siguientes puntos:\n", + "\n", + "- Enunciar la hipótesis: hipótesis nula y alternativa.\n", + "- Verifica los supuestos: Menciona los supuestos que deben cumplirse para poder aplicar el ANOVA.\n", + "- Realizar la prueba ANOVA.\n", + "- Analizar las conclusiones:Basándote en el p-valor, decide si rechazas o no la hipótesis nula y explica el significado del resultado del ANOVA.\n", + "- Prueba post-hoc: Si la prueba ANOVA indica una diferencia significativa, ¿cómo puedes saber cuál fertilizante es mejor? Realiza un test post-hoc (como la prueba de Tukey HSD) y explica su conclusión." + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "id": "eecad0ca", + "metadata": {}, + "outputs": [], + "source": [ + "# TODO" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.11.4" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/notebook/problems.ipynb b/notebook/problems.ipynb index 117f1ef5..ac1e2942 100644 --- a/notebook/problems.ipynb +++ b/notebook/problems.ipynb @@ -1 +1,130 @@ -{"cells":[{"cell_type":"markdown","id":"d9d036c6","metadata":{},"source":["# Hypothesis testing problems"]},{"cell_type":"markdown","id":"d157c4db","metadata":{},"source":["## Exercise 1\n","\n","You are a nutritionist researching two different types of diets to see if there is a significant difference in weight loss after one month. You choose two random groups of people; one group follows the first diet, and the other follows the second. At the end of the month, you record the weight loss (in kg) for each person:\n","\n","| Diet 1 | Diet 2 |\n","|:-------|:-------|\n","| 2.0 | 3.0 |\n","| 2.5 | 3.2 |\n","| 3.0 | 3.1 |\n","| 2.8 | 2.9 |\n","| 2.3 | 2.8 |\n","| 2.7 | 3.0 |\n","| 2.5 | 3.2 |\n","\n","With these data, it seeks to answer the following question: Is there a significant difference in average weight loss between people who followed the first diet and those who followed the second diet?\n","\n","To draw conclusions, follow the points below:\n","\n","- State the hypothesis: null and alternative hypothesis.\n","- Perform the test to test the hypothesis. You can use a Student's t-test.\n","- Analyze the conclusions."]},{"cell_type":"code","execution_count":1,"id":"8e0f3cf0","metadata":{},"outputs":[],"source":["# TODO"]},{"cell_type":"markdown","id":"f432e18e","metadata":{},"source":["## ANOVA\n","\n","**ANOVA** (*Analysis of Variance*) is a statistical technique used to compare the measures of two or more groups. The idea behind ANOVA is to decompose the total variability in the data into two components: between-group variability and within-group variability:\n","\n","- **Between-group variability**: This variability refers to the differences between the group means. If this variability is considerably larger than the within-group variability, it could be an indication that at least one of the group means is different.\n","- **Within-group variability**: This variability refers to the dispersion of the data within each group. If all groups have similar variability, then any noticeable difference in group means could be considered significant.\n","\n","Hypotheses in ANOVA typically include:\n","\n","- **Null hypothesis** ($H₀$): The means of all groups are equal.\n","- **Alternative hypothesis** ($H₁$): At least one of the group means is different.\n","\n","If the ANOVA test result is significant (e.g., a p-value less than a threshold such as 0.05), this suggests that at least one group mean is different."]},{"cell_type":"markdown","id":"fc5f3721","metadata":{},"source":["## Exercise 2\n","\n","A farmer decides to test three different types of fertilizers to determine if one is superior in terms of corn production. The farmer plants corn on 15 identical plots and uses all three fertilizers (5 plots for each type). At the end of the season, he measures the corn yield (in kg) of each plot, with the following result:\n","\n","| Fertilizer 1 | Fertilizer 2 | Fertilizer 3 |\n","|:-------------|:-------------|:-------------|\n","| 20 | 22 | 24 |\n","| 21 | 21 | 23 |\n","| 20 | 23 | 22 |\n","| 19 | 22 | 23 |\n","| 20 | 21 | 24 |\n","\n","With this data, he seeks to answer the following question: Is there a significant difference in average corn yield between the three types of fertilizers?\n","\n","To help you, follow the points below:\n","\n","- State the hypothesis: null and alternative hypothesis.\n","- Perform the ANOVA test.\n","- Analyze the conclusions.\n","- If one fertilizer is better than another, how can we know it?"]},{"cell_type":"code","execution_count":2,"id":"eecad0ca","metadata":{},"outputs":[],"source":["# TODO"]}],"metadata":{"interpreter":{"hash":"9248718ffe6ce6938b217e69dbcc175ea21f4c6b28a317e96c05334edae734bb"},"kernelspec":{"display_name":"Python 3.9.12 ('ML-BOOTCAMP')","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.11.4"}},"nbformat":4,"nbformat_minor":5} +{ + "cells": [ + { + "cell_type": "markdown", + "id": "d9d036c6", + "metadata": {}, + "source": [ + "# Hypothesis testing problems" + ] + }, + { + "cell_type": "markdown", + "id": "d157c4db", + "metadata": {}, + "source": [ + "## Exercise 1\n", + "\n", + "You are a nutritionist researching two different types of diets to see if there is a significant difference in weight loss after one month. You choose two random groups of people; one group follows the first diet, and the other follows the second. At the end of the month, you record the weight loss (in kg) for each person:\n", + "\n", + "| Diet 1 | Diet 2 |\n", + "|:-------|:-------|\n", + "| 2.0 | 3.0 |\n", + "| 2.5 | 3.2 |\n", + "| 3.0 | 3.1 |\n", + "| 2.8 | 2.9 |\n", + "| 2.3 | 2.8 |\n", + "| 2.7 | 3.0 |\n", + "| 2.5 | 3.2 |\n", + "\n", + "With these data, it seeks to answer the following question: Is there a significant difference in average weight loss between people who followed the first diet and those who followed the second diet?\n", + "\n", + "To draw conclusions, follow the points below:\n", + "\n", + "- **State the hypothesis**: null hypothesis ($H₀$) and alternative hypothesis ($H₁$).\n", + "- **Check the assumptions**: Before performing the test, mention the assumptions that must be met for the Student's t-test to be valid (independence, normality, and homogeneity of variances).\n", + "- Perform the test to test the hypothesis. You can use a Student's t-test.\n", + "- **Analyze the conclusions**: Based on the p-value, decide whether to reject or not reject the null hypothesis and explain what this means in the context of the problem." + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "id": "8e0f3cf0", + "metadata": {}, + "outputs": [], + "source": [ + "# TODO" + ] + }, + { + "cell_type": "markdown", + "id": "f432e18e", + "metadata": {}, + "source": [ + "## ANOVA\n", + "\n", + "**ANOVA** (*Analysis of Variance*) is a statistical technique used to compare the measures of two or more groups. The idea behind ANOVA is to decompose the total variability in the data into two components: between-group variability and within-group variability:\n", + "\n", + "- **Between-group variability**: This variability refers to the differences between the group means. If this variability is considerably larger than the within-group variability, it could be an indication that at least one of the group means is different.\n", + "- **Within-group variability**: This variability refers to the dispersion of the data within each group. If all groups have similar variability, then any noticeable difference in group means could be considered significant.\n", + "\n", + "Hypotheses in ANOVA typically include:\n", + "\n", + "- **Null hypothesis** ($H₀$): The means of all groups are equal.\n", + "- **Alternative hypothesis** ($H₁$): At least one of the group means is different.\n", + "\n", + "If the ANOVA test result is significant (e.g., a p-value less than a threshold such as 0.05), this suggests that at least one group mean is different." + ] + }, + { + "cell_type": "markdown", + "id": "fc5f3721", + "metadata": {}, + "source": [ + "## Exercise 2\n", + "\n", + "A farmer decides to test three different types of fertilizers to determine if one is superior in terms of corn production. The farmer plants corn on 15 identical plots and uses all three fertilizers (5 plots for each type). At the end of the season, he measures the corn yield (in kg) of each plot, with the following result:\n", + "\n", + "| Fertilizer 1 | Fertilizer 2 | Fertilizer 3 |\n", + "|:-------------|:-------------|:-------------|\n", + "| 20 | 22 | 24 |\n", + "| 21 | 21 | 23 |\n", + "| 20 | 23 | 22 |\n", + "| 19 | 22 | 23 |\n", + "| 20 | 21 | 24 |\n", + "\n", + "With this data, he seeks to answer the following question: Is there a significant difference in average corn yield between the three types of fertilizers?\n", + "\n", + "To help you, follow the points below:\n", + "\n", + "- State the hypothesis: null and alternative hypothesis.\n", + "- Check the assumptions: Mention the assumptions that must be met to apply ANOVA.\n", + "- Perform the ANOVA test.\n", + "- Analyze the conclusions: Based on the p-value, decide whether to reject or not reject the null hypothesis and explain the meaning of the ANOVA result.\n", + "- Post-hoc test: If the ANOVA test indicates a significant difference, how can you find out which fertilizer is better? Perform a post-hoc test (such as Tukey's HSD test) and explain its conclusion." + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "id": "eecad0ca", + "metadata": {}, + "outputs": [], + "source": [ + "# TODO" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.11.4" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/notebook/solutions.ipynb b/notebook/solutions.ipynb index 266a85e4..a7660d37 100644 --- a/notebook/solutions.ipynb +++ b/notebook/solutions.ipynb @@ -1 +1,280 @@ -{"cells":[{"cell_type":"markdown","id":"d9d036c6","metadata":{},"source":["# Hypothesis testing solutions"]},{"cell_type":"markdown","id":"f7a22d13","metadata":{},"source":["## Exercise 1"]},{"cell_type":"markdown","id":"09354ed2","metadata":{},"source":["### 1. State the hypotheses\n","\n","- **Null hypothesis** ($H₀$): There is no difference in average weight loss between the two diets.\n","- **Alternative hypothesis** ($H₁$): There is a difference in average weight loss between the two diets."]},{"cell_type":"markdown","id":"1a288221","metadata":{},"source":["### 2. Perform Student's t-test"]},{"cell_type":"code","execution_count":1,"id":"5cea63a1","metadata":{},"outputs":[{"name":"stdout","output_type":"stream","text":["t-value: -3.5383407969933938\n","p-value: 0.004083270191713912\n"]}],"source":["import scipy.stats as stats\n","\n","diet_1 = [2.0, 2.5, 3.0, 2.8, 2.3, 2.7, 2.5]\n","diet_2 = [3.0, 3.2, 3.1, 2.9, 2.8, 3.0, 3.2]\n","\n","# Student's t-test\n","t_value, p_value = stats.ttest_ind(diet_1, diet_2)\n","\n","print(f\"t-value: {t_value}\")\n","print(f\"p-value: {p_value}\")"]},{"cell_type":"markdown","id":"b5776261","metadata":{},"source":["### 3. Conclusion\n","\n","By obtaining a p-value of less than a chosen significance level of 5% (0.05), we would reject the null hypothesis and conclude that there is a significant difference in weight loss between the two diets.\n","\n","If the p-value had been greater, we would not reject the null hypothesis, suggesting that there was no significant difference."]},{"cell_type":"markdown","id":"b1f85e81","metadata":{},"source":["## Exercise 2"]},{"cell_type":"markdown","id":"957845e9","metadata":{},"source":["### 1. State the hypotheses\n","\n","- **Null hypothesis** ($H₀$): The corn yield averages of the three fertilizers are equal.\n","- **Alternative hypothesis** ($H₁$): At least one of the fertilizer corn yield averages is different."]},{"cell_type":"markdown","id":"d1466a32","metadata":{},"source":["### 2. Perform ANOVA test"]},{"cell_type":"code","execution_count":3,"id":"90f4d3a6","metadata":{},"outputs":[{"name":"stdout","output_type":"stream","text":["f-value: 20.315789473684188\n","p-value: 0.00014047824793190475\n"]}],"source":["import scipy.stats as stats\n","\n","fertilizer_1 = [20, 21, 20, 19, 20]\n","fertilizer_2 = [22, 21, 23, 22, 21]\n","fertilizer_3 = [24, 23, 22, 23, 24]\n","\n","# ANOVA test\n","f_value, p_value = stats.f_oneway(fertilizer_1, fertilizer_2, fertilizer_3)\n","\n","print(f\"f-value: {f_value}\")\n","print(f\"p-value: {p_value}\")"]},{"cell_type":"markdown","id":"8e8e04cb","metadata":{},"source":["### 3. Conclusion\n","\n","By obtaining a value less than the established significance level (0.05), we reject the null hypothesis and conclude that there is a significant difference in the mean corn yield between at least two of the fertilizers."]},{"cell_type":"markdown","id":"56265c27","metadata":{},"source":["### 4. Obtaining the best fertilizer\n","\n","After performing the ANOVA test, it can be seen that there are significant differences between the groups. The next step is to identify which fertilizer is better, for which we will perform a post-hoc test. One of these tests is the Tukey test, which is based on the following:\n","\n","1. If the first fertilizer is significantly different from the second fertilizer, but there is no significant difference between the first and the third, and no significant difference between the second and the third, we could conclude that the first fertilizer is the best or the worst, depending on the direction of the difference.\n","2. If all fertilizers are significantly different from each other, we would compare the means to determine which is the best."]},{"cell_type":"code","execution_count":4,"id":"fcaece07","metadata":{},"outputs":[{"name":"stdout","output_type":"stream","text":["Multiple Comparison of Means - Tukey HSD, FWER=0.05\n","==================================================\n","group1 group2 meandiff p-adj lower upper reject\n","--------------------------------------------------\n"," F1 F2 1.8 0.0099 0.4572 3.1428 True\n"," F1 F3 3.2 0.0001 1.8572 4.5428 True\n"," F2 F3 1.4 0.0409 0.0572 2.7428 True\n","--------------------------------------------------\n"]}],"source":["import numpy as np\n","from statsmodels.stats.multicomp import pairwise_tukeyhsd\n","\n","data = np.concatenate([fertilizer_1, fertilizer_2, fertilizer_3])\n","labels = [\"F1\"] * 5 + [\"F2\"] * 5 + [\"F3\"] * 5\n","\n","# Tukey test\n","result = pairwise_tukeyhsd(data, labels, alpha = 0.05)\n","print(result)"]},{"cell_type":"markdown","id":"e0e6c106","metadata":{},"source":["Interpreting this table is very simple. Look at the `reject` column. If the value is `True` it means that there is a statistically significant difference between those two groups. If there is a significant difference, we need to look at the `meandiff` column to see how much the means of the two groups differ. If the value is positive, it means that `group2` has a higher mean than `group1`. If it is negative, the other way around.\n","\n","From this information, we can determine which fertilizer is the best in terms of the metric we are measuring: the average yield (in kg) of corn. In this case, all fertilizers differ significantly from each other. The third fertilizer has the highest average, followed by the second and ending with the first. Therefore, **according to these results, the third fertilizer would be the best**."]}],"metadata":{"interpreter":{"hash":"9248718ffe6ce6938b217e69dbcc175ea21f4c6b28a317e96c05334edae734bb"},"kernelspec":{"display_name":"Python 3.9.12 ('ML-BOOTCAMP')","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.11.4"}},"nbformat":4,"nbformat_minor":5} +{ + "cells": [ + { + "cell_type": "markdown", + "id": "d9d036c6", + "metadata": {}, + "source": [ + "# Hypothesis testing solutions" + ] + }, + { + "cell_type": "markdown", + "id": "f7a22d13", + "metadata": {}, + "source": [ + "## Exercise 1" + ] + }, + { + "cell_type": "markdown", + "id": "09354ed2", + "metadata": {}, + "source": [ + "### 1. State the hypotheses\n", + "\n", + "* **Null Hypothesis ($H_0$)**: There is no significant difference in the average weight loss between the two diets. That is, the population means are equal: $\\mu_1 = \\mu_2$.\n", + "* **Alternative Hypothesis ($H_1$)**: There is a significant difference in the average weight loss between the two diets. That is, the population means are not equal: $\\mu_1 \\neq \\mu_2$." + ] + }, + { + "cell_type": "markdown", + "id": "d0e7e27f", + "metadata": {}, + "source": [ + "### 2. Check the Assumptions\n", + "\n", + "For the t-test for independent samples to be valid, the following assumptions should ideally be met:\n", + "\n", + "* **Independence**: The observations within each group are independent, and the groups themselves are independent. This is met as the groups were randomly selected.\n", + "* **Normality**: The data in each group should be approximately normally distributed. With small sample sizes ($n=7$), it's difficult to verify this assumption visually or with a formal test, but for the purpose of this exercise, we assume it's met. In a real-world scenario, you might use a **Shapiro-Wilk test** to check for normality.\n", + "* **Homogeneity of Variances**: The population variances of the two groups should be approximately equal. The `scipy.stats.ttest_ind` function assumes equal variances by default (`equal_var=True`). A **Levene's test** could be used to verify this assumption." + ] + }, + { + "cell_type": "markdown", + "id": "1a288221", + "metadata": {}, + "source": [ + "### 3. Perform Student's t-test" + ] + }, + { + "cell_type": "code", + "execution_count": 25, + "id": "5cea63a1", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "t-value: -3.5383407969933938\n", + "p-value: 0.004083270191713912\n" + ] + } + ], + "source": [ + "import scipy.stats as stats\n", + "\n", + "diet_1 = [2.0, 2.5, 3.0, 2.8, 2.3, 2.7, 2.5]\n", + "diet_2 = [3.0, 3.2, 3.1, 2.9, 2.8, 3.0, 3.2]\n", + "\n", + "# Student's t-test\n", + "t_value, p_value = stats.ttest_ind(diet_1, diet_2)\n", + "\n", + "print(f\"t-value: {t_value}\")\n", + "print(f\"p-value: {p_value}\")" + ] + }, + { + "cell_type": "markdown", + "id": "b5776261", + "metadata": {}, + "source": [ + "### 4. Conclusion\n", + "\n", + "We compare the p-value with a chosen significance level, typically $\\alpha = 0.05$.\n", + "\n", + "* Since the **p-value (0.0041)** is less than the significance level **(0.05)**, we **reject the null hypothesis ($H_0$)**.\n", + "* This suggests that there is a **statistically significant difference** in the average weight loss between the two diets. In other words, the difference observed is unlikely to be due to random chance." + ] + }, + { + "cell_type": "markdown", + "id": "b1f85e81", + "metadata": {}, + "source": [ + "## Exercise 2" + ] + }, + { + "cell_type": "markdown", + "id": "957845e9", + "metadata": {}, + "source": [ + "### 1. State the hypotheses\n", + "\n", + "* **Null Hypothesis ($H_0$)**: The average corn yield for all three fertilizers is equal. That is, $\\mu_1 = \\mu_2 = \\mu_3$.\n", + "* **Alternative Hypothesis ($H_1$)**: At least one of the fertilizer average corn yields is different from the others. That is, at least one $\\mu_i \\neq \\mu_j$." + ] + }, + { + "cell_type": "markdown", + "id": "77a5477f", + "metadata": {}, + "source": [ + "### 2. Check the Assumptions\n", + "\n", + "For a one-way ANOVA to be valid, the following assumptions should be met:\n", + "\n", + "* **Independence**: The samples from each group (fertilizer) are independent. This is met, as the plots are distinct from each other.\n", + "* **Normality**: The data within each group should be approximately normally distributed. With only 5 plots per fertilizer, it's difficult to confirm, but we assume it holds.\n", + "* **Homogeneity of Variances**: The population variances of the three groups are equal. This can be tested using Levene's test." + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "id": "5425faf6", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Levene's test p-value: 0.8039599174006208\n" + ] + } + ], + "source": [ + "import scipy.stats as stats\n", + "\n", + "fertilizer_1 = [20, 21, 20, 19, 20]\n", + "fertilizer_2 = [22, 21, 23, 22, 21]\n", + "fertilizer_3 = [24, 23, 22, 23, 24]\n", + "\n", + "# Levene's test for homogeneity of variances\n", + "levene_test = stats.levene(fertilizer_1, fertilizer_2, fertilizer_3)\n", + "print(f\"Levene's test p-value: {levene_test.pvalue}\")" + ] + }, + { + "cell_type": "markdown", + "id": "d1466a32", + "metadata": {}, + "source": [ + "### 3. Perform ANOVA test" + ] + }, + { + "cell_type": "code", + "execution_count": 23, + "id": "90f4d3a6", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "f-value: 20.31578947368421\n", + "p-value: 0.000140478247931904\n" + ] + } + ], + "source": [ + "# ANOVA test\n", + "f_value, p_value = stats.f_oneway(fertilizer_1, fertilizer_2, fertilizer_3)\n", + "print(f\"f-value: {f_value}\")\n", + "print(f\"p-value: {p_value}\")" + ] + }, + { + "cell_type": "markdown", + "id": "8e8e04cb", + "metadata": {}, + "source": [ + "### 4. Analyze the Conclusions\n", + "\n", + "* The **p-value (0.00014)** is less than the significance level **(0.05)**.\n", + "* Therefore, we **reject the null hypothesis ($H_0$)**.\n", + "* This indicates that there is a **significant difference** in the average corn yield between at least two of the three fertilizers. However, the ANOVA test alone **does not tell us which specific pairs of fertilizers are different**." + ] + }, + { + "cell_type": "markdown", + "id": "56265c27", + "metadata": {}, + "source": [ + "### 5. Post-hoc Test: Obtaining the Best Fertilizer\n", + "\n", + "To find out which fertilizer is superior, we need to perform a **post-hoc test**. Tukey's HSD (Honestly Significant Difference) is a common choice for this." + ] + }, + { + "cell_type": "code", + "execution_count": 24, + "id": "fcaece07", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Multiple Comparison of Means - Tukey HSD, FWER=0.05\n", + "==================================================\n", + "group1 group2 meandiff p-adj lower upper reject\n", + "--------------------------------------------------\n", + " F1 F2 1.8 0.0099 0.4572 3.1428 True\n", + " F1 F3 3.2 0.0001 1.8572 4.5428 True\n", + " F2 F3 1.4 0.0409 0.0572 2.7428 True\n", + "--------------------------------------------------\n" + ] + } + ], + "source": [ + "import numpy as np\n", + "from statsmodels.stats.multicomp import pairwise_tukeyhsd\n", + "\n", + "data = np.concatenate([fertilizer_1, fertilizer_2, fertilizer_3])\n", + "labels = [\"F1\"] * 5 + [\"F2\"] * 5 + [\"F3\"] * 5\n", + "\n", + "# Tukey test\n", + "result = pairwise_tukeyhsd(data, labels, alpha = 0.05)\n", + "print(result)" + ] + }, + { + "cell_type": "markdown", + "id": "e0e6c106", + "metadata": {}, + "source": [ + "**Interpreting the Results:**\n", + "The `reject` column shows whether there is a statistically significant difference between a pair of groups.\n", + "\n", + "* `F1` vs `F2`: `reject=True`. There is a significant difference.\n", + "* `F1` vs `F3`: `reject=True`. There is a significant difference.\n", + "* `F2` vs `F3`: `reject=True`. There is a significant difference.\n", + "\n", + "All fertilizer groups are significantly different from each other. Now, we look at the `meandiff` column to see the magnitude and direction of the difference.\n", + "\n", + "* The average yield of F2 is 1.8 kg higher than F1.\n", + "* The average yield of F3 is 3.2 kg higher than F1.\n", + "* The average yield of F3 is 1.4 kg higher than F2.\n", + "\n", + "Based on these results, **Fertilizer 3** provided the highest average yield, making it the most effective of the three fertilizers tested." + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.11.4" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +}