- EMOtional- Table of content
We all know that children are very emotional and do not hesitate to express their emotion in all the activities they do. The goal of our project is to objectively measure the emotions of preschoolers following the interaction with some educational activities. At a very young age, they still cannot give written feedback, so the best way to measure user satisfaction is observation. Our project allows the login of a preschooler in the application and start some small interactive games after which a report of emotions is established. This report will be automatically sent to the educator.
An LBP based facial recognition algorithm was used as the login procedure and for the recognition of emotions we used a Convolutional Neural Network together with Haar Cascade Classifiers.
The interractive application is developed in C# + VueJS 3 and integrates these previously presented smart models.
The training was performed on the CAFE database. This database contains 1192 photographs of 154 children. The name of each photograph contains the emotion expressed along with a unique identifier given to each participant in the experiment.
The 7 basic emotions included in the dataset ("sadness", "happiness", "fear", "anger", "surprise", "disgust", "neutral") are not balanced, which can be seen from the following histogram .
Moreover, the features of the children are not as pronounced as those of the adults and we encountered problems in determining the rectangle of the face with the Haar classifier in openCV. To solve this problem we have chosen to manually decrease the brightness of the images before they are given for facial detection.
| Original image | Processed image |
|---|---|
 |
 |
And finally, after applying the haar classifier and resizing, an input image will look like this:
For the facial recognition part, we chose Local Binary Patterns Histogram - an algorithm that involves labeling each pixel of an image by applying a threshold on the neighbors of a pixel and transforming the result into a binary number. The black and white image is then divided into regions. The extraction of the histogram corresponding to each region is based on the creation of a histogram with 256 positions whose value represents the number of occurrences of each value corresponding to the pixels. The histograms thus obtained are concatenated and for the facial recognition part they are compared. The easiest way to compare them is to make a Euclidean distance.
To detect emotions we chose to use a CNN with the following architecture that receives as input each image processed and resized to 64x64 pixels :
Unfortunately, the results are not satisfactory enough, although in the last training we added Early Stopping to prevent overfitting.
| Accuracy metrics | Loss | Confusion matrix |
|---|---|---|
 |
 |
 |
- numpy
- opencv
- tensorflow
- keras
- skimage
After the child/ educator starts the application, a gif appears on the screen to keep the child's attention.
After successful authentication, the application sends you to the main page. To the left of the main window is a list of options that contain links to interactive games
As the child interacts with an activity, screenshots are sent back every few seconds and the predominant emotion returns. At the end of the activity a diagram is made from these emotions and it will be exported to the educator.