This program allows people with motor function disabilities to control a computer using head movement and facial gestures. The head tilt enables the user to move the cursor, while eye blinks are used for clicking. A virtual keyboard allows the user to type using both eye blinking and head movement.
This program uses Arduino Nano and an Inertial Measurement Unit to collect data on the head tilt. The data is then processed and converted into coordinates that are used by the Python program to move the cursor to a specific location on the screen.
The eye-blink-detection program uses MediaPipe Face Mesh to identify key points on the face. Since there are 468 points detected by the Face Mesh module, the program only uses the points around both eyes. The program determines the height and width of both eyes and calculates the eye ratios (height/width). The eye ratios prevent the program from detecting false positive blinks when the user moves their head away from the screen. When the user blinks, the eye ratio decreases below a set threshold, causing the program to detect a blink.
The virtual keyboard uses Tkinter. The Tkinter module creates a Graphical User Interface (GUI) in which there are buttons corresponding to a specific letter. When the user moves the cursor and clicks on the button, the program types the letter in the GUI window. However, this process is too slow for the user since it requires typing letters using the cursor. To speed up this typing process, I integrated GPT-2 into my code to predict the next letter/word the user would type. This allows users to type words with one click rather than typing out several letters.
All these features run in real time, providing a great experience for the user. The goal of this project was to ensure that the combined software and hardware costs were below $50. The Arduino and IMU combined cost below $50, and the program was written on open-source software, making the project meet the criteria. This project won fourth place at the California Science and Engineering Fair.