flick.gg was envisioned as a tool to help bad players (me) improve their aim and game sense by providing instant replays so they could analyze their performance and learn from their mistakes. Based on the core belief that the best way to improve is to spend time watching yourself die, a lot.
flick.gg.mp4
The project was developed using Python, OpenCV, and Keras/TensorFlow. The model was trained on a dataset of ~5,000 images of enemies and non-enemies. The dataset was created by hooking into the game and capturing frames and labels in real-time.
Here's me after dying in a CS:GO silver game.
I'm old now, I swear I used to be good at video games, I breathed them as a kid, but now I'm just not fast enough. So I needed to do something about it. I needed to get good. My nephews are teenagers and asking me to play COD with them
This project was originally envisioned to run via a service you would stream to. This meant the machine-learning model was designed to be lightweight and fast, so it could run in real-time on a single GPU. We needed object detection in less than 33ms for 30fps stream and YOLOv3 was the only model even close.
Here's a demo of what the user would see immediately after dying.
gunfight1.mp4
The model got good eventually, but it was quite a road to get there.
model-got-good.mp4
It's worth mentioning that the user doesn't see the detections on their screen, the detections are visible in the replays only and help score the user's performance and detect clippable portions of their gameplay (death, kill, etc.).
There were some not so great models along the way, this one was pretty sure our gun was an enemy.
gun-is-enemy.mp4
A big problem with the original approach was I trained the model on full quality pngs, but ran the prediction against the compressed stream with quantized jpeg artifacts. Doh!
Your training set needs to resemble your prediction case as much as possible.
I'm not even sure what's going on with this model, everything is a terrorist, even HUD elements.
bad.mp4
Anyway, after many training iterations, revising the data test/train sets, asking different questions, we ended up with a nicely trained model which balanced object detection speed and accuracy well enough to clip user's gameplay in real-time and give them objective scores for their performance on metrics like crosshair placement, tracking, flicking, and many others.
In the end, I still am terrible at CS:GO, but this project was a ton of fun.