Monday, June 19, 2017

How to train your face tracking

Last week I was talking to a colleague at work about different facial tracking software and thought that it can't be that massively complicated to write a basic software that tracks pixels.
"All it needs to do is compare a bunch of colour values right?"

Sure it can probably be much more complicated than that but I thought I'd give it a shot and after a week of endless nights I have a first result that I am pretty happy with.

The solution I went for was essentially just looking for sub-images (a region defined by tracker points) in larger images (a larger area around the tracker points) over and over again. Each possibility gets an error rating and after trying every possible combination the software will pick the solution with the smallest error rate. Once a solution is picked it will do this again but now with the pixel values of the new region.

I first tried an external tool called ImageMagick for the tracking itself but found it to be very slow. Luckily this forced me into having to learn absolutely everything from scratch and its actually much faster and accurate now. I used the PIL library to do the pixel walking and image manipulation required, everything else is just using python standard libraries.

As you can see in the video its still quite jittery, and the area around the mouth is especially hard to track as it can accelerate really quickly, stop very abrupt and change shape significantly.

Mostly I am just tapping in the dark, as I don't have any expertise on this topic but I am hoping to improve the tracking by prediction the future position over several frames, doing a constant error check on that and widen the search area based on that error rate. Another thing I had in mind was maybe constantly comparing the current sub-image to the original one and once the difference between them is quite large then maybe use a combination of the prediction error rate and stabilisation through the neighbours.

If anyone reading this has any ideas on what I should do different, I would love to hear about them, please give me a shout!

No comments:

Post a Comment