kemorig Webcam WIP

Hey everyone, webcam tracking is working! 🎉

This is a quick work-in-progress video showing the current state. It's using Google MediaPipe (as alluded-to in the last post), and runs inside the kemorig app.

Right now you're seeing it track head and shoulders. Finally, we can see avatars move around instead of rotating in-place! It will also be able to track eyes, and influence some (but not all) face blendshapes.

Now, don't get too excited by the words "shoulder tracking". Although MediaPipe can technically track the whole body, it's very rough, and for the most part the only stable data we get is for the face. On top of that, depth (i.e. detecting what body parts are closer to the camera) is extremely rough, so rough that it's virtually unusable data. So, think of this as head rotation and position, with a tiny bit of shoulder information sprinkled on-top.

Combining Webcam with iPhone ARKit

You will be able to use webcam tracking at the same time as existing ARKit tracking. This is very cool! I plan to use this in my own setup, to get great face tracking from ARKit and get more body motion from the webcam.

There's going to be a new menu that lets you select which tracking methods affect Body, Eye, and Face tracking independently.

Easier Avatar Setup

Did you find it annoying manually entering bone orientations to setup your avatar? Me too, it sucked! With the webcam update, that's gone and the animation code now auto-detects this.

Code Refactor

I've spent the last couple weeks doing a lot of work reorganizing kemorig's code to support multiple tracking methods. Turns out, many of my assumptions about how to structure the app were challenged with the introduction of webcam tracking.

This refactor is basically done now. This is behind-the-scenes stuff, but the way tracking data flows through the app is vastly improved.

Thanks everyone, can't wait to get the next build in your hands!