Status Update - July 21

Hello!

I hope the v0.2 kemorig update has been fun to play around with.

First I'm going to share some cool stuff being done with kemorig, then go over what I'm working on right now.

Cool kemorig Stuff!

• Here's some clips of other streamers using kemorig!

  • Djano has a super-detailed wolf using fur rendering: https://www.twitch.tv/djano__/clip/ArtsyPlacidButterflyWutFace-i1NaiAq9UZcuqQX3

  • Kofu has a stylized borzoi that they've done some really neat shader work with: https://www.twitch.tv/kofu/clip/ChillyImpossibleTermiteDxAbomb-ojV7AshFSaFnlwxC

• Have a clip of what you're doing in kemorig? Post it down below!

Personal Site

After the v0.2 kemorig release, I went on a 2-week side quest to fix up my personal website: https://protowlf.com 

This is where I occasionally post tutorials, and now I've updated it to host documentation for my Unreal plugins. It also functions as a home-base for me, and I might expand it in the future. (Speaking of, I really need to post more tutorials...).

kemorig Webcam Tracking

I started looking into solutions for webcam tracking in kemorig. I've learned some things about the current landscape of this:

• There appear to be 3 main technologies out there that could realistically be used:

  • OpenSeeFace (this is what VSeeFace uses) - a small machine learning project using a relatively small (hundreds of thousands) dataset of images. Good at tracking where your head is, can only track rudimentary expressions.

  • MediaPipe - Google's machine learning stuff. Pretty sophisticated face expression tracking (still not as good as a depth sensing camera, but not a surprise). This is what most other apps other than VSeeFace are using.

    • However, Google's API is indecipherable, their documentation sucks (and a lot of it has been deleted), and it's really hard to build this into a C++ application. Hard as in: I am a professional software engineer and I have no idea where to start.

    • There are some open source projects and that implemented an older iteration of MediaPipe face/hand tracking into C++ apps, and I suspect this is what the apps we're seeing are using.

  • NVIDIA - NVIDIA is inching into face tracking with their machine learning stuff. I don't know a lot about it, but it turns out iFacialMocap has created a desktop app that uses it. Of course, this requires an NVIDIA gpu to use.

I think going forward with kemorig, these are the 2 things that will probably happen:

• 1) Implement an open source MediaPipe solution into kemorig, and support it natively.

• 2) Add support for iFacialMocap's desktop app, which just so happens to support Unreal out of the box.

  • This isn't an idea solution, as it requires running an extra app alongside kemorig. But I want to support iFacialMocap's iPhone app anyway, and if implementing support for the desktop app is easy, that's a win-win.

WHEN will webcam tracking come to kemorig?

Short answer: I'm not sure. But probably in August.

If it starts really taking a long time, I might add iFacialMocap support first, because that's undoubtedly an easier task.

A Note on Head Position Tracking

One of kemorig's current problems it that the Live Link iPhone app does not track head position (only expressions and rotation).

iFacialMocap's iPhone app does track head position, making it a better solution that Epic's Live Link app (note that iFacialMocap is not free, but it's pretty cheap). I plan to add support for this app at some point.

I might also look into a webcam mode that only tracks your head/torso, and you still use an iPhone for face expression. By combining the two, that might produce the highest fidelity motion. I don't have concrete plans for this, but it's on my mind.

Alright, long post! Thanks for reading everybody, and I hope you have a good day.

-PROTO