Feature Request: Support for AR-style face filters (masks, glasses, hats, avatars)

[Mirzahmet]

Hello @lukasIO @1egoman!
First of all, thank you for providing such a well-designed library for video track processing.
Currently, track-processors-js supports background effects (blur, replacement), which works great.
I would like to ask whether you plan to extend this framework to support AR-style face filters — similar to what you see in apps like Snapchat or Instagram — for example:
Overlaying 2D/3D elements (glasses, hats, beard, mustache)
Adding animated textures or stickers that follow facial landmarks
Optionally, full face replacement / avatar rendering (e.g., replace user’s face with a robot or animal mask)
This would open many creative use cases for LiveKit apps (virtual classrooms, entertainment, games, etc.)

[1egoman]

@Mirzahmet I'll chat with some folks internally at LiveKit and see what the priority of doing something like this is. My guess though is that it's probably not super high.

Depending on what you might be trying to do (assuming this isn't just a theoretical ask), and assuming this is in a call with an ai agent, you might be able to use agent avatars to accomplish something similar.

That being said, here are some notes in case anybody is thinking about trying this out on their own:

• There is a mediapipe face detection api: https://ai.google.dev/edge/mediapipe/solutions/vision/face_detector. Most likely it could be dropped into a copy of the BackgroundTransformer and operate in a roughly similar fashion as the existing mediapipe background segmentation worker.
• The output of that face detection could then be fed into a webgl shader (similarly to the current BackgroundTransformer) - maybe sending uniforms for the key points? And then those uniforms could control a transformation of some sort (maybe compute a transform matrix from the key points?) to texture blit operation to draw some sort of overlay.
• It probably starts to get more complicated if you need to render more than just a singular static image (ie, like a looping video, or like a 3d model). Here's a potentially promising way to render a looping video.