Habr AI showed how the virtual camera for Shorts and Reels with face tracking works

Habr AI demonstrated how a virtual camera for vertical video works, converting standard horizontal footage into Shorts or Reels without the jerky auto-crop typical of such conversions. The author describes a production pipeline where the camera tracks the face as a separate dynamic system, rather than following a single bounding box.

Why Auto-Crop Fails

On paper, the task seems straightforward: take a 16:9 frame, extract a 9:16 window, and keep the person centered. In real video, this approach quickly breaks down. The face can shift left or right, a second person may appear in frame, and the detector itself will introduce noise, lose the target on head turns, reflections, or gestures. The result is a vertical video that looks like old autofocus: the image twitches nervously and constantly mishandles composition.

The author proposes viewing the problem not as auto-crop, but as a virtual camera with its own inertia, speed constraints, reaction delay, and fallback logic. The system can't simply "follow the center of the box." It must exist in time, tolerate imprecise data, not panic during brief face losses, and maintain meaningful motion even in scenes where no faces appear at all. This is critical for longer clips.

"A camera is not a coordinate, but a dynamic system."

How the Pipeline Works

The core solution uses a three-level face detection chain: MediaPipe → YuNet → Haar Cascade. The first detector serves as the primary workhorse, the second acts as a backup ONNX backend via OpenCV, and the third remains the last line of defense when everything else is unavailable or performs poorly. Together they produce a unified data format: face center, size, confidence, and bounding box. This way, the rest of the system is independent of the specific detector and can work stably even in degraded mode.

On top of detection comes simple but practical tracking: the system compares the face center from the previous frame to current candidates and selects the nearest object. If the face disappears briefly, a grace period activates—the camera trusts the last known position for a short time to avoid jerking on a single error. After this, the signal is cleaned in two additional steps and gets a separate fallback for scenes without faces.

• Anti-jerk removes excessively large coordinate jumps between frames
• Low-pass filter smooths residual noise and micro-vibrations
• Ken Burns fallback handles the scene if the face is lost or absent
• Path interpolation converts 8 FPS analysis into smooth motion for final output at 30 or 60 FPS

At the final stage, the system doesn't just store discrete camera states—it builds a continuous path. Then for each frame, it takes the center and zoom, calculates the ROI, extracts the needed region, and scales it to the final vertical format. This keeps the pipeline computationally reasonable: analysis can be done on a reduced copy of the video, while the final crop works with the original, without losing movement smoothness in the final render.

How the Camera Becomes an Operator

The most interesting part is the movement model. Instead of instant jumps to the target point, the camera is described as a damped oscillator: it has "mass," stiffness, damping, acceleration limits, and maximum velocity. This scheme provides inertia and makes movement feel authentic. If a person starts moving faster, you can add predictive lead so the camera looks slightly ahead rather than only chasing the target. Paradoxically, a slight artificial delay—human lag—also improves the result: perfectly precise reactions often look less natural than slightly delayed ones.

On top of the physics, the author applies compositional rules. The camera can offset the face toward rule-of-thirds lines, raise the focus point closer to eye level, ignore micro-movements within a dead zone, and maintain a safe face margin to avoid cutting ears, hair, and gestures at the frame edges. For talking-head scenes, there's a dedicated single-face mode with more conservative speed and enhanced stabilization, and for videos without faces—a smooth transition to Ken Burns with panning and zoom instead of a "frozen" center.

What This Means

This breakdown shows where AI video editing tools are headed. Quality here isn't born from a single "magical" model, but from engineering assembly: fallback detectors, tracking, filtering, movement physics, and composition work together. For content creators and teams that mass-produce long videos cropped into Shorts and Reels, this is no longer a toy auto-crop, but a practical production camera template that can be replicated and adapted for different shooting styles.