An Agent Builds a Motion Studio From 20 Open-Source Libraries, From Remotion to Whisper

Twenty libraries, no "video model" among them, and a finished mp4 at the end. The Telegram channel «Точки над ИИ» (Dots over AI) has put together a set of open-source tools that a coding agent can use to turn a text brief into an animated video with physics, voiceover and subtitles. According to the authors, the video in the post itself was also built with code. The libraries are ordinary JavaScript and Python, and nothing in them is designed specially "for agents." But Claude Opus-class models know their APIs well enough to write working code without long debugging.

Why it is easier for an agent to write code than to generate pixels

Generative video models produce good-looking but poorly controllable output: fixing a single frame or nudging a title by 12 pixels is almost impossible. Code works differently. A scene is described with components, timelines and parameters, so the agent changes one line and re-renders the video. Version history lives in git, and a template can be run as a series.

The second advantage is that these libraries come with huge bodies of documentation and examples. All of it has long since made its way into training data, so there are fewer hallucinations in API calls than with new, niche tools.

All 20 tools by group

Video and rendering

• Remotion — video in React. Frames are described as components, and rendering outputs a file. This is the backbone of the whole setup.
• HyperFrames — HTML to video. The project lives in HeyGen's repository and turns web layouts into a video.
• OpenCut — an open-source video editor for manual editing and polishing what the agent produced.
• OBS Studio — screen recording. It is needed when a real interface or demo has to go into the video.
• FFmpeg — final export, conversion and concatenation. Almost any video pipeline ends up depending on it.

Animation

• Anime.js — timelines that sequences of motion are built on.
• React Spring — animation based on spring physics. Motion looks alive rather than linear.
• GL Transitions — shader-based transitions between scenes.
• tsParticles — particles: confetti, snow, sparks, background effects.

2D and generative graphics

• PixiJS — GPU-accelerated 2D graphics. It handles many sprites without losing speed.
• p5.js — generative visuals and creative coding.
• Paper.js — vector paths and operations on them.
• Matter.js — 2D physics: collisions, gravity, falling objects.

3D

• Three.js — the main library for 3D scenes in the browser.
• Rapier — a physics engine for 3D.
• OGL — a minimal WebGL wrapper for those who write their own shaders.
• glTF Transform — 3D model optimization: compression, size reduction, asset cleanup.

Audio and subtitles

• Chatterbox — an open-source voice model that generates voiceover from text.
• Tone.js — synthesizers and sound effects in the browser.
• faster-whisper — a fast Whisper implementation that turns speech into subtitles with timecodes.

How to assemble the pipeline and where it breaks

A practical setup looks like this. The agent receives a brief and writes a script. Chatterbox voices it, faster-whisper returns word timings, and Remotion or HyperFrames lay out the scenes based on those timings. Three.js, Rapier, Matter.js and PixiJS supply the graphics and physics, transitions come from GL Transitions, and FFmpeg stitches everything into an mp4. If manual edits are needed, the file opens in OpenCut.

There are limits, too. Heavy 3D rendering is slow on a weak machine. The libraries have different licenses, so it is worth checking the terms in advance for a commercial video. Voice models such as Chatterbox call for care when it comes to cloning other people's voices.

The authors of the roundup state the main limitation themselves: the libraries will not come up with what to film. The idea, the story and the taste remain with the human, and that layer now determines the quality of the video.