The Frontier

AI 101 — How do AI video generators work?

Editorial

Editorial

— 5 min read
Share
AI 101 — How do AI video generators work?

An AI video generator is a model that starts with a grid of random noise and repeatedly refines it into a moving picture that matches your prompt — the same trick image generators use, stretched across time. Nothing is filmed, and nothing is stitched together from clips the model has seen. The pixels in the finished video were invented, frame by frame, all at once.

Why it matters right now

Kling 4.0, Kuaishou's flagship video model, went fully live on October 15 with up to 30 seconds of video in a single native pass, 10-bit HDR output at 4K and 1080p, up to 15 reference assets per task, and ten keyframes for pinning exact moments; repeated extension stretches a shot to two minutes. The company's market is real, not a demo: Kuaishou's Q2 2026 filing put Kling revenue above RMB 850 million for the quarter, up more than 200 percent year over year.

What is notable is what Kling chose to brag about. Not prettier frames — length and control. A single beautiful second has been solved for a while. Holding a character's face, shirt, and the physics of a thrown ball consistent across hundreds of consecutive frames is the part that took years, and it is the part that decides whether a generated shot can survive in a real edit.

The mental model

Almost all of today's video generators share the same three-part anatomy, and OpenAI's technical report on Sora laid it out plainly.

First, a compressor. Raw video is far too big to model directly, so an encoder (a 3D VAE) squeezes each clip in space and time into a small numerical summary called a latent. The compression is aggressive: the CogVideoX paper describes a factor of 8× in each spatial direction and 4× along the time axis, which is what makes a 10-second clip small enough to work with.

Second, the denoiser. The latent is cut into small three-dimensional cubes — OpenAI calls them spacetime patches, and treats them as tokens, exactly as a language model treats words. A transformer is then trained on a simple game: take a real video's patches, scramble them with noise, and ask the model to predict the clean version. Do that across millions of clips and the model learns what plausible video looks like. Generating a new one runs the game backwards: start with patches of pure noise arranged in a grid, and let the model clean them up over a few dozen passes. This is the same denoise-and-refine loop our What is a diffusion language model? explainer describes for text.

Third, a text encoder, which turns your prompt into the conditioning signal that steers every denoising step, so the cleaning-up follows your description instead of drifting off on its own. Plus a decoder at the end to turn the finished latent back into pixels.

The analogy: a flip book drawn out of static

Picture 120 pages of television static. You hand them to an illustrator and ask her to erase, on every page, the specks that clearly do not belong to a scene of a dog catching a ball — and to keep the specks she is confident about. She does that fifty times, each pass tidying what the last one left. The picture resolves out of the noise.

That is roughly what the model does. The twist that separates video from still images is that the pages cannot be drawn independently. Frame 40 has to agree with frame 41 about where the dog is and what it looks like, or the result flickers like an old tape. Modern models never handle frames one at a time — they denoise a latent that includes time, so the whole sequence sharpens together.

Why it is harder than it looks

Hugging Face's Diffusers team lists everything a video model has to get right at once: prompt adherence, realism, aesthetics, motion quality, spatio-temporal consistency, frame rate, and duration. Image models only face the first three. The cost shows up in memory — the team measured HunyuanVideo, CogVideoX and LTX-Video needing roughly 60 GB, 36 GB and 18 GB respectively for a 121-frame, 512×768 clip at 50 denoising steps. Tencent's HunyuanVideo, at over 13 billion parameters, was the largest open video model published when it landed in December 2024.

Most of these models also use a VAE that is causal in time: each frame's compressed representation depends only on the frames before it, not the ones after. That single choice is what lets a model trained on short clips encode long ones in chunks, and it is why "extend this shot" is a feature rather than a retraining run.

Common misconceptions

"It generates each frame separately." That was the older approach, and it shows up as jitter. Today's models refine every frame at once inside a compressed latent that has a time axis.

"Longer just means more frames." Length is the expensive axis, not a slider. Thirty seconds in one native pass is a milestone precisely because five to ten seconds was the norm for years.

"It is a physics engine." OpenAI's report framed video models as "world simulators," and that claim is contested — the same report concedes Sora "does not accurately model the physics of many basic interactions, like glass shattering." Generators predict what looks like a plausible next state, which is a weaker thing than simulating one.

"The prompt is the whole product." Control is where the competition now sits: reference images for a character, keyframes for exact moments, composition boxes, first and last frames. A 30-second prompt-only shot remains a slot machine; a 30-second shot with ten fixed keyframes does not.

Where to learn more

OpenAI's Sora report is the clearest short walkthrough of patches, latent compression, and the reasoning behind the architecture. Hugging Face's Diffusers write-up is the practical companion — what each open model needs, and where it breaks. For the architecture itself, the CogVideoX paper is readable and explicit about why compressing time matters.

Related reading: What is a world model? on what a video model would have to do to count as a simulator · What is multimodal AI? on models that read and produce several kinds of media at once · Kling 4.0 goes live with 30-second native video at 10-bit HDR for the launch this piece is framed around.

If a model can only hold a scene together for 30 seconds, is generated video ready for anything longer than a social clip? Tell us in the comments.

Read more

Deep Dive — America's top open model still trails China's best

Deep Dive — America's top open model still trails China's best

Meta released the weights for Muse Spark Open on Thursday and called it "the most capable US open-weight model to date." Independent measurement agrees with the first half of that sentence and complicates the second: on Artificial Analysis's Intelligence Index the model scores 39, first among American open weights and behind GLM-5.3 at 45, Kimi K3 at 44 and Alibaba's Qwen3.8 at 40. The qualifier in Meta's headline is doing the work. This is a real release, a real engineering achievement, and not

Houmo moves compute inside memory — and A20 enters production

Houmo moves compute inside memory — and A20 enters production

Beijing chipmaker Houmo used its partner conference this week to lay out two compute-in-memory routes at once — and to say the first chip built on the newer one is already in production. Houmo's A20, its first chip built on a 3D compute-in-memory architecture, has entered production. The part supports up to 16 chips fully interconnected, claims a 50% improvement in compute utilization over the previous generation, and covers models from 3B to 30B parameters, according to co-founder and product

Kling 4.0 goes live with 30-second native video at 10-bit HDR

Kling 4.0 goes live with 30-second native video at 10-bit HDR

Kuaishou's Kling AI has spent two years becoming the video model people actually finish projects with. Its new flagship, Kling 4.0, went fully live on October 15 — and this round is less about prettier frames than about the two things that make a generated shot usable in a real edit: length and control. Kling 4.0 generates up to 30 seconds in a single native pass, outputs 10-bit HDR at 4K, 1080p and 21:9, and takes up to 15 reference assets in one task. That budget splits across ten images, fi

Brain organoids classified Braille at 83% — in a trio, not alone

Brain organoids classified Braille at 83% — in a trio, not alone

Two stories today: human brain tissue on a chip decoded a tactile signal, and the data-centre company behind some of the largest AI contracts in Europe turns out to be six months behind on its flagship site. Human forebrain organoids grown on a microelectrode array classified Braille letters with 61 percent accuracy on their own and 83 percent when the responses of three organoids were combined, in a peer-reviewed study from the University of Bristol. The team took event-based tactile readings