Who's Doing What to Whom: Teaching AI to Draw a Crowd
The Setup: A Simple, Devastating Question
Ask an image generator for a photo of a person, and it delivers, usually beautifully. Ask for two people, and it still mostly works. But ask for two people actually doing something together — a handoff, a tackle, an argument, a dance — and something breaks down. Limbs blur into the wrong bodies. Nobody's clearly reaching for anybody. The models can paint a crowd, but they can't quite explain what the crowd is doing. Why does adding interaction, not just adding people, break the illusion?
The Breakthrough: Building the Scene One Person at a Time
Here's the approach a Cornell research group actually built, stripped of the jargon.
Led by doctoral student Wenxuan Peng, with professors Hadar Averbuch-Elor and Bharath Hariharan, the team set out to fix multi-person image generation by rethinking how the model constructs a scene in the first place. Instead of generating everyone at once and hoping the interaction holds together, their method works iteratively: it predicts the first person's pose, generates that person, then feeds the resulting pose forward as a condition for predicting the second person, and so on. Each new figure is generated in response to the ones already placed, rather than independently.
The model uses FLUX, an image generator released by Black Forest Labs in 2024, as its backbone, and was trained on "Who's Waldo," a large-scale vision-language dataset of multi-person interaction images that Averbuch-Elor and colleagues built at Cornell in 2021. To handle the language side, the researchers paired pose detection with a multimodal large language model that produces aligned descriptions, poses, and spatial regions for each person, ordered starting with the primary actor and working outward to background figures. Given a prompt describing an Australian Rules Football play — one player driving forward with the ball as an opponent strips it and a teammate closes in for support — the model produced a coherent three-person image reflecting that exact sequence of roles.
To measure whether it actually worked, the team built a new benchmark called DrawWaldoWorlds, designed specifically to check not just whether multiple people appear in an image, but whether the model correctly captures who is doing what to whom. In a user study with 20 participants, the group's method was preferred over two versions of the standard FLUX generator by roughly two to one.
Why It's Bigger Than It Looks
The obvious headline is "AI gets better at drawing groups," but the more interesting shift is architectural. Most image generation research pushes toward doing everything in a single forward pass — faster, more parallel, more end-to-end. This approach deliberately goes the other way, building a scene sequentially and using each completed piece as a constraint on the next. That's a meaningfully different bet: that some kinds of visual coherence — specifically, coherence between multiple interacting subjects — are easier to get right by conditioning on what already exists than by trying to generate it all simultaneously.
That has implications well past group photos. Interaction is the hard part of almost any generative scene: two hands on the same object, eye contact between characters, a ball mid-transfer between players. A method that handles pose-to-pose conditioning well is a method that's chipping away at one of the genuinely unsolved problems in image synthesis, not just adding more people to the frame.
The Part Nobody Talks About: The Model Had to Learn What Wasn't Given to It
The quieter but arguably more important detail is what the model no longer needs from the user. As Averbuch-Elor points out, prior methods weren't ignoring pose — they were just requiring the user to supply it, which becomes slow and impractical the moment more than one person is involved. This system's real contribution is internalizing that step: the model infers pose on its own, rather than depending on precise input the average user has no easy way to provide. That's a usability shift as much as a technical one — it changes who can actually use pose-aware generation, not just how well it performs when a skilled user feeds it the right input.
The Meta-Twist: The Dataset Was Older Than the Model That Needed It
There's a small irony sitting at the center of this work. "Who's Waldo," the dataset that made this training possible, was built by Averbuch-Elor and Cornell colleagues back in 2021 — years before FLUX existed and years before this particular problem, realistic multi-person interaction, became a bottleneck worth solving. The dataset wasn't built with this generation method in mind; it was general-purpose vision-language data that happened to already contain thousands of images of people interacting, exactly the raw material a later, unrelated architecture would eventually need. Sometimes the missing ingredient for a 2026 breakthrough was sitting in a 2021 dataset the whole time, waiting for someone to ask the right question of it.
Conclusion: Interaction Was the Real Benchmark All Along
Generating a person was never really the hard problem — generating a relationship between people was. By building scenes iteratively, letting each figure's pose inform the next, and creating a benchmark that actually measures who's doing what to whom, this work reframes what "realistic" should mean for multi-person images. It's not just anatomical accuracy anymore; it's whether the story the image is trying to tell actually reads as one.
References:
https://news.cornell.edu/stories/2026/07/strike-pose-creating-more-realistic-multi-person-images
https://arxiv.org/abs/2605.23178
https://doi.org/10.1145/3799902.3811129
https://cornell-vailab.github.io/PeopleComposer/https://github.com/Cornell-VAILab/PeopleComposer